If I use GPL code in my Apache project, I have to use it unmodified; I could not, for example, refactor GPL code into an Apache licensed file.

That is because of Apache and GPL license conditions colliding, and as I stated GPL can be mixed with the majority of permissively licensed code, not all permissively licensed code, although IIRC GPLv3 projects can use Apache code but not vice versa.

I also force downstream users of my code to accept the restrictions of the GPL.

Depending on your downstream users, those 'restrictions' may not be considered as such, since they ensure access to source code and the right to copy, modify, run and further distribute said code, including derivatives.

If your downstream users are developers who want to use it in a proprietary fashion then certainly it is a unwelcome restriction.

Forcing developers to release open source code is the means to achieve the end of protecting user freedoms.

There really is no other way to give and preserve those end user freedoms, of course no one forces developers to use GPL licensed code to begin with.

Other code come with other conditions, proprietary code typically come with the condition of monetary compensation and is very restrictive if not prohibitive when it comes to distribution and modification (which would have to be done at binary level).

That is an untenable restriction for many developers,

Then don't use GPL licensed software, again you are not being forced to do so, copyleft exists to grant and preserve a set of end user rights, if you do not want to give these rights to your end users, then you can't use GPL licensed code.

I have no problem with developers saying they won't use GPL licensed code, however I can't understand those who whine about other people licensing their code under the GPL.

It's pure hypocrisy, you (not YOU specifically) demand the right to be able to set any conditions on your code you want, but others should not ? It boggles my mind, no one owes you free code to use as you wish.