To me the ability to update firmware or the OS in a way that bypasses cryptographic protections means that the cryptographic protections suck.

To my knowledge, the OS data itself is encrypted, but with a key that's protected by a hardware-specific unique identifier, not by the passcode. So that means you can't just access the flash memory directly and see the contents, but the device can boot without a passcode - and could potentially perform an update without a passcode.

This contrasts with user data which is encrypted with keys that are protected by the passcode in addition to the hardware ID. Apple doesn't have any way to get at this without knowing your passcode.

There are a number of reasons why different types of data get different levels of security on iOS. But importantly, remember that the iPhone does not protect the integrity of the OS by encrypting the partition so that no one else can read/write to it. That's not why system data is encrypted in the first place. The iPhone ensures the integrity of the OS by making sure that each part of the boot process is code-signed by Apple's certificate.

This means that if you are able to provide a valid version of iOS that is properly code-signed by Apple, the device will happily oblige and replace the system data with the new version. There are no encryption keys required for this process - it's a question of signature validation.

As far as your question...

"Why would Apple allow forcing a custom OS without wiping encrypted data?"

Remember, the iPhone has no idea this is a custom OS - this is an official build, verified with Apple's code-signing certificate. This is just as normal of an upgrade as iOS 8 to iOS 9, as far as the device is concerned.

The reason why they can't prevent valid upgrades without a passcode is because it would simply be a software failsafe and nothing more. The FBI could just simply add a demand to remove this failsafe from iOS. In other words, the FBI would just demand that in addition to the weakened version of iOS + brute forcing tool that they want created, they also want Apple to remove the code which clears user data keys if you can't provide a passcode on OS upgrade.