It's truly surprising how much of Curiosity is off-the-shelf stuff. Even the radiation-hardened computer is a commercially-available, radiation-hardened variant of what they used in Macintosh G3 computers. To put it another way, the software used on Curiosity could run, without modification, on the board in a PowerMac G3, or the early iBooks. Granted it would have trouble with the fact that it didn't have any of the peripherals or firmware, but if you included those (or figured out a way to fake them) it would.

In fact, I would be really surprised if they used the radiation-hardened variant for writing and testing new software for it. The RAD750 is a lot more expensive than the regular PPC750 and, since they're completely compatible with each other there's no reason to use the expensive one unless you absolutely have to (ie, because it's being run in a high-radiation environment, such as space).

As for operating systems, NASA has been using something called VxWorks for a while now and seems pretty invested in it (they've used it for most of its rovers and probes built in the last twenty years).

I think it would be really cool to see someone use RTLinux in a spacecraft. Assuming it was in Earth orbit, and you had a radio dish you could pretty much just ssh into your cubesat. In fact, if you included a satellite internet transceiver and kept paying the bill you wouldn't need the radio dish.

Crap, now I want to build a cubesat with a mirror and satellite internet connection and have it flash morse code at people on the ground. You could have a website where people could pay you money to have your satellite flash a message in morse code. If it were popular enough you might even make the $8000 launch cost back and break even.