I've heard all these arguments before and I simply don't find them compelling. The 'best' formations of the IJA/Kwangtung army had already been lost or dispersed throughout the pacific during the previous years of fighting.

No, they weren't. Just to list, for example, the service areas of Japan's main tank divisions by the end of the war:

1st Tank Division-One regiment to Saipan, Three regiments in Operation Ichi-Go, China

2nd Tank Division-Half in Manchuria and North China, some units in the Kurile Islands, other units destroyed in the Philippines.

3rd Tank Division-One regiment stuck in Rabaul, one stuck in Seoul, rest stuck in China

4th Tank Division-Training Unit, based in Japan.

One can see at least half of these tank units were in China or Korea, and were effectively doomed by the Soviet advance.

And of course I am speculating; the deliberations of the Japanese aren't really all that available.

Um, yes they are. The deliberations of the Privy Council have been studied. See Frank's Downfall: The End of the Japanese Empire.

But from the fact of the matter is that the Soviet army was never an existential threat to the continued existence of the Imperial family or the state of Japan. The Sea of Japan was an insurmountable barrier for the Red Army. It wasn't for the bombers of the USAF. The US had just promised to destroy every single city on the home islands, and demonstrated that they could do it. And I'm supposed to find credible the threat of the Red Army, a force that already exceeded it's supply lines getting into Korea?

You need to understand just how strong the Japanese ruling party's fear of Communism was. The Japanese were willing to essentially crush the Taisho Democracy and its reforms in favor of stomping out communist and socialist sympathizers (see: the Peace Preservation Law).