My theory till now is that the nine realms are not just planets.. these are entire parallel worlds. The planets usually associated with these realms are focal points or centres of the respective parallel worlds. i.e. Earth along with all other planets including Xandar, Knowwhere etc. all lie in midgard with earth just being a focal point or referring point when viewing the Yggdrasil tree.

Travelling between these realms/worlds requires some higher form of technology like the Einstein-rosen bridge etc. which super-advanced beings like Asgardians, Jotunheimians possess. So, it doesn't matter how far Starlord flies in his ship in any direction, he can't reach any other realm. He'll always be in Midgard. Milky way, andromeda all lie in midgard. Maybe the Asgard realm has its own structure of the universe and laws pertaining to that universe which are not applicable to midgard. Maybe the realm of Asgard just consists of a single planet and that is why realm Asgard and planet Asgard are one and the same thing.

The Infinity stones are spread across all the 9 realms and since Thanos being in Midgard has knowledge about them, this means that he might be the only being in midgard who knows how to travel across realms.

This is just my theory I have developed watching the MCU. And nothing shown till now disproves this. Maybe it is correct. Maybe it is wrong.