ESH at least

"I just wanted him to go to his room or hang out in the game room"

Were those the terms when you INVITED him to stay with you and that he agreed to pay for, because if not and you think that he just should infer that and comply with it just from you passively-aggressively munching your food at your party, that's on you.

Also:

"couldn’t take living with him so we moved into a place of our own. Well a year later I wanted to get a house with a yard for our dog and I knew it would be cheaper with him living there too. She’s really good friend with him"

You created this situation and now are bothering both him and your girlfriend. Admit the problem is yours, not his. And stop making their lives tiresome.