Sounds like a sore situation.
Perhaps he didn't know what he was getting into? Living with 4 other people might sound like fun at first but if I was at his place, I'd get very stressed out soon.
He might be aware of the talk that is going on about him (=others considering his behavior strange) and the pressure being put on him (=he should become less strange), and it's not probably making it any better. He might shut more and more inside his shell if he feels like he has permanently messed up the whole thing.
But it's not like you could do anything about it if he's not taking the initiative to meet you halfway. People will talk and there's nothing wrong in that - it's that when it's at the level when other people talk behind your back and think you don't notice, it's really hard to try to better the relationship with them. It feels unpure and permanently damaged.
Think about being at his place. Is he confirmed gay? If not, then think about other people suggesting behind your back that you're gay and you have a crush on the only person you're feeling comfortable around at the moment. He probably knows he's acting weird but it doesn't have to be that weird. If he's anything like me, he feels sorry for his weird behavior and wishes he could do something but he doesn't know what. He might even feel like he's making it better for everyone if he isolates himself.
It's clear you're the only person he feels comfortable around at the moment (it probably has nothing to do with romantic love). Living with 4 other (previously unknown, except you?) people is really harsh for somebody who prefers one-to-one relationships. Again - if he's anything like me, he doesn't care much about "hanging out together" if there's no meaningful relationship behind it. And usually it's not easy to share meaningful things about ourselves in a group. One-to-one relationships are better for that. He might need to get to know each one of you better before being able to be comfortable in group.
But yeah. Pretty bad situation. It sounds like he didn't know how things would turn out. He's probably not doing whatever he is doing out of being mean or hating anybody - he's probably very insecure about the situation as well. Being aware of what is happening and still not being able to take an action doesn't help.. and I'm pretty sure he knows 100% what is happening. It's not nice for anybody.
Do you think there would be anything "easy" you guys could do together? Like watch a movie? Something that would not put much pressure on anybody but that would make it feel more natural for you to be living under the same roof in the meantime 'til he finds the courage to open up?
Or.. is there any possibility you could just stop caring, keep it casual-friendly and give him time? If the situation becomes unbearable, perhaps he will move out himself.
At least make him feel like he's okay as he is. If he can't feel accepted, he's never going to open up.. many INFJs don't trust people who are disrespectful towards them, and based on what you've told, he could be like that as well. Even if they didn't confront people directly about it, they'll think like that in the back of their head. It's like a quiet door-slam. I would stop making suggestions about him having crush on you etc. (if he's not confirmed gay?) behind his back. It will affect the overall interaction with him even though you think he can't notice it.
I don't know how much of this are just my guesses based on how I would feel in his place, but hope this helps even a little.