I understand what he's feeling. It's definitely hard to grasp. I don't have an answer for why I do it myself. I want to make friends, but I struggle to stay with them. I tend to push them away because for some reason if I get too close to someone I feel uncomfortable. Then I get sad because I'm lonely lol. I went to a counselor for two semesters and I talked to her about this issue. We came to the conclusion that I don't get close to people because I expect them to leave me. Most if not all of my friends from school have drifted away from me, so I expect that from people I meet. I don't know if your friend has trust issues, but maybe something happened to him in the past that made him this way? Or another thing to think about is perhaps your friend doesn't see himself worthy of having friends? Maybe he thinks that people won't like him so he cuts off before they do...

Also, you mentioned that he doesn't do well in groups of people, but with you. Maybe he wants friends, but only one at a time? Having too many friends at once can be overwhelming to him and maybe he can't really connect with many people at once. In my case , I usually decline group invites because I feel left out. Not because I'm actively being left out, but because the attention isn't solely on me. I'd rather prefer hanging out with one person because that person is only talking to ME. And because their attention is on me only, that makes me feel better for some reason. It sounds selfish. :c I don't know if this is exactly what your friend is going through, but it's something to consider. Good luck :)