YTA

I just get the feeling like you think of as an incapable child.

I feel like I’m protecting him by only putting him in situations he is comfortable

It sounds like he's more or less high functioning. You don't need to protect him the way you are. I'm on the spectrum and so are a couple other family members. I feel the same as your friend. I really don't like type of places you mentioned. I'm perfectly capable on deciding if I want to go or not. Anyways, it's good to get out of your comfort zone sometimes (even though it can be stressful) and sometimes it's worth it in order to be around friends.

Let him make decisions for himself.