I’m also very aware that this is most likely even more difficult for Chuck than it is for everyone else...I’m probably the asshole here.

You could be right. I don't know anyone who uses "man, bro, sir" so much in their daily vocabulary that having to drop the male-gendered words is such an exhausting and painful process. Your language usage, or people around you, might be more saturated with cis masculinity than you think, and Chuck is made painfully aware of this every time you use it.

You can do your best without having to beat yourself up if you do slip up. It's not like Chuck goes on a tirade and gets hysterical or anything. Just because you fail to reach your own standards of perfection and NiceGuy doesn't mean that other people are actually being terrible to you. Relax and just do your best. NAH