Non-INFJ here. Just my two cents from the other side.

When I do set up a boundary, it often catches them off guard and they see it as a light door-slam. This has caused many of my friends to get pissed off or friendships to just die.

This can be totally avoided by setting up boundaries in the beginning. Especially if you're dealing with people like me who value directness. Just let me know straight up in the beginning what I'm working with and we won't get upset when you need time and space to yourself. And if we don't respect your boundaries even after you've warned us in the beginning, we are the ones who need to work on their interpersonal skills/EQ. It wouldn't be your fault if we get ticked off in this case.

You might feel disinclined to do this because of your Fe, but be firm with your rules from the get go. There are people out there who appreciate it when others are open and direct about their boundaries. I wouldn't be offended or think you're mean. In fact, I'd respect you more because I'd then see you as being courteous enough to let me know what to expect.

I've had an INFJ friend suddenly disappear on me before when I thought we were really hitting it off. I did not appreciate that because I was not made aware of her rules before hand.