Could you give more concrete examples of the manipulation?
People might accuse you of generalizing because you use words such as "INFJ men are always". Always; no exception. I think people tend to use exaggerating words when they're pissed off so it's understandable, though. That's the reason why I try to search my texts for such words and erase them so the text will seem more sober and less emotional and prejudiced.
To the topic. I have an INFJ guy friend (I'm INFJ female) who might be sometimes trying to manipulate me. I'm not sure what it's about. E.g. he likes my posts on FB, but can also suddenly removes the likes. They're just FB likes, yup, but there's a pattern in how he gives the likes.
One of my posts had no likes. He liked it. When other people started to like the post as well, he removed his like. I think he felt bad seeing my post had no likes, but when the post became more popular, his like no longer had the function it used to have and he removed it. Understandable, but I find it kind of petty to go out of your way and remove likes, as if the likes would be some precious currency. They're just likes after all.
He has also made some comments that seem harmless on the surface. Considering our past interactions and the current context, though, the comments seem to have special meaning to them. It's like he has quietly hold grudges and sometimes let little pieces of it show.
Even though I sometimes feel like he's trying to control me with how he gives and takes affection, I think every individual should take responsibility of their relationships as well. Take his FB behavior for example. I've been wondering why he does what he does, but as long as I don't take any concrete action to e.g. talk about it with him, I allow him to do it. Given this, if you are aware of what's happening and you and them are equal in your mental abilities, you also have agency - and if it's bothering you to the point you talk ill of him, I think even moral responsibility - to change things. Changing things doesn't have to mean anything big. It can mean making it clear that you've removed them from your social circles.