NTA. Your sex life isn't your friends business and it shows that they are extremely judgemental people if they are willing to throw away a friendship over it. All I can say to that is get better friends.

I've also been in that situation with someone once and when I tell people I typically clarify with, "he never told me he had a wife." Had a few people make homewrecker jokes, but most often they rush to insult him instead. It isn't your fault if someone lies or omits information. it leaves you feeling crap after, absolutely, but ultimately it doesn't make you a bad or untrustworthy person. Your friends were awful to try and tell you otherwise.