I've listen to every single podcast of his and I want you to understand it's totally not, but of course it sounds kind of paranoid and offensive if you look at it simplistically. That one line you're quoting is boiling down thousands of hours of discourse and hundreds of thousands of words he has written on the topic. It's not black and white.
Think of every marriage that has ever existed that lasted 20-50 years. What percentage of them had neither spouse EVER experience sexual or romantic contact outside of their relationship. The data might surprise you.
His point is NOT that if you clearly define the relationship as monogamous that infidelity is ok. Dan Savage (the advice columnist) is in an open marriage, and he says that for many people this works. He is married, they have a kid, and they have their clearly defined rules, but his point is not that all monogamous relationships are doomed to fail and open relationships are the only healthy model. *THAT IS NOT THE POINT. *
The point is that some people buy into the idea that X number of years in a marriage should axiomatically ALWAYS be thrown away in the event of infidelity. There are many couples that would benefit from being given a "free pass" and different couples define their rules differently. There are also many couples where they would only consider their healthy dynamic exclusive monogamy.
But that couple that prefers exclusive monogamy, if they're still in love, want to continue their relationship, just because one time out of 20 years a partner breaks their agreement, are we so judgmental that we should say that infedility SHOULD ALWAYS be a dealbreaker?
I don't think anyone would say so. Often, and frequently--fuck yeah it should, it's a breach of trust. The main point is that trust, communication, and compatibility matter way more than what society tells you should be an arbitrary relationship-ending event.
It's not ridiculous, it's just also not boilerplate advice applicable to every single relationship ever. His point is exactly the opposite, that we shouldn't arbitrarily assume in every relationship it ought to be. This is why the relationship advice industry will never die. Because every case is different.