For me, I think I realized that my dad had multiple affairs (or at least tried to).

We were watching tv a while ago and he casually mentions that he finds one of the characters attractive, he says "I guess she's just my type". That's when the light bulb goes off. The character has the same body type as my mom, but also as a family friend my dad worked with years ago. They were close friends, until they had a really bad falling out (I was never given a reason why), and the higher ups at my dad's job decided to move him. At his new job, my dad works closely with someone else of the same body type and they become close friends. Sure enough a few years later they have a falling out as well, and we move again. I'd known for a while that my parent's relationship wasn't great, and my dad confessed to me on one of his rough days that he and my mom hadn't had sex in years (something NO kid wants to hear), but at that moment in front of the tv, it all came together.

I have no solid proof, and I love my father to death whether this is true or not, but I really hope that i'm wrong.