I'm in a similar spot. I married to an old high school girlfriend at 22 after about 8 different women before her (she was number 3) Got divorced at 32 and bumped the "number" to over 50 and now I'm 42 years old. Some were one nights, some were actual relationships. But my personal opinion of myself changed and made all the difference.

One point of contention (and I deeply regret my attitude, but it did not contribute to my divorce) in my marriage was that my wife was with many more than i was at the time we got together seriously. And since we can from small town Wisconsin I knew every guy either personally or through the grapevine. And it dug at me. We moved to the west coast and we had a new life without that baggage. I regret the times I slut shamed her when I really was feeling inadequate.