Sex wasn’t as frequent as he wanted but we still were affectionate toward one another.

No. What you experienced was your partner being affectionate on the outside in the HOPE that you'd actually progress towards having sex. You say it was 'not as frequent as he like but...'

Which is sounding an awful lot like saying that he didn't get what he wanted, but you still got what you wanted so it was ok.

For someone who needs sex to feel wanted, desired, and happy, constant denial is like you putting a knife in his soul and twisting it regularly.

As I said in an erlier post: if you want to fix this, you really should figure out how to fix your sexlife because that is what is killing him. This is not about this particular evening. It's about the many months that have gone before.