When my mom was pregnant with me, her boss asked her what she'd do if her and my dad broke up. They were together over a decade, they just never married and he was concerned they were having a child out of wedlock. Her answer was, "Let him see his kid as often as he wants". Period. The end. They separated when I was six but it was never acrimonious.
He was invited to all my birthday parties. The first two years were hard according to both of them, but they got over it. They both struggled with the feelings. But they got up and over, became friends, they both loved each other in a platonic way, and they even spent time together without me. Which is not to say you're wrong in not going. I 100% believe that is not the relationship you have with your ex, or could have with your ex, and you're not a bad father for not having it. I got the jackpot. I got two parents who put their romantic relationship behind them willingly and without malice. No cheating, no damage, just a recognition that they were better off separate than together. That's the unicorn of separations and I was lucky as shit, but it can't be that way for everyone and I know that better than anyone else. I am not suggesting you go that route, or even that it would be possible to try.
What I am trying to say is that your kid is fucking lucky. His parents are at least putting it out there that he can have both of you at his birthday. That you don't hate each other. That you can have cake and smile because that day is about him. He's outgrown the need for that, and that's why he told you not to do the song and dance this year, but there's so much love in that. He gets you both love him more than you dislike each other and that means so, so much. He might not need it at his birthday party, but he'll need it over and over again (graduations, weddings, a housewarming party, whatever) and he knows he has it. He knows you'll both show up with a smile and a hug and be polite.
A lot of kids DON'T have that. And yours does. I think he feels loved and special in ways you can't put words to. Congratulations on you and your wife for being f'mazing parents and raising an incredible kid.