We are actually really good together when it works

This is literally the textbook reason that domestic abuse victims give for staying with their abusers. I'm not saying your bf is abusive, it doesn't sound like it, but the parallel of a fundamentally unhealthy relationship is there. You stick through the awful shit for the good times. But the thing about a relationship that's actually healthy is that you simply don't have really nasty incompatibilites you have to suffer through in order to enjoy good times.

This is an issue of "core needs." You need frequent reassurances and affirmations in order to be happy. He needs to be free of that kind of obligation in order to be happy. These things are never going to change, and they cannot coexist with one another. It's not a matter of improving communication, because what is making it better for you makes it worse for him. The two of you are not compatible, and you will be happier if you break up and find partners who match your needs rather than trying to force things together that just can't fit.