Only you get to decide what's a deal-breaker - is everything else good? What's actually upsetting you, going without him or the fact he doesn't want to go?

It's completely reasonable to ask your boyfriend to go with you to see your family, especially for birthdays etc. It would likely be seen as rude if he doesn't go, and if it's important to you then he should really suck it up and put in the effort.

But if it doesn't actually impact your enjoyment if he's there or not, and in other behaviours he is respectful and the relationship is good, there's no reason this should be a deal-breaker. Might just need some careful handling with your family, and a bit of negotiation with him on attending for the Big Stuff (funerals, weddings and, yes, birthdays).

(Side note: don't let the allowances like this that you make for him stack up too much, or one day you'll wake up and think "What the hell are you for.")