If I really confront him about something he teases me about, he feels really bad and starts saying “I was joking, babe, come on, you know I joke all the time,” but I don’t think that’s an excuse because you don’t get to just be super mean and call it a joke. I try to be clear about when I’m actually upset so that he doesn’t feel like he can’t joke at all (I always laugh and don’t say a word when I’m not upset—meanwhile I instantly look/sound put-out when he jokes “too hard”).

You're 100% right about all this. He should know when to turn off the joking and be genuine with you.

Is he always in "joking" mode or is he often sincere as well? When you try to be clear about when you're actually upset, are there ever times when you're both joking around and you say similar things, that might lead him to not realize when you're really really serious?

Because my husband and I have a kind of "safe word". Not so much for situations exactly like this because that's not quite the way we mess around, but if we are messing with each other in other ways, we have a word that basically means "no, I'm actually super serious right now" and we never abuse it so it always means exactly what it means.