I don't know whether it's a red flag in general but it seems like it's a red flag to you. If it turns you off, then that's understandable.

I was on a date once and the guy pointed to a woman that passed us on the street wearing a white t shirt with no bra, and says " look that girl's not wearing a bra. Now all she needs to do is jump up and down" I was immediately turned off and felt he was really immature. He apologized when I told him I found it off putting but the damage was done. So if something turned you, I don't think it matters whether anyone on reddit will disagree with you or not. It seems that it's something that's off putting to you and it's left a bad taste in your mouth. My guess is that unless you see something in him that's redeeming, this will be in the back of hit mind. That was pretty much how I felt.

For what it's worth, I did try to move on from that joke and didn't end things immediately. Hung out with him many more times but it kept getting worse over time because he liked to randomly bring up certain actresses being hot. Just not my type and I just started to dread being around him so eventually things just fizzled out.