I might be totally wrong, but it might have come from his insecurity. He might be insecure about his package, and might have taken your remark as a direct hit to his junk. Maybe he thought you are testing him. And I am NOT justifying the racism, just looking from a perspective of a vulnerable woman on a date. When men get so frustrated about another male it's usually a sign of not only being a bigot, but also huge immaturity and overall unsuitableness for a romantic relationship.

You definitely dodged a bullet, and if youre having doubts and guilt because of how well he presented himself (healthy, smart, rich, whatever) you can be sure that its also a lie to compensate for his insecurities.