So, I was going to comment and noticed I can piggyback off of what /u/Starbuckingit mentioned and take it a step further to bring up a point about that guy's reactions not reflecting on your worth, but rather his own current state if development.
But, I don't want to detract from the relevant discussion that this brings up... I personally was kind of conflicted about dating my (MtF) ex, a natural apprehension due to certain societal pressures. The thing is, I found it within myself to work through those anxieties and grow into a stronger person - I became way more confident in my sexuality and in my ability, my maturity to assert my life choices against any 'uneducated/uninformed' or downright ignorant people. I saw what gave me anxiety and chose to be a bigger person after opening my eyes to my ex's experience.
And the way that played out with one if my close contacts is kind of ironic. Even having made the choice to not dance around the fact that I'm bi, or to be actively vocal about LGBTQ support... I had little pangs of anxiety when I thought about one of my friends meeting my then GF. I came to realize this programming doesn't reflect the life I want to live. I mean sure, it was scary thinking that a friend might reject me. But how can they reject me for something that's simply a part of my life? If they did, it would be a sign that maybe they weren't that great of a friend. Issues of morality aren't like a difference in preferences (coke or pepsi).
The ironic thing about my anxiety was that after I internalized a healthier mindset about it all, and then things didn't work out and my ex and I broke up, I caught up with a close friend I hadn't seen in a long time. He surprised me by having grown enough to not bat an eye about my past relationship, he was purely accepting and understanding. We both come from a culture that isn't typically accepting of LGBTQ. It was fucking beautiful ❤️ to know I had that support, as it should be.
That guy wasn't ready for, or isn't mature enough, or maybe he's just not compatible with progressive types of people due to having made the kind of life choice that in effect turns a blind eye to marginalization.
There's a hundred reasons why this guy wouldn't have been the right choice for you, it's valid that he's perhaps apprehensive of how his family or coworkers would react to relationship with a marginalized person. You will find someone who is actually a match for you. It's not just "it's valid to be scared", it's "he doesn't let other shit get in the way of being supportive to his partner".