Size matters to some people, it doesn't matter to others. Like guys that chat about wanting to hook up with "that 9/10 chick", looks and build are important to some guys, and penis size is important to some girls. No one person is going to be what all men or women look for. I think being confident and a person who others like to be around is vital for getting in the door and meeting more women, and looking for those who appreciate personality with a body may be higher yield for you. Quick story time:

My SO of 2 years was immediately attractive to me because he was confident, sexy, and smart. We went on a few dates, and he was remarkably not pushy and really let things go at our own pace. Eventually he invited me into his place, and then got really nervous. We both tried to play it off, but when I reached for his pants- nothing. Maybe 2", soft, and he was completely terrified. I immediately scaled back, and we cuddled for an hour and talked, before I asked him to walk me home. Later on, he told me he was terrified he would never hear from me again. I was freaking out he didn't find me attractive. Go figure. Never crossed my mind to hate on what he was packing. We ended up going out again, and things went a lot better when he was more comfortable. He is definitely a grower, so he's almost average size but I didn't let our initial experience deter me, and now we have the best sex of my life.

TL;DR: I would have missed out on someone amazing if I was caught up in the "big dick" hype some people love- if you're worried about being "good enough in bed"- he can do ridiculously amazing things with my whole body, give me the best orgasms I have ever had, and can turn me on at the drop of a hat. Besides that, he's a freakin amazing person who I love to be with. Maybe someone else has missed out in the past with a similar reaction to what you experienced, but its been my gain.