This isn't really advice because everyone's experience is unique but I know exactly what you're talking about. It's a strange state for sure.

I've spent almost as much time single (including casual dating, FWB, etc.) as I have in relationships since my first girlfriend, about twelve years ago. In that time I've had three serious relationships, and dated twenty-ish women.

I totally identify with this:

I've always been a quiet person who's done things on their own, never really had a huge amount of friends. I want to feel affection from a relationship, but I can't seem to hold on to the interest in them.

It took forever to understand this about myself, but it what it comes down to (again, at least for me), is that I'm just really unconsciously picky. Not specifically about looks or personality traits or anything in particular; there's no template. The three people I've fallen in love with are totally different from one another. The only major similarity is that they're weirdos; like legitimately strange, unique human beings.

And I don't mean outwardly, like stuff that would appear on an OkCupid profile. It's just the way their mind works, and how they operate, and how they form bonds.

For example with my most recent, we hung out for six months before anything happened. We both dated several people, and in fact, she had only ever been with women, so it really was platonic. Neither of us were in the market for a serious relationship. I honestly didn't know I had real feelings for her until I left to travel for a month. One morning about a week into it I woke up just totally consumed by it. We hadn't spoken at all since I left but she called me later that same day, and before I could finish saying hello, to my complete bewilderment, confessed her feelings to me.

I don't know why I just typed all that out, but my point is that we are all puzzle pieces, and there are more pieces who fit as friends, or flings, (or secret part-time lovers), than there are who fit as true partners. And when you find one, there will come a point — it could be at first sight, or after years as friends — where you'll know, and what you're thinking about right now will fly out the window. In the meantime, don't waste any energy worrying about it; it's the perfect time to focus on yourself.