I'm a man, I'm slightly inebriated, let me tell you three stories.
Story 1. I was young and she was dating someone else, I loved her more than anyone. We got close, we got intimate. We talked online or on the phone almost every day but I was in a machismo period of my adolescences (15). I was also on drugs (acid, Ecstasy, dextrine I didn't need as speed, some cocaine) . I acted like I was manipulating her; I acted irrational towards her more than likely due to a manic depression that I had at the time and until now that I haven't dealt with (and drugs, and an abusive dad). We never dated but I can tell you I probably loved her more than anyone I've ever know or will know; we also got intimate, non-penetrative. I stopped talking to her because someone I know didn't approve of how she looked at the time. The ironic thing was, she got much better looking and the asshole I thought was my friend that influenced my opinion hooked up with someone that was, well, that wasn't great looking. I shouldn't blame him, though, I should have been stronger willed. I don't judge him because I'm sure he's in love. He shouldn't have judged me and, when he did, I shouldn't have listened. Young and dumb.
Story 2. There was this girl, she worked at a coffee shop that I frequented. I never acted on feelings I had towards this girl but I would go to the coffee shop just to see her; brief moments, just while I ordered we'd talk. I talked to friends, asked what I should do, they encouraged me, I still did nothing. I should mentioned, at this point, that I work out regularly and almost obsessively at some times. In my mind, I morphed my feelings towards her into her wanting someone more buff, someone more fit. I worked out more and more, I thought of her, I didn't do anything on my feelings, and I sunk into a deep depression. Obsessive (not stalker-y, I'm talk "self infliction" due to my perceived inadequacy of being unable to pursue someone I liked) behavior followed and my infatuation turned inwards. I won't mention the turmoil I put myself through further than that, but I can tell you I stressed about a relationship I didn't have.
Story 3. I'm a grown man. I make 6 figures. There's a girl at my dentists that I think about, she's a dental assistant. She's pretty. She's intense. I feel like, when she looks at me, she looks into me. I'm a brick wall but I feel like she looks into that wall and sees sunshine on the other side. I admit, I've creeped her online to see what kind of life she lives. She's dating another dude and I think, in my head, I'd be better for her; I wouldn't be, though, not until I get myself straight, which I won't. I'm not good. My life isn't good. She should stay with him.
My point in these three stories is infatuation with someone you think you love is a black hole of despair. If you're with someone, you should love them for who they are and what they are to you; you should care about that person because they love you, and you, probably, love them. If you don't love them, though, it's only fair to admit that, firstly to yourself, and then to them. Do not ever feel bad for not loving someone and leaving them for that; sentimentality will only hurt you both and disregarding sentimentality will save you both from years of heart ache and hurt. If you yearn for someone, you should tell them, because not telling them manifests in ways that creates obsessive and repulsive behavior that will damage your sanity and your healthy psyche. Any illusions you have for someone you love, secretly, but don't actually know, will only amplify interior pain and delusion.
Loving someone you don't know or don't open yourself up to is a mirror of your own internal inadequacies. You use your feelings for those people to self reflect, and not on the good parts of you, the bad parts. You think about what they might like, but those conceptions of what they might like are problems you see in yourself and are only your own projections (ie: not being fit enough). You think about what other people might think of your love towards that person and you dismiss your feelings based on those opinions (ie: they don't meet your buddy's standards). Sometimes, those people are just the Ferraris of your world (ie: you think they'd be great but they will always be out of reach).
My advice to you would be to first reflect on your current relationship, if you're in one. Determine if you love that person and, if you do, understand that outward desire is a projection of a perfect life that everyone manifests in their own head that could or may not ever exist. Everyone drives a ford focus that they chose to buy but thinks about the Ferrari sometimes; the Ferrari that can go 0 - 10 in under 6 but breaks down and costs too much and can't be driven in the rain and won't help you get groceries and isn't comfortable. Maybe the person you desire isn't even a Ferrari, maybe they're just a different brand -- the point is, what you don't drive will always have more horsepower, will be shiny-er, will have more features, because they're a projection of what you want. The reason the grass is always greener is because the grass that's greener is the grass that you don't have, which means you can be delusional about that grass, while the grass you're standing on you know every nook and cranny of.
Second, realize that people can change, but idealism typically doesn't. Someone will always be there to tell you that you could do better. Your own brain will be the one that tells you, you can do better. Your close friends will say you can do better. Don't listen to any of that. Do you care about the person you're with? Do you care about the person you're with regardless of their faults? If you do, that person should be the person you're dating, not some effigy that you manifestly perceive as ideal. Care about the one you're with. If you don't care about them, leave them, but not because of idealism, but because you love yourself and you have consideration for their feelings and both would do better without a inadequate pairing.
Third, there may be two people in this world that would be the best couple ever, but chance didn't bring them together. You need to realize that you could have a more perfect match than the one you're with, but that person either may not feel the same way about you or may have lead a different path than you, that doesn't lead to you. You could know someone that you think would change your life, but they're in their head, and inside their own head may be a completely different scenario that may not or does not involve you. The problem that every human has is that we can only live our lives; if that life leads to a person that we feel deeply about, we assume that the other person's path lead to the same crossroads, but there's a very real possiblity that, that crossroads is only ours, not anyone else's. With that, we need to choose a path, by ourselves, and walk the lonely road, by ourselves.
I'm sorry, this is ridiculously long. Like I said, I'm inebriated and I obviously have a perverse ego. You asked, "is getting over someone you haven't dated sometimes harder than someone you have dated?" I apologize if that's a simplified version, but I'm "straw-manning" your question to make the question more answerable. The answer is: always. Why? The person you haven't dated is a manifestation of YOURSELF. The person you have dated isn't theoretical, they're real, they're black and white, so you can't project a random idealism onto them, you can only reckon with what they are or are not.