I went through something very similar to this. I was very happy with somebody for a very long time, but couldn't quite knock those little fantasies of this person I'd known a long time ago. Years down the track, I'd only fantasise rarely, but he came into the place I worked suddenly. He had just moved to the area and that was his local shop. So every now and then he'd come in and I'd go bright red thinking about all the times I'd thought of him fucking me. I wasn't certain if he had feelings back, but had an inkling we were usually flirting. We added each other on Facebook and chatted over the next few months or so, quite briefly, but it was enough to send me over the edge and I was constantly thinking of him. I was so confused, my partner at the time was great and we had such a great time together, but these 'what if' thoughts consumed me.
Ultimately, after gruelling months of contemplating whether my feelings were real I decided to leave my partner. I didn't leave him specifically FOR this other man, I wasn't even sure if he was interested, but I just left for 'something' else. I knew I wasn't entirely happy regardless of how great my partner was.
Best decision of my life. I've now been with the man I was fantasising about for the last 4 years, and I see it all now. I see what I was missing, because I have it. At first it was actually a little like how I'd fantasised, it was hot and heavy, but I think that's what happens in every new relationship. Now we're just a regular happy couple who can't wait to spend time with each other!
I'm not saying this is what you should do, I just saw this story and I felt I could relate with it quite strongly. The man I left was an amazing person, I loved him, but I just wasn't in love with him. We're still friends today, I didn't leave him in some rage of passion for somebody else. It took a lot of time to come to the decision to leave.
TL;DR very similar thing happened to me, but I did actually leave my partner and ended up happily with the man I innocently fantasised about.