I can relate. Although I didn't grow up in the small city me and my ex lived in, I had family there and it was the first place that felt like it could be "home" to me. I really wasn't looking to leave anytime soon, but I did end up moving far away (not just because of the breakup, but a myriad of other factors too). I've visited a few times since then, and honestly, I mourn the city more than that relationship lol.
It may be a little weird the first time you go back. I had moved on completely from the relationship so I wasn't super emotional, but it can be strange to walk through the streets I used to stroll down every day, drive past my old apartment, his old apartment, the bar where we met. It felt like a snapshot of my life. Maybe if it had been 10 years from then I could be reminiscent about it, but when I visited it was only less than a year-- so it wasn't a long time ago, but felt forever ago. You can't help but think, "how would things be right now if I never left?"
We are lucky that we don't have to face the constant reminders, and moving away can offer fresh perspective, new opportunities, etc. But remember it's your hometown too. I was a bit nervous that I would run into my ex or his friends during my visits, but then I thought, why should I be walking on eggshells here? When you go back, visit those places with friends and family and make new memories! Don't let one person from your past keep you from doing what you want. He's not that powerful. I don't know your situation, but I promise you, it gets so much better with time. Good luck <3