Okay, I have lots of thoughts about this and I hope they'll help!
First, I'm proud of you for making a hard decision and ending a long-term relationship. That shit is really hard, especially when you've been with them through so many formative experiences that occur when you're a young adult.
With casual sex and other low-commitment styles of para-relationships, like... if you're finding that you're not feeling good coming out of those weekends, girl, it's not for you, and that's okay. Your emotions are nothing more or less than what they are - they just are, and they're neither good nor bad, just something to be noted and adjusted for accordingly. Does that make sense? Like if anyone's gonna try to tell you "well you shouldn't be feeling like this," that shit can go right out the window. If you're trying to tell yourself that you should or shouldn't be feeling some kind of way, that can go right out the window too. It's unproductive at best, battling your emotions like this. Your feelings are yours to feel, whatever they are. If you come out of a weekend of good sex feeling anxious (and boy can I sympathize! I get the same way!) and bad, how're you gonna avoid that in a healthy, realistic way in the future? (I stipulate "healthy" and "realistic" here because walling yourself off and steeling yourself against the world and trying to give yourself a big dose of Buck Up, Buttercup is neither healthy nor realistic!)
When I was in my early 20s, I was in college and doing the whole casual sex thing, but god did I feel fucking awful. I didn't feel empowered or liberated or sexy; I felt used and discarded and uncared-for. It took me a couple of years to figure out that the way I operate and the way I want to engage in a relationship is not compatible with casual flings - I needed something more dependable and supportive, somewhere I'd feel safe, and the flings just weren't providing that for me. It took me all that time to figure out that my feelings about relationships were valid, and that I didn't have to do what my friends were doing.
Take some time to drill down on why you're not feeling good about the sex you're having with this dude. Do you need more attention from him when you're not physically together, and does the lack of attention make you feel like you don't matter? Do you feel that sex should be more special than it feels with him even though it's good sex? Do you want someone to hang out with after work and make dumb jokes with when you're doing the dishes, and you're definitely not getting that from him? These are just some possible examples; your reasoning may be similar or way different, but whatever it is, it's valid.
Sending love. You're gonna be okay!
These are wise wise words