And then he uses him not asking me out and him breaking up with me as ammo when shit happens.

As in like he throws it in your face?! If so, cut him off because that's manipulative as fuck and you don't deserve that shit.

I went through something very similar with my ex and in retrospect, it's very clear to me that it was a "he's just not that into you" situation. Of course, my situation doesn't automatically mean yours is the same... but he used to ask me for space constantly and I was always giving it to him and then I ended up always feeling like shit because I would say to myself, "why does this person always need a break from me?"

Maybe he's just having personal issues or something, but if a guy is into you, you usually don't have to question it. And you shouldn't have to feel so shitty. Again, I don't want to overstep bounds here because I don't know your whole situation, but as someone who went through something (seemingly) very similar.... it took me a long time to realize that relationships should make you feel good more than than make you feel bad. When there is a shitty situation at hand and someone isn't treating you right, it's very important to know when to leave that situation. If someone isn't appreciating you or treating you well, as much as I know how you feel with being the only person waiting, wishing, and hoping and that hurts like hell, there will definitely be other people who actually want you and will make you happy.