You are most definitely not crazy; I wanted to share my story since I was (and somewhat still am) in a similar situation. I disliked my job. It was mentally and physically exhausting; I was basically forced to work 100+ hours (on average) each week during my first job after graduating university. For some perspective, that means sleeping ~3-4 hours per night if you are lucky, spending literally every other waking hour at work. The only good thing was being able to sleep in on the weekends, but other than that I literally lived at work (I considered moving out of my place to live at the office to save on rent, wouldn't that be something).

Anyway, I resigned last year and I am still unemployed. I gave up a job that paid a $80k salary and something like a $40-$70k salary (~$130k all in comp). Keep in mind also that this was my first job out of school. I am finding out that not only is the job hiring process extremely slow, the salary I am likely to receive is <50% of the job I left. On top of that, these jobs are not in the area that I was planning on moving into; they are not ideal whatsoever.

So why does this matter? Ask yourself, and be truthful, what would you rather be doing. Continuing to work at the job you absolutely hated enough to leave? Is it possible you are just feeling this way because "the grass is always greener?" I know that I felt this way for a few months after leaving. However, when I really thought about it, I would always say "well, I would be miserable at work right now if I was still there". Anyway, its something to think about. Sometimes having more time to yourself and actually enjoying your job (and this could mean "not hating" your job) can actually be very beneficial. Free time can be used for whatever you want. Consider starting a business on the side, or improving yourself such that you get to that place you want to be.

I hope this helps!