Changed jobs twice in one year. First time it was due to terrible working conditions, which caused a lot of stress. Looking back, I would have probably had a breakdown if I stayed for another year.
The second job was less stressful, with better pay and more free time. The project didn't pan out, I was offered a position on an existing team, but I declined - mostly because I wanted to game development and the offered position would imo pidgeonhole me into a certain job with fewer future prospects.
Got a new job in the gaming industry, moved to another country for it and so far it's ok. Not doing exactly what I want, but moving in the direction, so it feels like progress.
Compared to a year ago, when I left the first job, I'm definately feeling better (though not all of the stress came from the job, it was a large factor), I'm making more money and most importantly - I feel like I'm actually moving towards my goals. That said, no job is perfect. There will always be stress, there will always be people that you cant really get along with etc. Only you can decide if it's worth it to you, as we don't know your circumstances. But for me, it was the right step.