Imo, a degree in Physics can take you many places, even get you into careers that you didn't even considered before (I know fellow physics graduates that work for banks or insurance companies). And this is mostly due to the rigorous coursework (at least in my country, physicists don't have much difficulty when it comes to employment).

I changed from another course to Physics (+ a master's in Engineering Physics). And I still don't know if I made the right choice after 3 years. I felt/feel the same things that you are feeling (being overly tired, feeling way too dumb for the course, etc). I don't think you will ever truly know if what you're doing with your life is the right thing.

What I do know is that if the work you are doing is not making you at least a bit fulfilled or if your negative feelings towards the subject are becoming increasingly harder to deal with: you should consider other options. But you shouldn't do it lightly. Actually consider factors like future job prospects, evaluate if an "easier course" is actually what you desire or if these feelings are just coming from the initial shock in difficulty (in my experience, most people get used to it when second year rolls around).

In the end, this is a decision you have to make on your own. And don't worry too much. Your situation is perfectly normal.