It depends on what you want to do with your degree.

If you have a super good GPA you could probably get into a physics program at an R1 but like a state school. You could get a PhD in theory.

But, if you want to become a professor you need to go to an Ivy league and be super impressive. To get into an Ivy you would probably need the full ream of college courses and a first author paper. (As a professor I am super annoyed at the recent requirement that undergrads have a first author paper but it is what it is.)

But you should know that even if you decided to add two years to your degree, do the physics coursework, work your ass of in a research position to get a paper, and then got into an Ivy, and did an amazing phd you are still not guaranteed a faculty job. In fact the odds of anyone getting a faculty job to teach theoretical physics are devastatingly low.

But if you want to work in research/government sector/etc a PhD at a small state school (but one where you still did good research) is enough to get you a postdoc at a national lab or something where you could be hired on.

I should also mention all of this advice is contingent on you doing interesting and applicable research. So lean more towards something like quantum computing and very very far away from something like string theory.