Yeah what u/King-Azzer13 said. no need to start over.

Some perks you can't even get until you're around lvl 50 anyways.

Then there is legendary perks after that too so you'll probably want to re spec then too, i didn't really spec my character for any type of ''OP'' build until i was around lvl 100 i think, before that i just picked what i thought sounded useful for me at that time.

Had 5 weapons, machete, revolver, shotgun, handmade auto rifle, and a hunting rifle as a sniper up until about lvl 60-70 id say. then i started to make my build for 2 weapons only. and only then i started caring about legendary weapons and armor. before that was all basic workshop weapons and armor.

i will say i wish i didn't spec for that many weapons though lol, had to move a lot of points and cards around.