I would suggest: 1. Don’t constrain yourself to prefab, if budget is a deal breaker. It was a mistake for me to assume default/non-custom spec, prefab or modular were cheaper than fully custom. I found a builder locally who had better access to subs who would work for cheaper - and they did great work.
Get as many bids as you can on a plan you like. I was surprised at the range I got back. Everything from $320/sq ft to $125/sq ft on the same plan with mid-range finishes as the assumption.
Goes without saying but you didn’t mention size. A custom plan is ideal for tailoring to your must-haves. We scaled down our plan size dramatically, reached our budget and love living in it. Lots of wasted space McMansion plans, imo. Even if there isn’t waste, there is likely to space that isn’t practical to you and how you live.