We're still in our "starter home" after 30 years. It's a little under 2,000 square feet and about 80 years old.
Couldn't be happier.
No mortgage. Even when it had a mortgage, it was small, so if the economy went to hell, I could read about it in the news and think "Hmmm. Glad I don't have a big mortgage" and know that I could pay the whole thing off in cash if necessary.
Things that cost a fortune in big houses came for free. The entire place is old hardwood with nice grain, and great walls that don't transmit sound. There's really nice molding and doors. No HOA. Low taxes. Easily walkable area.
It's an established friendly neighborhood where the neighbors all know each other and throw block parties in the warm weather.
The biggest thing is that instead of paying for an expensive house (and expensive insurance and expensive mortgage and expensive taxes) we've been saving for retirement, so as we get older, we can actually retire comfortably and enjoyably.