This isn't about malice towards lower income people, but a problem of effort: It takes about the same amount of effort to build a $300,000 house as it does to build a $500,000 house.
Close, but not quite. We've got low consumer inflation - an avocado today costs the same as it has for decades - but rampant asset inflation. Stocks, homes, anything major costs a ton of money, and cheap debt with low supply of existing homes means that you're better off buying a more expensive home with debt than a old home with debt.
However! ... if cheap debt went away, the population of people able to take out those loans go with it. Builder would have to build more affordable homes because there's nobody left to purchase high-margin expensive homes.