In many residential areas there is just not anywhere to add commercial building near by. So all you get is more people closer together but no place close to work, so they need cars still.

This is by definition not the case in single-family neighborhoods. There are large swathes of quite useless space, both horizontally and vertically.

For this to work at all you need a plan. Planned mass transit, planned new schools, planned businesses. The developers who will be buying single family lots and building making them denser, do not care about any of that. They do not care about the infrastructure needed to support increased density. They just want to make a quick buck.

I mean, a plan can be helpful (especially in regards to transit & schools), but the most desirable neighborhoods are all essentially built without a plan, which is what makes them great. If a development plans for a certain business to move in, what happens when that business fails? The entire plan is disrupted. The benefit of looser zoning is that it allows for adaptability in commerce & residences. A generic 4-story apartment building can hold nothing but residential units - or the first floor can hold a grocery store while the top floor holds an art gallery or literally any conceivable combination of uses. A single-family house is never going to be anything but a single-family house. A Walmart is never going to be anything but a Walmart. If the city does not have Walmart in that space, then that's several miles of useless development that has to be demolished and rebuilt, or - more likely - left to rot. Developers are never going to care about infrastructure if they don't have to - which is why higher density is required so that infrastructure becomes far more cost-effective for cities to maintain & expand.

Removing zoning without a plan and control of the build out will lead to a mess.

Luckily, there is not a single living person who thinks we should 'remove zoning'.