Not going wide enough is the usual problem. Specializing planets early makes even tomb worlds breakeven quite early for the basic worker stratum. People get super scared of low habitability planets being colored red and dont colonize when they should.
You should always colonize if you can at least break even on the pops. The most important thing for you to do is, based on current colony development bonuses, get a feel for how long it'll take for you to get to a point (via techs, planet designation, %output buildings like energy grid, etc...) where the planet will break even. Then colonize ASAP. Some of the least efficient advice you could give someone for this game is to not colonize 50% or higher habitability planets starting year 0.
If you can calculate that you wont have enough specialist stratum on that planet by the time tier 3 habitability mod tech is available, there's absolutely 0 reason not to colonize immediately, assuming you can afford the colony ship.
To help illustrate the point here is a super simple worker stratum spreadsheet. This is for standard empire, organic pops, no traits, basic energy/food/minerals only as a baseline. If you have techs or bonuses that modify base values for output/upkeep or %, then change yellow highlighted cells as needed. Add whatever % penalty you want to account for inefficiencies in jobs like clerks or less specialization bonus as the planet gets more populated, but even at 0% output bonus and a measly 25% stability, you can see how quickly pops breakeven.