The Chaos Dwarfs are supposed to not be very numerous. There's been speculation on why that is, but the main reason just seems to be that they started out smaller. Both the regular and chaotic dwarfs started out as members of a world-spanning dwarf empire, but only a sparsely-populated region of that empire ended up turning to Chaos.
I'm not sure that there's much concrete information on how their population growth is; I think a lot of that is just speculation. The Chaos Dwarfs are generally portrayed as patiently sitting in the Dark Lands, growing their influence over time, while the Dwarfs are basically just constantly under siege by Greenskins and Skaven, so it does makes sense to a degree. On the other hand, Chaos Dwarf society seems to be pretty cutthroat, and the Dark Lands themselves are portrayed as being one of the more dangerous places in the setting.