I think, to understand you need to understand two premises in the 3/5th compromise:
that the residents of a state who can vote represent the interests of those who aren't. At the time of the founding, voting was very restricted. Black people weren't the only people deprived of the right to vote, though it is hard to imagine a worse situation than those enslaved.
that when represented, the population of slaves should only count toward representation as 3/5 of everyone else.
you, understandably, are pointing to bullet 1. I think pretty much everyone in the US agrees that slave holders' votes should not be viewed as representing the views of slaves.
The people who are pointing to the 3/5th compromise agree with you on point 1, but they are pointing to bullet 2 as well. For the most part, the people in nonslaveholding states weren't fighting for slaves rights either in 1776. Undermining political power of slave states, while certainly wouldn't have hurt Black slaves, wouldn't have been likely to help them at that point in our history.