I don't agree with you, though I don't entirely disagree either.
I think it can make the conversation more difficult in situations where there's a stronger likelihood of changing attitudes. For example, someone who holds a well-intentioned but false (or even true but damaging/self-reinforcing) belief might be able to be persuaded to change their views, but are less likely to do so if you start the conversation by calling them a racist.
This is especially true if you have no way of differentiating people holding ignorant/naive/"well-intentioned" racial biases from the David Duke nut cases who are out there calling for genocide.
So with that in mind, I wouldn't advocate for throwing away the word "racism", but I would exercise caution before using it to shut down a conversation that might otherwise be productive.