Romney was rightfully tagged as a rich white guy who was out of touch with the poor, minorities, and women. All Republican candidates get tagged with that and it is because the Republican party is, for the most part, comprised of rich white guys who are out of touch with the poor, minorities, and women.
Nobody significant believed he himself was a racist.
However, his party absolutely was at that time and that has only worsened. Romney rolled out the red carpet for an endorsement from Trump and Trump's entire political identity at that time was "President Obama isn't an American citizen." He was King Birther and the Romney camp knew it and they wanted to appeal to other birthers. Romney refused to condemn Trump's conspiracy theory, and bringing him out onto the national political stage as a serious person helped legitimize Trump both as a politician and a Republican.
Likewise, the Romney campaign and the Republican party let this nonsense run rampant throughout the party, with various surrogates for the campaign and party repeating it. Romney himself made a comment about nobody ever asking him for his birth certificate that his campaign claimed was a joke, but was clearly a remark to rev up the extreme wing of his party.
That's racist. The concept that Barack Obama is not a true American is a belief with zero evidence supporting it. It is based entirely on his skin color. No white President has endured that, even white candidates who weren't born in the USA like McCain and Romney's own father were not subject to any substantial witch hunt over it.
It was entirely built around the belief that the only true Americans are white people, and that's racist horseshit.
Mitt Romney is a genuinely good person with good intentions who allowed his campaign to exploit a racist lie about his opponent, the sitting President, for votes. Just abiding that kind of behavior is racist enough for many of us.
Is Mitt Romney a racist? I seriously doubt it. Was the Romney campaign racist? Yeah, kinda. Is the Republican party racist? Absolutely.
It is ridiculous to me that people think that not only we should be letting this crap go, but that we should be actively making an effort to do it less just in case Hitler runs for President. If we do that, nobody ever gets called on anything, and racism is not always as blatant as a burning cross.
Trump was not the line in the sand. At least, not Trump the Presidential candidate.
No, the line in the sand was Trump the birther. That was unacceptable, that was beyond offensive, and it was disgusting that we as a nation allowed that bullshit to be treated like a legitimate opinion and not a racist, career-ending belief. He should have been done as a public figure on that day. But no, people shrugged, mumbled something about PC culture, and NBC kept airing Celebrity Apprentice. Republicans in the House, in the Senate, and on TV repeatedly questioned the authenticity of the first black president based on absolutely nothing. And to this day, Trump has never apologized for it. None of them have.
That wasn't a freakout, that was Americans expecting more from the country and its leaders.