"All that is necessary for the triumph of evil is that good men do nothing."- Edmund Burke

Yes your behavior is defined as cowardice, not sure I'd do any different in your situation. Just remember though, the more and more persecuted groups you don't speak up for, the fewer and fewer groups opposing the people with whom you disagree...

"First they came for the Socialists, and I did not speak out— Because I was not a Socialist.

Then they came for the Trade Unionists, and I did not speak out— Because I was not a Trade Unionist.

Then they came for the Jews, and I did not speak out— Because I was not a Jew.

Then they came for me—and there was no one left to speak for me."

Martin Niemöller discussing the shame he felt at the cowardice of his own religion and the intellectuals of Germany not defending vulnerable fellow citizens before and during the Holocaust. (after he himself spent 8 years in Dachau for speaking out against the regime once it was too late to have an influence/no other groups had enough strength to do anything. So i'd say he has an interesting perspective on these types of situations)