but what if the people are wrong?

which people? The trick with institutions like fascism is that they cannot exist without subjugating out-groups. It is antithetical to its nature, and it can never exist with the consent of the governed because by design it needs to suppress and kill one section of the governed.

The confederacy could not have existed without slaves. Slavery was integral to the identity of the confederacy. You are making a false equivocation between majority rule/oppression and consent of the governed, i.e. how social contracts are formed.

People desiring agency over their physical space, the public land they live around and pay for, by removing statues, for instance, is an excellent example of this dichotomy. A government that refuses to even allow people to determine what they do with the parks in their neighborhood is not ruling by the consent of the governed. It is a fallacious comparison to then draw an analogy to a majority that desires oppression; by definition and construction, governance by that majority does not have consent of the governed, if their governance necessitates suppression of the governed.