If actions are criminal, the justice system should hold the actor accountable.

If actions are unpopular politically, the voters should hold the actor accountable, by not voting for said individual.

If actions are disliked by members of the ruling class present in the house and senate, they should not have the right to impeach someone. Imagine, by fluke, some Libertarian becomes president, but the largest parties remain the democratic and republican parties. Don't you think they would use this political process to try and stop this Libertarian President from running again?

So should we just get rid of the power to impeach entirely?

I don't believe so. I believe it should be used if the person in office is a present danger to the country. When they are removed, protection should be the motivation, not any kind of punishment, like removing the ability for the individual to run again. If someone is not in office, they are not an immediate threat.