You're misinformed about the numerical problem with the EC. The problem is not now that there are electors who actually cast the votes that count. In almost all cases those electors are legally required to vote as their states' voters indicate they should.

The problem is that states get a minimum of three electors, even if they don't have a population large enough to have two congressional districts. Larger states get one for each congressional district, plus two more. This means that residents of smaller states have more presidential voting power than residents of larger states, and this is the problem that has twice given us disagreements between the EC and the popular vote in 20 years.