I would personally prefer to let players control a "rubber ghost". When you punch, the arm of the colossus moves at a realistic pace to catch up. That way you can even change direction during an attack - as long as your momentum allows it, obviously. The player would see a "ghost" of the colossus' limbs with the true body following it as they move. (Hence the "rubber")

The drawback is that you don't get the satisfaction of real-time punching little humans, but the benefit is that a colossus could move realistically and combat becomes much more tactical than if you could move the colossus freely - which is important, since SotC is more than just a fight simulator.