It's never been that you vote in a person,

Again, this is wrong.

You vote in an MP. That is your person.

That MP may support (and be supported) by a party. They may also vote their conscious, for example if a clown 'leader' they didn't support tries to do something completely against the national interest.

Preventing MPs from voting (and also meeting, i.e. the mechanics of Parliament outside the in-session days) takes away the voice of the person you voted in, even if they are a Conservative.

This is why Conservative MPs like Philip Hammond and Bercow (also the Speaker of the House) are calling it an outrage.

The fact that a few people go along with it says more about those people than about the process.