This could be reasonably rectified by making a meter that fills based on how much G's you are forcing on yourself. Let just use 1-10 for relative ease of explanation.

1 G on the scale is obviously very bottom. You can sustain 1 G indefinitely. As you sustain 5 G's you will fill your meter at a 5G pace until you hit 5 on the meter. If you were under 7g's you would fill it faster, but stopping earlier might end up with less than 5g's. If you were at 5 g's or 7g's for the same amount of time you would have more meter filled from the 7g's because it was filling faster.

Now you can stack these effects of visual impairment by level of G induced and you can even add a verticle and negative G section of the meter. Since you have a resting natural recovery rate you can add that to the G's experienced when you go back and forth between negative and positive G's.

So if you fly in negative G's your recovery rate to zero is the sum of positive G's plus your recovery rate until you hit zero. Then you just move at your G rate filling in the positive sector.

Likewise on top of this you can set your G safe limit to whatever you want and the ship can then keep you under 5g's of medium impairment or if you want to raise it you can bump it up to 9. If you want to keep it on 5 but randomly pull off a fancy maneuver you can then set it to 5 but flip it off at certain times to pull one off manuevers. Thus you will only be able to pull 5G max maneuvers with it on but it creates a reasonably realistic, but still predictable, playful, and all around engaging G force system in the game. It is not even necessary to show this meter to the pilots, or maybe only have it in certain ships, or an upgrade for those that want to configure their HUD to show it.