I believe the OP doesn't mean just using Charisma to hit, but also to damage, which you would not get on your off-hand attack in your example. The Hex Warrior ability lets you use Charisma instead of Strength or Dexterity. Which means, for both the Hex Warrior weapon and the separate pact weapon you can use Charisma to hit, but not add your Charisma modifier to the off-hand's damage because of how Two-Weapon Fighting works. The exception to this would be if you had the two weapon fighting style.

Two-Weapon Fighting (PHB 195) reads:

When you take the Attack action and attack with a light melee weapon that you’re holding in one hand, you can use a bonus action to attack with a different light melee weapon that you’re holding in the other hand. You don’t add your ability modifier to the damage of the bonus attack, unless that modifier is negative.