Typically in a Hardware Security Module (HSM), probably backed up to a bunch of smart cards in one or more safes. An HSM is a beefed-up smart card, usually attached by PCI(e). The goal of either a smart card or HSM is to let appropriately authenticated users perform some operation with a key, but not to reveal the key even to an attacker with physical access. Often they have auditing capabilities as well, so that if the host-side software gets hacked you can work out what to revoke.

I'm not an expert in HSMs, but here are my recollections from working with them.

HSMs often (but not always) have countermeasures against physical side channels, such as algorithmic power-analysis countermeasures and power-supply filtering (both to prevent side channel attacks and to prevent glitching; glitching can also be mitigated with an alarm circuit). They usually have potting to make physical attacks harder, though this gives them thermal problems. They may also have temperature and light sensors, and a wire mesh that detects if the card's case has been breached, so that they can clear the secret parts of memory. Usually that memory is SRAM with a battery backup so that it can be cleared within microseconds. Within the HSM, the root key is likely stored in flash, encrypted with another key that's in that fast-clearing battery-backed SRAM.

There is usually some way to back up the keys on an HSM to smart cards, perhaps in an M-of-N secret sharing system, or to replicate them to other HSMs. The HSM may have a built-in port for a smart card reader to limit the attack surface from the host. With such a backup system, you don't lose the keys if your one HSM fails, or your data center is half a degree too cold and it triggers the HSM's alarm circuit. You can keep the smart cards in safes, on- or off-site.

A smart card has the fraction of the above that fits into a credit-card form factor. Often no battery, no wire mesh, limited potting, not many sensors, but still decent protection against glitching and power and electromagnetic side channel analysis. On other hand, the smart card spends most of its time offline, in a safe, and encrypted with a password. The HSM is likely to be online and powered all the time.