In simple terms, you would usually use it in place of a hash, except for data that you want to be able to access again at a later date.

I'll give you an example that I have used in real life.

A company approaches my company, and they want to use our services, which means we need to take their customers data, do some business stuff to it, and then give it back to them. Now, these customers are identified by a number, maybe it's a back account number, or a credit card number, or a social security number... what ever it is, they don't want my company to know what the number actually is, they just want to give us a unique identifier that we can both use to identify their customers.

So they give us a token instead. They create their own key to tokenize and detokenize the IDs. For instance if 1=G9 2=T4 3=H7... then the ID 123 = G9T4H7. They keep this key secret, so when the send us data about customer 123, all we know is customer G9T4H7. When we send them data back about customer G9T4H7, they know who we're talking about.

In reality, this can happen all very quickly and efficiently. In the scenario I described above, these customers were actually making online financial transactions that were routed through my system. The tokenization, processing and detokenization all took a matter of milliseconds.

There's plenty of other scenarios where you might want to use a token. For instance, the ID might be a name instead of a card number... Anyway, since I work in PCI, we followed the PCI SSC guidelines on how to do it.

Here is some whitepaper

https://www.pcisecuritystandards.org/documents/Tokenization_Guidelines_Info_Supplement.pdf

https://www.pcisecuritystandards.org/documents/Tokenization_Product_Security_Guidelines.pdf