This is a common mistake. When setting up a brand NEW drive, choose a good journaling file system like NTFS and do a quick format, it's done in the matter of minutes, then copy the data to it. It only marks the rest of the "disk" as formatted and does not actually do it until it needs to write a file there.
If doing an OLD drive that contains data, copy the data off to another drive and do like above, quick format, then copy the data back. I use a sync tool that's fast. This does require another drive, but they are cheap nowadays. You really need a backup of your data anyway, especially since you are encrypting it. If one of the drive VC encrypting or the hardware break, you have a backup.
There is some concern when you are encrypting a drive that previously held data that you are concerned about, doing the quick format may leave traces of the old files in the 'yet to be formatted' space of the old drive. There is also a bit of a performance hit when adding files to a quick formatted partition or drive as it has to actually format the section being written to and then write the file.
A simple solution for both the above situations is to go ahead and do a quick format, then copy your files to it. You are essentially done as far as your data is concerned. Then (for Windows) download and use a utility called sdelete. I usually wait until before going to bed, then I hook up the drive, mount it and use sdelete to write zeros to all the empty space (technically the 'unformatted' space on the disk). This not only formats it, it also VC encrypts it as the zeros are being written. By morning the drive is fully encrypted and 100% formatted and I spent little time prepping it, just the long slow (and not entirely needed) sdelete took place while I was sleeping.
The command line for sdelete to zero out the empty space is sdelete -z X: , where X: is the drive letter VC assigned to it.
The decrypt IS excruciatingly slow. So I copy the data to another drive, reformat the drive (again, quick format) and copy it back.