Today DigitalOcean lost our entire server

TL;DR: If your data is really important, back it up everywhere.

I learned this lesson the hard way recently. I had always been paranoid about my server architecture so not only did I have servers replicating off each other, but I had other servers in other geographic regions both replicating and backing up from them as well.

One day, I received an alert telling me that every server and site I hosted was down. Slightly panicked, I fired off an e-mail to the provider, and I received a cold response about an unspecified terms of use violation. After several hours of me trying to get more information out of them, it turned out that someone sent them an e-mail stating that one of my websites contained pedophilia and so they were no longer giving me access to my servers. Now that was cause for some major panic because I did have backups but the backups were on servers hosted by them! I then spent the next few hours trying to explain to disbelieving technicians that yes, we were a real company and that no, we do not host pornography, let alone pedophilia, and that they had just taken down an Alexa top 15,000 website, not to mention multiple other unrelated sites.

I finally managed to convince the provider that we were running a legitimate operation and to give us back access to our servers. It turned out that the post the e-mail referred to was of some user of the site stating that their posted content (all text) contained an explanation of pedophilia. Upon further investigation, no pedophilia was actually posted and none was found. Not only was the server running that site shut down, every server with all of my other sites were shut down because of it. In the end, they agreed to leave a note on my account in case someone sends them an e-mail again, which states that we have our own processes to remove content from our sites and to forward all complaints to us. The damage was done though and all of our sites were down for that day along with the terror at the prospect of losing all of our data.

So other than our replication and backup strategies distributed within a datacenter as well as to multiple geographic regions, we now replicate and back up to several other external providers along with multiple forms of payment to ensure complete redundancy. It would now take a worldwide armageddon for us to lose our data but then we'd have more urgent problems.

Edit: The server provider in this story was not Digital Ocean.

/r/webdev Thread Link - murze.be