Everyone should be doing backups, right? Especially if you run a SaaS business and store customers' data.
Coming up with a backup strategy can be complicated. Not only it can be terabytes of data. It can also be highly heterogeneous - meaning it's not just files and folders, but databases, transaction logs, key-value storages, full-text search indexes, cloud-storage...
But that's not the biggest problem. The biggest mistake I've seen people make is - protecting themselves from the wrong disaster.
When someone says "data loss" engineers usually think this is what happened:
Or, at least, this:
We subconsciously focus on "movie-like" threats, like some evil hacker bringing down a server from his dark basement. Or hijacking and then deleting important data by using a secret NSA/CIA backdoor in the operating system. Or an earthquake destroying your datacenter.
While in real life it's probably something much more prosaic: untested code, a tired sysadmin, a user deleting his own file, a developer commiting changes to a wrong branch, an angry ex-employee deleting stuff after being let go...
We've been running a SaaS helpdesk app for almost 10 years and we had our share of data losses. While no data was actually lost forever (we managed to recover every time), our humble stats goes like this:
Those #4 and #5 are the ones you should prepare yourself for. Those are going to be your biggest threat. Not a hurricane, not an earthquake, not a hardware failure and not a hacker attack. And the most popular request from a customer is going to be "help, I just deleted something".
Now, I'm not saying you shouldn't protect yourself from viruses, attacks, shouldn't patch your servers or encrypt the data. I'm just saying your backup/restore procedure should be designed accordingly. Your data will probably be deleted by legit users in a legit workflow, and most backup/restore strategies are not ready for this.
Like, here's the phrase I heard countless times:
"I don't need to backup Amazon S3 because Amazon S3 is like 99.99999999-something durable"
Yeah, right. And because your users never delete anything by accident. And because your developers make zero mistakes when writing code.
by Alex. CEO, founder