In order to guarantee the availibility of your websites or applications it is important to have a good backup and restore policy in place.
Either a user accidentally deletes a file, your website gets infected with malicious code or if the instance hosting the website gets corrupted.
Chances are that at some point in time you will need a backup of your data. When that time comes, it is reassuring to know that you will be able to restore that backup.
In this three part blog post we will cover the follwing topics:
- Backup restore tests (this article)
- Which method of backup to choose for your VPS or instance
- Types of backup at CloudVPS
In this part we will focus on monitoring and testing your backups so you know you are protected in case you need them.
Who should test their backups
Everyone. Seriously, you hope that you will never need to use your backup offcourse. But when you do, it better work. Without testing if you can actiually restore, your backups should not be trusted, regardless of the promises of your backup vendor.
When to test your backups
The most important part of backing up your data is to test if your the backup is actually usable to restore everything you might need. There are many factors that come in to play when making a functional backup. The only way to know 100% certain if your backup has actually succeeded is to restore every backup that is made. This might be possible if you automate this process, however this would be a complex and custom setup tailored for your application and might be a bit too much for your needs. Another option, more commonly used, is to do a restore test at a set interval so you don't have to test every backup. The chance of backups not being sufficient in between tests is smaller, and might be a risk you are willing to accept. Keep in mind that the interval you choose should be logical for your buisiness needs. Consider what your RPO (Recovery Point Objective) is for the application you're backing up. If you have a high volume webshop losing a day of order information will probably be unacceptable. Losing a week of order information might lead to inrecovvireble reputation damges. A minimal interval in which to test your backups might be one week if you are accepting the risk that intermediate backups might have failed. If the website does not contain customer data and only code changes need to be backed up it might be enough to check your backup once each month. In that case it might be more effective to accept the risk of loossing a developer's work for one month than to spend a lot of time testing each backup. You should also to take into account your backups retention. It might be acceptable to loose a month of changes. However if you were to only test the backup once a month while your backup retention is 2 weeks you are at risk of not being able to recover at all.
How to test your backups
In order to test your backups you will first have to define what is the desired end result of the test. Most of the time the disered end result should be a fully functional server with all of your applications or websites functining and all data complete up to the moment the backup was made. It could be sufficient for you to only be able to restore the customer data, for example when the application or website itself is deployed from a version control system. Another good thing to do is to time the restore test. You can use this information to determine your actual recovery rime en to check if your RTO (Recovery Time Objectvive) can be met with the chosen backup strategy.
Step one, start from scratch
When testing your backups always start from scratch with a new machine. This way you will be sure to cover every scenario including the worst case, losing the entire instance. Starting from scratch also tests if any software neccessary to restore your website or application is still available. This is esecially important when using legacy software and operating systems such as CentOS 5 and older. You can order a new VPS with CloudVPS. CloudVPS works with daily contract so you only pay for the days the vps is used. If possible you could also order a OpenStack instance wich is billed by the hour, minimising the resource costs of your restore test to a bare minimum.
Step two, install all the necessary software
Once the new VPS or instance is ready you will need to install any additional software you might need. If you are using a image with minimal modifications this step should not take up too much of your time. If you do need to install additional software or need to modify the instance al lot we recommend using a deployment tool of your choice. It will save you time and also make the restore process faster when needed and makes the restore test more relyable and repeatable. We prefer to use Ansible for this, but really any tool will do.
Step three, install & configure the tools neccessary to restore the backup
This step really depends on your backup method. If, for example, you have directadmin uploading its own backups to a ftp account, all you will need is access to that account and a running directadmin setup. If that is all that you require in a restore you are ready to start the actual test. Do keep in mind that you might be missing system files such as authentication logs. If you need them you might want to consider taking the next steps as well. If you have a regular backup method making backups on a system wide level you will need to configure the approriate backup client and account information there. This could be automated with the same deployment tools used to install the other software.
Step four, restore your backup and test
Restore the files and/or databases from your backup. For more details on the restore procedure refer to the documentation of your backup software. Make sure they are placed correctly and available to the correct user. After that you should be able to access your website or application. Chances are that you will need to configure specifc settings. You will however only find those settings by testing. Finally make sure to really test your restore. Don't just accept a working front page but check recent transactions and all functions.
Monitoring your backups
Now that we are sure that we can restore the backups, it is important to know that they actually run and complete. Most backup solutions provide email functionality on completion and failure. We often see these emails automatically moved by a email filter or ending up in a spam box. This is a big risk, even when you only filter out succesfull backups. This way you will not notice when the emails stop, indicating that something is wrong with your backup. If you do not always want to do a full restore test, it might be a good idea to have some quick indicators you can use as a spot check. If there are files that change daily or database dumps you can import and check you might be able to do a quick check more regualarly. Don't forget to do a full test every now and then.
To summ it all up
- Make sure that the backup method chosen fits your RPO/RTO
- When testing your restore, always start from scratch
- Monitor your backups daily
- Do regular spot checks
- Don't forget to do a full test every now and then.