Knowledge Base


« Backup & Snapshots

CloudVPS Backup Script: Configuration and Restoring Backups

Below you can find the relevant parts of the CloudVPS Backup Script documentation. Download the full backup script documentation (PDF)

The following knowledgebase pages about the CloudVPS Backup Script could also be relevant:
CloudVPS Backup Script Introduction
Configuration and Restoring Backups

The testing utility

In case you suspect that the backup operation is not functioning normally, you can use this simple utility to see what's working and what's not.

The tool will check the following:

  • SSH connection to the backupserver
  • Quota requirements fits (only for CloudVPS backupservers)
  • MySQL connectivity (if MySQL dumps are enabled)

Upon performing  the test, the results should appear as below (in our test case everything was working as we expected):

 
 
 
 
 
 
 
 
The quota calculator

The quota calculator can be used to calculate the total backup space needed based on the retention you entered in the 2 fields:

 

 

 

 

By default the script shows the current retention you have setup during the installation of the backup script (in our case that is 0 weeks/months). This tool is also available during the pre-installation; during installation all the fields are empty.

You can change the content of the fields, which will calculate the backup space you need on the backup server:


 

 

 

Please note that the calculated retention is the absolute minimal space that is needed by the backup script should all weeks and months of retention have passed. The calculation you see here is a rough estimate of what you need, in general use this can be more or less depending on the file usage on your server.

Clear backup log files

With this tool you can clear all the available log files that are present in /var/log/backups/.

The script will start asking for confirmation whether you are sure that you want to clear all the logs:

 

 

 

 

When you agree, the script will remove all log files that are present within the log directory /var/log/backups/.

Command line cleanup:
Commandline cleanup is also possible, see page 37 for more information.

Show account quota

This option shows the current account quota with the used, free and total space.  Selecting it will result in a window like this:

 

 

 

 

In this case we have used a 250 Gb account (because of the setup of the accounts around 1% less is shown) which has 189 GB left in space and we have used 58 GB (24%) so far.

Command line quota report:
Command line quota reporting is also possible, see page 37 of the full manual for more information.

Data recovery

The data recovery function is a simple but powerful tool to recover your data. Currently only files/folders and MySQL databases can be recovered.  Upon starting the function, you will be greeted with this screen:

 

 

 

 

Please note that the actual showed options may differ depending on your configuration details. If MySQL dumps are disabled, this option will not be shown.

We will explain both recovery options on the next pages.


File/folder recovery

Select option 1 and select “OK”. You will be presented the following menu:

 

 

 

 

Please note that depending on your retention options, not all options are shown. In our case we only have the current week with no extra retention set up, therefore only the “daily pool” is shown.

Choose the pool you want to restore your files/folders from, and select the day like seen below:

 

 

 

 

 

Next, enter the absolute location to the file or folder that you want to restore:


 

 

 

 

Note that inputs ending with a / will be treated as a folder and without a / as a regular file.

Next, we see what the script will recover for you and from which day the backups are pulled:


 

 

 

In our testing case we want to recover the folder “/root/” from the daily pool and we want to pull the backups made last monday.

After confirming the recovery procedure, the script will start pulling the backup from the backup server. This may take a while depending on the amount of data that has to be recovered.

Once the recovery is finished, a recovery log will be placed in /var/log/backups/recovery/.

MySQL database recovery

Select option 1 and select “OK”. You will be presented the following menu:

 

 

 

 

Please note that depending on your retention options, not all options are shown. In our case we only have the current week with no extra retention set up, therefore only the “daily pool” is shown.

Choose the pool you want to restore your files/folders from, and select the day like seen below:

 

 

 

 

 

Next, enter the name of the database that you want to recover and select “OK”.

On the next window you may choose to what database name it has to be recovered. You may decide if it has to be recovered to another database or the same database.

Please note that when you want to recover to the same database, the local database will be dropped first and then the import will be put back into place. Make sure that you have a copy of the database before overwriting it! If the script detects this, you are asked to confirm this recovery.

Next, a confirmation window will be shown with a small overview of the selected and entered options like shown below:

In our testing case we are recovering the database “test” to itself and want it pulled from the daily pool from Monday.

Confirm the recovery with “Yes” and let the script recover the database for you. This may take up some time depending on the size of the database.

Once the database is recovered, the database should be working again.

Manually run the backupscript

Normally, the backup script is being executed once a day. This should be sufficient for normal use.

In case you want to run the backupscript manually this can be done using this option. The output will be exactly the same as shown in the report emails you receive (when enabled).

Exchange SSH keyfile

In rare cases the keyfile on the backupserver or on the server itself may become damaged, unreadable or even be removed. If you want to have the keyfile exchanged again with the backupserver, use this option.

A window like below will be shown:

Please enter your password (entry is masked with a *) and select “OK”.

Now, the script will connect with the backup server and place the local public key into the authorized_keys file in the .ssh folder on the backup server.

This can be tested using any of the functions that require communication with the backup server, like the quota calculator. If the backup server does not ask for a password and returns the quota, the exchange was successful.

Mail recipients list

The script sends alerts, notifications and error messages via email to the address that has been set up during the installation. It is possible to extend this recipient list using this tool.

You are presented with 2 options:

View list:
This will show the list of recipients that are currently setup, like shown below:

Selecting “EXIT” will return you to the commandline.

Edit list:
This will open the embedded nano editor in which you can remove, change or add email addresses like shown below:

Navigation:
In case you are not familiar with the nano editor, a small explanation of keys and shortcuts can be  found below:

Arrow keys = Navigate throughout the editor and move the cursor
[CTRL] + [K] = Deletes the current line where the cursor is placed
[CTRL] + [X] = Saves your changes to the file, confirm this change with “Y” or “N” and press enter upon filename selection to keep the filename the script has just edited.

Rsync excludes list

Some files and/or folders don't have to be backupped. These include some system directories like /dev, /proc etcetera. These files and/or folders are excluded in the special exclude file:
/etc/cloudvps/excludes.rsync

It is possible to extend this excludes list using this tool.

You are presented with 2 options:

View list:
This will show the list of excludes that are currently setup, like shown below:


Please note the green arrow pointing downwards, this indicates that there is more to show than is currently being displayed (currently 83% of the excludes is shown as seen on the right). You can navigate in this window using the up and down arrows.

Selecting “EXIT” will return you to the command line.

Edit list:
This will open the embedded nano editor in which you can remove, change or add files and/or directories like shown below:


Navigation:
In case you are not familiar with the nano editor, a small explanation of keys and shortcuts are explained below:

Arrow keys = Navigate throughout the editor and moves the cursor
[CTRL] + [K] = Deletes the current line where the cursor is placed
[CTRL] + [X] = Saves your changes to the file, confirm this change with “Y” or “N” and press enter upon filename selection to keep the filename the script has just edited.

Note:
When the backup root is not the default /, excludes should be based on directories within the current backup root. So if it is set to /home/ and you wish to exclude /home/user1/, you need to exclude /user1/ instead of /home/user1/, because /home/ is not found within the /home/ backup root.

Tip: Paths can be absolute or relative; note that using relative paths, like “cache/”, will ignore all “cache” folders found within the backup root directory and is preferable when you want to exclude all cache folders on the server.

Backup explorer

The backup explorer is an advanced feature of the backup tooling that allows you to browse, recover or delete the backups that are currently present on the back upserver.

We have embedded the Midnight Commander (similar like the old Norton Commander for DOS) in this tool to provide an easy to use interface.

Upon starting this function, you are presented with an interface like this:


Tip: Some terminal clients support mouse interaction in terminal windows. If you use a client like Putty or the default Linux terminal (gpm support), all menu related items are also click-able with the mouse. This does not work however for the default OS X terminal.

There are 2 panes available, the left one shows the content on the backup account and the right pane shows the content of the server.

From here you can perform operations on the backup account like browsing, deleting or recovering backups. We will not cover the in-depth usage of this tool, since is generally used by expert users with specific knowledge about the backup process and related actions.

In addition, it's also possible to store data directly into this account, but note that these will not be added to your incremental backups and cannot be used for recovery using the recovery feature.

Navigation:
Here are some handy keys and combinations which help you navigate through the Midnight Commander:

[TAB] = Switch between the active pane, all actions that you start are run from the active pane.
[F5] = Copy files or folders from the active pane to the other pane.
[F6] = Move files or folders from the active pane to the other pane.
[F8] = Delete the current file/folder which is selected in the active pane (note that removal is recursive for folders!).
[F10] = Quit the Midnight Commander
[+] = Opens window in which items can be marked using the pattern you provide here.

Tips:
In order to copy or move data from one to the other pane, make sure that the destination pane is already in the correct folder, or else the files will be placed in the directory that is currently active in the destination pane.

When moving or copying folders that have subfolders make sure to check the checkbox “Dive into subdirs”.  All directories below the selected one will be copied or moved as well.

On some keyboards the function keys are mapped to other actions, like lowering the volume. In this case you have to press the “Fn” key on your keyboard together with the function key you wish to use, so for opening the copy dialog, issue <Fn> + <F5>.

Installer command line options

The script is provided with some command line options  This can be very useful for automation purposes.

The installer currently accepts the following:

--install
This will install the backupscript using the same technique as the easy installer and will make all necessary decisions for you.

This uses a predefined syntax and an example is found below:

--install [account] [password] [server] [weekly ret.] [monthly ret.] [sql dumps] [reports email] [send reports]

An example:
--install myaccount mypassword mybackupserver 0 1 Y my@email.nl N

--update
This will update the backup script on the given server; this will issue the same steps as with the graphical update, but shows less output.

--uninstall
This will completely uninstall the backup script from the server including the cronjob.

--show-quota
This will show the current quota usage statistics.

The quota report function will only work for CloudVPS backup servers.

--clear-logs
This will clear all available logs that reside in /var/log/backups/ and can save some space if big logs reside within this directory.

--edit-config
This will open and embed the nano editor with the backup config file opened, making it editable.

Please note that this option is for expert users only and requires understanding of the config file elements and their behavior.  Inappropriate settings may result in unsuccessful backups.

Backup command line options

The backup script itself also accepts certain command line options to manipulate it's working process:

--no-dumps
This starts the backup script and disables MySQL dumps for that session. This can be useful when you want to create or test the backup functionality where dumps are not needed.

--debug
This will start the script in debug mode showing all steps that are taken throughout the script. This is  useful when you are having trouble using the backup script or when errors are present.

Only one of the switches can be used with the script; if you use more than one, the first one given will be used. If you want to use both flags, then run bash in verbose mode by issuing:

bash -x /usr/local/bin/cloudvpsbackup --no-dumps
Logging

All provided backup scripts write all of their actions in the general syslog that can generally be found at the following 2 locations (depending on the OS):

/var/log/messages
/var/log/syslog

The following names are used for the scripts:

backup_script: the backupscript
backup_installer: the backup installer / configuration tool
backup_restore: the backup restore script
backup_patcher: the backup patching script

These names can be used to grep the whole backup log out of the syslog, see the example below for the backup script:

root@networkspace2:~# grep backup_script /var/log/messages
Feb 21 13:20:09 networkspace2 backup_script: Started the CloudVPS Backup Script version 2.1.0
Feb 21 13:20:09 networkspace2 backup_script: Loaded the backup configuration file
Feb 21 13:20:09 networkspace2 backup_script: Created backup lockfile
Feb 21 13:20:09 networkspace2 backup_script: Started MySQL dump procedure
Feb 21 13:20:10 networkspace2 backup_script: MySQL connection successful
Feb 21 13:20:10 networkspace2 backup_script: Cleared the MySQL dump directory: /var/sqlbackups/
Feb 21 13:20:11 networkspace2 backup_script: Database mysql dumped with a size of 136KB
Feb 21 13:20:12 networkspace2 backup_script: Database performance_schema dumped with a size of 1.8KB
Feb 21 13:20:13 networkspace2 backup_script: Database test dumped with a size of 444B
Feb 21 13:20:13 networkspace2 backup_script: Database test2 dumped with a size of 444B
Feb 21 13:20:13 networkspace2 backup_script: MySQL dump procedure completed, total size of the dumps is 152KB
Feb 21 13:20:15 networkspace2 backup_script: Starting rsync backup procedure
Feb 21 13:21:39 networkspace2 backup_script: There were no errors during the rsync backup procedure
Feb 21 13:21:39 networkspace2 backup_script: Backup lockfile removed
Feb 21 13:21:39 networkspace2 backup_script: The CloudVPS Backup Script has finished

As you can see, all actions are logged in the syslog. When backups are made every day, this log will provide a great insight in the overall backup process.

Glossary

Below is a list of common used terms and their explanation.

Retention
Retention is simply defined as how long the backup server will keep your data. In the backup script you have at least one whole week, with options to enable extra weekly and monthly retention for a maximum of 52 weeks and/or 12 months.

Incremental / Differential backup
The incremental style repository aims to make it more feasible to store backups from more points in time by organizing the data into increments of change between points in time. This eliminates the need to store duplicate copies of unchanged data. This starts out with a full backup (of all files) and after that, any number of incremental or differential backups are made for the current week and/or extra retention that is given.

SSH (public) key
SSH keys serve as a means of identifying oneself to an SSH server using public-key cryptography and challenge-response authentication. One immediate advantage this method has over traditional password authentication is that you can be authenticated by the server without ever having to send your password over the network. Anyone eavesdropping on your connection will not be able to intercept and crack your password because it is never actually transmitted. Additionally, using SSH keys for authentication virtually eliminates the risk posed by brute-force password attacks by drastically reducing the chances of the attacker correctly guessing the proper credentials.

Crontab / cronjob
Cron is a powerful job scheduler for GNU/Linux and many other operating systems. It automates recurring tasks by executing commands at a given time. It has a wide range of potential applications; most simple recurring tasks, from backups to e-mail retrieval, can be automated using cron, saving users time.

Bash
Bash is a Unix shell created as a free software replacement for the older Bourne shell (sh). With it's age of 24 years (released in 1989) it's widely available for almost every Unix/Linux platform including OS X.

Rsync
Rsync is a software application and network protocol for Unix-like systems with ports to Windows that synchronizes files and directories from one location to another while minimizing data transfer by using delta encoding when appropriate. Quoting the official website: “rsync is a file transfer program for Unix systems. rsync uses the 'rsync algorithm' which provides a very fast method for bringing remote files into sync.” An important feature of rsync not found in most similar programs/protocols is that the mirroring takes place with only one transmission in each direction. Rsync can copy or display directory contents and copy files, optionally using compression and recursion.

Midnight Commander (mc)
The Midnight Commander is a clone of the well-known Norton Commander that was commonly used in the days where DOS was still used. It features 2 panes where in each pane a different directory can be set. From there it's possible to delete, copy or move data between panes.

Midnight Commander is based on versatile text interfaces, such as Ncurses or S-Lang, which allow it to work on a regular console, inside an X Window terminal, over SSH connections, RS-232 interface (for embedded devices) and all kinds of remote shells.


Bugs, issues and feature requests

Bugs and issues:
Although we have spent a lot of time building and testing the installer and it's tools, it’s always possible that some bugs or uncovered errors are still present in the code.

Should you have found an issue or bug on which you cannot use the script as expected, please let us know by sending an email to support@cloudvps.nl with an explanation about when the error occurs. Please see to it that the system requirements  have been checked before sending in your report to make sure all dependencies are met.

When reporting errors, please supply the following items:
⁃ The actual error
⁃ Your operating system
⁃ What type of controlpanel you use (if any)
⁃ Version of the backupscript


Feature requests:
The tools can always be improved by expanding its usability and we always encourage users to send in their suggestions and tips. Please send them to support@cloudvps.nl and we will put your suggestion on the wish list.

Change logs:
We provide change logs online which indicate fixed bugs, added new features etc. Before filing in a bug or feature request, make sure to check the appropriate change log first to see if it's not already addressed in a released update.

The change logs can be found here:
http://download.cloudvps.com/pub/files/scripts/backup/

Support

CloudVPS Customers:
If you are looking for support regarding the installation, maintenance or recovering your data, check this manual and the available help in the installer first.

Should your question not be covered in these documents, please send an email to support@cloudvps.nl regarding your issue. Also check page 42 for more information on filing issues/bugs to us.

When emailing our support desk, please make sure that you explain in detail what the issue is regarding the backup, if it's possible to make a copy-paste or screenshot of the error that would be appreciated.

Non CloudVPS Customers:
We offer support for non CloudVPS customers, but is limited to the following:

  • Installation issues caused by our software.
  • Any type of bugs found in the script (only applicable if used on a supported system, see the software requirements).

For these types you can send an email to support@cloudvps.nl explaining your issue, see page 42 for more information on filing issues/bugs to us.

What we do not support:

  • Installation support on non CloudVPS (backup)machines.
  • Any misconfiguration errors caused by the user.  For support in this area a fee will be asked.
Errors and warnings

The script can produce several warnings and errors to state that some attention is required regarding the backup, although sometimes this is purely informational.

Some of these warnings are explained below with their appropriate solution:

rsync: failed to read xattr user.rsync.%stat for "<location>": Permission denied (13)

This error is being produced when that given file could not be read during the transfer. Usually this is safe to ignore when this only occurs once or twice. Should this notice be given every day it might be worth having a look at that folder or file to see whether there is something wrong. If the file is not needed for the backup then it's a good decision to place it on the exclude list.

--link-dest arg does not exist: <location>

This warning will appear when the script is used for the first time. Basically it's saying that it cannot find the incremental backup directory from yesterday, which is correct during the first run. This also may occur for the extra monthly and weekly retention if this has not been backupped at that time.

rsync: readlink_stat("<location>") failed: Permission denied (13)

This will occur when certain files during the backup could not be read by rsync. The most common errors are:
⁃ /etc/.shadow
⁃ /etc/.passwd
⁃ /home/<user>/.gvfs
If one of the above appears, please ignore it or place an exclude via the exclude management offered by the Installer.

/usr/local/bin/cloudvpsbackup: line 630: host: command not found

In this case you don't have the DNS utilities package installed, this can be fixed by installing the “bind-utils” for CentOS or “dnsutils” for Debian / Ubuntu.

FAQ

Is the backupscript / installer IPv6 compatible?
Yes, the script will connect using IPv6 if IPv6 is properly configured on the server where the script is used on. Also make sure that the used backup server is also capable of IPv6 connectivity.

The CloudVPS backup servers are all configured with IPv6.

Is it possible to only backup the SQL dumps?
Yes, to arrange for this you can use the advanced installer and point the backup root directory to “/var/sqlbackups/”.

Can I backup PostgreSQL databases?
Currently, we do not support PGSQL dumps within the backup script, however, the script will backup the raw PGSQL files if that directory is within the backup root directory.

Disclaimer

The scripts are provided as is and we cannot be held responsible for any data loss caused by improper usage of the scripts or a faulty configuration. You use these tools at your own risk.

Modification of the scripts is allowed, but we cannot guarantee that any future updates we release for our scripts will not overwrite your modifications.  If you rely on these, we strongly recommend disabling the automatic update function.

VPS Bestellen
VPS Bestellen