Server:housekeeping:backups
From Linux How-To Repository
Contents |
The Script + Mutt Routine
Backing Up Data
Good back up programs, such as backupPC, are freely available to you on Ubuntu, but you can also create solid and reliable back up routines using simple, easy-to-manage scripts. These are quick and easy to set up, are run automatically, and can back up your files remotely--provided they are not too large--by sending them to you as an email attachment. Sound like what you are looking for? Then, follow these easy steps and you'll never be caught out, in despair saying, "if only ..."
The Case Scenario
For demonstration purposes, let’s look at a scenario where we want a back up routine that runs automatically every day, backs up specific SQL databases, and then zips them up and emails them. OK, no problem. We’ll make a script that creates some SQL file dumps and moves them into a common folder. Then, it will compress that folder, and also copy it to another directory on the hard disk as a further back up precaution. Finally, it will use the Mutt email client to send off the compressed folder as a file attachment, so that it can be stored remotely in an email account.
For this to work, you need to have a mail service running, such as Postfix.
First Install Mutt
Install the Mutt client that will transfer emails via the command line:
sudo apt-get install mutt
You should now configure Mutt so that it does not create an ever growing "sent" file in your home or root directory, a file that will eventually max out your partition. So, create a .muttrc configuration file in your /home/user directory:
sudo gedit .muttrc
In that put something like the following, altered for your specifications:
set realname="Linux Geek"
set from="linuxgeek@example.com set use_envelope_from=yes
set mbox_type=maildir set mbox="~/muttmail/inbox/" set spoolfile="~/muttmail/inbox/" set folder="~/muttmail/" set record="~/muttmail/sent/" set postpone=ask-yes recall=no postponed=~/muttmail/postponed/
Save the file and copy it to /root. The script will be run under root, so Mutt will look for /root/.muttrc. But when testing, you could just run the script under your user account, in which case Mutt will look for ~/.muttrc. Either way, with .muttrc in both locations, all of Mutt’s sent mail will go to the one folder you nominate.
We don't need to configure anything else because we are only using Mutt for back up purposes here. Actually, if you put in other configuration settings, they will be ignored because Mutt is going to be run from a script, which is equivalent to running it from the command line, and Mutt seems to ignore command line arguments. I learned that the hard way after much trail and error. The basic stuff here, nominating the directories, is enough and will work.
Then create the directories that you nominated in .muttrc, plus, in each of those, you need to create directories called "cur," "tmp" and "new," for example, like this
sudo mkdir ~/muttmail/sent/cur sudo mkdir ~/muttmail/sent/new sudo mkdir ~/muttmail/sent/tmp
Actually, if you don't create a sent folder, Mutt will automatically create one for you when it sends its first email. But you at least need the "muttmail" directory, or whatever you want to call it.
If you already have email setup, for instance, if you have Courier installed, you could nominate your existing mail directories in .muttrc and not have to create any new ones. Another option, which you’ll need to find information on elsewhere, is to configure Mutt to use IMAP and POP folders on a remote server.
If you decide to use existing email directories in your home directory, be aware that "sent" items created from a cron job will have restrictive root permissions, and this may mess up the webmail program you are using by rendering sent items invisible. It did that to me when using Roundcube. I fixed that by making the script change the permissions in my "sent" directory as a final command:
chmod 777 -R /home/user/Maildir/.Sent
But you may not even care to have a record of sent items. After all, these will accumulate day after day, and might have large attachments. In that case, get rid of the “set record . . .” line in .muttrc and replace it with the following:
unset record
Before we move onto the main script, we can first run a little test script to see if everything is working with Mutt. Modify it for your email address and copy this little script into a file:
#!/bin/sh # Mutt Test # EMAIL="ugeek@example.com" MESSAGE='Hey, your mutt test has been run!' echo $MESSAGE echo $MESSAGE | mutt $EMAIL -s "Mutt Test"
Save the file under any name you like-- it doesn't have to have a "sh" extension. Then run it from the command line to test it, like this:
sudo sh filename
Or
sh filename
If it came through, it’s time to configure your back up script.
The Script
Now copy the SQL Back Up Script below into a file. Modify the variables, databases and directory names to suit. Of course, you’ll need your SQL details handy.
# !/bin/sh # SQL Back Up Script # # Backs up your SQL databases and emails them to you # ## DBUSER=username DBPASS=password DBNAME1=joomla DBNAME2=drupal DBNAME3=wordpress EMAIL="linuxgeek@example.com" DATE=`date +%Y%m%d` mysqldump --opt -u $DBUSER -p$DBPASS $DBNAME1 > $DBNAME1.sql mysqldump --opt -u $DBUSER -p$DBPASS $DBNAME2 > $DBNAME2.sql mysqldump --opt -u $DBUSER -p$DBPASS $DBNAME3 > $DBNAME3.sql mv *.sql /home/user/database_backups tar cvzf /home/user/databases-$DATE.tgz /home/user/database_backups echo 'Hey, see attached for your databases' | mutt -a /home/user/databases-$DATE.tgz $EMAIL -s 'Your database back up has been run' mv /home/user/databases-$DATE.tgz /var/local/backups
As with the little test script above, save and name the script file whatever you like, then test it from the command line.
The steps in this script are straight forward. SQL dumps are created and moved to a back up folder in your home directory. The whole folder is then zipped up (using the format tar cvzf output input). Next, the tgz file, with the current date in the file name, is emailed to the email address you nominate at EMAIL. Lastly, the tgz file is moved to another location on your hard disk as a further back up.
So, altogether, this script actually creates 3 back ups, two local and one remote. Apart from the tgz file you copy and send, you are left with the SQL files in your database_backups folder. These will be overwritten with new versions the next time the script is run.
The beauty of this script is that you can change the variables at the top or easily add databases (remembering to also add corresponding mysqldump lines) and everything will just work.
As with the little test script above, save and name the script file whatever you like, then test it from the command line. Once your script is doing everything you want it to, copy it into your /etc/cron.daily directory. Make sure it has executable permissions.
sudo chmod 755 filename
That’s it. Your back-up is set to go.
Some Notes On Anacron
A program called Anacron handles the running of the scripts in the /etc/cron.* directories. Each time scripts are run, if there is any output, Anacron sends an email with details about it to root. If no output, no email. With our script, there is basic output for some processes, so an email is sent to root and ends up in the main user account. This is because, by default, the /etc/aliases file contains a redirection that ensures any system mail to root goes to the main user’s account. But Anacron’s emails can be stopped or redirected.
The easiest way is probably to edit the /etc/anacrontab file. Open up the /etc/anacrontab file and at the top of it under PATH add a “mail to” direction with your email address, like this:
MAILTO=ugeek@gmail.com.
All Anacron emails will then go to that email address. Thus, it is possible to email your back ups to one address and receive an alert about it at another.
If you want to stop all Anacron emails, I’ve found that redirecting output to /dev/null works best for me. In your /etc/anacrontab file, leave out any reference to MAILTO and instead put “>&/dev/null” after the cron.daily line (or whatever line is applicable to your setup):
1 5 cron.daily nice run-parts --report /etc/cron.daily >& /dev/null
By the way, if you want to redirect not just Anacron’s but all of root’s email, then ignore /etc/anacrontab and just create a /root/.forward file with your preferred email address specified in that.
Other Things to Do
- Other SQL Commands
The basic SQL command we used could be varied. Here are some examples:
mysqldump [options] db_name [tables] mysqldump [options] --databases db_name1 [db_name2 db_name3...] mysqldump [options] –all-databases
You could create and compress SQL back ups directly too, using something like this command:
mysqldump --opt -u $DBUSER -p$DBPASS $DBNAME1 > $DBNAME1.sql | bzip2 -c > $DBNAME1.bz2
- Other Back Up Ideas
The SQL Back Up Script could also be used as the basis for back ups that zips up individual files or whole directories to another location.Of course, if you decide to compress individual files and directories and send them with emails, I know you’ll check what size they will be first, since you don’t want 20 MB email attachments!
- Incorporating Rsync
You could also “cp -a” directories to another part of your hard disk, another machine on your network, an external hard disk, or a USB device—you name it. Alternatively, you could use Rsync to synchronize directories. I have one script that does nearly everything--full of rsync commands, one after the other, that synchronizes my main box’s directories in various ways with 3 other hard disk locations on my local network. A script could also be configured to back up to remote machine using SSH and Rsync orscp.
See below for comprehensive details on the excellent Rsync.
- Keeping Things Tidy
Here's a tip for getting rid of old tgz files from your hard disk automatically. In the script above, add the following line below the DATE variable line:
EARLIER=`date -d "-2 day" +%Y%m%d`
You don't have to call it EARLIER. Call it whatever you want, as long as you are consistent throughout. Now, after the last line, add the line below, or its equivalent to suit your setup. This line will ensure that the back up file created 2 days earlier is deleted:
rm /var/local/backups/databases-$EARLIER.tgz
You can change it to 3 or 4 days, or a couple of weeks earlier, if you want, depending on how far back you want to keep things.
- Messing Around
Webmail programs like Google’s Gmail have filters, so it’s possible to archive backups as soon as they arrive. In Gmail, you can create a filter that labels and puts your backups in the Archives folder. You don’t have to worry about them. Speaking of Google, here’s something you might want to try, although it doesn’t seem of any practical use to me: it is possible to configure a script to email documents of the appropriate type to Google Docs or Zoho Writer, where they can be automatically converted to online documents, on a regular basis.
What you can do is all up to you. Using the example script I have given as a base, you can configure a whole range of back up routines to work with Ubuntu's cron.daily, cron.weekly and cron.monthly directories. If you know how to set up individual cron jobs, you can also ensure that your scripts are run at precise times. Ultimately, you can have everything covered, all your data safely backed up, so that in the event of some failure, what might have been disaster is little more than a minor inconvenience.
All About Rsync
Rsync speeds up file transfer by copying the differences between two files rather than copying an entire file every time. Rsync copies the differences only. Another benefit is that rsync can preserve permissions and ownership information, and copy symbolic links.
Rsync should already be installed on Linux by default.
Basic Rsync Use
Based on an article from Linux Magazine: Joe Brockmeier, July 17, 2007
Rsync Syntax
The basic syntax for rsync is simple enough -- just run rsync [options] source destination to copy the file or files provided as the source argument to the destination.
So, for example, if you want to copy some files under your home directory to a USB storage device, you might use this:
rsync -a /home/user/dir/ /media/disk/dir/.
By the way, "/home/user/dir/" and "/home/usr/dir" are not the same thing to rsync. Without the final slash, rsync will copy the directory in its entirety. With the trailing slash, it will copy the contents of the directory but won't recreate the directory. If you're trying to replicate a directory structure with rsync, you should omit the trailing slash.
In this example, the (-a) option combines several rsync options. It combines the recursive and copy symlinks options, preserves group and owner, and generally makes rsync suitable for making archive copies. Note that it doesn't preserve hardlinks; if you want to preserve them, you will need to add the hardlinks option (-H).
Another option you'll probably want to use most of the time is verbose (-v), which tells rsync to report lots of information about what it's doing. You can double and triple up on this option -- so using -v will give you some information, using -vv will give more, and using -vvv will tell you everything that rsync is doing.
rsync will move hidden files (files whose names begin with a .) without any special options. If you want to exclude hidden files, you can use the option
--exclude=".*/".
You can also use the --exclude option to prevent copying things like automatic backups (.bak) created by some programs.
Making local copies
Suppose you have an external USB or FireWire drive, and you want to copy data from your home directory to your external drive. A good way to do this would be to keep all your important data under a single top-level directory and then copy it to a backup directory on the external drive using a command like:
rsync -avh /home/usr/dir/ /media/disk/backup/
If you want to make sure that local files you've deleted since the last time you ran rsync are deleted from the external system as well, you'll want to add the --deleted option, like so:
rsync -avh --delete /home/user/dir/ /media/disk/backup
Be very careful with the delete option.
Making remote copies
What if you want to copy files offsite to a remote host? No problem -- all you need to do is add the host and user information. So, for instance, if you want to copy the same directory to a remote host, you'd use:
rsync -avhe ssh --delete /home/user/dir/ user@remote.host.com:dir/
If you want to know how fast the transfer is going, and how much remains to be copied, add the --progress option:
rsync --progress -avhe ssh --delete /home/user/dir/ user@remote.host.com:dir/
If you don't want to be prompted for a password each time rsync makes a connection -- and you don't -- make sure that you have rsync set up to log in using an SSH key rather than a password. To do this, create an SSH key on the local machine using ssh-keygen -t dsa, and press Enter when prompted for a passphrase. After the key is created, use ssh-copy-id -i .ssh/id_dsa.pub user@remote.host.com to copy the public key to the remote host.
What if you need to bring back some of the files you copied using rsync? Use the following syntax:
rsync -avze ssh remote.host.com:/home/user/dir/ /local/path/
The z option compresses data during the transfer. If the file you are copying exists on the local host, then rsync will just leave it alone -- the same as if you were copying files to a remote host.
Wrapping it up with a script
Once you've figured out what directory or directories you want to sync up, and you've gotten the commands you need to sync everything, it's easy to wrap it all up with a simple script. Here's a short sample:
rsync --progress -avze ssh --delete /home/user/bin/ user@remote.host.com:bin/ rsync --progress -avze ssh --delete /home/user/local/data/ user@remote.host.com:local/data/ rsync --progress -avze ssh --delete /home/user/.tomboy/ user@remote.host.com:/.tomboy/
Use the --progress option if you're going to be running rsync interactively. If not, there's no need for it.
Rsync Extras
Based on article by Jem Matzan, Apr 17, 2006. Thanks Jem!
Transferring Single Objects
The following applies if you have SSH running. You can copy files over a secure connection by using scp. The latter works just like the regular cp command does, except you have to give it an address for at least one of the files:
scp picture.jpg 192.168.1.101:/home/user/pictures/
In the above example, a file called picture.jpg is transferred to the network machine with the address 192.168.1.101 in the /home/user/pictures/ directory. By default, scp copies the target file or directory to the home directory of whatever user you're logged in as; you can specify a different location by adding the path after the colon following the address.
By using your /etc/hosts file, you can create a nickname for the remote machine you're copying files to. Just start a new line in the file, type in the IP address of the machine you want to nickname, then press the tab key once and type in the name you'd like to call it:
192.168.1.101 laptop
In the next example, we'll use the laptop nickname instead of the address, specify a different user than the one currently logged into the terminal we're copying from, and copy an entire directory instead of just one file:
scp -r /home/user/pictures/ user2@laptop:/home/user2/
The -r switch means recursive, which tells scp to copy the directory and everything in it. The above command will create a pictures directory to the /home/user/ directory on the laptop computer, and copy all of the contents of the local machine's /home/user/pictures/ directory to it. So what do you do if the remote machine already has a /home/user/pictures/ directory, but you still want to copy everything in the local machine's pictures directory? You use a wildcard:
scp /home/user/pictures/* laptop:/home/user/pictures/
Complex transfers
Rsync is a lot like scp, except it's designed for doing complex transfers. Like with OpenSSH, in order to use rsync for file transfers, you must start a daemon on the remote machine. The command is rsyncd, and you can of course add it to your operating system's startup script if you like. The examples below do not require the rsync daemon; they use OpenSSH for the file transfers instead, so you'll have to run the SSH daemon. You can specify which software to use for the file transfers by using either one (SSH) or two (rsync) colons after the remote host name or IP address. The examples that follow all use one colon.
The simplest way to use rsync is to synchronize the /home directories of two computers that have the same users and file system structures:
rsync -arvuz /home/user/ 192.168.1.101:/home/user/
Just like with SSH, you can use /etc/hosts to create a nickname for the IP address of the remote machine, and you can also specify other users with the @ symbol. The -arvuz flags mean (in order) that you're going to keep user and group file permissions; recursively copy the /home/user/ directory and all files and directories therein; verbosely show which files are transferred or updated; ignore identical files that have the same timestamp; and compress the data to use less bandwidth over the network.
What if you only want to transfer important files and directories -- not the whole /home/user/ directory? There are two ways. If you're copying the same directories in your /home dir each time, you can create a simple script to save yourself all of the typing. Open up your favorite text editor and create a file called sync_laptop.sh and put this into it, substituting your own directories for the ones in the example:
rsync -arvuz '/home/user/pictures /home/user/documents /home/user/jokes' laptop:
When it's saved, make it executable with chmod +x. The above command won't work with directories that are more than one level in, so /home/user/pictures/summer/ won't work. Neither can you copy files to anywhere other than the remote /home directory using the above example. You could add a new line to the script for each buried directory that you want to transfer, but there's a more efficient way to do it.
First, create a directory in your user's home dir called sync. Change to the sync directory and create symlinks to all of the directories that you want to transfer. Make sure you treat the sync dir as though it were your home directory when you create the destination links. You may have to create directories within the /home/user/sync/ dir if they are more than one level in:
ln -sf /home/user/documents ./documents mkdir .gconf mkdir .gconf/apps ln -sf /home/user/.gconf/apps/evolution ./.gconf/apps/evolution ln -sf /home/user/.evolution ./.evolution mkdir pictures mkdir pictures/summer ln -sf /home/user/pictures/summer ./pictures/summer
Now create the script that will do the transfers; call it sync_laptop.sh and put it in your user's home directory:
# This script syncs a remote computer to this one
cd /home/user/
# Uncomment the next command if you'd like to copy all of
# the files (not directories) in your home dir to the remote machine.
# cp * ./sync/
rsync -arLuvz /home/user/sync/ laptop:/home/user
The -L switch tells rsync to treat your symlinks as though they were real directories.
Save the script, then make it executable by using chmod +x on it. When you run the script, your laptop will be updated with all of those files and directories from your desktop machine. The first time you do this it will take a while, but each subsequent sync will take less time because rsync will not overwrite files that have not changed.
Alternative Methods
Method 1
Please note that I haven't tried this alternative yet so I can't vouch for it. I forget where I got it from. In any case, I use the above method.
Create the backup directory:
mkdir /home/backups
Copy these 2 scripts on your root directory or your user directory:
#!/bin/sh
# Script name : auto_mysql_dump.sh
# Backup the dbname database
dir=`date +%Y-%m-%d`
dbname=`mybase`
if [ -d /home/backups ]; then
mkdir /home/backups/$dir
mysqldump -B --user=user_of_my_base --password=pwd_of_my_base --host=host_of_my_base $dbname > /home/backups/$dir/$dbname.sql
fi
# End of script auto_mysql_dump.sh
Second script:
#!/bin/sh
# Script Name : auto_mail_dump.sh
# Sends an email with the dump realized before
dir=`date +%Y-%m-%d`
dbname=`mybase`
mutt -s "Today backup" -a /home/backups/$dir/$dbname.sql user@tosend.com < /dev/null
# End of script auto_mail_dump.sh
Don't forget to change the access to make them executable:
chmod 700 auto_mysql_dump.sh chmod 700 auto_mail_dump.sh
Edit the CronTab to schedule the execution of the two scripts.
gedit crontab -e
We consider that the 2 scripts are in the /root directory
The following will execute the dump at 8.30 everyday and send you an email at 9.00 everyday Add these 2 rows after the existing lines:
Hit the "i" to insert new characters ...
30 8 * * * /root/auto_mysql_dump.sh > /dev/null 00 9 * * * /root/auto_mail_dump.sh > /dev/null
Save the crontab
This might be helpful one day
When you need to import the data from a mysqldump, instead of using "shell>mysql < dump.sql" using "mysql> source dump.sql" is much better.
This way of importing the data avoids problems with language specific characters being turned into garble.
