Differences

This shows you the differences between two versions of the page.

--- backup [2015/07/14 04:15]
slackermedia
+++ backup [2021/06/03 19:48] (current)
@@ Line 1: / Line 1: @@
-[[{arrowp.png|border:0;background:none;width:0;display:inline-block;position:absolute;top:0;left:0;}sox|]]
-[[{arrown.png|border:0;background:none;width:0;display:inline-block;position:absolute;top:0;margin-left:2.5em;}rescue|]]
-/*
+/* if they are here, we dont need to convince them to backup
 ======System Backups======
 The reason so many appreciate "the cloud" today is because of the assurance that their data is safe. It's not safety from the NSA or computer attacks that they appreciate, it's the safety from dying hard drives, because everyone knows that the cloud is redundant, and the cloud is automated. It just backs stuff up, all the time.
@@ Line 17: / Line 16: @@
 */
-=====rdiff-backup=====
+=====Backing Up with Rdiff-Backup=====
+<WRAP info>
+**TL;DR** \\
+. [[drives|Format a big drive]] as UDF. \\
+. Install ''rdiff-backup'' \\
+. ''rdiff-backup --exclude-globbing-filename $HOME/.excludes $HOME /path/to/drive'' \\
+. Automate that command with ''cron'' \\
+</WRAP>
 There are several good backup applications aimed at local and network backups. Some are geared toward sys admins managing entire VFX houses, others are geared toward the advanced home user, and still others are aimed at regular, humble, everyday artists like us. No matter the audience, the best solutions can be //automated// so that you don't have to remember to do it, or wait for huge data dumps when you finally do remember to backup. A steady, automated, "always on" backup scheme tends to be the smartest backup plan.
 The most straight-forward and flexible backup application available is [[http://rdiff-backup.nongnu.org|rdiff-backup]], available from [[http://slackbuilds.org]]. It has one dependency, ''librsync'', which is also available on slackbuilds.org.
+<WRAP important>
+This section assumes that you are interested in backing up your personal data, not the entire system, since the OS is free and can be re-constructed. \\
+\\
+There is a case for backing up the entire system (quicker recovery in the event of a full system disaster being the main one), and this //can// be done; just replace ''/home/$USER'' in this chapter with ''/'' and run your backups as root. See [[http://rdiff-backup.nongnu.org/docs.html]] for more information on what directories you should exclude when doing a full system backup.
+</WRAP>
 Rdiff-backup is a terminal command only; it has no graphical user interface. If you are not comfortable using the terminal, it's still ok to use rdiff-backup because the point of it is that you don't have to use it much; you automate it. That's what we'll do together here, and once you get through this chapter, you will never have to use rdiff-backup directly again, except to recover data but under those circumstances, you probably won't mind learning a new command or two.
@@ Line 28: / Line 41: @@
-======Rdiff-backup Recipes======
+======Rdiff-backup Plans======
 There are a few likely scenarios for backing up:
@@ Line 38: / Line 51: @@
   * [[#bkpcherrypick|You only want to backup a few important files as you generate them.]]
-Here are recipes on how to create and use each of those backup schemes in such a way that it is as automated as possible.
@@ Line 81: / Line 93: @@
 <code>
-LABEL=safehouse     /safehouse    udf   auto,user 0 0
+LABEL=safehouse     /safehouse    udf   user,rw  0 0
 </code>
@@ Line 87: / Line 99: @@
 </WRAP>
-**3.** Create a directory for your backups on the drive.
+**3.** Perform an initial backup.
 <WRAP indent>
-In addition to a backup directory, create an empty file that you can use as a test to make sure that the drive is present (this will be used later, during the [[#automate|automation]] phase).
+The first backup is the largest and slowest backup because it is copying everything over to your backup drive. Subsequent backups will be much much smaller, because it will only copy changes to that base layer.
-<code>
+It is vital that this initial backup is done the exact same way it's going to be done on all future backups, for many reasons, but essentially because you need to get confirmation that everything has been set up correctly //before// you try to automate it. If you do a backup with a drive connected to your computer, and then try to switch to a networked backup, you might do user permissions wrong, or file paths might be inconsistent, or any number of problems, and you won't ever realise it because you'll have automated the process, never knowing that all you have automated is failure.
-$ mkdir /safehouse/$USER
-$ touch /safehouse/go
-</code>
-At this point, you have created:
+Do not do that.
-  * A standard location for your backup destination: ''/safehouse'')
+Make your first backup the initial test of your backup process. If it fails, refine your process, troubleshoot, and try again. When you get it right, you'll know it, and you'll know exactly what to automate.
-  * A test file to help detect if the drive is accessible: ''/safehouse/go''
-  * A directory to hold your backups: ''/safehouse/$USER'' (where $USER has been converted by the computer to your username)
-</WRAP>
+The basic command to backup your home directory (where all of your personal data lives):
-**4.** Perform an initial backup.
+For a locally attached drive:
-The first backup is the largest and slowest backup because it is copying everything over to your backup drive. Subsequent backups will be much much smaller, because it will only copy changes to that base layer.
-The basic command to backup your home directory (where all of your personal data lives):
 <code>
-$ rdiff-backup -v 9 $HOME /safehouse/$USER
+$ rdiff-backup --terminal-verbosity 8 --print-statistics \
+--exclude-globbing-filelist /home/klaatu/.excludes /home/klaatu/ \
+/safehouse/klaatuWork/
 </code>
@@ Line 119: / Line 124: @@
 </WRAP>
-This command backs up your entire home directory to the ''/safehouse'' location. It will take a long time, so just let it work.
+Or for a server:
+<code>
+$ rdiff-backup --terminal-verbosity 8 --print-statistics \
+--exclude-globbing-filelist /home/klaatu/.excludes /home/klaatu/ \
+klaatu@10.1.1.11::/safehouse/klaatuWork/
+</code>
+<WRAP alert>
+Notice that the server path contains //two// ''::'' between the hostname and the destination path.
+</WRAP>
+This command backs up your entire home directory to the ''/safehouse'' location. Do not bother creating the destination file (in this example's case, the ''/safehouse/klaatuWork'' folder); ''rdiff-backup'' will [attempt] to create that for you and warn you if it cannot write to the drive.
+No matter what, your first backup will take a long time, so just let it work.
 </WRAP>
@@ Line 145: / Line 164: @@
 </code>
-If nothing is listed, then the drive has not mounted successfully. Verify that you have labelled it properly with the ''vid'' flag (if using UDF as recommended; if you are using ext{2-4}, try using ''e2label'', and also question yourself about why you are not heeding Slackermedia's advice about UDF).
+If nothing is listed, then the drive has not mounted successfully. Verify that you have labelled it properly with the ''vid'' flag (if using UDF as recommended; if you are using ext{2-4}, try using ''e2label'').
-**2.** Verify that you have access to the drive:
+**2.** Change a file or add a new test file, and then run the your backup command again.
 <code>
-$ touch /safehouse/go || echo "Nope"
+$ rdiff-backup -v 8 /home/klaatu/ /safehouse/klaatuWork/
 </code>
-Silence indicates success. If you get "Nope" in response, then your ''/etc/fstab'' entry is incorrect, or you are not using UDF as recommended.
+This time, the process is relatively fast, since it has only copied over the changes.
-**3.** Now that everything is set, try changing a file or adding a new test file, and then run the your backup command.
-<code>
-$ rdiff-backup -v 9 $HOME /safehouse/$USER
-</code>
-This time, the process is quite fast, since it has only copied over the changes.
 ☛ You might have exclusions that you don't want to have backed up (for instance, do you ever want your trash bin backed up?). For this, read the section on [[#excludes|exclude]] options.
@@ Line 194: / Line 204: @@
 The danger of using this easy method is that if you have your computer set to backup every 4 hours, for example, and you attach your drive but only leave it attached for 3.75 hours, then you miss a backup cycle. Keep doing that, and you may end up with far fewer backups than expected. It practise, it's better than nothing, and as long as you remember to leave your laptop on and awake overnight with your backup drive attached, you are sure to hit one or two backup cycles (depending on how often you sleep).
+{{anchor:udev}}
 ====udev (the "I spent my weekend with udev" solution)====
@@ Line 207: / Line 218: @@
 <code>
-#!/usr/bin/bash
+#!/bin/sh
-echo $date > /tmp/udev.log
+date > /tmp/udev.log
 </code>
@@ Line 234: / Line 245: @@
 With ''udevadm'' monitor running, plug in your backup drive and watch as all kinds of information is spewed out onto your screen. Notice, particularly, that the type of event is an ''ADD'' event. That's a good way of identifying what type of event we want.
-The ''udevadm'' monitor provides a lot of good info, but we can see it with prettier formatting with the command ''udevadm info'', assuming you know where your drive is currently located in your ''dev'' tree. If not, unplug and then plug your thumbdrive back in and then immediately issue this command:
+The ''udevadm'' monitor provides a lot of good info, but we can see it with prettier formatting with the command ''udevadm info -a -n'', assuming you know where your drive is currently located in your ''dev'' tree. If not, unplug and then plug your thumbdrive back in and then immediately issue this command:
 <code>
@@ Line 248: / Line 259: @@
 This will return a wealth of information. Focus on the first block of info, first.
-The feedback from ''udevadm info'' starts with the device specified by the ''devpath'', and then walks up the chain of parent devices. It prints for every device found, all possible attributes in the ''udev'' rules key format.
+The feedback from ''udevadm info -a -n'' starts with the device specified by the ''devpath'', and then walks up the chain of parent devices. It prints for every device found, all possible attributes in the ''udev'' rules key format.
 A ''udev'' rule must contain one attribute from one single "parent" device; parent attributes are things that describe a device from the most basic level, such as "it's something that has been plugged into a physical port" or "it is something with a size of XX" or "this is a removable device".
-That info, just under the ''udevadm info'' introductory paragraph, looks a little something like this:
+That info, just under the ''udevadm info -a -n'' introductory paragraph, looks a little something like this:
 <code>
@@ Line 295: / Line 306: @@
 If you see the date and time returned from ''/tmp/udev.log'', then the script has worked.
 </WRAP>
 **Refining the Rule into Something Useful**
@@ Line 338: / Line 350: @@
 Once you can comfortably confirm that your script is triggered by ''udev'' when you attach your backup drive, turn your attention to the functionality of the script. Right now it is useless, doing nothing more than logging the fact that it has been executed.
-Here is an example backup-on-attach script. You can use it exactly as is EXCEPT for the last command; you need to change the path of the directory you want backed up (unless your username is also ''klaatu'') and the destination:
+Here is an example backup-on-attach script. You can use it exactly as is EXCEPT for the last command; you need to change the path of the directory you want backed up (unless your username is also ''klaatu'') and the destination (unless you also named your destination folder ''klaatu''):
 <code>
@@ Line 350: / Line 362: @@
     sleep 5
     if [ -b /safehouse ]; then
-        if [ -w /safehouse/go ]; then
+        if [ -w /safehouse/klaatu ]; then
             # everything is ok
             COUNT=9
@@ Line 395: / Line 407: @@
 If you have a home server, or a server away from home that you can physically access, you might prefer to do your backups remotely. The advantage to remote backups is that you neither get tied down by having a hard drive attached to your computer, nor do you have to remember to attach it if you like a mobile lifestyle. If your server is off-site, then you have some insurance against disaster, since the likelihood of both your personal computer and your off-site server facing destruction is less likely when they are not physically located in the same building.
-You should have gone through the setup steps before starting this process, so you should have a hard drive in the UDF format with the initial backup of your data on it. Your server should have a ''/safehouse'' mount point on it, and the drive should be set as mountable in ''fstab''.
+You should have gone through the setup steps before starting this process, so you should have a hard drive in the UDF format with the initial backup of your data on it. Your server should have a ''/safehouse'' mount point on it, and the drive should be set as mountable in ''fstab''. Both your client and your server must have ''rdiff-backup'' installed.
+<WRAP alert>
+For rdiff-backup to work remotely, //both// your client (the machine whose data is being backed up) and the server (the computer receiving the backup data for storage) must have ''rdiff-backup'' installed. That's **two** installs of ''rdiff-backup''.
+</WRAP>
 **1.** Mount the drive on your server:
@@ Line 412: / Line 428: @@
 <code>
-$ rdiff-backup -v 9 $HOME username@your-server.info::/safehouse/username
+$ rdiff-backup -v 8 $HOME username@your-server.info::/safehouse/username
 </code>
@@ Line 427: / Line 443: @@
 =====Backup to Someone Else's Server=====
-Backing up to a server that you do not own or do not have physical access to can be a little tricky, because you may not have access to it in such a way that remote backups are allowed. On the other hand, you might; it depends on what you are paying for, or what the sys admin is providing you.
+Backing up to a server that you do not own or do not have physical access to can be tricky, because you may not have access to it in such a way that remote backups are allowed. Besides, the remote server must also have ''rdiff-backup'' installed on it, and it may not have that.
+On the other hand, you might; it depends on what you are paying for, or what the sys admin is providing you.
 If you have ''ssh'' access to the server, then you can treat it the same as [[#bkpserver|your own private server]] in terms of setup.
-Other methods of accessing the server will likely not work with ''rdiff-backup'' because of the way ''rdiff-backup'' diffs changes and records metadata about these changes. You might look into other backup solutions, like [[http://luckybackup.sourceforge.net]].
+Other methods of accessing the server will likely not work, since ''rdiff-backup'' must be on the remote server, and needs ssh access.
+You might look into other backup solutions, like [[http://luckybackup.sourceforge.net]].
@@ Line 437: / Line 457: @@
 =====Backup a la Carte=====
-Some people generate more data than others. If you are a writer or an artist dealing in small file formats and find that it's easier to just backup files as you create them, then that's a valid backup option.
+Some people generate more data than others. If you are, for example, a writer or any kind of artist dealing in small file formats, and you find it's easier to just backup files as you create them, then that's a valid backup option...as long as you remember to do it.
-Consider purchasing a large-capacity thumbdrive and using it as either your master work directory or as you [[#bkpconnect|plug-and-backup]] destination.
+Consider purchasing a large-capacity thumbdrive and using it as either your master work directory or as your [[#bkpconnect|plug-and-backup]] destination. That way, every time you plug your drive into your computer, it gets backed up; you no longer have to remember to do it manually!
+If you are using rdiff-backup, you can backup a single file between backup script occurrences:
+<code>
+$ rdiff-backup --include $HOME/foo.txt --exclude '**' $HOME /safehouse
+</code>
+Another option is to wield the power of the Internet, and use a good git service. There are several simple clients for git, including the very appealing [[http://sparkleshare.org|SparkeShare]] or, if you have access to a server and the time to configure it, [[http://owncloud.org|ownCloud]]. These both have Dropbox-style sync folders.
@@ Line 495: / Line 522: @@
 <code>
-*/6 * * * rdiff-backup --exclude-globbing-filelist /home/klaatu/.excludes /home/klaatu/ klaatu@example.local:/safehouse/klaatu/
+*/6 * * * rdiff-backup --exclude-globbing-filelist /home/klaatu/.excludes /home/klaatu/ klaatu@example.local::/safehouse/klaatu/
 </code>
@@ Line 547: / Line 574: @@
 When prompted to create a passphrase, **leave the passphrase empty**. If you give you ssh key a passphrase, then you are defeating the purpose of using an ssh key for this use case.
+If you already have an ssh key on your source machine, then you can use that. If, however, that key requires a password, then it's useless for automated backups. Create a new key, exclusively for the backup server. For example:
+<code>
+$ ssh-keygen
+Generating public/private rsa key pair.
+Enter file in which to save the key (/home/klaatu/.ssh/id_rsa): /home/klaatu/.ssh/safehouse_rsa
+Enter passphrase (empty for no passphrase): LEAVE THIS EMPTY
+Enter same passphrase again: LEAVE THIS EMPTY
+Your identification has been saved in /home/klaatu/.ssh/safehouse_rsa.
+</code>
+Now configure ''ssh'' to use your new ''safehouse_rsa'' key whenever you attempt to login to your backup host.
+Open ''~/.ssh/config'' in your favourite text editor. If that file does not exist (it probably won't) then create it. Here are two examples of valid configurations, one for a server with a fully-qualified domain name, and one for an internal home server:
+<code>
+host slackermedia.info
+        hostname slackermedia.info
+        identityfile /home/klaatu/.ssh/safehouse_rsa
+        protocol 2
+host 10.1.1.11
+     hostname 10.1.1.11
+     identityfile /home/klaatu/.ssh/safehouse_rsa
+     protocol 2
+</code>
 Once your ssh key has been generated, copy it to your server:
 <code>
-$ ssh-copy-id klaatu@example.com
+$ ssh-copy-id -i /home/klaatu/.ssh/safehouse_rsa klaatu@example.com
 </code>
-Now you have password-less entry into your server.
+Test it out by ssh'ing into the server: You should not be prompted for a password.
+<code>
+$ ssh klaatu@10.1.1.11"
+</code>
+Assuming success, you now have password-less entry into your backup server (and so does rdiff-backup, since it will be running under your username).
@@ Line 597: / Line 656: @@
 rdiff-backup /safehouse/klaatu/rdiff-backup-data/increments/novel.2014-11-24T02:21:41-07:00.diff.gz ~
 </code>
+=====Dummy Checks=====
+It's not a bad idea to check in on your backups every now and again, just to verify that what you think is happening is really happening.
+To check your backups, log into your backup server or attach your backup drive, and have a look at the latest ten (or so) backup files created:
+<code>
+$ find /path/to/backup/dir -type f -printf "%C+ %p\n" | sort -rn | head -n 10
+</code>
+Adjust your backup scheme as necessary, depending on what you discover.
@@ Line 604: / Line 676: @@
 There are several good article on ''rdiff-backup''. [[http://www.kirya.net/articles/backups-using-rdiff-backup]] is quite good.
-[[{arrown.png|border:0;background:none;width:0;display:inline-block;float:right;}rescue|]][[{arrowp.png|border:0;background:none;width:0;float:right;}sox|]]
+<WRAP centeralign>
+<wrap fa>[[sox|R]]</wrap> <wrap fa>[[start|S]]</wrap> <wrap fa>[[rescue|Q]]</wrap>
+</WRAP>