[ Date Index ] [ Thread Index ] [ <= Previous by date / thread ] [ Next by date / thread => ]
On 28/04/14 23:48, Julian Hall wrote:
On 28/04/14 18:09, bad apple wrote:Another excellent tutorial I have copied into a text file for further reading and/or future reference - as I did with the Mint network upgrade one you did a while ago.On 27/04/14 23:13, Julian Hall wrote:On 27/04/14 20:10, Julian Hall wrote:Hi All, I've been trying different GUI based programs for backup and had most luck with 'luckyBackup', however it's not /that/ lucky because no matter what Exclusions I give it, it keeps trying to add the content of drives that are currently mounted when I've told it to excludethem. I had a go with dd tonight but got this error after a few minutes:sudo dd if=/dev/sda8 of=/media/julian/ARTEMIS/systembackup.img dd: reading ‘/dev/sda8’: Input/output error 17130448+0 records in 17130448+0 records out 8770789376 bytes (8.8 GB) copied, 257.292 s, 34.1 MB/s /dev/sda8 is the root of my Mint installation, ARTEMIS is a USB external HD. Should I try 'luckyBackup' again but with other drives unmounted? Kind regards, JulianMany thanks to everyone who has replied to this, Grant, Kevin, Simon, Gordon and bad apple. The situation at present is; bad apple was right.. I am/was running dd from the active installation, and ermm.. no it didn't /seem/ to work. The block comments from Gordon explain why the img that did result is 8.8Gb when I don't have anything near that used.. 8.8Gb is nearer the size of the partition not the amount of data. So, the USB drive now has an 8.8Gb img file that /may/ have my system in it, and two copies of my Mint installation, one from luckyBackup - I did try doing that with the other partitions unmounted. One copy in the backup.root directory that rsync created. However, given that I was executing rsync from the running installation I'm not sure even now if any of these are actually valid. I'll try bad apple's advice booting with a Live CD with Gordon's rsync instructions and see what happens. Kind regards, JulianOk chief, hang on a sec here - you're kind of going off half cocked on this, what you probably want to do is settle down for a bit, have a good think about what you're actually trying to achieve and then make a plan. At the moment you're sort of flailing around without a good understanding of your tools or what your end goal is. Tools first: "dd" is a program designed to be used offline to take sequential, bit for bit copies of an entire partition or disk (it actually stands for "disk dump"). You need to run it from a live bootable media or from a dual boot OS that resides outside of the disk or partition you intend to run dd from. dd is stupid, old fashioned and doesn't understand things like compression or free space: if you dd an entire 120Gb SSD, the resulting file will also be exactly 120Gb because dd will blindly copy all the zeroes as well as the ones - this is by design. dd is less commonly used as a backup tool and more in the creation and movement of disk images for VMs, image deployment, sysprepping, etc. Rsync is a much newer and more sophisticated version of "cp" - it most certainly can be run from within the system you are backing up whilst 'online' and understands things like file locking semantics, hard and soft links, incremental versions, etc. In backup strategies, rsync is usually the chief executable involved and there are countless tutorials, scripts and howtos all over the internet on how to backup your system(s) using rsync. It's almost definitely what you want to use for sane, incremental file backups that are easily searchable and located at somewhere convenient like /media/julian/ARTEMIS/my-rsync-backup-folder. Analogous to this Windows has VSS (Volume Shadow-copy Services) and Apple have TimeMachine which use pretty much identical methods for user backups. Pretty much all backup tools, commercial or FLOSS, use a combination of these two approaches (frequently, even in the proprietary ones, they are using *exactly* these two tools). For example, the commonly used Acronis backup/imaging tool will offer to create either one-off bootable images (dd) or standard differencing incremental filesystem backups that are non-bootable but that contain complete copies of your user-defined file trees to pluck individual files from at will (rsync). Whilst the details might differ slightly, every single backup tool operates on these two principles (excepting tar, the Tape ARchiver, which I will strategically ignore here because there is no way you have a £2g+ LTO unit at home). Here is how you want to use these two tools. I'll use my current workstation here as an example, just because it's easy. Before too long, I will have to stop being lazy and upgrade my Ubuntu 13.10 OS to 14.04. Before I do that, as per usual I will take my backups (it's the one time I relax my admittedly laissez faire attitude toward my own not-actually-that-important stuff). In this workstation, I have: sda: 120Gb SSD (GPT, one / partition, no swap, no separate /home) sdb: 1Tb HDD (GPT, 4 partitions for win8.1 and minimal debian OS) sdc: 1Tb HDD (GPT, Mac 10.9 install) sdd: 2Tb HDD (MSDOS, 1 linux ext4 data partition for "stuff") iscsi: multiple LUNs on SAN in garage, 30Tb+ usable Before doing an in-place upgrade, which will probably not work too well, I will boot my workstation from the decidedly minimal Debian OS that lives on my machine at /dev/sdb[3,4]. From there, I will use dd to dump an entire bit-for-bit perfect copy of my entire SSD containing Ubuntu to either my data drive or the server downstairs: sudo dd if=/dev/sda of=/$BACKUP_PATH/failbot-ssd-28.04.2014.img Because the SSD at /dev/sda isn't running an active OS at this point, this will create for me a complete snapshot of my working system that can be copied to other machines, cloned back to the SSD (thus overwriting it) if my upgrade completely borks or run through a P2V, etc process. As my boot drives' partitioning structure is unusually simplistic (most people will have a /boot partition, swap space and maybe a separate /home for a total of at least 4 partitions on their linux install) the resulting image is actually surprisingly accessible: to access files from it I can simply mount the entire image as a loopback device and it will present itself as a single 120Gb filesystem at an arbitrary mount point. For people with traditional, more complex partitioning (aka: suckers), let alone people like yourself with dual or tri-boot systems that may have 8 or more partitions on a single disk, you can still access all the available files easily enough from your dd-created image by using a tool such as kpartx that will automate the correct loopback mount points: sudo kpartx -l /media/ghost/btrfs/ssd-ubuntu-13.04-270114.dd loop0p1 : 0 217665536 /dev/loop0 2048 loop0p2 : 0 2 /dev/loop0 217669630 loop0p5 : 0 16771072 /dev/dm-1 2 >From here, I can call mount to access the relevant loop0pX device and get at my old data, although this is not what one might call user-friendly or intuitive. Also bear in mind that I'm dealing here with big fat all or nothing 120Gb entire disk images - unless you're made of money, taking entire massively inefficient non-versioned images of your entire boot drive is going to destroy your backup media capacity in very short order. Really, you only ever want one on disk at any given point. Of course, if you're not using a smallish SSD (and if not, why not?) as your bootable OS drive you're immediately going to have bigger problems - if you boot from a 4Tb HDD, obviously you're going to need at least 4Gb of backup space to hold that massive 4Gb dd image file, which is not really a good plan. You can of course just dd individual partitions but then you have the headache of making them bootable, remembering which sdaX is which, etc. So, on to rsync for more 'normal' backups. Rsync is an endlessly versatile tool with a lot of options and a great man page that you should *really* read before you set to work with it. On Unix, pretty much every backup tool leverages it behind the scenes for the heavy lifting so you might as well just cut out the middleman and use it directly. Here is a one of the better tutorials on how to use it: https://wiki.archlinux.org/index.php/Full_system_backup_with_rsync Rsync is ultimately just a clever version of "cp" that you can point at almost any source and destination that will duplicate a file structure for you. The result won't be a fully bootable complete image but will be trivially searchable, easy to access without complex stuff like loopback mounting and called from a script or cron, will take care of itself with no user interaction and definitely no monkey business like booting from a USB stick or backup OS install. This makes it, generally speaking, a much more appealing solution than dd for average backup purposes. Particularly via scripting or an apt-gettable backup program like bacula, etc, rsync can even be fed a list of locations (/dev/pts), filetypes (.tmp), mountpoints and so on that it should ignore completely to avoid wasting space attempting to duplicate stuff you don't want or need in your backups. This is definitely the path you want to go down from here on in to maintain backups for your system. Here is another psuedo-random link to a scripted, drop-in rsync approach to full system backups:http://www.pointsoftware.ch/howto-local-and-remote-snapshot-backup-using-rsync-with-hard-links/This one is clever enough to follow user-defined space limitations, do full incremental copies, save space using hard/soft links and scavenge space by deleting old or duplicated files. You could do a lot worse than having a good read through it, dropping it into testing for a week or two and making sure you've got a reasonable handle on exactly what it's doing before you make the leap to trusting it completely with your somewhat critical task of saving your data from disaster. In your case, you would probably want to nuke *all* of your old, crappy, probably not functional 'backups' from your external USB drive to free up space, create a new $BACKUP folder and then start synchronising via rsync the important folders from your various operating systems (rsync can certainly also backup your XP/Win7 files safely too by the way, as long as they are mounted and accessible from the Mint install where you are running the rsync script from). You probably really only want the contents of your /home and maybe /etc and /var directories at most to be copied - as long as your backups are good, in the case of disaster it's almost inevitably quicker for a relatively experienced user to just reinstall the OS from scratch and then copy the relevant files back into position from backups than messing around with complete disk images. Right, I hope all that makes sense - I didn't intend to write a screed on backups but hey ho, I seem to have anyway. Some closing notes: 1: Easy, GUI-style backup tools are crap. Always. Don't use them. 2: The exception to rule 1 is Apple TimeMachine, which you can't use. 3: TEST YOUR BACKUPS BY RESTORING THEM. This should be rule 0 actually. That's it really. There is actually another very highly used backup method which I have purposely completely ignored here as it's much more complicated, requires a ton more hardware and experience and is highly unlikely to be of immediate use to you. In the enterprise, we use LVM, BTRFS/ZFS or the built-in, usually highly proprietary methods available in commerical filers, SANs, etc to do something called snapshotting. Unless you want the headaches associated with modern-era COW filesystems, commercial crapware like EMC and frighteningly ballooning space and RAM requirements you should feel free to completely ignore this path. As usual, if you have any specific questions feel free to ask. CheersMany thanks for that! You're right about my partitions too - currently I have: Partition 1; /dev/sda1 - Windows XP Partition 3 (Extended partition); /dev/sda5 - Windows 7/dev/sda6 - Windows 'My Documents' partition.. saves losing the lot or having to muck about pulling the disk out and shoving it in a caddy if Windows faceplants and there is no choice but to reinstall /dev/sda7 - Used for video files - not particularly needed to be backed up./dev/sda8 - Mint / /dev/sda9 - Mint /home1.4Mb space (? No idea what that's doing there but it's negligible so not worth messing about resizing partitions to gain it unless I'm doing something else as well)Partition 3: dev/sda2 - Linux swap space As for where /dev/sda4 is - ummm... that blank space? Not a clue.In a nutshell then it would seem sensible to backup /dev/sda1, 5, 6, 8 and 9?And yes.. they're all on one 1Tb hard disk. Probably not the greatest idea, therefore getting them backed up takes on a wee bit of urgency.I hadn't thought of backing up the Windows installations from Mint - why I'm not quite sure, it seems obvious now :)Kind regards, Julian
Hi All,Following all your advice and bad apple's detailed explanation above I have managed to backup my Mint root /, /home, and the Windows XP partition. The Windows 'My Documents' partition is currently backing up now.
However, the Windows 7 partition is not going so well. After several hours backing up, it terminated with the following error:
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1070) [sender=3.0.9]
Error 23 seems to be 'partial transfer due to error', but what I need to understand is /what/ error? There is plenty of space on the partition it's backing up to - I purchased a SATA 2.5" 1Tb HD and caddy, partitioned it in half and am backing up Windows to 1 500Gb partition and Mint to the other, the Windows partition has about half free; I say 'about half' due to the current backup mentioned above still being underway; it won't be full even when that's finished, but as Windows 7 was backed up /before/ this one it had even more space available when it crashed. The rsync error says 'see previous errors', where would I look for that please?
Kind regards, Julian -- The Mailing List for the Devon & Cornwall LUG http://mailman.dclug.org.uk/listinfo/list FAQ: http://www.dcglug.org.uk/listfaq