Shrink-backup: a backup tool

Edit: now also works on ArchLinuxARM for rpi

I am very proud to present my absolute first contribution on github.
With a little help from some members here on the forum I finally got it ready to share.

shrink-backup is a very fast utility for backing up your SBC:s into minimal bootable img files for easy restore with autoexpansion at boot

_ I made this script because I wanted a universal method of backing up my SBC:s into small img files as fast as possible (with rsync), indepentent of what os is in use._

Autoexpansion tested on Raspberry Pi os, Armbian, Manjaro-arm and ArchLinuxARM for rpi with ext4 root partition.

Very fast restore because of minimal size of img file.

Can back up any device as long as root is ext4
Default device that will be backed up is detected by scanning what disk-device root resides on.
This means that if boot is a partition, that partition must be on the same device as root.
Backing up/restoring to/from usb-stick (/dev/sda) with Raspberry pi os has been tested and works.
Wrinting a sd-card img to a usb-stick and vice versa has also been tested and works.

Ultra-fast incremental backups to existing img files

Don’t forget to make the script executable if you git clone

To restore a backup, simply “burn” the img file to a device using your favorite method.

Usage:

sudo shrink-backup -h
Script for creating an .img file and subsequently keeing it updated (-U), autoexpansion is enabled by default
Directory where .img file is created is automatically excluded in backup
########################################################################
Usage: sudo shrink-backup [-Uatyelh] imagefile.img [extra space (MB)]
  -U         Update the img file (rsync to existing backup .img), no resizing, -a is disregarded
  -a         Let resize2fs decide minimum space (extra space is ignored), disabled if using -U
  -t         Use exclude.txt in same folder as script to set excluded directories
             One directory per line: "/dir" or "/dir/*" to only exclude contents
  -y         Disable prompts in script
  -e         DO NOT expand filesystem when image is booted
  -l         Write debug messages in logfile shrink-backup.log in same directory as script
  -h --help  Show this help snippet
########################################################################
Example: sudo shrink-backup -a /path/to/backup.img
Example: sudo shrink-backup -e -y /path/to/backup.img 1000
Example: sudo shrink-backup -Ut /path/to/backup.img

The folder where the img file is created will ALWAYS be excluded in the backup.
If -t option is selected, exclude.txt MUST exist (but can be empty) within the directory where the script is located or the script will exit with an error.

Use one directory per line in exclude.txt.
/directory/* = create directory but exclude content.
/directory = exclude the directory completely.

If -t is NOT selected the following folders will be excluded:

/lost+found
/proc/*
/sys/*
/dev/*
/tmp/*
/run/*
/mnt/*
/media/*
/var/log.hdd
/var/swap

Rsync WILL cross filesystem boundries, so make sure you exclude external drives unless you want them included in the backup.

Use -l to write debug info into shrink-backup.log file in the same directory as the script.

Applications used in the script:

  • fdisk (sfdisk)
  • dd
  • parted
  • e2fsck
  • truncate
  • mkfs.ext4
  • rsync

Info

Theoretically the script should work on any device with maximum 2 partitions (boot and root).
The script can handle maximum 2 partitions, if there are more than that on root device the script will fail with an error.
Even if you forget to disable autoexpansion on a non supported system, the backup will not fail. :slight_smile:

Order of operations - image creation

  1. Reads the block sizes of the partitions
  2. Uses dd to create the boot part of the system + a few megabytes to include the filesystem on root (this can be a partition)
  3. Removes and recreates the root partition, size depends on options used when starting the script
  4. Creates a new ext4 filesystem with the same UUID and LABEL as the system you are backing up from
  5. Uses rsync to sync both partitions (if more than one)

Added space is added on top of df reported “used space”, not the size of the partition. Added space is in MB, so if you want to add 1GB, add 1024.

The script can be instructed to set the img size by requesting recomended minimum size from e2fsck by using the -a option.
This is not the absolute smallest size you can achieve bit is the “safest” way to create a “smallest possible” img file.
If you do not increase the size of the filesystem you are backing up too much, you can most likely keep it updated with the update function (-U) of the script.

Smallest possible image

To get the absolute smallest img file possible, do NOT set -a option and set “extra space” to 0

Example: sudo shrink-backup /path/to/backup.img 0

This will instruct the script to get the used space from df and adding 192MB “wiggle room”.
If you are like me, doing a lot of testing, rewriting the sd-card multiple times. The extra time it takes each time will add up pretty fast.

Example:

-rw-r--r-- 1 root root 3.7G Jul 22 21:27 test.img # file created with -a
-rw-r--r-- 1 root root 3.3G Jul 22 22:37 test0.img # file created with 0

Disclaimer:
Because of how filesystems work, df is never a true representation of what will actually fit on a created img file.
Each file, no matter the size, will take up one block of the filesystem, so if you have a LOT of very small files (running docker f.ex) the “0 added space method” might fail during rsync. Increase the 0 a little bit and retry.
This also means you have VERY little free space on the img file after creation.
If the filesystem you back up from increases in size, an update (-U) of the img file might fail.

Order or operations - image update

  1. Probes the img file for information about partitions
  2. Mounts root partition with an offset for the loop
  3. Checks if multiple partitions exists, if true, loops the boot with an offset and mounts it within the root mount
  4. Uses rsync to sync both partitions (if more than one)

To update an existing img file simply use the -U option and the path to the img file.
Changing size in an update is not possible at the moment but is in the todo list for the future.

Thank you for using my software <3

A backup is not really a backup until it has been restored.

6 Likes

Update:

The script can now be used on other devices by using the -d option. (used to be the debug option, new option for that is -l
Experimental so keep that in mind. I do not have great hardware to test this functionality on at the moment.

And also woks on ArchLinuxARM, but not the autoexpansion, I am still working on it but it is close.

Information in OP is edited.

Attention, attention, version 0.9.1 released!! come get your free copy! xD

A lot has happened:

  • device being backed up is what root resides on, ie works on any device, not just mmcblk. no need to specify, script is smart enough to figure it out.
  • ArchLinuxArm autoexpansion now supported without the need for any AUR downloads, only parted is needed.
  • Tested to update with same image though kernel and boot changes on ManjaroArm and no issues detected.
  • Generally a more user friendly interface and the debug function is beyond explanatory. hint hint

Hope you find it useful!

2 Likes

Big thanks for this one! So far I’ve been using Image File Utilities but it always choked on /var/lib/docker. This one however works flawlessly.

1 Like

That was the one I was using too, but Armbian pushed me into creating my own script. I have been using a smaller version of this script on my sbc:s for a very long time, just that when I also included ManjaroArm I pushed myself to actually clean up the code and make a release and implement all functions into one script rather than a few.

As far as I can find, the closest application to this is the one you link to and is the inspiration for this project, but as you state, this covers more.
Almost all other backup apps does full dd and then compresses the img with zip or smthn making the process very slow compared to this. Also hard to write an image to a card that is smaller than the initial backup.
I have been thinking of implementing compression in this, but It would slow things down significantly each time I want to run an update on the img, it would have to be uncompressed and compressed again. I feel the user can manage that if they want to, because compression on these images really does magic! (rar is by far the most efficient AND fastest in all my testing on these images)

The next step is to implement shrinking/expanding of an existing img file, you can find the development in the testing branch if you are curious. Nothing is done yet (I released this version today), but will start soon so keep an eye there if you are interested.

I’m really glad you found use for this, after all, the reason to share is to help out. :hearts:

I agree, I, for one, certainly prefer faster updates. I wish you best of luck in your future endeavors!

1 Like

A long time has passed since I updated this post and a lot has happened.

I figured I post an update here since the thing I’m working with right now is btrfs, and I use Manjaro-arm as the base for this.

I will just copy/paste what is said about btrfs here. I can no longer edit the original post so some of the info might be incorrect in OP.
Please see GitHub - UnconnectedBedna/shrink-backup: A utility to backup SBC:s (like Raspberry pi) into minimal bootable img files for correct description of the script.

Or directly to testing branch if you want to test the btrfs functionality: GitHub - UnconnectedBedna/shrink-backup at testing

Hopefully someone might find it useful. Btrfs functionality only on testing branch.

btrfs

This is still in experimental stage so ideas & feedback is HIGHLY appreciated!
The subvolumes are mounted with default compression: compress=zstd (default means zstd:3)

I am working against Manjaro-arm to create this functionality and the standard install creates root (/@) and home (/@home) subvolumes, so the script assumes this is the situation on ALL btrfs systems as of now.

The backup img is NOT a clone. Snapshots are NOT used to create the backup.
The UUID will change on the created img filesystem (btrfs is way more picky than ext4 about this), but in the case of Manjaro (and raspberry pi too for that matter), that does not matter since PARTUUID is used in mounting, and that stays the same, but users should be aware.
Subvol id:s are NOT guaranteed to be the same.

Instead of using btrfs send/recieve I opted for rsync, quck and dirty.
Both in creation of a new img and when keeping it updated with -U.
My resoning for this is that this script is primarily for creating bootable img files, NOT to create perfectly cloned backups. Speed is also a strong argument here.

The goal in developement of this script is ALWAYS to: as fast as possible create an img file that you can write directly to a sd-card and boot. That goal does NOT mix well with also creating a perfectly cloned backup.
This does mean the script cares MORE about the file integrity rather than the disk integrity. The compression f.ex might be different than on your root filesystem. Subvol id:s might change etc etc.
But the main goal stays the same, the backup must contain ALL REQUESTED FILES, ie a bootable file backup. I do NOT want to be responsible for people loosing their data when using this script, hence this decision. :slight_smile:

All of this might change in the future though. Not the rsync part (I value speed very high), but the subvol id:s, compression and such is on my mind.
F.ex if more subvols (or less) than root and home is used I want the script to be able to handle that.

1 Like