Shrink-backup: a backup tool

Edit: now also works on ArchLinuxARM for rpi

I am very proud to present my absolute first contribution on github.
With a little help from some members here on the forum I finally got it ready to share.

shrink-backup is a very fast utility for backing up your SBC:s into minimal bootable img files for easy restore with autoexpansion at boot

_ I made this script because I wanted a universal method of backing up my SBC:s into small img files as fast as possible (with rsync), indepentent of what os is in use._

Autoexpansion tested on Raspberry Pi os, Armbian, Manjaro-arm and ArchLinuxARM for rpi with ext4 root partition.

Very fast restore because of minimal size of img file.

Can back up any device as long as root is ext4
Default device that will be backed up is detected by scanning what disk-device root resides on.
This means that if boot is a partition, that partition must be on the same device as root.
Backing up/restoring to/from usb-stick (/dev/sda) with Raspberry pi os has been tested and works.
Wrinting a sd-card img to a usb-stick and vice versa has also been tested and works.

Ultra-fast incremental backups to existing img files

Don’t forget to make the script executable if you git clone

To restore a backup, simply “burn” the img file to a device using your favorite method.

Usage:

sudo shrink-backup -h
Script for creating an .img file and subsequently keeing it updated (-U), autoexpansion is enabled by default
Directory where .img file is created is automatically excluded in backup
########################################################################
Usage: sudo shrink-backup [-Uatyelh] imagefile.img [extra space (MB)]
  -U         Update the img file (rsync to existing backup .img), no resizing, -a is disregarded
  -a         Let resize2fs decide minimum space (extra space is ignored), disabled if using -U
  -t         Use exclude.txt in same folder as script to set excluded directories
             One directory per line: "/dir" or "/dir/*" to only exclude contents
  -y         Disable prompts in script
  -e         DO NOT expand filesystem when image is booted
  -l         Write debug messages in logfile shrink-backup.log in same directory as script
  -h --help  Show this help snippet
########################################################################
Example: sudo shrink-backup -a /path/to/backup.img
Example: sudo shrink-backup -e -y /path/to/backup.img 1000
Example: sudo shrink-backup -Ut /path/to/backup.img

The folder where the img file is created will ALWAYS be excluded in the backup.
If -t option is selected, exclude.txt MUST exist (but can be empty) within the directory where the script is located or the script will exit with an error.

Use one directory per line in exclude.txt.
/directory/* = create directory but exclude content.
/directory = exclude the directory completely.

If -t is NOT selected the following folders will be excluded:

/lost+found
/proc/*
/sys/*
/dev/*
/tmp/*
/run/*
/mnt/*
/media/*
/var/log.hdd
/var/swap

Rsync WILL cross filesystem boundries, so make sure you exclude external drives unless you want them included in the backup.

Use -l to write debug info into shrink-backup.log file in the same directory as the script.

Applications used in the script:

  • fdisk (sfdisk)
  • dd
  • parted
  • e2fsck
  • truncate
  • mkfs.ext4
  • rsync

Info

Theoretically the script should work on any device with maximum 2 partitions (boot and root).
The script can handle maximum 2 partitions, if there are more than that on root device the script will fail with an error.
Even if you forget to disable autoexpansion on a non supported system, the backup will not fail. :slight_smile:

Order of operations - image creation

  1. Reads the block sizes of the partitions
  2. Uses dd to create the boot part of the system + a few megabytes to include the filesystem on root (this can be a partition)
  3. Removes and recreates the root partition, size depends on options used when starting the script
  4. Creates a new ext4 filesystem with the same UUID and LABEL as the system you are backing up from
  5. Uses rsync to sync both partitions (if more than one)

Added space is added on top of df reported “used space”, not the size of the partition. Added space is in MB, so if you want to add 1GB, add 1024.

The script can be instructed to set the img size by requesting recomended minimum size from e2fsck by using the -a option.
This is not the absolute smallest size you can achieve bit is the “safest” way to create a “smallest possible” img file.
If you do not increase the size of the filesystem you are backing up too much, you can most likely keep it updated with the update function (-U) of the script.

Smallest possible image

To get the absolute smallest img file possible, do NOT set -a option and set “extra space” to 0

Example: sudo shrink-backup /path/to/backup.img 0

This will instruct the script to get the used space from df and adding 192MB “wiggle room”.
If you are like me, doing a lot of testing, rewriting the sd-card multiple times. The extra time it takes each time will add up pretty fast.

Example:

-rw-r--r-- 1 root root 3.7G Jul 22 21:27 test.img # file created with -a
-rw-r--r-- 1 root root 3.3G Jul 22 22:37 test0.img # file created with 0

Disclaimer:
Because of how filesystems work, df is never a true representation of what will actually fit on a created img file.
Each file, no matter the size, will take up one block of the filesystem, so if you have a LOT of very small files (running docker f.ex) the “0 added space method” might fail during rsync. Increase the 0 a little bit and retry.
This also means you have VERY little free space on the img file after creation.
If the filesystem you back up from increases in size, an update (-U) of the img file might fail.

Order or operations - image update

  1. Probes the img file for information about partitions
  2. Mounts root partition with an offset for the loop
  3. Checks if multiple partitions exists, if true, loops the boot with an offset and mounts it within the root mount
  4. Uses rsync to sync both partitions (if more than one)

To update an existing img file simply use the -U option and the path to the img file.
Changing size in an update is not possible at the moment but is in the todo list for the future.

Thank you for using my software <3

A backup is not really a backup until it has been restored.

6 Likes

Update:

The script can now be used on other devices by using the -d option. (used to be the debug option, new option for that is -l
Experimental so keep that in mind. I do not have great hardware to test this functionality on at the moment.

And also woks on ArchLinuxARM, but not the autoexpansion, I am still working on it but it is close.

Information in OP is edited.

Attention, attention, version 0.9.1 released!! come get your free copy! xD

A lot has happened:

  • device being backed up is what root resides on, ie works on any device, not just mmcblk. no need to specify, script is smart enough to figure it out.
  • ArchLinuxArm autoexpansion now supported without the need for any AUR downloads, only parted is needed.
  • Tested to update with same image though kernel and boot changes on ManjaroArm and no issues detected.
  • Generally a more user friendly interface and the debug function is beyond explanatory. hint hint

Hope you find it useful!

2 Likes

Big thanks for this one! So far I’ve been using Image File Utilities but it always choked on /var/lib/docker. This one however works flawlessly.

1 Like

That was the one I was using too, but Armbian pushed me into creating my own script. I have been using a smaller version of this script on my sbc:s for a very long time, just that when I also included ManjaroArm I pushed myself to actually clean up the code and make a release and implement all functions into one script rather than a few.

As far as I can find, the closest application to this is the one you link to and is the inspiration for this project, but as you state, this covers more.
Almost all other backup apps does full dd and then compresses the img with zip or smthn making the process very slow compared to this. Also hard to write an image to a card that is smaller than the initial backup.
I have been thinking of implementing compression in this, but It would slow things down significantly each time I want to run an update on the img, it would have to be uncompressed and compressed again. I feel the user can manage that if they want to, because compression on these images really does magic! (rar is by far the most efficient AND fastest in all my testing on these images)

The next step is to implement shrinking/expanding of an existing img file, you can find the development in the testing branch if you are curious. Nothing is done yet (I released this version today), but will start soon so keep an eye there if you are interested.

I’m really glad you found use for this, after all, the reason to share is to help out. :hearts:

I agree, I, for one, certainly prefer faster updates. I wish you best of luck in your future endeavors!

1 Like