Backup helper for btrfs send/receive

It is easy to use btrfs with its send/receive to backup one snapshot to an external drive. But it is real work, if you want do do this for several snapshots repeatedly. Even when you want to make these snapshots differential. …

I did program this for my own backup, but it may be useful for others.


[HowTo] Backup btrfs snapshots with send/receive

Goals:

Simply a backup of a complete system (only the snapshots)

  • Back up all snapshots
  • Differential backup of each snapshot (fast)
  • Use as little storage space on the backup medium as possible (compressed)
  • Differential backup (when repeating the backup after a few days/weeks/months)
  • The backup medium can be used for backups of different computers

No goal:

  • Automatic management of backups by age
  • Automatically delete old backups when there is not enough space
  • Backup of the current state of the subvolume

Requirements:

  • BTRFS on both, the computer and the backup media
  • snapper layout of snapshots
  • Java 11 or newer on the computer
  • Recommended: pv installed

Back snap:

The Java program Backsnap saves ALL snapshots that have been specified to a backup medium. Therefore it uses btrfs send and btrfs receive

The 1st passed parameter points to the SOURCE path where the snapshots
are reachable. Snapper puts all snapshots in directories with ascending numbering. The actual snapshot inside this directory is simply called “snapshot”.

  • /.snapshots
  • /home/.snapshots

The 2nd parameter points to where the snapshots are to be saved. To do this, the backup medium needs to be mounted"special (subvol=/)". It needs a subvolume called @snapshots and a directory named for this PC (for example the hostname). The path to this directory is specified as the TARGET path for the backup.

  • /mnt/BACKUP/@snapshots/manjaro
  • /mnt/BACKUP/@snapshots/manjaro.home
  • /mnt/BACKUP/@snapshots/notebook
  • /mnt/BACKUP/@snapshots/notebook.home

Backsnap goes through all the numbered directories in the source path in ascending order and checks wether this directory already exists at the destination. If not, the snapshot will be send there. If possible, a previous snapshot is used as a “parent”.

Each time the program is called, all snapshots of ONE subvolume can be backed up corresponding to ONE configuration of snapper.

Recommendations:

  • Create a btrfs Volume on your external drive with a subvolume called @snapshots
  • Create a shell script called /usr/local/bin/backup that handles the entire backup.
  • Mount the backup volume with options subvol=/,compress=zst:9

I am willing to adapt this to support timeshift if desired :footprints:

P.S. The responsibility for backups never lies with a program, but always with the user!

1 Like

The sourcecode as java-shellscript :wink: (under GPL):

#!/bin/java --source 11
// © 2022 Andreas Kielkopf --> License GPL
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStreamReader;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.TreeMap;

public class Backsnap {
   static String               parent=null;
   static String               dest;
   final static ProcessBuilder pb    =new ProcessBuilder();
   public static void main(String[] args) {
      String source=(args.length > 0) ? args[0] : "/.snapshots";
      dest=(args.length > 1) ? args[1] : "/mnt/BACKUP/@snapshots/manjaro";
      System.out.println("Backup Snapshots from " + source + " to " + dest + " ");
      try {
         TreeMap<String, File> sfMap=getMap(source);
         TreeMap<String, File> dfMap=getMap(dest);
         for (String sourceName:sfMap.keySet())
            try {
               backup(sourceName, sfMap, dfMap);
            } catch (NullPointerException n) {
               n.printStackTrace();
            }
      } catch (FileNotFoundException e) {
         e.printStackTrace();
      }
   }
   static void backup(String sourceName, TreeMap<String, File> sfMap, TreeMap<String, File> dfMap)
            throws FileNotFoundException {
      String name=sfMap.get(sourceName).getName();
      if (dfMap.containsKey(sourceName) && dfMap.get(sourceName).toPath().resolve("snapshot").toFile().exists()) {
         parent=sourceName;
         return;
      } else
         if (!Paths.get(dest, name).toFile().mkdirs())
            throw new FileNotFoundException("Could not create dir: " + name);
      System.out.print("Backup of " + name);
      StringBuilder cmd=new StringBuilder("/bin/btrfs send ");
      if (parent != null) {
         System.out.print(" based on " + parent);
         cmd.append("-p ");
         cmd.append(sfMap.get(parent).getPath());
         cmd.append("/snapshot ");
      }
      System.out.println();
      cmd.append(sfMap.get(sourceName).getPath());
      cmd.append("/snapshot ");
      if (Paths.get("/bin/pv").toFile().canExecute())
         cmd.append("| /bin/pv -f ");
      cmd.append("| /bin/btrfs receive ");
      cmd.append(Paths.get(dest, name).toFile().getPath());
      execute(cmd.toString());
      parent=sourceName;
   }
   static void execute(String cmd) {
      try {
         final ArrayList<String> command=new ArrayList<>();
         command.add("/bin/bash");
         command.add("-c");
         command.add(cmd);
         try (BufferedReader br=new BufferedReader(
                  new InputStreamReader(pb.command(command).redirectErrorStream(true).start().getInputStream()))) {
            br.lines().forEach(System.out::println);
         }
      } catch (final IOException e) {
         e.printStackTrace();
      }
   }
   final static TreeMap<String, File> getMap(String name) throws FileNotFoundException {
      File dir=Paths.get(name).toFile();
      if (!dir.isDirectory())
         throw new FileNotFoundException(name);
      TreeMap<String, File> fileMap=new TreeMap<>();
      for (File file:dir.listFiles())
         if (file.isDirectory()) {
            System.out.print(file.getName() + " ");
            String s=".".repeat(10).concat(file.getName());
            fileMap.put(s.substring(s.length() - 10), file);
         }
      System.out.println();
      return fileMap;
   }
}

Save this under /usr/local/bin/Backsnap and make it executable. Do not name it Backsnap.java !

chmod a+x /usr/local/bin/Backsnap

Make a backup-script at /usr/local/bin/backup like the following example:

#!/bin/sh
# Backup to USB-disk
UUID=03417033-3745-4ae7-9451-efafcbb9124e 
BACKUP=/mnt/BACKUP 
# mount /mnt/BACKUP 
mount -o subvol=/,compress=zstd:9 /dev/disk/by-uuid/$UUID $BACKUP
# use Hostname
NAME=$HOST
# is this dir available ?
[ -d $BACKUP/@snapshots/$NAME ] || { echo "Das mounten war wohl nicht erfolgreich"; exit; }
# backup snapshots of / 
Backsnap /.snapshots $BACKUP/@snapshots/$NAME
sync
# Backup snapshots of /home
Backsnap /home/.snapshots $BACKUP/@snapshots/$NAME.home 
sync
umount $BACKUP

Have fun with your backups :wink:

Thanks for your work!

There is btrbk for BTRFS backup using btrfs send | btrfs receive.

  • Its config in /etc/btrbk/btrbk.conf is more flexible and supports any different Btrfs layouts.
  • Enable systemd timer btrbk.timer for automatic backup every day or every week.
  • It automatically deletes outdated backups, that depends on your config.

It is available in AUR.

1 Like

btrbk seems to be a really good solution for btrfs snapshots and backups.

It’s a bit heavy for me

I’m already running snapper, which manages the snapshots with its own strategy per configuration.
It was about quickly backing up ALL the local snapshots onto an external drive that is momentary connected.

  • There should be no duplicates,
  • the backup drive should be able to be used for several computers and, above all,
  • it should be able to hold as many snapshots as possible. (compression)
    :footprints:

@andreas85 Cheers so much for the initiative. I would love to see it set for timeshift.

I recently stubbled upon something for KDE, although not as sophisticated as what you are proposing.

The KDE service is this one: BTRFS Subvolume Manager Service Menu for Dolphin

Just mentioning that here because it may be useful somehow.