[HowTo] Helper to backup btrfs snapshots with send/receive

andreas85 · 12 September 2022 06:45

It is easy to use btrfs with its send/receive to backup one snapshot to an external drive. But it is real work, if you want do do this for several snapshots repeatedly. Even when you want to make these snapshots differential. …

I did program this for my own backup, but it may be useful for others.

[HowTo] Backup btrfs snapshots with send/receive

Goals:

Simply a backup of a complete system (only the snapshots)

Back up all snapshots
Differential backup of each snapshot (fast)
Use as little storage space on the backup medium as possible (compressed)
Differential backup (when repeating the backup after a few days/weeks/months)
The backup medium can be used for backups of different computers

No goal:

Automatic management of backups by age
Automatically delete old backups when there is not enough space
Backup of the current state of the subvolume

Requirements:

BTRFS on both, the computer and the backup media
snapper layout of snapshots
Java 11 or newer on the computer
Recommended: pv installed

Back snap:

The Java program Backsnap saves ALL snapshots that have been specified to a backup medium. Therefore it uses btrfs send and btrfs receive

The 1st passed parameter points to the SOURCE path where the snapshots
are reachable. Snapper puts all snapshots in directories with ascending numbering. The actual snapshot inside this directory is simply called “snapshot”.

/.snapshots
/home/.snapshots

The 2nd parameter points to where the snapshots are to be saved. To do this, the backup medium needs to be mounted"special (subvol=/)". It needs a subvolume called @snapshots and a directory named for this PC (for example the hostname). The path to this directory is specified as the TARGET path for the backup.

/mnt/BACKUP/@snapshots/manjaro
/mnt/BACKUP/@snapshots/manjaro.home
/mnt/BACKUP/@snapshots/notebook
/mnt/BACKUP/@snapshots/notebook.home

Backsnap goes through all the numbered directories in the source path in ascending order and checks wether this directory already exists at the destination. If not, the snapshot will be send there. If possible, a previous snapshot is used as a “parent”.

Each time the program is called, all snapshots of ONE subvolume can be backed up corresponding to ONE configuration of snapper.

Recommendations:

Create a btrfs Volume on your external drive with a subvolume called @snapshots
Create a shell script called /usr/local/bin/backup that handles the entire backup.
Mount the backup volume with options subvol=/,compress=zst:9

I am willing to adapt this to support timeshift if desired

P.S. The responsibility for backups never lies with a program, but always with the user!

andreas85 · 12 September 2022 07:01

The sourcecode as java-shellscript (under GPL):

#!/bin/java --source 11
// © 2022 Andreas Kielkopf --> License GPL
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStreamReader;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.TreeMap;

public class Backsnap {
   static String               parent=null;
   static String               dest;
   final static ProcessBuilder pb    =new ProcessBuilder();
   public static void main(String[] args) {
      String source=(args.length > 0) ? args[0] : "/.snapshots";
      dest=(args.length > 1) ? args[1] : "/mnt/BACKUP/@snapshots/manjaro";
      System.out.println("Backup Snapshots from " + source + " to " + dest + " ");
      try {
         TreeMap<String, File> sfMap=getMap(source);
         TreeMap<String, File> dfMap=getMap(dest);
         for (String sourceName:sfMap.keySet())
            try {
               backup(sourceName, sfMap, dfMap);
            } catch (NullPointerException n) {
               n.printStackTrace();
            }
      } catch (FileNotFoundException e) {
         e.printStackTrace();
      }
   }
   static void backup(String sourceName, TreeMap<String, File> sfMap, TreeMap<String, File> dfMap)
            throws FileNotFoundException {
      String name=sfMap.get(sourceName).getName();
      if (dfMap.containsKey(sourceName) && dfMap.get(sourceName).toPath().resolve("snapshot").toFile().exists()) {
         parent=sourceName;
         return;
      } else
         if (!Paths.get(dest, name).toFile().mkdirs())
            throw new FileNotFoundException("Could not create dir: " + name);
      System.out.print("Backup of " + name);
      StringBuilder cmd=new StringBuilder("/bin/btrfs send ");
      if (parent != null) {
         System.out.print(" based on " + parent);
         cmd.append("-p ");
         cmd.append(sfMap.get(parent).getPath());
         cmd.append("/snapshot ");
      }
      System.out.println();
      cmd.append(sfMap.get(sourceName).getPath());
      cmd.append("/snapshot ");
      if (Paths.get("/bin/pv").toFile().canExecute())
         cmd.append("| /bin/pv -f ");
      cmd.append("| /bin/btrfs receive ");
      cmd.append(Paths.get(dest, name).toFile().getPath());
      execute(cmd.toString());
      parent=sourceName;
   }
   static void execute(String cmd) {
      try {
         final ArrayList<String> command=new ArrayList<>();
         command.add("/bin/bash");
         command.add("-c");
         command.add(cmd);
         try (BufferedReader br=new BufferedReader(
                  new InputStreamReader(pb.command(command).redirectErrorStream(true).start().getInputStream()))) {
            br.lines().forEach(System.out::println);
         }
      } catch (final IOException e) {
         e.printStackTrace();
      }
   }
   final static TreeMap<String, File> getMap(String name) throws FileNotFoundException {
      File dir=Paths.get(name).toFile();
      if (!dir.isDirectory())
         throw new FileNotFoundException(name);
      TreeMap<String, File> fileMap=new TreeMap<>();
      for (File file:dir.listFiles())
         if (file.isDirectory()) {
            System.out.print(file.getName() + " ");
            String s=".".repeat(10).concat(file.getName());
            fileMap.put(s.substring(s.length() - 10), file);
         }
      System.out.println();
      return fileMap;
   }
}

Save this under /usr/local/bin/Backsnap and make it executable. Do not name it Backsnap.java !

chmod a+x /usr/local/bin/Backsnap

Make a backup-script at /usr/local/bin/backup like the following example:

#!/bin/sh
# Backup to USB-disk
UUID=03417033-3745-4ae7-9451-efafcbb9124e 
BACKUP=/mnt/BACKUP 
# mount /mnt/BACKUP 
mount -o subvol=/,compress=zstd:9 /dev/disk/by-uuid/$UUID $BACKUP
# use Hostname
NAME=$HOST
# is this dir available ?
[ -d $BACKUP/@snapshots/$NAME ] || { echo "Das mounten war wohl nicht erfolgreich"; exit; }
# backup snapshots of / 
Backsnap /.snapshots $BACKUP/@snapshots/$NAME
sync
# Backup snapshots of /home
Backsnap /home/.snapshots $BACKUP/@snapshots/$NAME.home 
sync
umount $BACKUP

Have fun with your backups

Zesko · 12 September 2022 14:09

Thanks for your work!

There is btrbk for BTRFS backup using btrfs send | btrfs receive.

Its config in /etc/btrbk/btrbk.conf is more flexible and supports any different Btrfs layouts.
Enable systemd timer btrbk.timer for automatic backup every day or every week.
It automatically deletes outdated backups, that depends on your config.

It is available in AUR.

andreas85 · 12 September 2022 15:31

btrbk seems to be a really good solution for btrfs snapshots and backups.

It’s a bit heavy for me

I’m already running snapper, which manages the snapshots with its own strategy per configuration.
It was about quickly backing up ALL the local snapshots onto an external drive that is momentary connected.

There should be no duplicates,
the backup drive should be able to be used for several computers and, above all,
it should be able to hold as many snapshots as possible. (compression)

clmbtti · 12 September 2022 18:29

@andreas85 Cheers so much for the initiative. I would love to see it set for timeshift.

I recently stubbled upon something for KDE, although not as sophisticated as what you are proposing.

The KDE service is this one: BTRFS Subvolume Manager Service Menu for Dolphin

Just mentioning that here because it may be useful somehow.

andreas85 · 26 December 2022 10:48

I made it a github project

[HowTo] Backup btrfs snapshots with send/receive

Prologue

A snapshot is not a backup! see@ btrfs docs

I’ve read that before. Correct in principle, but a btrfs snapshot is as good as any other In-system backup. Btrfs can bring a high level of security against data loss for home users. With some additional work, an external backup for btrfs can also be achieved.

Some improvements:

command-line only
only 2 parameters to do a backup
supports backup of other computers on the network over ssh
no config necessary
batchfile for easy backup suggested
batchfile example included
executable jar (when java is installed )

(It is written in java not because java is the best, but because i do “speak” java)

andreas85 · 10 February 2023 07:06

A post was split to a new topic: I have a ideal :

apptainer · 10 February 2023 07:30

I think send/receive like http is for files, send/receive like btrfs is for directory . The purpose is not only for backup, ít can become www version 2.0

eugen-b · 10 February 2023 08:29

That applies if a snapshot is on the same drive. Moving it to a different drive makes it a backup.

andreas85 · 10 February 2023 11:05

Send/receive for btrfs does create a readonly snapshot after it is finished.
Send/receive for btrfs can use a “parent” snapshot to reduce the size of the data to be transfered

So this is no replacement for rsync see@btrfs docs

The backup(-snapshot) has a new UUID, but is also linked to its orginal-snapshshot by UUID !

andreas85 · 19 June 2023 17:25

Now it is time to anounce:

A new Betaversion of BackSnap is avaliable for usage !

It supports timeshift as was suggested

Andreas