Proper Backup Strategy

I’m running a triple-boot machine with Windows 10, Linux Mint, and Manjaro on a single, 2TB HDD.
Both Linux distros are sharing a separate, 70GB HDD for Timeshift snapshots.
I have another 500GB HDD I would like to use as a glorified /home directory, but one that is available to every OS I happen to have installed on the 2TB HDD.

  1. Is this even possible?
  2. The HDD currently has an older version of Mint installed on it. I assume I would need to wipe the disk and reformat it, but do I need to format it to particular filesystem for the use I have in mind?

Assuming this can be done, I would like to create incremental backups of the 500GB HDD to an external, 1TB HDD or two to protect against disk failure. OSs can be reinstalled if push comes to shove, but at least my personal data will be safe from anything affecting the OS drive or the entire rig. From what I understand, incremental backups usually assume a continuous connection to allow for scheduled syncs, but ideally, I would only connect my external whenever I wanted to sync the backup.

  1. What backup software would be best for this purpose?
  2. Am I going about this okay?

EDIT----------------------------------------------------------------------------------------------------------
Turns out I may have been going about this all wrong after all. I’ve discovered that I can actually access files on other OSs directly through the file manager. So creating a shared data partition would seem to be unnecessary. The question becomes how can I back up my data without creating 3 separate backup archives or is that precisely what I should do? I know that good backup policy is to have at least 3 copies of every file worth backing up, but I reckon each copy is meant to be saved to a different location (in my case, one internal HDD and two external). So how would I merge the backups from each OS without creating unnecessary duplications?

A point to note is that I avoid using the common folders (Documents, Videos, etc.) for long-term storage because my files are organized by association (School, Family, Work, Fanfiction, etc.), not filetype. I keep my data in a separate home directory called “Groups” on all three installations. Would it be easier to create a synced backup by just backing up the Group directory?

1 Like

You can make a big data partition, and/or symlink it to a folder like Downloads … but its is exceedingly unwise to try and share a HOME across different OS’s.

So I can’t treat it like a giant flashdrive?

No, thats perfectly fine. Something like a big Data drive.
I mean you shouldnt use it to share one HOME.

Okay. I meant just sharing the data, not an actual /home partition.

If Windows is involved ntfs would be recommendable.

You can keep separate HOME directories, but you can remap common folders like Video/Documents/Downloads/Music etc to keep all your private documents separate/shared.

I’m not sure about sharing with Windows. I’d probably set up another backup to rsync my ext4 (documents etc) directories on an NTFS partition every now and then.

  1. There is a way of sharing home directories between OSes but you need:

    • to know the ntfs-3g advanced settings (so NTFS disk only)
    • can only share Documents, Downloads, Videos, … folders (so not any of the hidden directories)
    • Need to change permissions from Windows only.
    • Need to keep all of your users and passwords synchronised.

    Basically, you want to become a professional NTFS-3G permissions admin. :+1:

  2. Read this for the perfect backup solution:

The example uses borg but you can use timeshift using the same philosophy although the examples are for BorgBackup (You will be assimilated and all that)

:innocent:

1 Like

So the cold system backup is for restoring the system, while data backups are for saving user files? It was my understanding that Timeshift was for system restores and not for data backups; which is why they exclude /home by default? Also, wouldn’t testing a cold system backup be just as dangerous as not if you do something wrong? I don’t have a second, test computer on hand.

yes

Timeshift requires a bootable system and is similar to Windows Restore Points. If you want to restore a totally crashed system with TimeShift, you need to re-install a minimal system first, then install Timeshift, then restore the latest backup.

A cold system backup requires you to keep a bootable CloneZilla USB disk around for the worst-case scenarios. (I.E. Totally crashed disk)

I’ve tested this for you and can guarantee that a restore will work even after you replace your computer’s disk with another one (equal or larger than the crashed one)

:man_shrugging:

Okay, so let me get this straight:

  1. I install CloneZilla to the disk.
  2. I make a disk image and save it to an external drive.
  3. In the event I need to restore the disk, it will be an exact copy of the disk at the time I imaged it.
  4. If my disk is damaged, presumably my CloneZilla installation is also gone. So I just create a bootable flashdrive, reinstall, and import the image onto the new disk.

Assuming I have that right, you say that the restore will be of a non-running system. Can you explain what you mean by that?

You also say that unimportant files should be

…symlinked to a drive/partition that does not get backed up so that you still back up your /home 's important configuration files and data files using CloneZilla.

What are these big, unimportant files exactly? I’m assuming they’re not the user config files. Unless you mean the user files like Pictures, Docs, Videos…? I can understand putting those on a separate drive (where the stuff about ntfs-3g comes into play) and symlink them to /home, while leaving the configs in place? Then you use a data backup to backup of the symlinked files to a separate external disk. Do I have that right.?

Change “reinstall” to “boot it” (that’s all you have to do: just boot the CloneZilla Live USB and then restore your image: once that’s done everything, including CloneZilla is back!)

Videos:I can always remaster them from the original DVD (though it’s a pain in the butt)

Nope, I don’t back up the videos at all as I still have the original DVDs… (It’s over 1 TB)

:innocent:

I apparently don’t have permission to save or move files to /opt/ and there was no prompt for granting root access.

Also, when I tried to initialize a Borg repository on an external HDD, it returned this error message:
[*user*@*device* ~]$ borg init --encryption=none /run/media/*user*/Elements
There is already something at /run/media/*user*/Elements.

And yes, I already reformatted the drive. It’s completely empty. I also took out “–stats” because:
usage: borg [-V] [-h] [--critical] [--error] [--warning] [--info] [--debug] [--debug-topic TOPIC] [-p] [--log-json] [--lock-wait SECONDS] [--bypass-lock] [--show-version] [--show-rc] [--umask M] [--remote-path PATH] [-remote-ratelimit RATE] [--consider-part-files] [--debug-profile FILE] [--rsh RSH]
<command> ...
borg: error: unrecognized arguments: --stats

I tried taking out “/run” since all the examples I’ve seen start at “/media,” and this was Borg’s response:
The parent path of the repo directory [/media/crossgear/Elements] does not exist.

sudo cp /path/to/file /opt

as /opt is owned by root:

ll /opt
total 24K
drwxr-xr-x  5 root root 4.0K Mar 28 21:32 ./
drwxr-xr-x 18 root root 4.0K Jul 18 12:20 ../
drwxr-xr-x  2 root root 4.0K Feb 20  2020 Live-ISOs/
drwxr-xr-x  4 root root 4.0K Mar 28 21:32 nbfc/
drwxr-xr-x 20 root root 4.0K Aug 18 04:38 zoom/

that’s a Dolphin / Nautilus / nemo auto mount: mount the drive using mountt. ((not in fstab as you don’t want it to be available when ransomware strikes, so that drive should be only available when backing up)

have a look here for more info:

It looks like /run/media/*user*/Elements is the base folder of your external drive, which does not seem to be completely empty, while borg expects an empty folder for its use.

1 Like

if @Tullius would be correct (90% chance), just create a bakup directory and borg init to that.

i.e.:

borg init --encryption=none \
          /run/media/*user*/Elements/Backup/

So it looks like Tullius was correct. I went ahead and created a directory in Elements called “backup,” then attempted to create the repository. It was successful, but threw an error.


[*user*@*device* Elements]$ borg init --encryption=none /run/media/*user*/Elements/backup
Local Exception
Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/borg/archiver.py", line 4565, in main
exit_code = archiver.run(args)
File "/usr/lib/python3.8/site-packages/borg/archiver.py", line 4497, in run
return set_ec(func(args))
File "/usr/lib/python3.8/site-packages/borg/archiver.py", line 161, in wrapper
with repository:
File "/usr/lib/python3.8/site-packages/borg/repository.py", line 188, in __enter__
self.create(self.path)
File "/usr/lib/python3.8/site-packages/borg/repository.py", line 272, in create
with open(os.path.join(path, 'README'), 'w') as fd:
PermissionError: [Errno 13] Permission denied: '/run/media/*user*/Elements/backup/README'

Platform: Linux bruiser 5.4.58-1-MANJARO #1 SMP PREEMPT Tue Aug 11 15:46:30 UTC 2020 x86_64
Linux: Unknown Linux
Borg: 1.1.13 Python: CPython 3.8.5 msgpack: 0.5.6
PID: 3755 CWD: /run/media/*user*/Elements
sys.argv: ['/usr/bin/borg', 'init', '--encryption=none', '/run/media/*user*/Elements/backup']
SSH_ORIGINAL_COMMAND: None


Not knowing what this meant, I attempt to create my first backup. This was my “create” code.


[*user*@*device* Elements]$ borg create --stats --progress --compression lzma,9 \
> --exclude ".audacity-data" \
> --exclude ".bash_history" \
> --exclude ".bash_logout" \
> --exclude ".bash_profile" \
> --exclude ".bashrc" \
> --exclude ".cache" \
> --exclude ".config" \
> --exclude ".deepin-screen-recorder" \
> --exclude ".dir_colors" \
> --exclude ".dmrc" \
> --exclude ".gnupg" \
> --exclude ".googleearth" \
> --exclude ".gphoto" \
> --exclude ".gtk-recordmydesktop" \
> --exclude ".ICEauthority" \
> --exclude ".lesshst" \
> --exclude ".local" \
> --exclude ".lynxrc" \
> --exclude ".mozilla" \
> --exclude ".pki" \
> --exclude ".profile" \
> --exclude ".sc_history" \
> --exclude ".sc-iminfo" \
> --exclude ".thunderbird" \
> --exclude ".viminfo" \
> --exclude ".w3m" \
> --exclude ".Xauthority" \
> --exclude ".Xclients" \
> --exclude ".xinitrc" \
> --exclude ".xsession-errors" \
> --exclude ".xsession-errors.old" \
> /run/media/*user*/Elements/backup::backup-{user}-{now} /home/*user*/


Suffice to say it failed:


Failed to create/acquire the lock /run/media/crossgear/Elements/backup/lock.exclusive ([Errno 13] Permission denied: '/run/media/crossgear/Elements/backup/lock.exclusive')

Your user does not have write rights to the /run/media/*user*/Elements/backup directory as it can’t create the file README. (probably owned by root instead of your user)

Please read up on file permissions:
https://wiki.archlinux.org/index.php/File_permissions_and_attributes

and adapt the permissions so your user has some.

:innocent:

P.S. Your borg command itself looks correct! :+1:
P.P.S. In the future, when providing output, please copy-paste it in-between 3 backticks at the beginning and end of the code/text ``` so that the output looks like this:

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

instead of like this:

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

(as that makes our life much easier so you get helped more quickly and efficiently)

:innocent:

For anyone who consults this thread later, I used sudo chown to change ownership of the directory of the external drive. That allowed me to initialize a repository with no issues.

I did have some trouble with Borg backing up the hidden files/directories I had explicitly told it to exclude (reference my create code above), until I realized I was using relative pathnames. I had cded to the backup directory in question to initialize the repository and start the backup, so I needed to use absolute pathnames for Borg to recognize the excludes. Probably best practice to use absolutes by default.

Also, Borg uses single quotation marks for excludes, whereas I was initially using double quotes. That change didn’t seem to make a difference, but I used it anyway as all the exclude examples in the manual use single quotes. Just something to keep in mind.

Lastly, if you have a large amount of data you are seeking to backup (>200GB in my case), the first backup is going to take a LOOOOOOOOONG TIME. I started this backup at 11pm and let it run all through the night. It’s only a little over half-way through. So be prepared to wait a while.

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.