Is there an open source software doing incremental, encrypted, and deduplicated backups that lets users show a file revision history across archives?
I would like to be able to find which archives contains the last 3 modifications (for example) of a given file with one command (terminal) or click (GUI).
To clarify, for whose who are familiar with git, I would like to be able to do something similar to git log --max-count=3 -- file.ext (ie. ‘find the last 3 commits where the file file.ext was modified’).
I am not able to find a backup software with such requirements at the moment but maybe one of you knows about one.
Thanks.
EDIT: “track file changes” was replaced by “show a file revision history” to avoid confusion on what I am trying to achieve
However, if you break down your requirements, you might find what you’re looking for.
A. A system set up on an encrypted block device
B. A system that allows differential backups
C. A system that allows the desired search for changes to individual files
A. I think there are several
B. btrfs can do this with send/receive
C. btrfs can do this with any btrfs-volume (including the backup volume)
But of course, this isn’t “pre-built” for you; it requires you to get your hands dirty.
Regarding encryption, I would rather avoid block device encryption with my current hardware to have a better write speed. Indeed, the write speed is around 25 Mio/s when the external storage is encrypted with LUKS while it is 14 times higher when it is not.
Regarding using BTRFS to find which snapshots contain the last 3 revisions of a specific file, would you know how to do this?
Because I think I understand how to use snapshots to have “archives” and subvolumes to find differences between two snapshots but I don’t see yet how to collect the names of the 3 most recent snapshots where a given file is different from the current version.
The closest I was able to find would be to use btrfs subvolumefind-new or list but I would stil need to know which generations I need to look at while I don’t know between which ones the file I am interested in was actually modified.
If I understand correctly the lingo of BTRFS, my goal would be to find these generations for a file.
NB: The write speed and the ratio w/o LUKS is on a EXT4 file system. The performances depend on CPU (instruction sets, …), RAM (quantity, …), drive (speed, …), protocol (revision, …), … and maybe also of the file system type. I have personally not researched this last aspect. Maybe the performance impact of LUKS for BTRFS would be lower than for EXT4 on my current hardware (CPU, RAM, drive, protocol) and in general. I have not tested and cannot without buying another drive.
How would you show the revision history of a given file with Borg then?
One would indeed need to execute borg diff repo::archive_1 archive_2 for all the pairs of archives, from the most recent to the least recent, and collect the least recent archive names in each pair when the output of borg diff contains the name of the file of interest.
Or is there a built-in (ie. efficient) way of getting the revision history of a given file ?
(I have replaced “track file changes” by “show a file revision history” in the original post title and body to avoid confusion on what I am trying to achieve as it seems “revision history” defines the goal better)
I saw it with Snapper. There, all changes between two snapshots (but these are Btrfs snapshots, and Snapper itself doesn’t record anything significant about them) can be displayed. The command would be:
snapper -c home status 27984..27990
where -c home is the snapper config home (which helps him finding the place of these btrfs snapshots)
and 27984..27990 is the range of names for snapshots generated by snapper (which helps to find the individual snapshots first and last)
But just as Snapper can do this with its own snapshots (which are btrfs-snapshots plus a small info-file) , it would be possible with any Btrfs snapshot/volume. (The program just has to be written first. )
So you would need to add a filter like
grep -E '/etc/pacman.conf'
And of course the list would need to contain the date-time and snapshot-number
Thank you @andreas85 for your take on what I am trying to achieve. I think I see where you are going and what would be the needed next steps (apart from switching to BTRFS from EXT4 ).
The two suggested approaches (BTRFS and Borg) rely on the efficiency of repeating tens or hundreds of times snapper status x y or borg diff x y each time I need to see a file revision history.
It is strange most cloud file-hosting service (eg. Google Drive, Dropbox) offer this feature (ie. showing file versions) but there would be no existing (backup) software offering this locally
)
).
