I randomly execute journalctl -k --priority 3 --boot 0 from time to time to check for important information I may need to pay attention to… and today it honed me in on a cluster of journal entries related to backintime…
"Look for the two `EXT4-fs error` lines"; about 12 lines in
Feb 01 13:59:36 AM4-5600X-Linux plasmashell[1022867]: INFO: Lock
Feb 01 13:59:36 AM4-5600X-Linux python[1022867]: backintime (<userid>/1): INFO: Lock
Feb 01 13:59:36 AM4-5600X-Linux plasmashell[1022867]: INFO: Take a new snapshot. Profile: 1 Main profile
Feb 01 13:59:36 AM4-5600X-Linux python[1022867]: backintime (<userid>/1): INFO: Take a new snapshot. Profile: 1 Main profile
Feb 01 13:59:36 AM4-5600X-Linux plasmashell[1022867]: INFO: Call rsync to take the snapshot
Feb 01 13:59:36 AM4-5600X-Linux python[1022867]: backintime (<userid>/1): INFO: Call rsync to take the snapshot
Feb 01 13:59:43 AM4-5600X-Linux plasmashell[1022867]: INFO: Save config file
Feb 01 13:59:43 AM4-5600X-Linux plasmashell[1022867]: INFO: Save permissions
Feb 01 13:59:43 AM4-5600X-Linux python[1022867]: backintime (<userid>/1): INFO: Save config file
Feb 01 13:59:43 AM4-5600X-Linux python[1022867]: backintime (<userid>/1): INFO: Save permissions
Feb 01 13:59:49 AM4-5600X-Linux plasmashell[1022867]: INFO: Create info file
Feb 01 13:59:49 AM4-5600X-Linux python[1022867]: backintime (<userid>/1): INFO: Create info file
Feb 01 13:59:49 AM4-5600X-Linux kernel: EXT4-fs error (device nvme3n1p1): htree_dirblock_to_tree:1072: inode #38297640: comm rsync: Directory block failed checksum
Feb 01 13:59:49 AM4-5600X-Linux kernel: EXT4-fs error (device nvme3n1p1): ext4_empty_dir:3068: inode #38297640: comm rsync: Directory block failed checksum
Feb 01 13:59:49 AM4-5600X-Linux plasmashell[1022867]: WARNING: Command "rsync -a --delete -s /tmp/tmp4d6tsyhw/ /data/sn550/backintime/AM4-5600X-Linux/<userid>/1/20230115-230001-242" returns 23 | rsync: [generator] readdir("/data/sn550/backintime/AM4-5600X-Linux>
Feb 01 13:59:49 AM4-5600X-Linux plasmashell[1022867]: cannot delete non-empty directory: backup/home/<userid>/.npm/_cacache/index-v5/68/88
Feb 01 13:59:49 AM4-5600X-Linux plasmashell[1022867]: cannot delete non-empty directory: backup/home/<userid>/.npm/_cacache/index-v5/68
Feb 01 13:59:49 AM4-5600X-Linux plasmashell[1022867]: cannot delete non-empty directory: backup/home/<userid>/.npm/_cacache/index-v5/68
Feb 01 13:59:49 AM4-5600X-Linux plasmashell[1022867]: cannot delete non-empty directory: backup/home/<userid>/.npm/_cacache/index-v5
Feb 01 13:59:49 AM4-5600X-Linux plasmashell[1022867]: cannot delete non-empty directory: backup/home/<userid>/.npm/_cacache/index-v5
Feb 01 13:59:49 AM4-5600X-Linux plasmashell[1022867]: cannot delete non-empty directory: backup/home/<userid>/.npm/_cacache
Feb 01 13:59:49 AM4-5600X-Linux plasmashell[1022867]: cannot delete non-empty directory: backup/home/<userid>/.npm/_cacache
Feb 01 13:59:49 AM4-5600X-Linux plasmashell[1022867]: cannot delete non-empty directory: backup/home/<userid>/.npm
Feb 01 13:59:49 AM4-5600X-Linux plasmashell[1022867]: cannot delete non-empty directory: backup/home/<userid>/.npm
Feb 01 13:59:49 AM4-5600X-Linux plasmashell[1022867]: cannot delete non-empty directory: backup/home/<userid>
Feb 01 13:59:49 AM4-5600X-Linux plasmashell[1022867]: cannot delete non-empty directory: backup/home/<userid>
Feb 01 13:59:49 AM4-5600X-Linux plasmashell[1022867]: cannot delete non-empty directory: backup/home
Feb 01 13:59:49 AM4-5600X-Linux plasmashell[1022867]: cannot delete non-empty directory: backup/home
Feb 01 13:59:49 AM4-5600X-Linux plasmashell[1022867]: cannot delete non-empty directory: backup
Feb 01 13:59:49 AM4-5600X-Linux plasmashell[1022867]: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1336) [sender=3.2.7]
Feb 01 13:59:49 AM4-5600X-Linux plasmashell[1022867]: ERROR: Last rsync command failed with return code "23". See previous WARNING message in the logs for details.
Feb 01 13:59:49 AM4-5600X-Linux python[1022867]: backintime (<userid>/1): WARNING: Command "rsync -a --delete -s /tmp/tmp4d6tsyhw/ /data/sn550/backintime/AM4-5600X-Linux/<userid>/1/20230115-230001-242" returns 23 | rsync: [generator] readdir("/data/sn550/backint>
cannot delete non-empty directory: backup/home/<userid>/.npm/_cacache/index-v5/68/88
cannot delete non-empty directory: backup/home/<userid>/.npm/_cacache/index-v5/68
cannot delete non-empty directory: backup/home/<userid>/.npm/_cacache/index-v5/68
cannot delete non-empty directory: backup/home/<userid>/.npm/_cacache/index-v5
cannot delete non-empty directory: backup/home/<userid>/.npm/_cacache/index-v5
cannot delete non-empty directory: backup/home/<userid>/.npm/_cacache
cannot delete non-empty directory: backup/home/<userid>/.npm/_cacache
cannot delete non-empty directory: backup/home/<userid>/.npm
cannot delete non-empty directory:
Feb 01 13:59:49 AM4-5600X-Linux python[1022867]: backintime (<userid>/1): WARNING: CONTINUE: backup/home/<userid>/.npm
cannot delete non-empty directory: backup/home/<userid>
cannot delete non-empty directory: backup/home/<userid>
cannot delete non-empty directory: backup/home
cannot delete non-empty directory: backup/home
cannot delete non-empty directory: backup
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1336) [sender=3.2.7]
Feb 01 13:59:49 AM4-5600X-Linux python[1022867]: backintime (<userid>/1): ERROR: Last rsync command failed with return code "23". See previous WARNING message in the logs for details.
Feb 01 13:59:56 AM4-5600X-Linux plasmashell[1022867]: INFO: Unlock
Feb 01 13:59:56 AM4-5600X-Linux python[1022867]: backintime (<userid>/1): INFO: Unlock
Feb 01 13:59:56 AM4-5600X-Linux plasmashell[1022867]: Back In Time
Feb 01 13:59:56 AM4-5600X-Linux plasmashell[1022867]: Version: 1.3.3
Feb 01 13:59:56 AM4-5600X-Linux plasmashell[1022867]: Back In Time comes with ABSOLUTELY NO WARRANTY.
Feb 01 13:59:56 AM4-5600X-Linux plasmashell[1022867]: This is free software, and you are welcome to redistribute it
Feb 01 13:59:56 AM4-5600X-Linux plasmashell[1022867]: under certain conditions; type `backintime --license' for details.
I ran backintime manually a couple times and duplicated the error.
Thinking there might be an issue with the filesystem/disk, i ran…
e2fsck
$ sudo umount /dev/nvme3n1p1
$ sudo e2fsck -vf /dev/nvme3n1p1
e2fsck 1.46.5 (30-Dec-2021)
Pass 1: Checking inodes, blocks, and sizes
Inode 12 extent tree (at level 2) could be narrower. Optimize<y>? yes
Inode 11666897 extent tree (at level 1) could be shorter. Optimize<y>? yes
Inode 68685017 extent tree (at level 1) could be shorter. Optimize<y>? yes
Inode 89141649 extent tree (at level 1) could be shorter. Optimize<y>? yes
Inode 110626785 extent tree (at level 1) could be shorter. Optimize<y>? yes
Pass 1E: Optimizing extent trees
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
NVME_SN550_BAK: ***** FILE SYSTEM WAS MODIFIED *****
2247717 inodes used (1.84%, out of 122101760)
6367 non-contiguous files (0.3%)
3971 non-contiguous directories (0.2%)
# of inodes with ind/dind/tind blocks: 0/0/0
Extent depth histogram: 2142434/1541/1
386483084 blocks used (79.14%, out of 488377600)
0 bad blocks
71 large files
1200201 regular files
911882 directories
0 character device files
0 block device files
5 fifos
7420377 links
135620 symbolic links (103728 fast symbolic links)
0 sockets
------------
9668085 files
$ sudo mount /dev/nvme3n1p1
The “inode optimizations” seemed to correct the issue as backintime no longer triggered then EXT4-fs errors… but I’d like to ask some questions before i feel good about this “solution”:
- Is it possible the filesystem just required the fschk? (like maybe an issue cropped up during shutdown/unmounting?)
- Is it possible that what went on with the file system might actually be pointing to the disk? It’s not quite a year old, but you never know.
- maybe my manual
backintimeexecutions forced some “bad data” passed the retention period and were deleted
Any observations that would help confirm what I’ve done is “good”, or any additional checks I should make would be appreciated.