Corrupt filesystems

I got corrupt filesystem on different versions of Manjaro (with 5.4, 5.13, 5.14 kernels). I’m not sure that OS triggers them. I got swap partition installations with default disk encryption (which can be picked during installation), with swap in file. I supposed hibernation triggers that, but I have checked my btrfs system (Manjaro 21.1.3, 5.4 kernel with swap in file) and got errors in filesystem. I got this on ext4 and btrfs (others I just didn’t use). I checked disk with smartctl (long and conveyance), made 4 passes of memtest, made small test with stress CPU. Any guesses what can trigger this? Any diagnostic tips? Maybe USB, from wich OS was installed is broken?

Hi!
Welcome to community!


?


You did a storage and a CPU tests, but we still did not post your:
-) log excerpts of issue you got;
-) machine config.

Please check How to provide good information and also try [HowTo] Increase your chances of solving your issue and post your research results: may be for someone it will be a hint to suggest something.


Also you did S.M.A.R.T. test, but may be it is current run-related test results, but not shows previous issues/failures during earlier usage. What about to check the report of sudo smartctl -x /dev/sdX in term of what looks bad there?

May be over(under- ?)heating issues?

If you can easily find other cables, try to change SATA data cable.
In case of non-cable setup, try to change port you use for your storage. Before to do it clean the port with vacuum cleaner to get rid of dust as maximum as you can.

Check BIOS modes of your storage and what modes are actually supported by the storage model.

Also, electrical contacts are vulnerable to high electrical current: in the place of pins touches each other a high electrical current produces new chemical substance (oxides). While the effect related to high current, the low current equipment could also have it after some years of usage. The solution here is to eject the storage from port and to plug in into port and to do it 2-3 times: you will scratch oxide film on surface of all contacts (device’s and port’s for it) to restore pure metal-to-metal touch.

I didn’t posted logs, because they are a bit different every time and show only problem I described above. I just wanted to post new log of filesystem check, but it suddenly found no errors. Magic is real maybe.
But here is my smartctl logs: smartctl log - Pastebin.com
Here is my btrfs check log

btrfs

[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
root 257 inode 36656 errors 100, file extent discount
Found file extent holes:
start: 4096, len: 8192
root 257 inode 45746 errors 100, file extent discount
Found file extent holes:
start: 4096, len: 8192
root 257 inode 46903 errors 100, file extent discount
Found file extent holes:
start: 0, len: 20480
root 257 inode 46904 errors 100, file extent discount
Found file extent holes:
start: 0, len: 8192
root 257 inode 46905 errors 100, file extent discount
Found file extent holes:
start: 0, len: 8192
root 257 inode 46906 errors 100, file extent discount
Found file extent holes:
start: 0, len: 8192
root 257 inode 46907 errors 100, file extent discount
Found file extent holes:
start: 0, len: 8192
root 257 inode 46908 errors 100, file extent discount
Found file extent holes:
start: 0, len: 8192
root 257 inode 46909 errors 100, file extent discount
Found file extent holes:
start: 0, len: 8192
root 257 inode 46910 errors 100, file extent discount
Found file extent holes:
start: 0, len: 8192
root 257 inode 46911 errors 100, file extent discount
Found file extent holes:
start: 0, len: 8192
root 257 inode 46912 errors 100, file extent discount
Found file extent holes:
start: 0, len: 8192
root 257 inode 46913 errors 100, file extent discount
Found file extent holes:
start: 0, len: 8192
root 257 inode 46914 errors 100, file extent discount
Found file extent holes:
start: 0, len: 8192
root 257 inode 46915 errors 100, file extent discount
Found file extent holes:
start: 0, len: 8192
root 257 inode 46916 errors 100, file extent discount
Found file extent holes:
start: 36864, len: 4096
root 257 inode 46926 errors 100, file extent discount
Found file extent holes:
start: 4096, len: 4096
start: 16384, len: 8192
start: 28672, len: 4096
start: 36864, len: 4096
root 257 inode 46928 errors 100, file extent discount
Found file extent holes:
start: 0, len: 20480
root 257 inode 46929 errors 100, file extent discount
Found file extent holes:
start: 0, len: 20480
root 257 inode 46931 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 46932 errors 100, file extent discount
Found file extent holes:
start: 0, len: 131072
root 257 inode 46941 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 46942 errors 100, file extent discount
Found file extent holes:
start: 0, len: 98304
root 257 inode 46943 errors 100, file extent discount
Found file extent holes:
start: 0, len: 98304
root 257 inode 46945 errors 100, file extent discount
Found file extent holes:
start: 0, len: 28672
root 257 inode 46946 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 46947 errors 100, file extent discount
Found file extent holes:
start: 0, len: 221184
root 257 inode 46948 errors 100, file extent discount
Found file extent holes:
start: 0, len: 65536
root 257 inode 46949 errors 100, file extent discount
Found file extent holes:
start: 0, len: 53248
root 257 inode 46950 errors 100, file extent discount
Found file extent holes:
start: 0, len: 114688
root 257 inode 46952 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 46953 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 46954 errors 100, file extent discount
Found file extent holes:
start: 0, len: 176128
root 257 inode 46955 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 46958 errors 100, file extent discount
Found file extent holes:
start: 0, len: 1871872
root 257 inode 46959 errors 100, file extent discount
Found file extent holes:
start: 0, len: 450560
root 257 inode 46960 errors 100, file extent discount
Found file extent holes:
start: 0, len: 606208
root 257 inode 46961 errors 100, file extent discount
Found file extent holes:
start: 0, len: 692224
root 257 inode 46962 errors 100, file extent discount
Found file extent holes:
start: 0, len: 32768
root 257 inode 46963 errors 100, file extent discount
Found file extent holes:
start: 69632, len: 45056
root 257 inode 46969 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 46971 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 46972 errors 100, file extent discount
Found file extent holes:
start: 0, len: 208896
root 257 inode 46973 errors 100, file extent discount
Found file extent holes:
start: 4096, len: 36864
root 257 inode 46974 errors 100, file extent discount
Found file extent holes:
start: 24576, len: 20480
root 257 inode 46975 errors 100, file extent discount
Found file extent holes:
start: 0, len: 16384
root 257 inode 46976 errors 100, file extent discount
Found file extent holes:
start: 4096, len: 12288
start: 20480, len: 24576
root 257 inode 46977 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 46978 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 46979 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 46980 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 46981 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 46982 errors 100, file extent discount
Found file extent holes:
start: 4096, len: 24576
root 257 inode 46983 errors 100, file extent discount
Found file extent holes:
start: 12288, len: 4096
root 257 inode 46984 errors 100, file extent discount
Found file extent holes:
start: 4096, len: 4096
start: 12288, len: 20480
start: 36864, len: 20480
start: 61440, len: 45056
root 257 inode 46985 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 46987 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 46988 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 46989 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 46991 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 46992 errors 100, file extent discount
Found file extent holes:
start: 0, len: 8192
root 257 inode 46993 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 46994 errors 100, file extent discount
Found file extent holes:
start: 0, len: 8192
root 257 inode 46995 errors 100, file extent discount
Found file extent holes:
start: 0, len: 24576
root 257 inode 46996 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 46997 errors 100, file extent discount
Found file extent holes:
start: 36864, len: 28672
start: 69632, len: 28672
start: 102400, len: 28672
start: 135168, len: 28672
start: 167936, len: 28672
start: 200704, len: 28672
start: 233472, len: 28672
start: 266240, len: 28672
start: 299008, len: 28672
root 257 inode 46998 errors 100, file extent discount
Found file extent holes:
start: 8192, len: 8192
root 257 inode 46999 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47000 errors 100, file extent discount
Found file extent holes:
start: 69632, len: 126976
start: 200704, len: 520192
start: 724992, len: 360448
root 257 inode 47002 errors 100, file extent discount
Found file extent holes:
start: 0, len: 28672
root 257 inode 47003 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47004 errors 100, file extent discount
Found file extent holes:
start: 69632, len: 16384
root 257 inode 47007 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47008 errors 100, file extent discount
Found file extent holes:
start: 0, len: 16384
root 257 inode 47009 errors 100, file extent discount
Found file extent holes:
start: 0, len: 40960
root 257 inode 47011 errors 100, file extent discount
Found file extent holes:
start: 8192, len: 4096
start: 16384, len: 8192
start: 28672, len: 4096
start: 40960, len: 28672
start: 73728, len: 12288
root 257 inode 47012 errors 100, file extent discount
Found file extent holes:
start: 36864, len: 28672
start: 69632, len: 28672
start: 102400, len: 28672
start: 135168, len: 28672
start: 167936, len: 28672
start: 200704, len: 28672
start: 233472, len: 28672
start: 266240, len: 28672
start: 299008, len: 16384
root 257 inode 47013 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47014 errors 100, file extent discount
Found file extent holes:
start: 528384, len: 520192
root 257 inode 47015 errors 100, file extent discount
Found file extent holes:
start: 4096, len: 8192
start: 16384, len: 28672
start: 49152, len: 28672
start: 81920, len: 24576
start: 122880, len: 4096
start: 139264, len: 8192
start: 155648, len: 28672
start: 188416, len: 28672
start: 221184, len: 28672
start: 253952, len: 28672
start: 286720, len: 28672
start: 319488, len: 28672
start: 352256, len: 28672
start: 385024, len: 28672
start: 417792, len: 28672
start: 450560, len: 28672
start: 483328, len: 28672
start: 516096, len: 28672
start: 548864, len: 16384
root 257 inode 47016 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47017 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47018 errors 100, file extent discount
Found file extent holes:
start: 16384, len: 28672
start: 53248, len: 4096
root 257 inode 47019 errors 100, file extent discount
Found file extent holes:
start: 69632, len: 118784
root 257 inode 47020 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47021 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47022 errors 100, file extent discount
Found file extent holes:
start: 0, len: 12288
root 257 inode 47023 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47024 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47025 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47026 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47027 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47028 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47029 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47030 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47031 errors 100, file extent discount
Found file extent holes:
start: 36864, len: 12288
start: 53248, len: 24576
start: 81920, len: 12288
start: 102400, len: 12288
root 257 inode 47032 errors 100, file extent discount
Found file extent holes:
start: 69632, len: 323584
root 257 inode 47033 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47034 errors 100, file extent discount
Found file extent holes:
start: 0, len: 16384
root 257 inode 47035 errors 100, file extent discount
Found file extent holes:
start: 0, len: 36864
root 257 inode 47036 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47037 errors 100, file extent discount
Found file extent holes:
start: 36864, len: 12288
root 257 inode 47038 errors 100, file extent discount
Found file extent holes:
start: 0, len: 155648
root 257 inode 47040 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47041 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47042 errors 100, file extent discount
Found file extent holes:
start: 0, len: 8192
root 257 inode 47043 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47044 errors 100, file extent discount
Found file extent holes:
start: 32768, len: 28672
start: 65536, len: 28672
start: 98304, len: 28672
start: 131072, len: 28672
start: 163840, len: 28672
start: 196608, len: 28672
start: 229376, len: 28672
start: 262144, len: 32768
root 257 inode 47045 errors 100, file extent discount
Found file extent holes:
start: 69632, len: 258048
start: 331776, len: 520192
root 257 inode 47046 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47047 errors 100, file extent discount
Found file extent holes:
start: 0, len: 12288
root 257 inode 47048 errors 100, file extent discount
Found file extent holes:
start: 0, len: 32768
root 257 inode 47049 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47050 errors 100, file extent discount
Found file extent holes:
start: 0, len: 16384
root 257 inode 47051 errors 100, file extent discount
Found file extent holes:
start: 0, len: 40960
root 257 inode 47052 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47053 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47054 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47055 errors 100, file extent discount
Found file extent holes:
start: 0, len: 8192
root 257 inode 47056 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47057 errors 100, file extent discount
Found file extent holes:
start: 0, len: 8192
root 257 inode 47059 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47060 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47061 errors 100, file extent discount
Found file extent holes:
start: 0, len: 8192
root 257 inode 47062 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47063 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47064 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47065 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47066 errors 100, file extent discount
Found file extent holes:
start: 20480, len: 28672
start: 53248, len: 4096
root 257 inode 47067 errors 100, file extent discount
Found file extent holes:
start: 69632, len: 86016
root 257 inode 47068 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47069 errors 100, file extent discount
Found file extent holes:
start: 0, len: 12288
root 257 inode 47070 errors 100, file extent discount
Found file extent holes:
start: 0, len: 24576
root 257 inode 47072 errors 100, file extent discount
Found file extent holes:
start: 16384, len: 28672
start: 49152, len: 28672
start: 81920, len: 16384
root 257 inode 47073 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47075 errors 100, file extent discount
Found file extent holes:
start: 0, len: 12288
root 257 inode 47077 errors 100, file extent discount
Found file extent holes:
start: 0, len: 32768
root 257 inode 47078 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47079 errors 100, file extent discount
Found file extent holes:
start: 0, len: 8192
root 257 inode 47080 errors 100, file extent discount
Found file extent holes:
start: 0, len: 12288
root 257 inode 47081 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47082 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47083 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47084 errors 100, file extent discount
Found file extent holes:
start: 0, len: 61440
root 257 inode 47085 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47086 errors 100, file extent discount
Found file extent holes:
start: 0, len: 8192
root 257 inode 47087 errors 100, file extent discount
Found file extent holes:
start: 0, len: 8192
root 257 inode 47090 errors 100, file extent discount
Found file extent holes:
start: 0, len: 8192
root 257 inode 47091 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47092 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47094 errors 100, file extent discount
Found file extent holes:
start: 8192, len: 4096
start: 20480, len: 12288
root 257 inode 47095 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47096 errors 100, file extent discount
Found file extent holes:
start: 8192, len: 8192
root 257 inode 47097 errors 100, file extent discount
Found file extent holes:
start: 24576, len: 12288
start: 40960, len: 8192
start: 53248, len: 8192
start: 69632, len: 12288
start: 86016, len: 4096
start: 94208, len: 4096
start: 102400, len: 4096
start: 110592, len: 8192
start: 122880, len: 4096
start: 135168, len: 4096
start: 147456, len: 4096
start: 155648, len: 8192
start: 167936, len: 4096
start: 176128, len: 4096
root 257 inode 47099 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47100 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47101 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47103 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47106 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47107 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47109 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
root 257 inode 47114 errors 100, file extent discount
Found file extent holes:
start: 0, len: 4096
ERROR: errors found in fs roots
found 25397071872 bytes used, error(s) found
total csum bytes: 15645796
total tree bytes: 641138688
total fs tree bytes: 596344832
total extent tree bytes: 25673728
btree space waste bytes: 112011282
file data blocks allocated: 37672751104
referenced 35925909504

It happens after reboot. I checked filesystem yesterday it was ok. But today this happen again. I will check disk using badblocks today.

It’s a known BTRFS bug and it seems some of our options (maybe commit?) trigger it more often. We’re still in research but you might want to try 5.16 kernel prerelease where this bug is claimed to be fixed.


(from #btrfs IRC channel)

Just saw you have this also in ext4… Not sure what’s wrong here but still try 5.16 for btrfs case.

2 Likes

So after 3 weeks got corrupted fs again. Now with ext4 and kernel 5.4. Will try another distro and see if something will change.
Some errors from dmesg.

dmesg output

[ 91.983627] EXT4-fs error (device dm-0): ext4_lookup:1703: inode #2625221: comm NetworkManager: deleted inode referenced: 2621809
[ 93.727566] EXT4-fs (dm-0): Delayed block allocation failed for inode 2625313 at logical offset 262 with max blocks 1 with error 117
[ 93.727569] EXT4-fs (dm-0): This should not happen!! Data will be lost
[ 89.692930] EXT4-fs error (device dm-0): ext4_validate_block_bitmap:384: comm xauth: bg 833: bad block bitmap checksum
[ 18.229059] EXT4-fs error (device dm-0): ext4_validate_inode_bitmap:99: comm sddm-gr:disk$0: Corrupt inode bitmap - block_group = 480, inode_bitmap = 157
28656

Will update later in several weeks. Not sure it is hardware problem, as I wrote above, I tried smartctl tests, memtest and cpu stress testes.

Did you try badblocks -svn command ?
smartctr not so usefull with ssd yet …

Looks like hardware problem to me tbh.