I had an error that required replacing the BTRFS @ and @home subvolumes on my BTRFS system, but now I’m not able to boot the system (it boots in emergency mode).
The subvolumes are not read-only as verified by btrfs property get -t s SUBVOL_HERE
. This returns ro=false
. However, I cannot delete the subvolume, giving the strange error from the live USB:
[manjaro DI]# btrfs subvolume delete -C @
Delete subvolume (commit): '/mnt/DI/@'
ERROR: Could not destroy subvolume/snapshot: Read-only file system
The disk should not be read-only.
Stranger yet, there seems to be a nested subvolume for some reason:
[manjaro DI]# btrfs subvolume list .
ID 2067 gen 2563184 top level 5 path Snapshots/MidnightStarSign___@___09_06_2022___12_39
ID 2068 gen 2563186 top level 5 path Snapshots/MidnightStarSign___@home___09_06_2022___12_39
ID 2069 gen 2565191 top level 5 path Snapshots/MidnightStarSign___@___09_07_2022___13_27
ID 2070 gen 2565193 top level 5 path Snapshots/MidnightStarSign___@home___09_07_2022___13_27
ID 2071 gen 2567114 top level 5 path Snapshots/MidnightStarSign___@___09_08_2022___13_19
ID 2072 gen 2567116 top level 5 path Snapshots/MidnightStarSign___@home___09_08_2022___13_19
ID 2073 gen 2578406 top level 5 path Snapshots/MidnightStarSign___@___09_09_2022___13_19
ID 2074 gen 2578439 top level 5 path Snapshots/MidnightStarSign___@home___09_09_2022___13_19
ID 2076 gen 2578314 top level 5 path Snapshots/@BUGGY
ID 2077 gen 2578315 top level 5 path Snapshots/@BUGGY_HOME
ID 2078 gen 2578495 top level 5 path @
ID 2079 gen 2578375 top level 2078 path @/MidnightStarSign___@home___09_09_2022___13_19
ID 2080 gen 2578406 top level 2078 path @/MidnightStarSign___@___09_09_2022___13_19
ID 2081 gen 2578474 top level 5 path @home
I cannot delete the nested ones either:
[manjaro DI]# btrfs subvolume delete -C \@/MidnightStarSign___\@___09_09_2022___13_19/
Delete subvolume (commit): '/mnt/DI/@/MidnightStarSign___@___09_09_2022___13_19'
ERROR: Could not destroy subvolume/snapshot: Read-only file system
Here is my disk setup:
[manjaro DI]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 21.2M 1 loop
loop1 7:1 0 717.4M 1 loop
loop2 7:2 0 1.9G 1 loop
loop3 7:3 0 645.3M 1 loop
sda 8:0 0 3.6T 0 disk
└─sda1 8:1 0 3.6T 0 part
sdb 8:16 0 3.6T 0 disk
└─sdb1 8:17 0 3.6T 0 part
sdc 8:32 1 29.3G 0 disk /run/miso/bootmnt
├─sdc1 8:33 1 3.3G 0 part
└─sdc2 8:34 1 4M 0 part
nvme0n1 259:0 0 931.5G 0 disk
├─nvme0n1p1 259:1 0 512M 0 part
├─nvme0n1p2 259:2 0 864.8G 0 part
│ └─DI 254:0 0 864.8G 0 crypt /mnt/DI
└─nvme0n1p3 259:3 0 66.3G 0 part
nvme1n1 259:4 0 931.5G 0 disk
├─nvme1n1p1 259:5 0 731.5G 0 part
├─nvme1n1p2 259:6 0 16M 0 part
└─nvme1n1p3 259:7 0 200G 0 part
How can I resolve this?
EDIT 0:
The mount command reveals that it was mounted ro for some reason:
/dev/mapper/DI on /mnt/DI type btrfs (ro,relatime,ssd,space_cache,subvolid=5,subvol=/)
I unmounted using umount -a
and then remounted the system using -o rw
, but disk it STILL ro
:
[manjaro mnt]# mount -o rw /dev/mapper/DI DI/
[manjaro mnt]# mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
dev on /dev type devtmpfs (rw,nosuid,relatime,size=32850072k,nr_inodes=8212518,mode=755,inode64)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64)
/dev/sdc on /run/miso/bootmnt type iso9660 (ro,relatime,nojoliet,check=s,map=n,blocksize=2048,iocharset=utf8)
overlay on / type overlay (rw,relatime,lowerdir=/run/miso/sfs/livefs:/run/miso/sfs/mhwdfs:/run/miso/sfs/desktopfs:/run/miso/sfs/rootfs,upperdir=/run/miso/overlay_root/upper,workdir=/run/miso/overlay_root/work,index=off,xino=off)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=32877384k,nr_inodes=1048576,inode64)
tmpfs on /etc/pacman.d/gnupg type tmpfs (rw,relatime,mode=755,inode64)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=6575476k,nr_inodes=1643869,mode=700,uid=1000,gid=1000,inode64)
/dev/mapper/DI on /mnt/DI type btrfs (ro,relatime,ssd,space_cache,subvolid=5,subvol=/)
dmesg seems to think similarly:
2762.082072] item 230 key (2358919168 169 0) itemoff 8444 itemsize 33
[ 2762.082073] extent refs 1 gen 2450843 flags 258
[ 2762.082073] ref#0: shared block backref parent 2354380800
[ 2762.082074] BTRFS error (device dm-0): unable to find ref byte nr 2354757632 parent 0 root 1909 owner 2 offset 0
[ 2762.082075] ------------[ cut here ]------------
[ 2762.082075] BTRFS: Transaction aborted (error -2)
[ 2762.082079] WARNING: CPU: 12 PID: 11509 at fs/btrfs/extent-tree.c:3067 __btrfs_free_extent+0x699/0x990 [btrfs]
[ 2762.082098] Modules linked in: nilfs2 jfs btrfs blake2b_generic xor raid6_pq dm_crypt cbc encrypted_keys trusted asn1_encoder tee rfcomm nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) sg st sr_mod intel_rapl_msr cmac algif_hash algif_skcipher af_alg bnep intel_rapl_common snd_hda_codec_realtek snd_hda_codec_generic qrtr edac_mce_amd ns ledtrig_audio snd_hda_codec_hdmi snd_hda_intel amdgpu kvm_amd snd_intel_dspcfg snd_intel_sdw_acpi uvcvideo kvm btusb snd_usb_audio snd_hda_codec videobuf2_vmalloc videobuf2_memops btrtl videobuf2_v4l2 snd_usbmidi_lib btbcm irqbypass eeepc_wmi snd_hda_core btintel videobuf2_common snd_rawmidi snd_hwdep snd_seq_device snd_pcm asus_wmi bluetooth crct10dif_pclmul sparse_keymap videodev snd_timer sp5100_tco platform_profile ecdh_generic snd xpad mc joydev soundcore ff_memless mousedev crc16 rfkill gpu_sched ghash_clmulni_intel tpm_crb wmi_bmof i2c_piix4 ccp rapl tpm_tis k10temp pcspkr tpm_tis_core pinctrl_amd tpm mac_hid rng_core acpi_cpufreq crypto_user fuse
[ 2762.082119] ip_tables x_tables overlay squashfs isofs cdrom xfs libcrc32c crc32c_generic vfat fat hid_uclogic uas usb_storage usbhid radeon crc32_pclmul crc32c_intel aesni_intel mxm_wmi crypto_simd video cryptd drm_ttm_helper ttm igb dca xhci_pci wmi dm_snapshot dm_bufio dm_mod loop [last unloaded: nouveau]
[ 2762.082126] CPU: 12 PID: 11509 Comm: kworker/u64:4 Tainted: P W OE 5.15.60-1-MANJARO #1 cb6dece7f351a33c0ac9650a5b516aad0722dc40
[ 2762.082127] Hardware name: System manufacturer System Product Name/PRIME X570-PRO, BIOS 4021 08/09/2021
[ 2762.082128] Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs]
[ 2762.082147] RIP: 0010:__btrfs_free_extent+0x699/0x990 [btrfs]
[ 2762.082161] Code: 48 05 50 0a 00 00 f0 48 0f ba 28 03 0f 92 c0 41 58 84 c0 0f 85 dc 3a 0c 00 be fe ff ff ff 48 c7 c7 e0 f6 9f c1 e8 b4 c4 87 f0 <0f> 0b e9 c4 3a 0c 00 31 d2 4c 89 e6 4c 89 ef e8 83 4e 03 00 4c 89
[ 2762.082162] RSP: 0018:ffff963300a17ba8 EFLAGS: 00010246
[ 2762.082163] RAX: 0000000000000000 RBX: 000000008c5ac000 RCX: 0000000000000000
[ 2762.082163] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 2762.082164] RBP: 0000000000000775 R08: 0000000000000000 R09: 0000000000000000
[ 2762.082164] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[ 2762.082165] R13: 0000000000000000 R14: 0000000000000002 R15: ffff8b21fe539d20
[ 2762.082166] FS: 0000000000000000(0000) GS:ffff8b2ffed00000(0000) knlGS:0000000000000000
[ 2762.082166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2762.082167] CR2: 00007fd991a51000 CR3: 00000002906b8000 CR4: 0000000000750ee0
[ 2762.082167] PKRU: 55555554
[ 2762.082168] Call Trace:
[ 2762.082168] <TASK>
[ 2762.082169] ? btrfs_old_root_level+0x61/0x90 [btrfs 58160d4e4c68fb5dbbbe3decf3f6145defd12a37]
[ 2762.082184] __btrfs_run_delayed_refs+0x7e0/0x1150 [btrfs 58160d4e4c68fb5dbbbe3decf3f6145defd12a37]
[ 2762.082199] btrfs_run_delayed_refs+0x6d/0x1f0 [btrfs 58160d4e4c68fb5dbbbe3decf3f6145defd12a37]
[ 2762.082213] flush_space+0x41a/0x600 [btrfs 58160d4e4c68fb5dbbbe3decf3f6145defd12a37]
[ 2762.082226] ? psi_task_switch+0xbf/0x200
[ 2762.082228] ? __switch_to_asm+0x42/0x80
[ 2762.082229] ? finish_task_switch.isra.0+0x86/0x2b0
[ 2762.082230] btrfs_async_reclaim_metadata_space+0x1a4/0x2a0 [btrfs 58160d4e4c68fb5dbbbe3decf3f6145defd12a37]
[ 2762.082243] process_one_work+0x1ca/0x390
[ 2762.082245] worker_thread+0x4d/0x3a0
[ 2762.082246] ? process_one_work+0x390/0x390
[ 2762.082247] kthread+0x123/0x150
[ 2762.082247] ? set_kthread_struct+0x60/0x60
[ 2762.082248] ret_from_fork+0x22/0x30
[ 2762.082250] </TASK>
[ 2762.082250] ---[ end trace 50b3ba4c2799e7de ]---
[ 2762.082251] BTRFS: error (device dm-0) in __btrfs_free_extent:3067: errno=-2 No such entry
[ 2762.082252] BTRFS info (device dm-0): forced readonly
[ 2762.082253] BTRFS: error (device dm-0) in btrfs_run_delayed_refs:2149: errno=-2 No such entry
EDIT 1:
Just in case I have to nuke the partition, assuming I have a good backup scheme (I do on an UNRAID server downstairs), can I just use gparted to destroy and re-create the encrypted LUKS partition and then send @ and @home to it? Will this be bootable, or do I have to mess around with efibotmgr too? Or is it more complicated than that?