Unable to Login to KDE (BTRFS) | Read-Only Disk?

I had an error that required replacing the BTRFS @ and @home subvolumes on my BTRFS system, but now I’m not able to boot the system (it boots in emergency mode).

The subvolumes are not read-only as verified by btrfs property get -t s SUBVOL_HERE. This returns ro=false. However, I cannot delete the subvolume, giving the strange error from the live USB:

[manjaro DI]# btrfs subvolume delete -C @
Delete subvolume (commit): '/mnt/DI/@'
ERROR: Could not destroy subvolume/snapshot: Read-only file system

The disk should not be read-only.

Stranger yet, there seems to be a nested subvolume for some reason:

[manjaro DI]# btrfs subvolume list .
ID 2067 gen 2563184 top level 5 path Snapshots/MidnightStarSign___@___09_06_2022___12_39
ID 2068 gen 2563186 top level 5 path Snapshots/MidnightStarSign___@home___09_06_2022___12_39
ID 2069 gen 2565191 top level 5 path Snapshots/MidnightStarSign___@___09_07_2022___13_27
ID 2070 gen 2565193 top level 5 path Snapshots/MidnightStarSign___@home___09_07_2022___13_27
ID 2071 gen 2567114 top level 5 path Snapshots/MidnightStarSign___@___09_08_2022___13_19
ID 2072 gen 2567116 top level 5 path Snapshots/MidnightStarSign___@home___09_08_2022___13_19
ID 2073 gen 2578406 top level 5 path Snapshots/MidnightStarSign___@___09_09_2022___13_19
ID 2074 gen 2578439 top level 5 path Snapshots/MidnightStarSign___@home___09_09_2022___13_19
ID 2076 gen 2578314 top level 5 path Snapshots/@BUGGY
ID 2077 gen 2578315 top level 5 path Snapshots/@BUGGY_HOME
ID 2078 gen 2578495 top level 5 path @
ID 2079 gen 2578375 top level 2078 path @/MidnightStarSign___@home___09_09_2022___13_19
ID 2080 gen 2578406 top level 2078 path @/MidnightStarSign___@___09_09_2022___13_19
ID 2081 gen 2578474 top level 5 path @home

I cannot delete the nested ones either:

[manjaro DI]# btrfs subvolume delete -C \@/MidnightStarSign___\@___09_09_2022___13_19/
Delete subvolume (commit): '/mnt/DI/@/MidnightStarSign___@___09_09_2022___13_19'
ERROR: Could not destroy subvolume/snapshot: Read-only file system

Here is my disk setup:


[manjaro DI]# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
loop0         7:0    0  21.2M  1 loop  
loop1         7:1    0 717.4M  1 loop  
loop2         7:2    0   1.9G  1 loop  
loop3         7:3    0 645.3M  1 loop  
sda           8:0    0   3.6T  0 disk  
└─sda1        8:1    0   3.6T  0 part  
sdb           8:16   0   3.6T  0 disk  
└─sdb1        8:17   0   3.6T  0 part  
sdc           8:32   1  29.3G  0 disk  /run/miso/bootmnt
├─sdc1        8:33   1   3.3G  0 part  
└─sdc2        8:34   1     4M  0 part  
nvme0n1     259:0    0 931.5G  0 disk  
├─nvme0n1p1 259:1    0   512M  0 part  
├─nvme0n1p2 259:2    0 864.8G  0 part  
│ └─DI      254:0    0 864.8G  0 crypt /mnt/DI
└─nvme0n1p3 259:3    0  66.3G  0 part  
nvme1n1     259:4    0 931.5G  0 disk  
├─nvme1n1p1 259:5    0 731.5G  0 part  
├─nvme1n1p2 259:6    0    16M  0 part  
└─nvme1n1p3 259:7    0   200G  0 part

How can I resolve this?

EDIT 0:
The mount command reveals that it was mounted ro for some reason:
/dev/mapper/DI on /mnt/DI type btrfs (ro,relatime,ssd,space_cache,subvolid=5,subvol=/)

I unmounted using umount -a and then remounted the system using -o rw, but disk it STILL ro:

[manjaro mnt]# mount -o rw /dev/mapper/DI DI/
[manjaro mnt]# mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
dev on /dev type devtmpfs (rw,nosuid,relatime,size=32850072k,nr_inodes=8212518,mode=755,inode64)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64)
/dev/sdc on /run/miso/bootmnt type iso9660 (ro,relatime,nojoliet,check=s,map=n,blocksize=2048,iocharset=utf8)
overlay on / type overlay (rw,relatime,lowerdir=/run/miso/sfs/livefs:/run/miso/sfs/mhwdfs:/run/miso/sfs/desktopfs:/run/miso/sfs/rootfs,upperdir=/run/miso/overlay_root/upper,workdir=/run/miso/overlay_root/work,index=off,xino=off)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=32877384k,nr_inodes=1048576,inode64)
tmpfs on /etc/pacman.d/gnupg type tmpfs (rw,relatime,mode=755,inode64)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=6575476k,nr_inodes=1643869,mode=700,uid=1000,gid=1000,inode64)
/dev/mapper/DI on /mnt/DI type btrfs (ro,relatime,ssd,space_cache,subvolid=5,subvol=/)

dmesg seems to think similarly:

2762.082072]  item 230 key (2358919168 169 0) itemoff 8444 itemsize 33
[ 2762.082073]          extent refs 1 gen 2450843 flags 258
[ 2762.082073]          ref#0: shared block backref parent 2354380800
[ 2762.082074] BTRFS error (device dm-0): unable to find ref byte nr 2354757632 parent 0 root 1909  owner 2 offset 0
[ 2762.082075] ------------[ cut here ]------------
[ 2762.082075] BTRFS: Transaction aborted (error -2)
[ 2762.082079] WARNING: CPU: 12 PID: 11509 at fs/btrfs/extent-tree.c:3067 __btrfs_free_extent+0x699/0x990 [btrfs]
[ 2762.082098] Modules linked in: nilfs2 jfs btrfs blake2b_generic xor raid6_pq dm_crypt cbc encrypted_keys trusted asn1_encoder tee rfcomm nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) sg st sr_mod intel_rapl_msr cmac algif_hash algif_skcipher af_alg bnep intel_rapl_common snd_hda_codec_realtek snd_hda_codec_generic qrtr edac_mce_amd ns ledtrig_audio snd_hda_codec_hdmi snd_hda_intel amdgpu kvm_amd snd_intel_dspcfg snd_intel_sdw_acpi uvcvideo kvm btusb snd_usb_audio snd_hda_codec videobuf2_vmalloc videobuf2_memops btrtl videobuf2_v4l2 snd_usbmidi_lib btbcm irqbypass eeepc_wmi snd_hda_core btintel videobuf2_common snd_rawmidi snd_hwdep snd_seq_device snd_pcm asus_wmi bluetooth crct10dif_pclmul sparse_keymap videodev snd_timer sp5100_tco platform_profile ecdh_generic snd xpad mc joydev soundcore ff_memless mousedev crc16 rfkill gpu_sched ghash_clmulni_intel tpm_crb wmi_bmof i2c_piix4 ccp rapl tpm_tis k10temp pcspkr tpm_tis_core pinctrl_amd tpm mac_hid rng_core acpi_cpufreq crypto_user fuse
[ 2762.082119]  ip_tables x_tables overlay squashfs isofs cdrom xfs libcrc32c crc32c_generic vfat fat hid_uclogic uas usb_storage usbhid radeon crc32_pclmul crc32c_intel aesni_intel mxm_wmi crypto_simd video cryptd drm_ttm_helper ttm igb dca xhci_pci wmi dm_snapshot dm_bufio dm_mod loop [last unloaded: nouveau]
[ 2762.082126] CPU: 12 PID: 11509 Comm: kworker/u64:4 Tainted: P        W  OE     5.15.60-1-MANJARO #1 cb6dece7f351a33c0ac9650a5b516aad0722dc40
[ 2762.082127] Hardware name: System manufacturer System Product Name/PRIME X570-PRO, BIOS 4021 08/09/2021
[ 2762.082128] Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs]
[ 2762.082147] RIP: 0010:__btrfs_free_extent+0x699/0x990 [btrfs]
[ 2762.082161] Code: 48 05 50 0a 00 00 f0 48 0f ba 28 03 0f 92 c0 41 58 84 c0 0f 85 dc 3a 0c 00 be fe ff ff ff 48 c7 c7 e0 f6 9f c1 e8 b4 c4 87 f0 <0f> 0b e9 c4 3a 0c 00 31 d2 4c 89 e6 4c 89 ef e8 83 4e 03 00 4c 89
[ 2762.082162] RSP: 0018:ffff963300a17ba8 EFLAGS: 00010246
[ 2762.082163] RAX: 0000000000000000 RBX: 000000008c5ac000 RCX: 0000000000000000
[ 2762.082163] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 2762.082164] RBP: 0000000000000775 R08: 0000000000000000 R09: 0000000000000000
[ 2762.082164] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[ 2762.082165] R13: 0000000000000000 R14: 0000000000000002 R15: ffff8b21fe539d20
[ 2762.082166] FS:  0000000000000000(0000) GS:ffff8b2ffed00000(0000) knlGS:0000000000000000
[ 2762.082166] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2762.082167] CR2: 00007fd991a51000 CR3: 00000002906b8000 CR4: 0000000000750ee0
[ 2762.082167] PKRU: 55555554
[ 2762.082168] Call Trace:
[ 2762.082168]  <TASK>
[ 2762.082169]  ? btrfs_old_root_level+0x61/0x90 [btrfs 58160d4e4c68fb5dbbbe3decf3f6145defd12a37]
[ 2762.082184]  __btrfs_run_delayed_refs+0x7e0/0x1150 [btrfs 58160d4e4c68fb5dbbbe3decf3f6145defd12a37]
[ 2762.082199]  btrfs_run_delayed_refs+0x6d/0x1f0 [btrfs 58160d4e4c68fb5dbbbe3decf3f6145defd12a37]
[ 2762.082213]  flush_space+0x41a/0x600 [btrfs 58160d4e4c68fb5dbbbe3decf3f6145defd12a37]
[ 2762.082226]  ? psi_task_switch+0xbf/0x200
[ 2762.082228]  ? __switch_to_asm+0x42/0x80
[ 2762.082229]  ? finish_task_switch.isra.0+0x86/0x2b0
[ 2762.082230]  btrfs_async_reclaim_metadata_space+0x1a4/0x2a0 [btrfs 58160d4e4c68fb5dbbbe3decf3f6145defd12a37]
[ 2762.082243]  process_one_work+0x1ca/0x390
[ 2762.082245]  worker_thread+0x4d/0x3a0
[ 2762.082246]  ? process_one_work+0x390/0x390
[ 2762.082247]  kthread+0x123/0x150
[ 2762.082247]  ? set_kthread_struct+0x60/0x60
[ 2762.082248]  ret_from_fork+0x22/0x30
[ 2762.082250]  </TASK>
[ 2762.082250] ---[ end trace 50b3ba4c2799e7de ]---
[ 2762.082251] BTRFS: error (device dm-0) in __btrfs_free_extent:3067: errno=-2 No such entry
[ 2762.082252] BTRFS info (device dm-0): forced readonly
[ 2762.082253] BTRFS: error (device dm-0) in btrfs_run_delayed_refs:2149: errno=-2 No such entry

EDIT 1:
Just in case I have to nuke the partition, assuming I have a good backup scheme (I do on an UNRAID server downstairs), can I just use gparted to destroy and re-create the encrypted LUKS partition and then send @ and @home to it? Will this be bootable, or do I have to mess around with efibotmgr too? Or is it more complicated than that?

Try running sudo btrfs subvolume list / to mount it first and check the path of the subvolume, say., /@DI.

Then use sudo btrfs subvolume delete /@DI

I’m assuming that you mean chroot into DI/@ and run these?

I did that and it still says that its read-only and can’t remove it:

[manjaro mnt]# manjaro-chroot DI/@
sh-5.1# btrfs subvolume list /
ID 2067 gen 2563184 top level 5 path Snapshots/MidnightStarSign___@___09_06_2022___12_39
ID 2068 gen 2563186 top level 5 path Snapshots/MidnightStarSign___@home___09_06_2022___12_39
ID 2069 gen 2565191 top level 5 path Snapshots/MidnightStarSign___@___09_07_2022___13_27
ID 2070 gen 2565193 top level 5 path Snapshots/MidnightStarSign___@home___09_07_2022___13_27
ID 2071 gen 2567114 top level 5 path Snapshots/MidnightStarSign___@___09_08_2022___13_19
ID 2072 gen 2567116 top level 5 path Snapshots/MidnightStarSign___@home___09_08_2022___13_19
ID 2073 gen 2578406 top level 5 path Snapshots/MidnightStarSign___@___09_09_2022___13_19
ID 2074 gen 2578439 top level 5 path Snapshots/MidnightStarSign___@home___09_09_2022___13_19
ID 2076 gen 2578314 top level 5 path Snapshots/@BUGGY
ID 2077 gen 2578315 top level 5 path Snapshots/@BUGGY_HOME
ID 2078 gen 2578495 top level 5 path @
ID 2079 gen 2578375 top level 2078 path MidnightStarSign___@home___09_09_2022___13_19
ID 2080 gen 2578406 top level 2078 path MidnightStarSign___@___09_09_2022___13_19
ID 2081 gen 2578474 top level 5 path @home
sh-5.1# ls
MidnightStarSign___@___09_09_2022___13_19      bin   crypto_keyfile.bin  dev  home  lib64  mnt  proc  rootfs-pkgs.txt  sbin  sys  usr
MidnightStarSign___@home___09_09_2022___13_19  boot  desktopfs-pkgs.txt  etc  lib   media  opt  root  run              srv   tmp  var
sh-5.1# btrfs subvolume delete -C MidnightStarSign___\@___09_09_2022___13_19/
Delete subvolume (commit): '//MidnightStarSign___@___09_09_2022___13_19'
ERROR: Could not destroy subvolume/snapshot: Read-only file system
sh-5.1#

You missed an @ at the end

Try

sudo btrfs subvolume delete /run/MidnightStarSign___@___09_09_2022___13_19/@

The trick part is really the path. You can try /MidnightStarSign___@___09_09_2022___13_19/@ too.

There isn’t such a folder.

/run/MidnightStarSign___@___09_09_2022___13_19/ only contains the typical / directories.


[manjaro mnt]# ls DISK/\@/MidnightStarSign___\@___09_09_2022___13_19/
bin  boot  crypto_keyfile.bin  desktopfs-pkgs.txt  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  rootfs-pkgs.txt  run  sbin  srv  sys  tmp  usr  var

Do you mind testing

sudo btrfs subvolume delete /@MidnightStarSign___@___09_09_2022___13_19/@

If that does not work, we need to deal with the encryption.

sudo mount /dev/mapper/cryptroot -o subvolid=5 /mnt

And next
sudo btrfs subvolume delete /mnt/@MidnightStarSign___@___09_09_2022___13_19/@

Well, there is kinda a reason what make the module itself forceing to switch to RO. Preventing data damage …

If you have a backup, then you are lucky. That could definitely work. On LUKS encrpytion I would not even try a filesystem repair, although it might work.

BUT… I would try just this: copy @ and @home to another btrfs partition (btrfs send/recieve). Delete the whole btrfs partition. Copy the btrfs stream back.

1 Like

I reinstalled the entire OS and plan on moving the @ and @home data from the most recent backup to the @ and @home of the current OS.

The newly installed OS was encrypted with the same password, assuming that I changed the /etc/fstab to the new UUIDs, would this boot, or is there more that I have to copy over from the installed OS to get it to boot? I merged over the /.crypto_keyfile.bin file to the old data?

assuming it is a BIOS/legacy install and grub is used, you need to inform grub that the UUID has been changed. Otherwise I see nothing else what is needed.

1 Like

How would I do that?

It’s been a while since I’ve messed around with GRUB configurations.

Well update-grub do that… :slightly_smiling_face:

1 Like

It’s been a long time :skull:
:joy:

Having a strange issue where I can’t seem to find the kernels on update-grub I tried mounting the EFI partition to /boot/efi and then using grub-install, but when I update-grub, no kernels found:

sh-5.1# grub-install /dev/nvme0n1 --target=x86_64-efi --efi-directory=/boot/EFI
Installing for x86_64-efi platform.
Installation finished. No error reported.
sh-5.1# update-grub 
Generating grub configuration file ...
Found theme: /usr/share/grub/themes/manjaro/theme.txt
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
ERROR: mkdir /var/lock/dmraid
grub-probe: error: cannot find a GRUB drive for /dev/sdc1.  Check your device.map.
grub-probe: error: cannot find a GRUB drive for /dev/sdc1.  Check your device.map.
Adding boot menu entry for UEFI Firmware Settings ...
Detecting snapshots ...
Kernels not found.
If you think an error has occurred , please file a bug report at " https://github.com/Antynea/grub-btrfs "
Unmount /tmp/grub-btrfs.PFz4ECU4C5 .. Success
done
sh-5.1# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
loop0         7:0    0  21.2M  1 loop  
loop1         7:1    0 717.4M  1 loop  
loop2         7:2    0   1.9G  1 loop  
loop3         7:3    0 645.3M  1 loop  
sda           8:0    0   3.6T  0 disk  
`-sda1        8:1    0   3.6T  0 part  
  `-SDA     254:1    0   3.6T  0 crypt 
sdb           8:16   0   3.6T  0 disk  
`-sdb1        8:17   0   3.6T  0 part  
sdc           8:32   1  29.3G  0 disk  
|-sdc1        8:33   1   3.3G  0 part  
`-sdc2        8:34   1     4M  0 part  
nvme0n1     259:0    0 931.5G  0 disk  
|-nvme0n1p1 259:1    0   512M  0 part  /boot/efi
|                                      /boot
|-nvme0n1p2 259:2    0    12G  0 part  
`-nvme0n1p3 259:3    0   919G  0 part  
  `-DISK    254:0    0   919G  0 crypt /
nvme1n1     259:4    0 931.5G  0 disk  
|-nvme1n1p1 259:5    0 731.5G  0 part  
|-nvme1n1p2 259:6    0    16M  0 part  
`-nvme1n1p3 259:7    0   200G  0 part

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.