Suddenly my file system became read-only

Support
1

I am working from home and suddenly my FS became read only. I have access on very few folders but the rest is read-only.

I can use /home/username but I cannot use /home/username/Pictures, or Documents, or any other folder. Also I cannot use /
I checked journal for entries but I couldn’t find any particular related to a device. Also, I have multiple mounts, SSDs, M.2s, Samba drives and neither of them is accessible, therefore it shouldn’t be a single device, correct?

Steps taken so far:
Checked NTFS mount points (not automounted, used only from certain VMs)
Checked with other Kernels (5.10, 5.12, 5.15, 5.16)
Checked folder user/group access (there some strange thing there, like group owner was ‘videos’, changed it to the correct, still inaccessible)

I also booted with Live stick and ran " fsck -f /dev/sdxx" on all EXT4 partitions, which didn’t made any difference.

I can rwx on /home/username/Data which is mounted on a RAID (/dev/md127p1)
I cannot on /home/username/Pictures which is mounted on the system disk (/dev/nvme0n1p2)
I cannon on /home/username/VirtualMachines which is mounted on a M.2 disk (/dev/nvme2n1p2)

All of them are EXT4

Please advice as my “work PC” is a virtual machine and it fails to run cause of this issue, and I need to work…

inxi --full

System:
  Host: wizzy-manjaro Kernel: 5.10.93-1-MANJARO x86_64 bits: 64
    Desktop: KDE Plasma 5.23.5 Distro: Manjaro Linux
Machine:
  Type: Desktop System: Gigabyte product: X570 AORUS MASTER v: -CF
    serial: <superuser required>
  Mobo: Gigabyte model: X570 AORUS MASTER v: x.x
    serial: <superuser required> UEFI: American Megatrends LLC. v: F33a
    date: 01/22/2021
CPU:
  Info: 12-core model: AMD Ryzen 9 5900X bits: 64 type: MT MCP cache:
    L2: 6 MiB
  Speed (MHz): avg: 2551
    min/max: 2200/5994:6294:6442:6146:5846:5102:4950:5698:5550:5398:5250 cores:
    1: 2793 2: 2479 3: 2987 4: 2885 5: 2856 6: 2878 7: 2203 8: 2197 9: 2196
    10: 2190 11: 2196 12: 2198 13: 3608 14: 2875 15: 2880 16: 2875 17: 2876
    18: 2879 19: 2201 20: 2202 21: 2193 22: 2197 23: 2185 24: 2199
Graphics:
  Device-1: AMD Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] driver: vfio-pci
    v: 0.2
  Device-2: AMD Navi 14 [Radeon RX 5500/5500M / Pro 5500M] driver: amdgpu
    v: kernel
  Device-3: Logitech HD Pro Webcam C920 type: USB
    driver: snd-usb-audio,uvcvideo
  Display: x11 server: X.Org 1.21.1.3 driver: loaded: amdgpu,ati
    unloaded: modesetting,radeon resolution: 1: 2560x1440~60Hz
    2: 1200x1920~60Hz
  Message: Unable to show advanced data. Required tool glxinfo missing.
Audio:
  Device-1: AMD Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT]
    driver: vfio-pci
  Device-2: AMD Navi 10 HDMI Audio driver: N/A
  Device-3: Creative Sound Blaster GC7 type: USB
    driver: cdc_acm,hid-generic,snd-usb-audio,usbhid
  Device-4: Logitech HD Pro Webcam C920 type: USB
    driver: snd-usb-audio,uvcvideo
  Sound Server-1: ALSA v: k5.10.93-1-MANJARO running: yes
  Sound Server-2: PipeWire v: 0.3.43 running: yes
Network:
  Device-1: Intel Wi-Fi 6 AX200 driver: iwlwifi
  IF: wlp6s0 state: down mac: fa:1f:3e:73:5b:88
  Device-2: Intel I211 Gigabit Network driver: igb
  IF: enp7s0 state: up speed: 1000 Mbps duplex: full mac: b4:2e:99:f7:3e:40
  Device-3: Realtek RTL8125 2.5GbE driver: vfio-pci
  IF-ID-1: virbr0 state: down mac: 52:54:00:61:02:f0
Bluetooth:
  Device-1: Intel AX200 Bluetooth type: USB driver: btusb
  Report: rfkill ID: hci0 state: up address: see --recommends
RAID:
  Device-1: md127 type: mdraid level: mirror status: active size: 3.64 TiB
    report: 2/2 UU
  Components: Online: 0: sda1 1: sdb1
Drives:
  Local Storage: total: raw: 9.57 TiB usable: 5.93 TiB
    used: 998.74 GiB (16.4%)
  ID-1: /dev/nvme0n1 vendor: Samsung model: SSD 970 EVO 500GB
    size: 465.76 GiB
  ID-2: /dev/nvme1n1 vendor: Seagate model: XPG GAMMIX S50 Lite
    size: 953.87 GiB
  ID-3: /dev/nvme2n1 vendor: Samsung model: SSD 970 EVO 250GB
    size: 232.89 GiB
  ID-4: /dev/sda vendor: Toshiba model: HDWE140 size: 3.64 TiB
  ID-5: /dev/sdb vendor: Toshiba model: HDWE140 size: 3.64 TiB
  ID-6: /dev/sdc vendor: Samsung model: SSD 840 EVO 250GB size: 232.89 GiB
  ID-7: /dev/sdd vendor: Samsung model: SSD 850 EVO 500GB size: 465.76 GiB
Partition:
  ID-1: / size: 448.43 GiB used: 29.23 GiB (6.5%) fs: ext4
    dev: /dev/nvme0n1p2
  ID-2: /boot/efi size: 299.4 MiB used: 288 KiB (0.1%) fs: vfat
    dev: /dev/nvme0n1p1
Swap:
  ID-1: swap-1 type: partition size: 8.8 GiB used: 0 KiB (0.0%)
    dev: /dev/nvme0n1p3
Sensors:
  System Temperatures: cpu: 16.8 C mobo: N/A gpu: amdgpu temp: 57.0 C
  Fan Speeds (RPM): N/A gpu: amdgpu fan: 0
Info:
  Processes: 462 Uptime: 16m Memory: 62.74 GiB used: 19.62 GiB (31.3%)
  Shell: Zsh inxi: 3.3.12

fdisk -l

Disk /dev/nvme0n1: 465.76 GiB, 500107862016 bytes, 976773168 sectors
Disk model: Samsung SSD 970 EVO 500GB               
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: A182A0A7-7763-DD4A-8837-E0574FEF4D6E

Device             Start       End   Sectors   Size Type
/dev/nvme0n1p1      4096    618495    614400   300M EFI System
/dev/nvme0n1p2    618496 958313125 957694630 456.7G Linux filesystem
/dev/nvme0n1p3 958313126 976768064  18454939   8.8G Linux swap


Disk /dev/nvme2n1: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: Samsung SSD 970 EVO 250GB               
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C6A52053-3EE0-124E-8A59-046EF525C717

Device         Start       End   Sectors   Size Type
/dev/nvme2n1p2  2048 488396799 488394752 232.9G Linux filesystem


Disk /dev/nvme1n1: 953.87 GiB, 1024209543168 bytes, 2000409264 sectors
Disk model: XPG GAMMIX S50 Lite                     
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 5DEAF967-77A7-4D3A-9928-F5B4E69A5EF6

Device         Start        End    Sectors   Size Type
/dev/nvme1n1p1    34      32767      32734    16M Microsoft reserved
/dev/nvme1n1p2 32768 2000406527 2000373760 953.9G Microsoft basic data


Disk /dev/sda: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: TOSHIBA HDWE140 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: A5AC1B52-1E94-443A-AFF1-1FB481A6E584

Device     Start        End    Sectors  Size Type
/dev/sda1   2048 7814035455 7814033408  3.6T Linux filesystem


Disk /dev/sdb: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: TOSHIBA HDWE140 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: F46C6484-A401-496C-BC7A-5E5433F52509

Device     Start        End    Sectors  Size Type
/dev/sdb1   2048 7814035455 7814033408  3.6T Linux filesystem


Disk /dev/sdc: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: Samsung SSD 840 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 7BE60044-B88F-4791-92DC-CD9F023ACF38

Device         Start       End   Sectors   Size Type
/dev/sdc1       2048    206847    204800   100M EFI System
/dev/sdc2     206848    239615     32768    16M Microsoft reserved
/dev/sdc3     239616 487163903 486924288 232.2G Microsoft basic data
/dev/sdc4  487163904 488394751   1230848   601M Windows recovery environment


Disk /dev/sdd: 465.76 GiB, 500107862016 bytes, 976773168 sectors
Disk model: Samsung SSD 850 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 7043E3A4-DCA7-4D25-AEFF-F74318401036

Device         Start       End   Sectors   Size Type
/dev/sdd1       2048 209717247 209715200   100G Linux filesystem
/dev/sdd2  209717248 976773119 767055872 365.8G Linux filesystem


Disk /dev/loop0: 4 KiB, 4096 bytes, 8 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop1: 65.21 MiB, 68378624 bytes, 133552 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop2: 43.27 MiB, 45371392 bytes, 88616 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/md127: 3.64 TiB, 4000649838592 bytes, 7813769216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: F1F9A13E-9C5C-472D-AEC4-116B21BB0D28

Device       Start        End    Sectors  Size Type
/dev/md127p1  2048 7813767167 7813765120  3.6T Linux filesystem


Disk /dev/loop3: 42.17 MiB, 44220416 bytes, 86368 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop4: 219 MiB, 229638144 bytes, 448512 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop5: 55.49 MiB, 58183680 bytes, 113640 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop6: 55.51 MiB, 58204160 bytes, 113680 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
2

Before you do anything further, copy your important data with USB stick to external drive.

I recommend using a separate partition for /home.

2 Likes
3

I do have backup from 1-2 days before, both local and cloud, thank you.

4

I can’t offer any help here, but I do have a couple of questions for you or for anyone that might have an answer.

Would any useful info show up if you use something like GSmartControl? Maybe your disk is dying and has gone read only, similar to what happens with usb storage devices?

5

And you are right. It looks like one of my RAID1 members has errors:
/dev/sda/


Complete error log:

SMART Extended Comprehensive Error Log Version: 1 (64 sectors)
Device Error Count: 1
	CR     = Command Register
	FEATR  = Features Register
	COUNT  = Count (was: Sector Count) Register
	LBA_48 = Upper bytes of LBA High/Mid/Low Registers ]  ATA-8
	LH     = LBA High (was: Cylinder High) Register    ]   LBA
	LM     = LBA Mid (was: Cylinder Low) Register      ] Register
	LL     = LBA Low (was: Sector Number) Register     ]
	DV     = Device (was: Device/Head) Register
	DC     = Device Control Register
	ER     = Error register
	ST     = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 1 [0] occurred at disk power-on lifetime: 6 hours (0 days + 6 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  84 -- 41 00 d0 00 00 fb 23 c1 7f 40 00  Error: ICRC, ABRT at LBA = 0xfb23c17f = 4213424511

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  61 05 00 00 e0 00 00 fb 23 c6 00 40 00     06:24:21.075  WRITE FPDMA QUEUED
  61 04 80 00 d8 00 00 fb 23 c1 80 40 00     06:24:21.071  WRITE FPDMA QUEUED
  61 01 00 00 d0 00 00 fb 23 c0 80 40 00     06:24:21.066  WRITE FPDMA QUEUED
  61 05 00 00 c8 00 00 fb 23 bb 80 40 00     06:24:21.066  WRITE FPDMA QUEUED
  61 05 00 00 c0 00 00 fb 23 b6 80 40 00     06:24:21.062  WRITE FPDMA QUEUED

So, how can I isolate this one, until it is replaced? It is part of the raid.

6

Sorry I have no experience with RAID

7

Found how to remove the faulty disk from the array, but I am still unable to use the filesystem!

Edit: Completely broke the RAID, created and EXT4 partition on the /dev/sdb1 (now sda1) but still my FS is read only!

8

Haven’t been more frustrated in my life…!!!

I removed the faulty HDD (if that was the actual issue).
Broke the RAID, deleted the other disk and created an EXT4 partition.

Still not working.

Since I need to work tomorrow, cannot afford another day off, I did a clean Manjaro installation (deleted only the primary disk).

Everything looked like it was working ok, until now, where I tried to edit a text file with some notes, that is not even locally, it is on a NAS share and I got the same error!!!

After that, my FS is locked again!!!

HELP!

9

Solid State drives go read only when they fail, for security reasons. This happened to me in the past. Boot from a live iso pen drive and try to gain write permission to your drives. If you can’t, the drive is permanently broken. In my case is was still under warranty, so I returned it ans they gave me a new one. I was able to recover all the information.

10

Thank you for the reply. You are right, but the drive that had few errors was an HDD, part of a RAID.

Regardless, I posted before, that there are no errors in the logs, dmesg looks ok, fsck fixed few things on one disk, and still wasn’t working. Nobobody was suspicious of a certain error in the logs, to provide some keywords to filter by, to know what to look for.

And since it was urgent, I had to reinstall OS from scratch to be able to work. Therefore, no more troubleshooting, problem “solved”.

ps. the drive was a Toshiba X300 and when I contacted their support they replied “Unfortunately we can not help here, as we do not offer a repair service and do not sell any HDD parts.”. No more Toshiba for me.

1 Like
11

Toshiba is getting out of the laptop business. Probably from other computer components too.

Maybe the RAID failed because one of the filesystems got corrupted.

12

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.