Once more a NVMe related issue, I think

nvme

#1

I have been experiencing frequent OS crashes starting around September last year while still running Kubuntu which I wasn’t able to pin down. Filled several pages over at the Kubuntu Forum and spend hours with a Dell support guy but never got to the bottom of the issue.

Part of my quest to get this issue solved was moving to Manjaro and I got stuck here because I liked what I saw and it seemed to be a lot more resilient to this issue, a crash maybe once a week. And I only recently stumbled over the likely root cause, this is exactly what I see happening:

Once I found this I started to play around with the nvme_core.default_ps_max_latency_us kernel parameter but if anything things are worse than before, no matter if I set it to 0, 220, 5500 or leave it out.
Actually, today was the first time I was able to reliably reproduce the error(at least for a while) by running a backup to a second internal drive. The OS reliably (sic) crashed, no matter what I set in GRUB_CMDLINE_LINUX_DEFAULT

I can see the changes are picked up by running

sudo nvme get-feature -f 0x0c -H /dev/nvme0
get-feature:0xc (Autonomous Power State Transition), Current value:0x000001
Autonomous Power State Transition Enable (APSTE): Enabled
Auto PST Entries …
Entry[ 0]

Idle Time Prior to Transition (ITPT): 86 ms
Idle Transition Power State (ITPS): 3

Which gives me Enabled when a value >0 or nothing is set. When I set nvme_core.default_ps_max_latency_us=0 this will switch to Disabled

Kernel doesn’t seem to matter, I tried with 4.20 (my default), 4.19 and 4.14 today, got crashes with all of them.
I had probably 20 crashes today, most of them forced via the backup, but now things are stable again for 2 hours, including running a backup!?

The unpredictability of this bug is driving me mad, and I got a few presentations coming up where I can’t have a crashing system. I think I will open a case with Dell to get this nasty NVMe drive replaced but before I do so I wanted to quickly check if anyone here has got any other ideas of what to check.

And because I know you will ask :slight_smile:, here’s my Inxi:

inxi -D
Drives:    Local Storage: total: 2.29 TiB used: 1.04 TiB (45.6%) 
           ID-1: /dev/nvme0n1 vendor: Samsung model: SM961 NVMe 512GB size: 476.94 GiB 
           ID-2: /dev/sda vendor: Seagate model: ST2000LM015-2E8174 size: 1.82 TiB 
thomas@hermes:~$ inxi -b
System:    Host: hermes Kernel: 4.20.3-1-MANJARO x86_64 bits: 64 Desktop: KDE Plasma 5.14.5 Distro: Manjaro Linux 
Machine:   Type: Laptop System: Dell product: Precision 7510 v: N/A serial: <root required> 
           Mobo: Dell model: 0M1YNP v: A00 serial: <root required> UEFI: Dell v: 1.16.3 date: 09/12/2018 
Battery:   ID-1: BAT0 charge: 41.8 Wh condition: 66.5/91.0 Wh (73%) 
CPU:       Quad Core: Intel Core i7-6920HQ type: MT MCP speed: 825 MHz min/max: 800/3800 MHz 
Graphics:  Device-1: Intel HD Graphics 530 driver: i915 v: kernel 
           Device-2: NVIDIA GM107GLM [Quadro M1000M] driver: nouveau v: kernel 
           Display: x11 server: X.Org 1.20.3 driver: intel,nouveau unloaded: modesetting resolution: 1920x1080~60Hz 
           OpenGL: renderer: Mesa DRI Intel HD Graphics 530 (Skylake GT2) v: 4.5 Mesa 18.3.2 
Network:   Device-1: Intel Ethernet I219-LM driver: e1000e 
           Device-2: Intel Wireless 8260 driver: iwlwifi 
Drives:    Local Storage: total: 2.29 TiB used: 1.04 TiB (45.6%) 
Info:      Processes: 319 Uptime: 1h 32m Memory: 15.56 GiB used: 2.92 GiB (18.8%) Shell: bash inxi: 3.0.30 


#2

I don’t see anything obvious that we could advise without any error logs. Not that it will help, but just to say we tried… :wink:
Please post something…

inxi -Fxz
sudo inxi -apd
journalctl -b -p3 --no-hostname --no-pager
journalctl -b -1 -p3 --no-hostname --no-pager
journalctl -b -2 -p3 --no-hostname --no-pager

#3

Thanks petsam, the error logs were of no help to me because the error seems to result in the drive going into read only all of a sudden which means no error after the issue can get logged.
Actually, thinking about this, could I move the logs to a different drive? I have a built in HDD as well… but I will lose all mount points…not going to work I guess.

 inxi -Fxz
System:    Host: hermes Kernel: 4.20.3-1-MANJARO x86_64 bits: 64 compiler: gcc v: 8.2.1 Desktop: KDE Plasma 5.14.5 
           Distro: Manjaro Linux 
Machine:   Type: Laptop System: Dell product: Precision 7510 v: N/A serial: <filter> 
           Mobo: Dell model: 0M1YNP v: A00 serial: <filter> UEFI: Dell v: 1.16.3 date: 09/12/2018 
Battery:   ID-1: BAT0 charge: 29.8 Wh condition: 66.5/91.0 Wh (73%) model: LGC-LGC3.975 DELL RDYCT71 status: Discharging 
CPU:       Topology: Quad Core model: Intel Core i7-6920HQ bits: 64 type: MT MCP arch: Skylake-S rev: 3 L2 cache: 8192 KiB 
           flags: lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx bogomips: 46480 
           Speed: 800 MHz min/max: 800/3800 MHz Core speeds (MHz): 1: 800 2: 800 3: 800 4: 800 5: 800 6: 800 7: 800 8: 800 
Graphics:  Device-1: Intel HD Graphics 530 vendor: Dell driver: i915 v: kernel bus ID: 00:02.0 
           Device-2: NVIDIA GM107GLM [Quadro M1000M] vendor: Dell driver: nouveau v: kernel bus ID: 01:00.0 
           Display: x11 server: X.Org 1.20.3 driver: intel,nouveau unloaded: modesetting resolution: 1920x1080~60Hz 
           OpenGL: renderer: Mesa DRI Intel HD Graphics 530 (Skylake GT2) v: 4.5 Mesa 18.3.2 direct render: Yes 
Audio:     Device-1: Intel 100 Series/C230 Series Family HD Audio vendor: Dell Sunrise Point-H driver: snd_hda_intel v: kernel 
           bus ID: 00:1f.3 
           Sound Server: ALSA v: k4.20.3-1-MANJARO 
Network:   Device-1: Intel Ethernet I219-LM vendor: Dell driver: e1000e v: 3.2.6-k port: f040 bus ID: 00:1f.6 
           IF: enp0s31f6 state: down mac: <filter> 
           Device-2: Intel Wireless 8260 driver: iwlwifi v: kernel port: e000 bus ID: 02:00.0 
           IF: wlp2s0 state: up mac: <filter> 
Drives:    Local Storage: total: 2.29 TiB used: 1.04 TiB (45.6%) 
           ID-1: /dev/nvme0n1 vendor: Samsung model: SM961 NVMe 512GB size: 476.94 GiB 
           ID-2: /dev/sda vendor: Seagate model: ST2000LM015-2E8174 size: 1.82 TiB 
Partition: ID-1: / size: 444.25 GiB used: 262.80 GiB (59.2%) fs: btrfs dev: /dev/dm-0 
           ID-2: /home size: 444.25 GiB used: 262.80 GiB (59.2%) fs: btrfs dev: /dev/dm-0 
           ID-3: swap-1 size: 31.47 GiB used: 48.8 MiB (0.2%) fs: swap dev: /dev/nvme0n1p4 
Sensors:   System Temperatures: cpu: 41.0 C mobo: 41.0 C sodimm: 33.0 C gpu: nouveau temp: 37 C 
           Fan Speeds (RPM): cpu: 0 
Info:      Processes: 332 Uptime: 2h 28m Memory: 15.56 GiB used: 3.78 GiB (24.3%) Init: systemd Compilers: gcc: 8.2.1 
           Shell: bash v: 5.0.0 inxi: 3.0.30 
sudo inxi -apd
[sudo] password for thomas: 
Drives:    Local Storage: total: 2.29 TiB used: 1.04 TiB (45.6%) 
           ID-1: /dev/nvme0n1 vendor: Samsung model: SM961 NVMe 512GB size: 476.94 GiB block size: physical: 512 B 
           logical: 512 B 
           ID-2: /dev/sda vendor: Seagate model: ST2000LM015-2E8174 size: 1.82 TiB block size: physical: 4096 B logical: 512 B 
           Message: No Optical or Floppy data was found. 
Partition: ID-1: / raw size: 444.25 GiB size: 444.25 GiB (100.00%) used: 262.80 GiB (59.2%) fs: btrfs block size: 4096 B 
           dev: /dev/dm-0 
           ID-2: /boot/efi raw size: 512.0 MiB size: 511.0 MiB (99.80%) used: 30.5 MiB (6.0%) fs: vfat block size: 512 B 
           dev: /dev/nvme0n1p1 
           ID-3: /home raw size: 444.25 GiB size: 444.25 GiB (100.00%) used: 262.80 GiB (59.2%) fs: btrfs block size: 4096 B 
           dev: /dev/dm-0 
           ID-4: /home/thomas/.dropbox/ext4 raw size: 50.0 MiB size: N/A (hidden?) used: N/A (hidden?) fs: ext4 
           dev: /dev/loop0 
           ID-5: /home/thomas/data raw size: 1.72 TiB size: 1.72 TiB (100.00%) used: 771.80 GiB (43.7%) fs: btrfs 
           block size: 4096 B dev: /dev/dm-1 
           ID-6: /mnt raw size: 444.25 GiB size: 444.25 GiB (100.00%) used: 262.80 GiB (59.2%) fs: btrfs block size: 4096 B 
           dev: /dev/dm-0 
           ID-7: /run/media/thomas/Win10 raw size: 97.66 GiB size: 97.66 GiB (100.00%) used: 33.18 GiB (34.0%) fs: ntfs 
           block size: 4096 B dev: /dev/sda1 
           ID-8: /run/media/thomas/bf949040-3b6e-419b-b1a7-a8464a33a1ec raw size: 732.0 MiB size: 704.5 MiB (96.24%) 
           used: 1.4 MiB (0.2%) fs: ext4 block size: 4096 B dev: /dev/nvme0n1p2 
           ID-9: swap-1 size: 31.47 GiB used: 48.8 MiB (0.2%) fs: swap swappiness: 60 (default) cache pressure: 100 (default) 
           dev: /dev/nvme0n1p4 

The nouveau error message here started when I ran the 4.14. kernel, won’t show with 4.19 and 4.20

journalctl -b -p3 --no-hostname --no-pager
-- Logs begin at Sun 2019-01-20 16:34:34 CET, end at Wed 2019-02-06 19:49:15 CET. --
Feb 06 17:20:20 kernel: nouveau 0000:01:00.0: bus: MMIO read of 00000000 FAULT at 3c7e3c [ IBUS ]
Feb 06 17:20:20 kernel: nouveau 0000:01:00.0: bus: MMIO read of 00000000 FAULT at 10ac08 [ IBUS ]
Feb 06 19:48:27 sudo[6884]: thomas : a password is required ; TTY=pts/1 ; PWD=/home/thomas ; USER=root ; COMMAND=/sbin/nvme smart-log /dev/nvme0
Feb 06 19:48:27 sudo[6886]: thomas : a password is required ; TTY=pts/1 ; PWD=/home/thomas ; USER=root ; COMMAND=/sbin/hddtemp -nq -u C /dev/sda
thomas@hermes:~$ journalctl -b -1 -p3 --no-hostname --no-pager
-- Logs begin at Sun 2019-01-20 16:34:34 CET, end at Wed 2019-02-06 19:50:51 CET. --
-- No entries --

thomas@hermes:~$ journalctl -b -2 -p3 --no-hostname --no-pager
-- Logs begin at Sun 2019-01-20 16:34:34 CET, end at Wed 2019-02-06 19:51:36 CET. --
Feb 06 17:12:30 kernel: nouveau 0000:01:00.0: bus: MMIO read of 00000000 FAULT at 3c7e3c [ IBUS ]
Feb 06 17:12:30 kernel: nouveau 0000:01:00.0: bus: MMIO read of 00000000 FAULT at 10ac08 [ IBUS ]
Feb 06 17:16:22 systemd-cryptsetup[2574]: Failed to deactivate: Device or resource busy
Feb 06 17:16:22 systemd-cryptsetup[2575]: Failed to deactivate: Device or resource busy
Feb 06 17:16:23 kernel: watchdog: watchdog0: watchdog did not stop!


#4

It is possible to use another partition on another drive mounted at /var/log. Add to /etc/fstab to have it after reboot.

On topic, because of your FS is btrfs, my mind goes to the schedulers, maybe unrelated, but I am no expert.
You might want to check the Mighty Tutorial for Power Saving to get some ideas…

Some of Manjaro gurus may have more ideas that may help you. Maybe @FadeMind


#5

Just checking, You are on btrfs and you have encryption? LUKS?
Do you have LVM as well?

[EDIT]
You have a swap?!! For btrfs??!!

petsam - your information request - inxi - hard to read, hard to digest. I’m getting gastric attack! :laughing:


#6

It looks like the swap is on a dedicated partition, not inside a btrfs filesystem. Isn’t that the normal process when using btrfs?


#7

I think (> 50%) btrfs cannot use swap. swap by its nature is a separate partition anyway. (swap files are different). Also, there’s talk about allowing swap for kernel 5.0. Don’t know about that.
But meantime our kernels are 4.xx


#8

You got in the failed 50% I suppose…:sob:
BTRFS cannot use swapfile , as from Archwiki (I don’t know btrfs)

Take precautions when seeing my Avatar!! :rofl::rofl:


#9

Sorry guys, I am on the phone but see your responses coming in. :slight_smile: and wanted to quickly respond.
Yes, I do have swap, but on a dedicated partition because btrfs can’t handle swap files.
I do have luks but without lvm.

Edit:
And I went through your thought process back when on Kubuntu, I moved off luks, moved off btrfs until I had the most basic setup and this ■■■■■■ still crashed. Lots of fresh installs…


#10

Okay, I stand corrected on swap. Mixed swap with swap file. Thanks dalto and petsam.
But re: luks (encryption)
From this arch wiki

Btrfs has no built-in encryption support,

But as I dropped btrfs and I don’t use encryption (and lvm), hope I’m again wrong.:grinning:


#11

Correct again :slight_smile:
But you can run btrfs inside a luks container.
https://btrfs.wiki.kernel.org/index.php/FAQ#Does_btrfs_support_encryption.3F

There are several different ways in which a filesystem can interoperate with encryption to keep your data secure:

- It can operate on top of an encrypted partition (dm-crypt / LUKS) scheme.
- It can be used as a component of a stacked approach (eg. ecryptfs) where a layer above the filesystem transparently provides the encryption. 

#12

No crash whatsoever all day, including running a backup. And in the evening I was hardly able to complete the service ticket with Dell in between crashes, so strange.
I’ll keep you posted…

Wrote myself a little canary :slight_smile:

#!/bin/bash

while [ 1 ]; do
    cat /proc/mounts | grep "btrfs ro"
    if [ $? -eq 0  ]; then
        kdialog  --error "NMVe died!!"
         exit
    fi 
    sleep 1
done


#13

Have you reviewed your Schedulers setup, by the posted tutorial?
Have you tried any change?


#14

Sorry, must have missed this one. Can’t find a reference to schedulers here? Do you mean the mighty power saving one?

Edit:
Really good stuff in there! I will try a few of these settings in any case!

Edit2:
No response from Dell yet but things became so unbearable today that I installed Manjaro on a USB attached SSD. So far so good, no crash within 2 hours and the quickest “blank drive” to “back in business” in my professional live, really cool! :slight_smile:


#15

Question:
Is the SSD using the latest firmware? (I don’t know whether you get it from Dell or from Samsung).

THe only other idea I have is to patch the kernel code and add a quirk for that specific drive (the code has a single quirk for a Toshiba drive) - however it seems unlikely that such a problem with a well known drive wouldn’t have been detected earlier.


#16

I looked into the firmware first thing last year. The situation back then was that OEM drives like mine can’t be upgraded with Samsung firmware but one has to go through the vendor. I have been in contact with Dell in November 2018 and back then there was no update. Once I hear back from them this will be my first question.


#17

Ah yes, maybe you have one of those OEM drives from Samsung that are problematic. Some SM951 and PM951 in Dell laptops also have (or had) that problem.

What I would try (if I had a lot of time) is patching drivers/nvme/pci.c and include another “if” clause for your hardware, in the function check_vendor_combination_bug


#18

I think that’s well beyond my level of expertise :slight_smile:

Model Number:                       SM961 NVMe SAMSUNG 512GB
Serial Number:                      S34YNX0HC01903
Firmware Version:                   CXA74D0Q
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Total NVM Capacity:                 512.110.190.592 [512 GB]

#19

Well I’m just speculating :slight_smile:
But the kernel code already has some “exceptions” for specific manufacturer combinations, for example for the Dell XPS 15 9550 where the deepest sleep state is disabled.

For 4.19, see here, line 2418:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/nvme/host/pci.c?h=v4.19.20#n2418

EDIT: if you can get me the device ID and exact DMI data of the laptop (vendor etc.), I could try to include a patch in my custom kernel. But only if I find enough time.


#20

I really appreciate the offer! My machine is a Dell Precision 7510.
Looking at the code, I think I could handle the change, it’s just that I never compiled a kernel myself and hence don’t know how big a task this is. But maybe here’s the opportunity to find out :slight_smile:

For my case I guess all I need to do is to either add another || clause reflecting my machine or, even easier, simply change the existing 5510 to 7510?

	if (dmi_match(DMI_SYS_VENDOR, "Dell Inc.") &&
		    (dmi_match(DMI_PRODUCT_NAME, "XPS 15 9550") ||
		     dmi_match(DMI_PRODUCT_NAME, "Precision 5510")))
			return NVME_QUIRK_NO_DEEPEST_PS;