Inconsistent mount behaviour in USB drives

slimply · 16 August 2023 23:12

This is slowly driving me crazy. Three USB SSDs, automounted (fstab, systemd-automount) behave differently where behavior should be identical between two of them.

The hardware looks identical:

# lsusb -vt
...
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
    ID 1d6b:0003 Linux Foundation 3.0 root hub
        |__ Port 2: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
        ID 2109:0817 VIA Labs, Inc.
            |__ Port 1: Dev 4, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
            ID 152d:1561 JMicron Technology Corp. / JMicron USA Technology Corp. JMS561U two ports SATA 6Gb/s bridge
        |__ Port 2: Dev 5, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
            ID 152d:1561 JMicron Technology Corp. / JMicron USA Technology Corp. JMS561U two ports SATA 6Gb/s bridge
        |__ Port 4: Dev 6, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
            ID 152d:1561 JMicron Technology Corp. / JMicron USA Technology Corp. JMS561U two ports SATA 6Gb/s bridge
    |__ Port 3: Dev 3, If 0, Class=Hub, Driver=hub/4p, 5000M
            ID 2109:0817 VIA Labs, Inc.

The drives are 3x Samsung SSD 870 QVO 1TB. One of the SSDs was bought at a different time than the others. The three USB-SATA bridges have the same vendor:device id but are in different enclosures.
So it may be that the firmware of the enclosures, and/or of the SSDs, are different.

SMART data of all three SSDs show nothing but “never” in the “failed” column.

I tested with and without quirks in /etc/modprobe.d/disable-uas.conf – no difference.
[#] options usb-storage quirks=152d:1561:u,152d:0578:u

Two of the drives have nearly identical entries in fstab and crypttab; the third is unlocked and mounted manually.

# /etc/crypttab
backup-usb-1 PARTUUID=f29a9694-f2a1-7341-9871-c68f546e4899 /etc/keyfiles.d/usb-backup.keyfile luks,key-slot=1,headless=true,noauto,nofail,x-systemd.device-timeout=4
backup-usb-2 PARTUUID=585a389f-975a-db43-bcb3-f2c045e345b2 /etc/keyfiles.d/usb-backup.keyfile luks,key-slot=1,headless=true,noauto,nofail,x-systemd.device-timeout=4

Setting the timeout had made no difference.

### /etc/fstab
/dev/mapper/backup-usb-1 /run/media/backup-devices.d/backup1 btrfs x-systemd.automount,nofail,nouser,ssd,rw,noatime 0 0
/dev/mapper/backup-usb-2 /run/media/backup-devices.d/backup2 btrfs x-systemd.automount,nofail,nouser,ssd,rw,noatime 0 0

So what’s weird, is this:
Baseline: After boot, none of the SSDs are mounted, as expected.

# lsblk
...
sda            0    0 931,5G  0 disk
└─sda1         1    0 931,5G  0 part
sdb            16   0 931,5G  0 disk
└─sdb1         17   0 931,5G  0 part
sdc            32   0 931,5G  0 disk
└─sdc1         33   0 931,5G  0 part

A manual mount only mounts one SSD, but three times for good measure.

# mount -av
... # (boot device mounts)
/run/media/backup-devices.d/backup1: successfully mounted

Yes, that’s all.

# grep mapper /etc/mtab
...
/dev/mapper/backup-usb-1 /run/media/backup-devices.d/backup1 btrfs rw,noatime,ssd,space_cache=v2,subvolid=5,subvol=/ 0 0
/dev/mapper/backup-usb-1 /run/media/backup-devices.d/backup1 btrfs rw,noatime,ssd,space_cache=v2,subvolid=5,subvol=/ 0 0
/dev/mapper/backup-usb-1 /run/media/backup-devices.d/backup1 btrfs rw,noatime,ssd,space_cache=v2,subvolid=5,subvol=/ 0 0
...

The other two are not mounted and not unlocked.

# lsblk 
...
sda                   8:0    0 931,5G  0 disk
└─sda1                  1    0 931,5G  0 part
sdb                     16   0 931,5G  0 disk
└─sdb1                8:17   0 931,5G  0 part
sdc                          0 931,5G  0 disk
└─sdc1                  33   0 931,5G  0 part
    └─backup-usb-1  254:2    0 931,5G  0 crypt /run/media/backup-devices.d/backup1
                                               /run/media/backup-devices.d/backup1
                                               /run/media/backup-devices.d/backup1
...

After manually unlocking (GUI, KDE Daemon dialog asks for password) all are mounted automatically on the right places. (The inconsistent labelling is cruft, please ignore.)

# lsblk
...
sda                                               8:0    0 931,5G  0 disk
└─sda1                                            8:1    0 931,5G  0 part
    └─luks-3ad86390-5624-4e39-ab75-913d0c3e59f5 254:4    0 931,5G  0 crypt /run/media/me/backup-usb-2
sdb                                               8:16   0 931,5G  0 disk
└─sdb1                                            8:17   0 931,5G  0 part
    └─luks-a386b816-2230-472b-b2a5-f66a76c66f1d 254:3    0 931,5G  0 crypt /run/media/me/Backup 0  
sdc                                               8:32   0 931,5G  0 disk
└─sdc1                                            8:33   0 931,5G  0 part
    └─backup-usb-1                              254:2    0 931,5G  0 crypt /run/media/backup-devices.d/backup1
                                                                           /run/media/backup-devices.d/backup1
                                                                           /run/media/backup-devices.d/backup1

A second manual mount attempt hangs for a long time after mounting backup1, then reports a changed fstab. I’d expect three already mounted.

# mount -av
... # (boot device mounts)
/run/media/backup-devices.d/backup1: already mounted
mount: (hint) your fstab has been modifieotherd, but systemd still uses 
       the old version; use 'systemctl daemon-reload' to reload.

No mention of the other two filesystems but a ‘changed’ fstab? What gremlin is messing with my sanity?!? (Realistically, I have a typo somewhere, or <insert basic dumbness here>.)

cscs · 16 August 2023 23:15

slimply:

### /etc/fstab
/dev/mapper/backup-usb-1 /run/media/backup-devices.d/backup1 btrfs x-systemd.automount,nofail,nouser,ssd,rw,noatime 0 0
/dev/mapper/backup-usb-2 /run/media/backup-devices.d/backup2 btrfs x-systemd.automount,nofail,nouser,ssd,rw,noatime 0 0

Use UUIDs (or PARTUUIDs) … nothing else is reliable.

https://wiki.archlinux.org/title/Fstab

Olli · 16 August 2023 23:22

noauto

you’re using the “noauto” option in crypttab. this saves some time at boot because the mounting procedure is ignored. that might cause trouble in your case.

there is a note at chapter 3 in the link below

https://wiki.archlinux.org/title/Fstab

1546611326-uuidgen-t · 16 August 2023 23:46

Dont use hypens in combination with systemd, they have special meaning when they are used in unit names that get created, so try to use underscores instead, or CamelCase without hypens nor underscores.

Don’t mount under /run that is a tmpfs, use a non-tmpfs mount point. like fe. /media
Drop fstab usage and use direct systemd mount units only…
Another point to check is the GUID’s of the GPT partitions on all three SSD’s, which might play a role in the confusion if they are identical…
Each disk needs different UUID’s just like partitions…

In this case he is required to use the /dev/mapper names because of the encrypted volumes used…

cscs · 16 August 2023 23:55

Am I missing something? Does not luks have UUIDs?

All GNU/Linux filesystems (including swap and LUKS headers of raw encrypted devices) support UUID

https://wiki.archlinux.org/title/Persistent_block_device_naming#by-uuid

1546611326-uuidgen-t · 16 August 2023 23:58

yes it does, and so does he in his crypttab

But when those are decrypted, the decrypted volumes are exposed as names under /dev/mapper

slimply · 17 August 2023 00:22

I made sure the mapped names are unique. I could imitate the automatic naming, though, and use partition uuid as fs-name:

sdb                                            0
└─sdb1                                         0 crypto_LUKS a386b816-2230-472b-b2a5-f66a76c66f1d                        9f2e3a83-63d2-4ed6-bff8-3ccd48131cfd
    └─luks-a386b816-2230-472b-b2a5-f66a76c66f1d

Will give it a try.

@ Olli
I’ll also try without noauto.

@ 1546611326-uuidgen-t
Good advice re naming, I think I need to overhaul my setup accordingly.

The UUIDs are unique:

# lsblk -o name,ro,fstype,uuid,label,partlabel,partuuid,mountpoints
sda                                            0
└─sda1                                         0 crypto_LUKS 3ad86390-5624-4e39-ab75-913d0c3e59f5 BackupUSB2             597447f6-5ae0-4df4-a7f9-28bb6c99474d
    └─luks-3ad86390-5624-4e39-ab75-913d0c3e59f5  0 btrfs       1c9721ca-66f5-4b00-b6bf-18afedeca1e6 backup-usb-2                                                /run/media/me/backup-usb-2
sdb                                            0
└─sdb1                                         0 crypto_LUKS a386b816-2230-472b-b2a5-f66a76c66f1d                        9f2e3a83-63d2-4ed6-bff8-3ccd48131cfd
    └─luks-a386b816-2230-472b-b2a5-f66a76c66f1d  0 btrfs       86c77618-2c16-4536-ac68-3d8777bc8397 Backup 0                                                    /run/media/me/Backup 0
sdc                                            0
└─sdc1                                         0 crypto_LUKS 5624ae33-96a8-43e6-b5a1-15690c6bb82d                        f29a9694-f2a1-7341-9871-c68f546e4899
    └─backup-usb-1                               0 btrfs       b3c7b795-c2d8-4191-84c2-b1f3c05bb946  BackupUSB1                                                  /run/media/backup-devices.d/backup1
                                                                                                                                                                 /run/media/backup-devices.d/backup1
                                                                                                                                                                 /run/media/backup-devices.d/backup1

Re mounting under /run/..., I imitated the Tip from the Arch Wiki:

File systems can also be mounted with systemd-mount instead of mount. If the mount point is not specified, the file system will be mounted at /run/media/system/device_identifier/.

Udev also mounts in /run:

By default, udisks2 mounts removable drives under the ACL controlled directory /run/media/$USER/

If they mount there, it could not be so wrong, or so I dreamt…

megavolt · 17 August 2023 06:38

Hello @slimply

I have similar chipset from JMicron 152d:0567, but with 4 HDDs. All HDDs are spanned and there is only one UUID. Metadata and Data have the DUP profile.

Anyway I had problems with x-systemd.automount when the controller goes into standby. I couldn’t mount it anymore because systemd “forgot” it. Instead of reactivating it, it tried to mount it again. It seems to me that you have a similar problem.

I couldn’t solve the problem, but with udisk2ctl it works great. systemd-automount might work good when it is connected internally, but it has problems with USB and its standby or auto suspend. Just my experience.

File: /etc/udev/rules.d/99-udisks2.rules

# UDISKS_FILESYSTEM_SHARED
# ==1: mount filesystem to a shared directory (/media/VolumeName)
# ==0: mount filesystem to a private directory (/run/media/$USER/VolumeName)
# See udisks(8)
ENV{ID_FS_USAGE}=="filesystem|other|crypto", ENV{UDISKS_FILESYSTEM_SHARED}="1"

File: /etc/udisks2/mount_options.conf

# Storage, you can set here special mount options for each partition, can also be /dev/mapper/xy
[/dev/disk/by-uuid/uuid]
btrfs_defaults=autodefrag,compress=zstd:9,space_cache=v2

There is also an example file with default options in the same folder.

Or /etc/fstab with noauto and x-mount.mkdir.

xabbu · 17 August 2023 06:52

And of course these will have UUID’s and Part UUID’s. They can be used like a non encrypted device after the LUKS device is opened.

slimply · 17 August 2023 08:14

That would explain the multiple mounts.
A hot lead!

Makes me think of trying TLP, enable USB autosuspend there and put the troubling devices and upstream hubs into the autosuspend deny list. If nothing overwrites that, maybe…

See my lsblk upthread for an example (ctrl-F “UUIDs are unique”).

I may not find time to test before weekend; will report back. Thanks to everybody for now!

dmt · 17 August 2023 08:49

Of course it doesn’t matter as the device file is specified in cryttab, it will only change if the user changes it.

The device file is usually both shorter and more human readable (except when luks-<uuid> is used) so it should be preferred (unlike /dev/sda etc).