Nvme drive changing number

soundofthunder · 30 June 2024 03:49

This is what the Virtualbx manual has to say about raw hard disk access:

9.7.1.1. Access to Entire Physical Hard Disk:

On a Linux host, to create an image that represents an entire physical hard disk which will not contain any actual data, as this will all be stored on the physical disk, use the following command:
$ VBoxManage createmedium disk --filename path-to-file.vmdk --format=VMDK
 --variant RawDisk --property RawDrive=/dev/sda
This creates the path-to-file.vmdk file image that must be an absolute path. All data is read and written from /dev/sda.

Notice this states that the path-to-file.vmdk image must have an absolute path; but makes no mention of the physical disk itself requiring one.

The manual also suggests that raw hard disk access is possible for [both] entire disks and individual partitions - the only requirement is that a special VMDK image file which defines where the data will be stored is created.

So, if I’m understanding this correctly, a physical disk which only has one partition – /dev/nvme0n1p1 for example – could be used. By extension, this might also suggest that other path-to-partition notation can be used (UUIDs or Volume labels), but it doesn’t specifically state it.

With this logic, however, it might be possible (if the documentation is accurate) to target a specific partition using a mount point (symlink); something like:

--property RawDrive=/media/physical-partition-for-my-vm

So, in this way, the partition used can still be assigned an UUID (in /etc/fstab) as per usual, while the Virtualbox VM accesses the physical partition (using the special VMDK file) via its usual mount point.

These are only guesses; and untested guesses. Use at your own risk.

9.7.1. Using a Raw Host Hard Disk From a Guest

megavolt · 30 June 2024 08:59

Okay I tested it for you…

$ ls -l /dev/disk/* 
/dev/disk/by-id:
insgesamt 0
lrwxrwxrwx 1 root root 9 30. Jun 10:24 ata-QEMU_DVD-ROM_QM00001 -> ../../sr0

/dev/disk/by-partlabel:
insgesamt 0
lrwxrwxrwx 1 root root 10 30. Jun 10:24 root -> ../../vdb2

/dev/disk/by-partuuid:
insgesamt 0
lrwxrwxrwx 1 root root 10 30. Jun 10:30 3463f0cc-2658-494b-a135-ab22a19c1ea0 -> ../../vda1
lrwxrwxrwx 1 root root 10 30. Jun 10:24 50324670-b3d0-4187-9c66-ffa2da5e2fb1 -> ../../vdb2
lrwxrwxrwx 1 root root 10 30. Jun 10:24 c07133b6-c369-4da0-a107-48ed7c4375c6 -> ../../vdb1

/dev/disk/by-path:
insgesamt 0
lrwxrwxrwx 1 root root  9 30. Jun 10:24 pci-0000:00:1f.2-ata-1 -> ../../sr0
lrwxrwxrwx 1 root root  9 30. Jun 10:24 pci-0000:00:1f.2-ata-1.0 -> ../../sr0
lrwxrwxrwx 1 root root  9 30. Jun 10:24 pci-0000:04:00.0 -> ../../vda
lrwxrwxrwx 1 root root 10 30. Jun 10:30 pci-0000:04:00.0-part1 -> ../../vda1
lrwxrwxrwx 1 root root  9 30. Jun 10:24 pci-0000:08:00.0 -> ../../vdb
lrwxrwxrwx 1 root root 10 30. Jun 10:24 pci-0000:08:00.0-part1 -> ../../vdb1
lrwxrwxrwx 1 root root 10 30. Jun 10:24 pci-0000:08:00.0-part2 -> ../../vdb2
lrwxrwxrwx 1 root root  9 30. Jun 10:24 pci-0000:09:00.0 -> ../../vdc
lrwxrwxrwx 1 root root  9 30. Jun 10:24 virtio-pci-0000:04:00.0 -> ../../vda
lrwxrwxrwx 1 root root 10 30. Jun 10:30 virtio-pci-0000:04:00.0-part1 -> ../../vda1
lrwxrwxrwx 1 root root  9 30. Jun 10:24 virtio-pci-0000:08:00.0 -> ../../vdb
lrwxrwxrwx 1 root root 10 30. Jun 10:24 virtio-pci-0000:08:00.0-part1 -> ../../vdb1
lrwxrwxrwx 1 root root 10 30. Jun 10:24 virtio-pci-0000:08:00.0-part2 -> ../../vdb2
lrwxrwxrwx 1 root root  9 30. Jun 10:24 virtio-pci-0000:09:00.0 -> ../../vdc

/dev/disk/by-uuid:
insgesamt 0
lrwxrwxrwx 1 root root 10 30. Jun 10:24 187e6dba-d514-47e1-b27b-9c039d422c61 -> ../../vdb2
lrwxrwxrwx 1 root root 10 30. Jun 10:24 E434-E243 -> ../../vdb1

I will use /dev/vda here.

$ sudo VBoxManage createmedium disk --filename=somefile.vmdk --format=VMDK --variant RawDisk --property RawDrive=/dev/vda --property Partitions=1
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Medium created. UUID: 94f37f46-06b6-4c8e-88ba-ba10dcccb803

$ sudo cat somefile.vmdk
# Disk DescriptorFile
version=1
CID=db701f37
parentCID=ffffffff
createType="partitionedDevice"

# Extent description
RW 1 FLAT "somefile-pt.vmdk" 0
RW 2047 FLAT "somefile-pt.vmdk" 1
RW 41938944 FLAT "/dev/vda" 2048
RW 2048 ZERO 

# The disk Data Base 
#DDB

ddb.virtualHWVersion = "4"
ddb.adapterType="ide"
ddb.geometry.cylinders="16383"
ddb.geometry.heads="16"
ddb.geometry.sectors="63"
ddb.uuid.image="94f37f46-06b6-4c8e-88ba-ba10dcccb803"
ddb.uuid.parent="00000000-0000-0000-0000-000000000000"
ddb.uuid.modification="00000000-0000-0000-0000-000000000000"
ddb.uuid.parentmodification="00000000-0000-0000-0000-000000000000"

Nice, it points to /dev/vda. Good

sudo VBoxManage createmedium disk --filename=somefile1.vmdk --format=VMDK --variant RawDisk --property RawDrive=/dev/disk/by-path/pci-0000:04:00.0 --property Partitions=1
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Medium created. UUID: 76fcdba8-c4a8-4d35-81d5-fa94d7dd2d60

$  sudo cat somefile1.vmdk
# Disk DescriptorFile
version=1
CID=be47579f
parentCID=ffffffff
createType="partitionedDevice"

# Extent description
RW 1 FLAT "somefile1-pt.vmdk" 0
RW 2047 FLAT "somefile1-pt.vmdk" 1
RW 41938944 FLAT "/dev/disk/by-path/pci-0000:04:00.0" 2048
RW 2048 ZERO 

# The disk Data Base 
#DDB

ddb.virtualHWVersion = "4"
ddb.adapterType="ide"
ddb.geometry.cylinders="16383"
ddb.geometry.heads="16"
ddb.geometry.sectors="63"
ddb.uuid.image="76fcdba8-c4a8-4d35-81d5-fa94d7dd2d60"
ddb.uuid.parent="00000000-0000-0000-0000-000000000000"
ddb.uuid.modification="00000000-0000-0000-0000-000000000000"
ddb.uuid.parentmodification="00000000-0000-0000-0000-000000000000"

okay… it uses /dev/disk/by-path/pci-0000:04:00.0 just like /dev/vda. Both are equal. What’s the problem?

Or use the partuuid, if it is a partition. Here it is /dev/vda1:

sudo VBoxManage createmedium disk --filename=somefile2.vmdk --format=VMDK --variant RawDisk --property RawDrive=/dev/disk/by-partuuid/3463f0cc-2658-494b-a135-ab22a19c1ea0

$ sudo cat somefile2.vmdk 
# Disk DescriptorFile
version=1
CID=0b2795a7
parentCID=ffffffff
createType="fullDevice"

# Extent description
RW 41938944 FLAT "/dev/disk/by-partuuid/3463f0cc-2658-494b-a135-ab22a19c1ea0" 0

# The disk Data Base 
#DDB

ddb.virtualHWVersion = "4"
ddb.adapterType="ide"
ddb.geometry.cylinders="16383"
ddb.geometry.heads="16"
ddb.geometry.sectors="63"
ddb.uuid.image="31115c42-97f0-4b6a-90e6-9c579bcbf5e9"
ddb.uuid.parent="00000000-0000-0000-0000-000000000000"
ddb.uuid.modification="00000000-0000-0000-0000-000000000000"
ddb.uuid.parentmodification="00000000-0000-0000-0000-000000000000"

The VMDK file is just a pointer here to a physical disk. It can use symbolic links.

lanepark · 30 June 2024 17:56

The problem is, it appears to be related to nvme storage devices according to Gigabyte support, as the physical m.2 device doesn’t always come up with the same nvme0n1xx address, so by-id will still point to the wrong m.2 device. I believe the same is true for path. I’ve never had this issue when I was using SATA drives for storage, this issue has only surfaced when I move to m.2 devices for storing the rawdisk installs and wanted to run them as vbox clients. I have not tried to boot rawmetal (Via grub) after the devices (m.2) were assigned the wrong identifiers. I have discovered that the incident of switching identifiers has become less frequent so I simply check via gnome-disk to assure the correct physical nvme device contains the guest distros and reboot usually fixes the issue.

megavolt · 30 June 2024 18:12

Okay, you still don’t get the point. I try it more simple:

/dev/sda or /dev/vda or /dev/nvme0n1 can always change depending on which device initialize faster. The initialization process works parallel at the kernel, to speed it up. /dev/disk/by-whatever/ are symlinks with information about the device in its name, they are added afterwards by udev. That are reliable names then. If you have 2 identical NVME’s, then the ID would be probably the same, thus not reliable in your case.

Please give us a favor and share the output of:

ls -l /dev/disk/*

xabbu · 30 June 2024 18:13

No it doesn’t. The Links in by-id and in any other by-* folder are newly created on every boot to the nvme device. These links point always to the correct device depending on the by property. That is the whole purpose of it.

soundofthunder · 30 June 2024 18:19

Has anyone bothered to try (or read) my earlier proposal? I’m afraid I don’t have storage available to test it; referencing the mount point;

--property RawDrive=/media/physical-partition-for-my-vm

lanepark · 1 July 2024 07:02

Thanks for being patient with me, I am new at Linux, and have not delved into the source of the kernel.
I do have a thirst for knowledge though my mind is a bit sluggish most of the time.
I did create a VMDK to point to one of my clients using /dev/disk/by-id and vboxmanage accepted the RawDrive parm as such. So I waited until the 2 m.2 devices came up switched (As of lately it has not occured after 20 or so boots) and tried the VMDK file built with by-id, and it worked. IE Vbox found the m.2 device and booted the client. Looking at the client via gnome-disks the volumes do come up as sdax rather then nvme’anything’.

soundofthunder · 1 July 2024 07:20

Could you please provide the output requested by @megavolt :-

And also the output from:

cat /etc/fstab

Note: When using UUIDs it doesn’t matter how frequently the device node order (the order of disks) changes, as the disk (or partition) will always be referenced only by the UUID. This is part of the design. I hope this helps to understand a little better. Cheers.

lanepark · 1 July 2024 07:24

I’m working on that, as the drives came up switched and didn’t want to provide output from that boot. I’m only going to provide the by-id as we are getting into the workings of my system and posting it on an open forum.

lanepark · 1 July 2024 07:30

/dev/disk/by-id:
total 0
lrwxrwxrwx 1 root root  9 Jul  1  2024 ata-CT240BX200SSD1_1613F01900BD -> ../../sda
lrwxrwxrwx 1 root root 10 Jul  1  2024 ata-CT240BX200SSD1_1613F01900BD-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jul  1  2024 ata-CT240BX200SSD1_1613F01900BD-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jul  1  2024 ata-CT240BX200SSD1_1613F01900BD-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 Jul  1  2024 ata-CT240BX200SSD1_1613F01900BD-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jul  1  2024 ata-CT240BX200SSD1_1613F01900BD-part5 -> ../../sda5
lrwxrwxrwx 1 root root  9 Jul  1  2024 ata-CT500MX500SSD1_2044E4C2FD07 -> ../../sdb
lrwxrwxrwx 1 root root 10 Jul  1  2024 ata-CT500MX500SSD1_2044E4C2FD07-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Jul  1  2024 ata-CT500MX500SSD1_2044E4C2FD07-part2 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Jul  1  2024 ata-CT500MX500SSD1_2044E4C2FD07-part3 -> ../../sdb3
lrwxrwxrwx 1 root root 10 Jul  1  2024 ata-CT500MX500SSD1_2044E4C2FD07-part4 -> ../../sdb4
lrwxrwxrwx 1 root root 10 Jul  1  2024 ata-CT500MX500SSD1_2044E4C2FD07-part5 -> ../../sdb5
lrwxrwxrwx 1 root root 10 Jul  1  2024 ata-CT500MX500SSD1_2044E4C2FD07-part6 -> ../../sdb6
lrwxrwxrwx 1 root root  9 Jul  1  2024 ata-HL-DT-ST_BD-RE_WH14NS40_K91E7CA4443 -> ../../sr0
lrwxrwxrwx 1 root root 13 Jul  1  2024 nvme-CT2000P3PSSD8_2331E866F901 -> ../../nvme1n1
lrwxrwxrwx 1 root root 13 Jul  1  2024 nvme-CT2000P3PSSD8_2331E866F901_1 -> ../../nvme1n1
lrwxrwxrwx 1 root root 15 Jul  1  2024 nvme-CT2000P3PSSD8_2331E866F901_1-part1 -> ../../nvme1n1p1
lrwxrwxrwx 1 root root 15 Jul  1  2024 nvme-CT2000P3PSSD8_2331E866F901_1-part2 -> ../../nvme1n1p2
lrwxrwxrwx 1 root root 15 Jul  1  2024 nvme-CT2000P3PSSD8_2331E866F901_1-part3 -> ../../nvme1n1p3
lrwxrwxrwx 1 root root 15 Jul  1  2024 nvme-CT2000P3PSSD8_2331E866F901_1-part4 -> ../../nvme1n1p4
lrwxrwxrwx 1 root root 15 Jul  1  2024 nvme-CT2000P3PSSD8_2331E866F901_1-part5 -> ../../nvme1n1p5
lrwxrwxrwx 1 root root 15 Jul  1  2024 nvme-CT2000P3PSSD8_2331E866F901-part1 -> ../../nvme1n1p1
lrwxrwxrwx 1 root root 15 Jul  1  2024 nvme-CT2000P3PSSD8_2331E866F901-part2 -> ../../nvme1n1p2
lrwxrwxrwx 1 root root 15 Jul  1  2024 nvme-CT2000P3PSSD8_2331E866F901-part3 -> ../../nvme1n1p3
lrwxrwxrwx 1 root root 15 Jul  1  2024 nvme-CT2000P3PSSD8_2331E866F901-part4 -> ../../nvme1n1p4
lrwxrwxrwx 1 root root 15 Jul  1  2024 nvme-CT2000P3PSSD8_2331E866F901-part5 -> ../../nvme1n1p5
lrwxrwxrwx 1 root root 13 Jul  1  2024 nvme-CT2000P3SSD8_2311E6B9B101 -> ../../nvme0n1
lrwxrwxrwx 1 root root 13 Jul  1  2024 nvme-CT2000P3SSD8_2311E6B9B101_1 -> ../../nvme0n1
lrwxrwxrwx 1 root root 15 Jul  1  2024 nvme-CT2000P3SSD8_2311E6B9B101_1-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 16 Jul  1  2024 nvme-CT2000P3SSD8_2311E6B9B101_1-part10 -> ../../nvme0n1p10

lanepark · 1 July 2024 07:33

This is the value I used for by-id the serial number of the actual m.2 device is imbeded in the value.
nvme-CT2000P3SSD8_2311E6B9B101

soundofthunder · 1 July 2024 07:35

Let’s make that a little more readable. Edit your post and add ``` (three backtick characters) both before and after the pasted text; then it will appear in a scrollable text box, much like my example previously.

Actually, nevermind.

I’ll simply wish you luck.

Teo · 1 July 2024 08:53

As you can see, the by-id contains the serial number of the disk, so it will be unique. As by-label and by-uuid. At least 3 different ways.

lanepark · 1 July 2024 14:06

Yes, Agreed the serial number is embedded in the by-id. When doing the “ls” I can not find a by-uuid or by-label that represents the whole m.2 device. It’s my understanding that the vbox “createmedium” parm for “RawDrive” needs to point to the whole m.2 device, not a volume on the device. That is what the “partitions” parm is used for.

soundofthunder · 1 July 2024 15:48

As quoted earlier, directly from the Virtualbox manual;

This is the command structure for accessing a disk:

VBoxManage createmedium disk --filename path-to-file.vmdk --format=VMDK
 --variant RawDisk --property RawDrive=/dev/sda

And the command structure for accessing partitions:

VBoxManage createmedium disk --filename path-to-file.vmdk --format=VMDK
--variant RawDisk --property RawDrive=/dev/sda --property Partitions=1,5

So, using the PARTUUID (which is different on every disk - or, should be) we access either the dis or partition(s) using these formats:

--property RawDrive=/dev/disk/by-partuuid/xxxx-partuuid-here-xxxx
= entire disk
--property RawDrive=/dev/disk/by-partuuid/xxxx-partuuid-here-xxxx --property Partitions=1
= individual partition.

Given that both entire disks and individual partitions are in fact supported, all that remains is how you access the disk (whether by id, or uuid, etc); this is irrespective of whether or not you wish to access any partitions.

I note that @megavolt has already confirmed that using UUIDs will work.
It makes absolutely no difference whic device node is listed before another.

So, use this format for RawDrive and add the Partitions parameter (as well) only if you wish to access partitions.

Cheers.

lanepark · 1 July 2024 17:27

[lane@ManjaroLP ~]$ sudo vboxmanage createmedium disk --filename nvmeuuidtest1.vmdk --format=VMDK --variant RawDisk --property RawDrive=/dev/disk/by-partuuid/d556d536-170f-4a35-a0db-26d889a07c56 --property Partitions=7,8
0%...VBOX_E_FILE_ERROR
VBoxManage: error: Failed to create medium
VBoxManage: error: Could not create the medium storage unit '/home/lane/nvmeuuidtest1.vmdk'.
VBoxManage: error: VMDK: Image path: '/home/lane/nvmeuuidtest1.vmdk'. RTDvmMapOpen failed for '/dev/disk/by-partuuid/d556d536-170f-4a35-a0db-26d889a07c56' (VERR_NOT_SUPPORTED) (VERR_NOT_SUPPORTED).
VBoxManage: error: VMDK: could not create raw descriptor for '/home/lane/nvmeuuidtest1.vmdk' (VERR_NOT_SUPPORTED)
VBoxManage: error: Details: code VBOX_E_FILE_ERROR (0x80bb0004), component MediumWrap, interface IMedium
VBoxManage: error: Context: "RTEXITCODE handleCreateMedium(HandlerArg*)" at line 630 of file VBoxManageDisk.cpp
[lane@ManjaroLP ~]$ sudo vboxmanage createmedium disk --filename nvmeuuidtest1.vmdk --format=VMDK --variant RawDisk --property RawDrive=/dev/disk/by-partuuid/d556d536-170f-4a35-a0db-26d889a07c56
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Medium created. UUID: cac00344-f7c8-4f32-979e-91435050c3e8
[lane@ManjaroLP ~]$

I tried by-partuuid 2 different ways, one with the Partitions parm included and one wothout the Partitions parm. Above shows that vbox does not like by-partuuid when specifing Partitions. Without the Partitions parm vbox is happy.
(I added the tripple ticks fore and aft, so we’ll see if it scrolls.)

I use the partitions parm to isolate each client from the other clients on the same m.2 device, just incase…

soundofthunder · 1 July 2024 19:14

Then, you’re doing something wrong…

Except you used the wrong character. The backtick key is usually found under the Esc key (on a full keyboard); it looks like `.

You can also use the </> button before you paste the content.

From the text dump, it looks like you didn’t use a path to the .vmdk file (in the command). A path is required; absolute is probably best.

# The example
--filename path-to-file.vmdk
# So, something like
--filename /home/lane/my_nvme_test1.vmdk

Try again.

It shows you have an error; you don’t (yet) know exactly what the error means. Search internet for anything you can find about the error:

Try Virtualbox Raw Host VERR_NOT_SUPPORTED as the search term, for example.

lanepark · 1 July 2024 20:00

All I did was attempt to create the file with the Partitions parm specified, which failed, the recalled the same entry and delete the Partitions parm and it worked. I wanted vboxmanage to put the output file in the directory I was in, as the directory where I normally store .vmdk files does not need to be cluttered with tests.

Yes, I don’t know what the actual failure is. Will go look as you suggest, later.

Later:
Cant find anything that would explain the failure to createmedium short of removing the Partitions parm. I was able to createmedium using by-id instead of by-partuuid and include the Partitions parm successfully. So am suspicious that vbox doesn’t like partitions defined in both the RawDrive and Partitions parameters.

soundofthunder · 1 July 2024 20:11

In your blob of text was this:

sudo vboxmanage createmedium disk --filename nvmeuuidtest1.vmdk

Do you see that the .vmdk file has no path? It still needs a path defined in the command (regardless of the path appearing in errors; that’s different).

I think that might be part of the problem; possibly.

Otherwise, I don’t see what the problem is; at least, nothing jumps out with any certainty.

lanepark · 2 July 2024 00:11

soundofthunder:

In your blob of text was this:
sudo vboxmanage createmedium disk --filename nvmeuuidtest1.vmdk
Do you see that the .vmdk file has no path? It still needs a path defined in the command (regardless of the path appearing in errors; that’s different).

I think that might be part of the problem; possibly.

Otherwise, I don’t see what the problem is; at least, nothing jumps out with any certainty.

The blob shows 2 attempts to build a VMDK file, one fails, the other (After removing the Partitions parmeter) did not fail and built the vmdk file in the “current” directory, in this case, /home/lane.