Efi: There are differences between boot sector and its backup

When starting my system with an Stable Kernel, it lands into emergency shell.
Due this the system is not booting.:

sudo fsck.vfat -r -t /dev/nvme0n1p1
[sudo] Passwort für pheiduck: 
fsck.fat 4.2 (2021-01-31)
There are differences between boot sector and its backup.
This is mostly harmless. Differences: (offset:original/backup)
  65:01/00
1) Copy original to backup
2) Copy backup to original
3) No action
[123?q]? 2

How can I solve this?

System Info:

System:
  Kernel: 6.10.5-1-MANJARO arch: x86_64 bits: 64 compiler: gcc v: 14.2.1
    clocksource: tsc avail: hpet,acpi_pm
    parameters: BOOT_IMAGE=/@/boot/vmlinuz-6.10-x86_64
    root=UUID=2858d787-58d2-4b32-bfe2-dcd848223859 rw rootflags=subvol=@
    cryptdevice=UUID=ec00de08-393b-4d8c-a1ed-853a4de311ee:cryptroot quiet
    splash udev.log_priority=3 amd_pstate=active
  Desktop: Xfce v: 4.18.1 tk: Gtk v: 3.24.43 wm: xfwm4 v: 4.18.0
    with: xfce4-panel tools: light-locker,xfce4-screensaver dm: LightDM v: 1.32.0
    Distro: Manjaro base: Arch Linux
Machine:
  Type: Desktop Mobo: ASUSTeK model: PRIME X570-PRO v: Rev X.0x
    serial: <filter> part-nu: SKU uuid: f1be0f0b-83c9-788a-bfae-244bfe5cd0c0
    UEFI: American Megatrends v: 5013 date: 03/22/2024
Battery:
  Device-1: hidpp_battery_0 model: Logitech Wireless Mouse MX Master 3
    serial: <filter> charge: 100% (should be ignored) rechargeable: yes
    status: discharging
Memory:
  System RAM: total: 32 GiB available: 31.25 GiB used: 4.57 GiB (14.6%)
  Array-1: capacity: 128 GiB slots: 4 modules: 2 EC: None
    max-module-size: 32 GiB note: est.
  Device-1: DIMM_A1 type: no module installed
  Device-2: DIMM_A2 type: DDR4 detail: synchronous unbuffered (unregistered)
    size: 16 GiB speed: 3200 MT/s volts: curr: 1.2 min: 1.2 max: 1.2
    width (bits): data: 64 total: 64 manufacturer: Corsair
    part-no: CMK32GX4M2B3200C16 serial: N/A
  Device-3: DIMM_B1 type: no module installed
  Device-4: DIMM_B2 type: DDR4 detail: synchronous unbuffered (unregistered)
    size: 16 GiB speed: 3200 MT/s volts: curr: 1.2 min: 1.2 max: 1.2
    width (bits): data: 64 total: 64 manufacturer: Corsair
    part-no: CMK32GX4M2B3200C16 serial: N/A
CPU:
  Info: model: AMD Ryzen 9 3900X socket: AM4 bits: 64 type: MT MCP arch: Zen 2
    gen: 3 level: v3 note: check built: 2020-22 process: TSMC n7 (7nm)
    family: 0x17 (23) model-id: 0x71 (113) stepping: 0 microcode: 0x8701033
  Topology: cpus: 1x cores: 12 tpc: 2 threads: 24 smt: enabled cache:
    L1: 768 KiB desc: d-12x32 KiB; i-12x32 KiB L2: 6 MiB desc: 12x512 KiB
    L3: 64 MiB desc: 4x16 MiB
  Speed (MHz): avg: 2001 high: 4063 min/max: 550/4673 base/boost: 3800/4650
    scaling: driver: amd-pstate-epp governor: powersave volts: 1.1 V
    ext-clock: 100 MHz cores: 1: 3777 2: 3597 3: 3600 4: 3775 5: 3645 6: 4063
    7: 550 8: 550 9: 3500 10: 550 11: 550 12: 550 13: 550 14: 550 15: 550
    16: 550 17: 550 18: 550 19: 3599 20: 3937 21: 3566 22: 3828 23: 550 24: 550
    bogomips: 182451
  Flags: 3dnowprefetch abm adx aes aperfmperf apic arat avic avx avx2 bmi1
    bmi2 bpext cat_l3 cdp_l3 clflush clflushopt clwb clzero cmov cmp_legacy
    constant_tsc cpb cpuid cqm cqm_llc cqm_mbm_local cqm_mbm_total
    cqm_occup_llc cr8_legacy cx16 cx8 de decodeassists extapic extd_apicid
    f16c flushbyasid fma fpu fsgsbase fxsr fxsr_opt ht hw_pstate ibpb ibs
    irperf lahf_lm lbrv lm mba mca mce misalignsse mmx mmxext monitor movbe
    msr mtrr mwaitx nonstop_tsc nopl npt nrip_save nx osvw overflow_recov pae
    pat pausefilter pclmulqdq pdpe1gb perfctr_core perfctr_llc perfctr_nb
    pfthreshold pge pni popcnt pse pse36 rapl rdpid rdpru rdrand rdseed rdt_a
    rdtscp rep_good sep sev sev_es sha_ni skinit smap smca smep ssbd sse sse2
    sse4_1 sse4_2 sse4a ssse3 stibp succor svm svm_lock syscall tce topoext
    tsc tsc_scale umip v_spec_ctrl v_vmsave_vmload vgif vmcb_clean vme vmmcall
    wbnoinvd wdt x2apic xgetbv1 xsave xsavec xsaveerptr xsaveopt xtopology
  Vulnerabilities:
  Type: gather_data_sampling status: Not affected
  Type: itlb_multihit status: Not affected
  Type: l1tf status: Not affected
  Type: mds status: Not affected
  Type: meltdown status: Not affected
  Type: mmio_stale_data status: Not affected
  Type: reg_file_data_sampling status: Not affected
  Type: retbleed mitigation: untrained return thunk; SMT enabled with STIBP
    protection
  Type: spec_rstack_overflow mitigation: Safe RET
  Type: spec_store_bypass mitigation: Speculative Store Bypass disabled via
    prctl
  Type: spectre_v1 mitigation: usercopy/swapgs barriers and __user pointer
    sanitization
  Type: spectre_v2 mitigation: Retpolines; IBPB: conditional; STIBP:
    always-on; RSB filling; PBRSB-eIBRS: Not affected; BHI: Not affected
  Type: srbds status: Not affected
  Type: tsx_async_abort status: Not affected
Graphics:
  Device-1: AMD Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT]
    vendor: ASUSTeK driver: amdgpu v: kernel arch: RDNA-1 code: Navi-1x
    process: TSMC n7 (7nm) built: 2019-20 pcie: gen: 4 speed: 16 GT/s lanes: 16
    ports: active: HDMI-A-1 empty: DP-1,DP-2,DP-3 bus-ID: 0b:00.0
    chip-ID: 1002:731f class-ID: 0300
  Device-2: Logitech C922 Pro Stream Webcam driver: snd-usb-audio,uvcvideo
    type: USB rev: 2.0 speed: 480 Mb/s lanes: 1 mode: 2.0 bus-ID: 1-1:2
    chip-ID: 046d:085c class-ID: 0102 serial: <filter>
  Display: x11 server: X.Org v: 21.1.13 compositor: xfwm4 v: 4.18.0 driver:
    X: loaded: amdgpu unloaded: modesetting,radeon alternate: fbdev,vesa
    dri: radeonsi gpu: amdgpu display-ID: :0.0 screens: 1
  Screen-1: 0 s-res: 1920x1080 s-dpi: 96 s-size: 508x285mm (20.00x11.22")
    s-diag: 582mm (22.93")
  Monitor-1: HDMI-A-1 mapped: HDMI-A-0 model: Samsung SyncMaster
    serial: <filter> built: 2009 res: 1920x1080 hz: 60 dpi: 102 gamma: 1.2
    size: 477x268mm (18.78x10.55") diag: 587mm (23.1") ratio: 16:9, 15:9 modes:
    max: 1920x1080 min: 640x480
  API: EGL v: 1.5 hw: drv: amd radeonsi platforms: device: 0 drv: radeonsi
    device: 1 drv: swrast gbm: drv: radeonsi surfaceless: drv: radeonsi x11:
    drv: radeonsi inactive: wayland
  API: OpenGL v: 4.6 compat-v: 4.5 vendor: amd mesa v: 24.1.6-arch1.1
    glx-v: 1.4 direct-render: yes renderer: AMD Radeon RX 5700 XT (radeonsi
    navi10 LLVM 18.1.8 DRM 3.57 6.10.5-1-MANJARO) device-ID: 1002:731f
    memory: 7.81 GiB unified: no
Audio:
  Device-1: AMD Navi 10 HDMI Audio driver: snd_hda_intel v: kernel pcie: gen: 4
    speed: 16 GT/s lanes: 16 bus-ID: 0b:00.1 chip-ID: 1002:ab38 class-ID: 0403
  Device-2: AMD Starship/Matisse HD Audio vendor: ASUSTeK
    driver: snd_hda_intel v: kernel pcie: gen: 4 speed: 16 GT/s lanes: 16
    bus-ID: 0d:00.4 chip-ID: 1022:1487 class-ID: 0403
  Device-3: Logitech C922 Pro Stream Webcam driver: snd-usb-audio,uvcvideo
    type: USB rev: 2.0 speed: 480 Mb/s lanes: 1 mode: 2.0 bus-ID: 1-1:2
    chip-ID: 046d:085c class-ID: 0102 serial: <filter>
  Device-4: Logitech [G533 Wireless Headset Dongle]
    driver: hid-generic,snd-usb-audio,usbhid type: USB rev: 1.1 speed: 12 Mb/s
    lanes: 1 mode: 1.1 bus-ID: 3-2:2 chip-ID: 046d:0a66 class-ID: 0300
  API: ALSA v: k6.10.5-1-MANJARO status: kernel-api
    tools: alsactl,alsamixer,amixer
  Server-1: JACK v: 1.9.22 status: off tools: N/A
  Server-2: PipeWire v: 1.2.2 status: n/a (root, process) with:
    1: pipewire-pulse status: active 2: pipewire-media-session status: active
    3: pipewire-alsa type: plugin tools: pactl,pw-cat,pw-cli
Network:
  Device-1: Intel I211 Gigabit Network vendor: ASUSTeK driver: igb v: kernel
    pcie: gen: 1 speed: 2.5 GT/s lanes: 1 port: f000 bus-ID: 05:00.0
    chip-ID: 8086:1539 class-ID: 0200
  IF: enp5s0 state: up speed: 1000 Mbps duplex: full mac: <filter>
  IP v4: <filter> type: dynamic noprefixroute scope: global
    broadcast: <filter>
  IP v6: <filter> type: dynamic noprefixroute scope: global
  IP v6: <filter> type: noprefixroute scope: link
  IF-ID-1: virbr0 state: down mac: <filter>
  IP v4: <filter> scope: global broadcast: <filter>
  Info: services: NetworkManager,systemd-timesyncd
  WAN IP: <filter>
Bluetooth:
  Device-1: Cambridge Silicon Radio Bluetooth Dongle (HCI mode) driver: btusb
    v: 0.8 type: USB rev: 2.0 speed: 12 Mb/s lanes: 1 mode: 1.1 bus-ID: 1-2:3
    chip-ID: 0a12:0001 class-ID: e001
  Report: rfkill ID: hci0 rfk-id: 0 state: up address: see --recommends
Logical:
  Message: No logical block device data found.
  Device-1: cryptroot maj-min: 254:0 type: LUKS dm: dm-0 size: 419.26 GiB
  Components:
  p-1: nvme0n1p2 maj-min: 259:4 size: 419.26 GiB
RAID:
  Message: No RAID data found.
Drives:
  Local Storage: total: 1.82 TiB used: 118.85 GiB (6.4%)
  ID-1: /dev/nvme0n1 maj-min: 259:1 vendor: Samsung model: SSD 970 EVO 500GB
    size: 465.76 GiB block-size: physical: 512 B logical: 512 B speed: 31.6 Gb/s
    lanes: 4 tech: SSD serial: <filter> fw-rev: 2B2QEXE7 temp: 48.9 C
    scheme: GPT
  SMART: yes health: PASSED on: 11d 9h cycles: 2,093
    read-units: 6,823,828 [3.49 TB] written-units: 5,806,992 [2.97 TB]
  ID-2: /dev/nvme1n1 maj-min: 259:0 vendor: Samsung model: SSD 970 EVO 500GB
    size: 465.76 GiB block-size: physical: 512 B logical: 512 B speed: 31.6 Gb/s
    lanes: 4 tech: SSD serial: <filter> fw-rev: 2B2QEXE7 temp: 45.9 C
    scheme: GPT
  SMART: yes health: PASSED on: 7d 20h cycles: 2,093
    read-units: 8,653,937 [4.43 TB] written-units: 7,971,963 [4.08 TB]
  ID-3: /dev/sda maj-min: 8:0 vendor: Samsung model: SSD 870 QVO 1TB
    family: based SSDs size: 931.51 GiB block-size: physical: 512 B
    logical: 512 B sata: 3.3 speed: 6.0 Gb/s tech: SSD serial: <filter>
    fw-rev: 2B6Q temp: 33 C scheme: GPT
  SMART: yes state: enabled health: PASSED on: 48d 22h cycles: 1408
    written: 4.44 TiB
  Message: No optical or floppy data found.
Partition:
  ID-1: / raw-size: 419.26 GiB size: 419.26 GiB (100.00%)
    used: 75.76 GiB (18.1%) fs: btrfs block-size: 4096 B dev: /dev/dm-0
    maj-min: 254:0 mapped: cryptroot label: N/A uuid: N/A
  ID-2: /boot/efi raw-size: 512 MiB size: 511 MiB (99.80%)
    used: 27.7 MiB (5.4%) fs: vfat block-size: 512 B dev: /dev/nvme0n1p1
    maj-min: 259:2 label: N/A uuid: C380-DE14
  ID-3: /home raw-size: 419.26 GiB size: 419.26 GiB (100.00%)
    used: 75.76 GiB (18.1%) fs: btrfs block-size: 4096 B dev: /dev/dm-0
    maj-min: 254:0 mapped: cryptroot label: N/A uuid: N/A
  ID-4: /run/media/pheiduck/Daten raw-size: 931.51 GiB
    size: 931.51 GiB (100.00%) used: 43.06 GiB (4.6%) fs: btrfs
    block-size: 4096 B dev: /dev/sda1 maj-min: 8:1 label: N/A
    uuid: 3704c1e0-6710-4592-8b0b-28322fd45bbd
Swap:
  Kernel: swappiness: 60 (default) cache-pressure: 100 (default) zswap: yes
    compressor: zstd max-pool: 20%
  ID-1: swap-1 type: zram size: 15.63 GiB used: 0 KiB (0.0%) priority: 100
    comp: zstd avail: lzo,lzo-rle,lz4,lz4hc,842 max-streams: 24 dev: /dev/zram0
Unmounted:
  ID-1: /dev/nvme1n1p1 maj-min: 259:3 size: 16 MiB fs: N/A label: N/A uuid: N/A
  ID-2: /dev/nvme1n1p2 maj-min: 259:5 size: 418.41 GiB fs: ntfs
    label: Windoof uuid: 1E16606116603C41
  ID-3: /dev/nvme1n1p3 maj-min: 259:6 size: 778 MiB fs: ntfs label: N/A
    uuid: 6AA27E6AA27E3AA1
USB:
  Hub-1: 1-0:1 info: hi-speed hub with single TT ports: 6 rev: 2.0
    speed: 480 Mb/s (57.2 MiB/s) lanes: 1 mode: 2.0 chip-ID: 1d6b:0002
    class-ID: 0900
  Device-1: 1-1:2 info: Logitech C922 Pro Stream Webcam type: video,audio
    driver: snd-usb-audio,uvcvideo interfaces: 4 rev: 2.0
    speed: 480 Mb/s (57.2 MiB/s) lanes: 1 mode: 2.0 power: 500mA
    chip-ID: 046d:085c class-ID: 0102 serial: <filter>
  Device-2: 1-2:3 info: Cambridge Silicon Radio Bluetooth Dongle (HCI mode)
    type: bluetooth driver: btusb interfaces: 2 rev: 2.0
    speed: 12 Mb/s (1.4 MiB/s) lanes: 1 mode: 1.1 power: 100mA
    chip-ID: 0a12:0001 class-ID: e001
  Hub-2: 2-0:1 info: super-speed hub ports: 4 rev: 3.1
    speed: 10 Gb/s (1.16 GiB/s) lanes: 1 mode: 3.2 gen-2x1 chip-ID: 1d6b:0003
    class-ID: 0900
  Hub-3: 3-0:1 info: hi-speed hub with single TT ports: 6 rev: 2.0
    speed: 480 Mb/s (57.2 MiB/s) lanes: 1 mode: 2.0 chip-ID: 1d6b:0002
    class-ID: 0900
  Device-1: 3-2:2 info: Logitech [G533 Wireless Headset Dongle]
    type: audio,HID driver: hid-generic,snd-usb-audio,usbhid interfaces: 4
    rev: 1.1 speed: 12 Mb/s (1.4 MiB/s) lanes: 1 mode: 1.1 power: 100mA
    chip-ID: 046d:0a66 class-ID: 0300
  Device-2: 3-4:3 info: ASUSTek AURA LED Controller type: HID
    driver: hid-generic,usbhid interfaces: 2 rev: 2.0 speed: 12 Mb/s (1.4 MiB/s)
    lanes: 1 mode: 1.1 power: 16mA chip-ID: 0b05:18f3 class-ID: 0300
    serial: <filter>
  Hub-4: 4-0:1 info: super-speed hub ports: 4 rev: 3.1
    speed: 10 Gb/s (1.16 GiB/s) lanes: 1 mode: 3.2 gen-2x1 chip-ID: 1d6b:0003
    class-ID: 0900
  Hub-5: 5-0:1 info: hi-speed hub with single TT ports: 4 rev: 2.0
    speed: 480 Mb/s (57.2 MiB/s) lanes: 1 mode: 2.0 chip-ID: 1d6b:0002
    class-ID: 0900
  Hub-6: 5-3:2 info: Genesys Logic Hub ports: 4 rev: 2.0
    speed: 480 Mb/s (57.2 MiB/s) lanes: 1 mode: 2.0 power: 100mA
    chip-ID: 05e3:0608 class-ID: 0900
  Device-1: 5-3.2:4 info: Metadot Das Keyboard 4 type: keyboard,HID
    driver: hid-generic,usbhid interfaces: 2 rev: 1.1 speed: 12 Mb/s (1.4 MiB/s)
    lanes: 1 mode: 1.1 power: 100mA chip-ID: 24f0:204a class-ID: 0300
  Device-2: 5-4:3 info: Logitech Unifying Receiver type: keyboard,mouse,HID
    driver: logitech-djreceiver,usbhid interfaces: 3 rev: 2.0
    speed: 12 Mb/s (1.4 MiB/s) lanes: 1 mode: 1.1 power: 98mA
    chip-ID: 046d:c52b class-ID: 0300
  Hub-7: 6-0:1 info: super-speed hub ports: 4 rev: 3.1
    speed: 10 Gb/s (1.16 GiB/s) lanes: 1 mode: 3.2 gen-2x1 chip-ID: 1d6b:0003
    class-ID: 0900
Sensors:
  System Temperatures: cpu: 42.0 C mobo: 46.0 C gpu: amdgpu temp: 53.0 C
    mem: 50.0 C
  Fan Speeds (rpm): N/A gpu: amdgpu fan: 0
Info:
  Processes: 490 Power: uptime: 23m states: freeze,mem,disk suspend: deep
    avail: s2idle wakeups: 0 hibernate: platform avail: shutdown, reboot,
    suspend, test_resume image: 12.48 GiB services: upowerd,xfce4-power-manager
    Init: systemd v: 256 default: graphical tool: systemctl
  Packages: 1372 pm: pacman pkgs: 1357 libs: 386 tools: pamac pm: flatpak
    pkgs: 15 Compilers: clang: 18.1.8 gcc: 14.2.1 Shell: Sudo (sudo) v: 1.9.15p5
    default: Zsh v: 5.9 running-in: xfce4-terminal inxi: 3.3.35

Let the program decide:

sudo fsck.vfat -a /dev/nvme0n1p1

Okay thank you for this information. But how to fix this.
When I start my system with an stable Kernel the EFI part gets corrupted LTS Kernel do not have this issue.
System is turned off → Start Linux Kernel 6.11 → emergency shell
System is turned off → Start Linux Kernel 6.10.x → emergency shell
System is turned off → Start Linux Kernel 6.6.x → loginscreen

Stable kernels are considers as stable enough, not meaning that they are stable. It is probably a bug. If you are not able to debug the kernel, then my advice is just using a LTS kernel.

1 Like

Because you ask:

I may not understood what the solution was here.

Did you do the
fsck.vfat -a /dev/nvme0n1p1
or not?

could be done via boot from usb or from the running system by unmounting the efi partition before the check

I did it. Linux 6.6 is booting also when there are differences between boot sector and its backup.
As its saying: This is mostly harmless. but 6.10 and 6.11 may handle efi different?
Stick to 6.6 for now
I haven’t time to bisect kernels
I hope it will sort out from next LTS 6.12

Thank you for the feedback!
Strange situation - the efi partition apparently becoming corrupted / not cleanly unmounted with some kernels but not with others.

added fsck.repair=preen to Kernel Command line is fine now.
6.10 boots without any issue.

https://man.archlinux.org/man/systemd-fsck%40.service.8#KERNEL_COMMAND_LINE

Okay I removed apparmor=1 security=apparmor they have may caused this issue.

???

Are you sure?

The error is from the ESP not being unmounted/shutdown properly.

I suppose apparmor could contribute to this somehow or another - but its far from certain.

Simply powering off forcibly at the wrong time could cause this.

This could be in conjunction with somehow fragmented or broken kernels (maybe due to power loss?), as opposed to some ‘kernel bug’ in a particular series.

Without knowing the sequence of events, or the integrity of other parts of the system, its hard to say.

Though I would tend to think that if multiple kernels and you are seeing fsck-related messages … its probably broken partitions, incomplete packages, etc.

I updated the description with System Info.

I am not sure on this due it’s not happen on LTS Kernel. As it can be reproduced I will post the output from journalctl -xb