System crash, cannot reboot

sh-4.4# inxi -Fxxxz
12System:    12Host manjaro 12Kernel 4.19.23-1-MANJARO x86_64 12bits 64 12compiler gcc 12v 8.2.1 
           12Desktop KDE Plasma 5 12tk Qt 5.11.1 12wm kwin_x11 12dm N/A 12Distro Manjaro Linux 
12Machine:   12Type Laptop 12System Dell 12product XPS 13 9360 12v N/A 12serial <filter> 12Chassis 12type 9 
           12serial <filter> 
           12Mobo Dell 12model 0PF86Y 12v A00 12serial <filter> 12UEFI Dell 12v 1.3.7 12date 07/04/2017 
12Battery:   12ID-1 BAT0 12charge 39.5 Wh 12condition 39.9/60.0 Wh (66%) 12volts 8.2/7.6 
           12model LGC-LGC7.790 DELL RNP7277 12type Li-ion 12serial <filter> 12status Discharging 
12CPU:       12Topology Dual Core 12model Intel Core i7-7560U 12bits 64 12type MT MCP 12arch Kaby Lake 12rev 9 
           12L2 cache 4096 KiB 
           12flags lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx 12bogomips 19204 
           12Speed 658 MHz 12min/max 400/3800 MHz 12Core speeds (MHz) 121 755 122 718 123 705 124 720 
12Graphics:  12Card-1 Intel 12vendor Dell 12driver i915 12v kernel 12bus ID 00:02.0 12chip ID 8086:5926 
           12Display x11 12server X.org 1.19.6 12driver intel 12unloaded modesetting 12alternate fbdev,vesa 
           12compositor kwin x11 12tty N/A 
           12Message Advanced graphics data unavailable for root. Old System? 
12Audio:     12Card-1 Intel Sunrise Point-LP HD Audio 12vendor Dell 12driver snd_hda_intel 12v kernel 
           12bus ID 00:1f.3 12chip ID 8086:9d71 
           12Sound Server ALSA 12v k4.19.23-1-MANJARO 
12Network:   12Card-1 Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter 12driver ath10k_pci 
           12v kernel 12bus ID 3a:00 12chip ID 168c:003e 
           12IF wlp58s0 12state up 12mac <filter> 
           12Card-2 Qualcomm Atheros 12type USB 12driver btusb 12bus ID 1:5 12chip ID 0cf3:e300 
12Drives:    12Local Storage 12total 491.88 GiB 12used 389.50 GiB (79.2%) 
           12ID-1 /dev/nvme0n1 12vendor Toshiba 12model THNSN5512GPUK NVMe 512GB 12size 476.94 GiB 
           12speed 31.6 Gb/s 12lanes 4 12serial <filter> 12rev 5KDA4103 
           12ID-2 /dev/sda 12type USB 12vendor Generic 12model Flash Disk 12size 14.94 GiB 12serial <filter> 
           12rev 8.07 
12Partition: 12ID-1 / 12size 410.28 GiB 12used 389.50 GiB (94.9%) 12fs ext4 12dev /dev/dm-0 
           12ID-2 /boot 12size 511.0 MiB 12used 408 KiB (0.1%) 12fs vfat 12dev /dev/nvme0n1p1 
12Sensors:   12System Temperatures 12cpu 44.0 C 12mobo N/A 
           12Fan Speeds (RPM) N/A 
12Info:      12Processes 191 12Uptime 7h 19m 12Memory 15.55 GiB 12used 1.17 GiB (7.5%) 12Init systemd 12v 239 
           12Compilers 12gcc 8.1.1 12Client Unknown Client: systemd 12inxi 3.0.18
sh-4.4# journalctl -b -p3
-- Logs begin at Sat 2018-09-01 09:39:25 CST, end at Mon 2019-05-13 14:48:31>
-- No entries --

That's almost certainly chroot, else you would have got an error message.

1 Like

Looking under partition ID-1 the system thinks there is very little free space.

Try making some space by partially cleaning the package cache:

sudo paccache -rvk3

This will remove all but the last 3 versions of each package. It should give some feedback. Check again with inxi if there is now some more space and try the update again. If there still is not enough space you may need to make room in another way by moving some of your personal (i.e. non-system) files elsewhere.

Looking at the kernel version the machine has not been updated for some time and there may be multiple Gb of updates needed plus there will need to be space to unpack and overwrite these packages.

Assuming the update works, once it finishes reboot to the normal system and check again with journalctl -b -p3 (messages from current boot of error or above priority) to confirm that there are no outstanding issues.

1 Like

Update appeared to work but;

Cpu core temp above threshold looks pretty damning, is it a hardware issue?

Also saw "failed to start load kernel modules"

Should I have mounted the swap partition before the update too?

when i try to update grub seems it fails

sh-5.0# sudo update-grub
Generating grub configuration file ...
Found background: /usr/share/grub/background.png
Found linux image: /boot/vmlinuz-4.9-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-4.9-x86_64.img
Found initrd fallback image: /boot/initramfs-4.9-x86_64-fallback.img
Found linux image: /boot/vmlinuz-3.16-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-3.16-x86_64.img
Found initrd fallback image: /boot/initramfs-3.16-x86_64-fallback.img
grub-probe: error: cannot find a GRUB drive for /dev/sda1.  Check your device.map.
Found memtest86+ image: /boot/memtest86+/memtest.bin
/usr/bin/grub-probe: warning: unknown device type nvme0n1.
done
sh-5.0# 

For the grub problem try the instructions here: Using livecd v17.0.1 (and above) as grub to boot OS with broken bootloader. Read the first post carefully to figure out what your particular setup might need.

If/when that is solved and there are still other problems then repost the journalctl ouptut here. :slightly_smiling_face:

when i get to grub> as per the directions although using a USB rather than a livecd, i use

search.file /etc/manjaro-release root

and it says the device doesnt exist

I am unable to follow any of those directions. I fail at every first step aside from being able to confirm I am efi

Do you think maybe I need to mount while in grub because of the encrypted drive?

i can see i boot into efi, i dont think there is any other boot partition

You can see there I tried to go into the normal grub> (not the live media) and had different results as far as I actually got some output. "crypto0" and "hd1, gpt1" "hd1, gpt1"

I do not know enough about this. You will need to wait for someone who knows grub better.

Perhaps @gohlip or @AgentS might be able to help?

1 Like

Then 2 months later

How was your system running all this time?
No boot problems?
No update, or no update problems?

I suspect you suspect correctly. If I read correctly, you were mounting your esp (vfat partition 1) to /boot instead on /boot/efi. During standard installations, esp should be mounted at /boot/efi.
You may have read some tutorials about encrypted systems that say "Mount your boot partition at /boot". I don't have experience with encryption, though I think this is not what you would want to do in your case.
To see how your system is originally configured on partitioning, you should read your system's /etc/fstab.
My question is also, what happened and some day you had boot problems, while everything was fine?
I suspect some hardware/firmware reset, or incorrect fstab, but we can't be sure.
I would suggest you try a couple of things...

  • Shutdown your system for a while, 15 minutes or a day, so any firmware/UEFI issues may "unload/reset".
  • Try to boot as normal to your system. If you get to grub menu (after crypto password), edit grub entry (press ‹kbd›E‹/kbd› at grub menu) and replace "quiet" with "3", at the linux line.
    This should bring you to TTY-no graphics. Then collect info
cat /etc/fstab
sudo fdisk -l
efibootmgr -v
lsblk -f
journalctl -b -1 -p3 --no-pager
df -h
grep -E "^Server" /etc/pacman.d/mirrorlist

To post those, we would prefer text (not images). Please try to use a web pastebin service and post a link, or do your best.

There maybe more than one issues to solve, but the main issue is that, in order to troubleshoot system problems you need to have some study in manuals (about installation on UEFI, encryption, partitioning, mount, pacman etc. ). If you want to maintain your system in confidence, of course :smile:

3 Likes

Before coming back to the thread I had waited a day or so and when booting up again to try to solve the problem I missed going to the boot menu and the machine booted as normal.

When I came back to this thread after two months I had what looked to be the same exact problem, so waited 24 hours and tried to just let it magically fix itself again, to no avail.

From the second re-emergence of the original problem I followed grinner's directions and seemed to have updated something as when I chroot now it is "sh-5. 0#" rather than 4.4 like before

Other than that I noticed nothing out of the ordinary aside from never really being able to full system update successfully. I had always just updated one by one the few programs I needed to use.

I don't know how you mean to use paste bin from console, I used OCR and also post the pictures

Here we are;

[root@xxx~]# cat /etc/fstab
UUID-5229-7F82   /boot/efi   vfat   defaults,noatime 0 2
/dev/mapper/luks-8667d333-164f-4756-8050-d9b498c191f7   /   ext4 defaults,noatime 0 1
/dev/mapper/luks-106889b8-3e26-40f9-b837-f2a7b1ca3081   swap   swap defaults,noatime 0 0

sudo fdisk -l
Disk /dev/nvme0n1: 477 GIB, 512110190592 bytes, 1000215216 sectors 
Disk model: THNSN5512GPUK NVMe TOSHIBA 512GB 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes 
I/O size (minimum/optimal): 512 bytes / 512 bytes 
Disklabel type: gpt 
Disk identifier: 52277A5A-6E01-4563-97B9-B8D0258CE6B0

Device Start End Sectors Size Type 
/dev/nvme0n1p1 2048 1050623 1048576 512M EFI System 
/dev/nvme0n1p2 1050624 877326335 876275712 417.9G Linux filesystem 
/dev/nvme0n1p3 877326336 893710335 16384000 7.8G Linux filesystem 

Disk /dev/sda: 15 GiB, 1694320800 bytes, 31334499 sectors 
Disk model: Flash Disk 
Units: sectors of 1 * 512 = 512 bytes 
Sector size (logical/physical): 512 bytes / 512 bytes 
I/O size (minimum/optimal): 512 bytes / 512 bytes 
Disklabel type: dos 
Disk identifier: 0x00000000 

Device Boot Start End Sectors Size Id Type 
/dev/sda1 * 64 4144743 4144680 2G 0 Empty 
/dev/sda2  4144744 4152935 8192 4M ef EFI (FAT-12/16/32) 

Disk /dev/mapper/luks-0667d333-164f-4756-8050-d9b498c191f7: 417.9 GiB, 448651067392 bytes, 876271616 sectors 
Units: sectors of 1 * 512 = 512 bytes 
Sector size (logical/physical): 512 bytes / 512 bytes 
I/0 size (minimum/optimal): 512 bytes / 512 bytes 

Disk /dev/mapper/luks-10688968-3e26-40f9-b837-12a7b1ca3081: 7.8 GiB, 8386510848 bytes, 16379904 sectors 
Units: sectors of 1 * 512 = 512 bytes 
Sector size (logical/physical): 512 bytes / 512 bytes 
I/0 size (minimum/optimal): 512 bytes / 512 bytes 

[root@xxx~]# efibootmgr -v 
BootCUrreNt: 0002 
Timeout: 0 seconds 
BootOrder: 0009.000A.000C.000D. 0002.000E 
Boot-0000* Windows Boot Manager HD(2$GPT$271ab8138-6443-4bc2-bb31-0f6Z0131g9t19,0x01000,0x32000)/FileNtrismicresofttapowpootwo, ef imo 
IliootNielL• 
Boot-0001* Manjaro HD(1,GPT 9806a7ba-d43c-4f6f-9fc5-ea368f56408b, 0x800,0X100000 )/file(\ EFI\ Manjaro\grubx64.efi)
Boot0002* Manjaro PciRoot(0x0)/Pc (Ox1d,0x0)/Pci (0x0,0x0)/NVMe(0x1, 00-08-0D-03-00-12-B4-D9)/HD(1.GPT.9806a7ba-d43c-4f6f-9fc5-ea368f56408b,0x800, 0x100000)/File(\EFI\boot\grubx64.efi)
Boot0009* Diskette Drive BBS(Floppy. Diskette Drives,0x0)..BO 
Boot000A• M.2 PCIe SSD BBS(HD,PO: THNSM5512GPUK NVMe TOSHIBA 512G,0x0)..BO
Boot000C* USB Storage Device BBS(USB,Generic Flash Disk 8.07,0x0)..BO
Boot000D* CD/DVD/CD-RW Drive BBS(CD/DVD/CD-Re Drive,0x0)..BO 
Boot000E- UEFI: Generic Flash Disk 8.07, Partition 1  PciRoot(0x0)/Pci(0x14,0x0)USB(1,0)HD(1,MBR,0x0,0x3f3e68,0x2000).. BO

There are so many mirrors from the last output, so I figured it wasn't necessary to post about it

For the journalctl output, the last 3 lines do not occur normally when I boot from grub using 3 instead of quiet in the Linux line. I mean to say there aren't any lines in the output after "failed to mount /boot/efi"

Do you still go to "Emergency shell", or no prompt?
If you get to no prompt, try changing TTY (Ctrl+Alt+4).
This issue looks a cryptosetup related, which I can't help.
But it could be because of the full disk. You've been advised to free space on the drive, but I see it's going worse. What have you done, or not done?

The /boot/efierror, should not block your system, not fatal AFAIK.
Then graphics/Xorg, you should have other symptoms, like garbled reception TTY or error messages.

It may also be a bad mirror. That's what was my advice on mirrorlist.
Many factors to investigate. Fix one at a time. Just watching a black screen is not going to fix it...:wink:

Yes, I go to "emergency mode" after grub menu.

I also just used;
sudo paccache - rvk3
Freed up 1.42GB, if that's not sufficient please recommend how to make more space from this shell

And here is the mirror list

sudo paccache - rk1

About the mirrorlist, select manually your nearest servers and place them at the top.
I suggest you check which ones are up-to-date and select one of them, disable the rest and do a proper sync to allign your DataBases

sudo pacman -Syyu

As I said, solve one at a time and hope for an ecryption expert to stop by and help. :wink:

Is there a way to connect to wifi while in the emergency shell? Since I can't be sure I chroot into the correct place I'd like to try to update from the emergency shell

I don't think it's possible to update from emergency shell (which is not the same as the rescue shell) because root is mounted read-only at best.

You said I may not be chrooting correctly while on liveUSB, does that mean i can try to update after mounting nvme0n1p1 to /mnt/boot/efi instead of mnt/boot?

I also went in and deleted about 20gb of media but maybe it's still in the recycle bin cause it still doesn't reflect more space freed up

  • now there is 60gb (14%) free when checking partition manager

I used Sudo Pacman-mirrors -i to generate new mirrors I think.

After I refreshed mirrors (I think) and made more space I -Syyu'd again on after chrooting the same way as before in the thread, I saw a different update process, and recognized several packages that I'm guessing wouldn't have shown up if I didn't chroot correctly in the first place (that I also didn't notice in past attempts to update recently) .

Have you tried booting to TTY (edit grub meny, replace quiet with 3 and press F10 to boot)?
Does this send you to emergency shell again?

Yes, I do exactly that most every time since you mentioned it, and yes I end up at at emergency shell

My issue is unresolved I hope someone can give me a suggestion. Even if not solving the problem directly. I'm pretty crippled in my daily life right now without being able to use this laptop

If I can transfer my virtual machines and one or two of the programs with their associated data, I would be happy to do a fresh install without encrypting.

Am I able to make a back up of the system and reinstall nonencrypted using said back up? Or something?