Manjaro crash on update 08/16/20

[jkz067archmanj]

I am usually able to figure out these problems, however, not this one. My system crashed yesterday during a Manjaro update and I became unable to boot. I tried to restore GRUB using this link:
https://wiki.manjaro.org/index.php/Restore_the_GRUB_Bootloader
My attempts failed at the updating GRUB part, however I am certain that deleting grub part worked. I was not swift enough to restart holding shift, prior to trying to restoring the GRUB. I am running in BIOS even though my system is UEFI capable. and I tried the UEFI part of this link to be sure, it did not work. The Manjaro system partition I want to boot and/or fix is sdb1, below.

lsblk -l shows (running from live iso):

manjaro@manjaro-gnome Linux 5.6.15-1-MANJARO x86_64 20.0.3 Lysia
~ >>> lsblk -f                                                                                       
NAME   FSTYPE FSVER  LABEL              UUID                                 FSAVAIL FSUSE% MOUNTPOINT
loop0  squash 4.0                                                                  0   100% /run/miso/
loop1  squash 4.0                                                                  0   100% /run/miso/
loop2  squash 4.0                                                                  0   100% /run/miso/
loop3  squash 4.0                                                                  0   100% /run/miso/
sda    iso966 Joliet MANJARO_GNOME_2003 2020-06-06-09-50-25-00                              
├─sda1 iso966 Joliet MANJARO_GNOME_2003 2020-06-06-09-50-25-00                     0   100% /run/miso/
└─sda2 vfat   FAT12  MISO_EFI           57E8-A8ED                                           
sdb                                                                                         
├─sdb1 ext4   1.0                       UUID######   I changed the UUIDs             
├─sdb2 ext4   1.0    JK-Files           UUID######                
└─sdb3 ext4   1.0    JK-TimeShift       UUID######                
sr0       

I tried the following:
Repairing the system using Manjaro Architect, the system would not repair.
Reparing the system using the “Disks” application in Manjaro, resulted in error, also not mounting.
Restoring the partition with the Timeshift image for sdb1, would not mount the target sdb1.

So, I think (hopefully), I just need to reinstall grub to make the system boot, and continue the update. However, I appear to be unable to do that. The only thing that will boot on restart+shift is memtest.

I really don’t want to rebuild a new system install from scratch. And, I don’t mind installing a new system as long as I can keep the /Home and configuration and settings files that are on sdb1. I am certain I can access my home directory and files on sdb1. I am able to access the files on sdb2 and sdb3 with no problem. I may also be able to run timeshift (or maybe even testdisk) and back up /Home. If I install a clean version of Manjaro on sdb1 “alongside” can I also then tell the Installer to use the existing /Home, configurations and settings? Then remove the old version of Manjaro?

Is there any way I can just copy the needed files to the root directory of sdb1? Any help you can offer is greatly appreciated. …Just thinking out loud a bit here, I am not really an expert.

I have been using Linux for a long time, so I know my way around the OS pretty well. I am not a programmer or developer. I started with Manjaro in December 2019 and I am [mostly] enjoying it. Thank you.

[dr_pants]

I had a crash as well during the last update and my system broke severely so I had to do a clean install. Grub wasn’t deleted in my case, but I was getting an error saying I had to load the kernel first and setting it manually didn’t work. Have you tried to chroot into your system and updating or install grub again?

[stephane]

• how have you updated , with pamac or in TTY / terminal ?
• have you an USB iso manjaro ?

[jkz067archmanj]

Well, I am very pleased to report a successful recovery with no data loss. I attribute this success to the backup programs TimeShift and CloneZilla, the countless hours of time that the Linux Community has put into posting their experience and knowledge over many years and all the advice to backup, backup and finally, backup. With all of that, however, the recovery was still tricky, carefully thought through and carried out.

Here is what I offer:

1. I had another SSD that I blanked out.
2. Used CloneZilla to copy the entire non-booting SSD to an image, then used CZ to image each partition individually…just in case.
3. Used CZ to re-image the blank SSD, using the full disk image. This way, the original SSD remains in tact. The re-imaged 2nd SSD did not boot, as expected. Which is okay in this case. CZ did try to fix the boot sector, however message said it failed. Like I said, it was not booting.
4. With the second SSD, now imaged and plugged in, I used the Manjaro Live USB as root to run TimeShift Restore and restored the first partition (which is sdb1, above). Not having run TimeShift ever before for this purpose, I watched very carefully as the process was displayed across the terminal screen. On process end, I shut it all down.
5. Removed the Live USB, reconnected the 2nd SSD, reboot and walla, it booted! The result was a SSD/os running a very close approximation to the crashed disk!
   Seriously, how exciting is that! It all worked. And reality says, duh, it worked as intended. I digress.
6. After booting and poking around, I noticed a few things were missing, gnotes, firefox bookmarks, downloads, and a few other things in /Home. So, I did a little homework on where the missing items were kept in the file system and wrote down a few things. Then I rebooted the Live USB, rooted through the terminal and went digging around the original-SSD’s file system for the missing pieces. I admit, It felt as if I was rolling the dice a bit here, however, it all worked out. No lost data. Once the missing pieces were located, I just copied them to the correct destinations on the restored disk.

A Few Takeaways…
1 You may notice that my Personal Files and TimeShift are kept in their own partitions. This is intentional and in this case, these partitions definitely made my recovery much easier. It prevented me from having to dig around the filesystem and cross-check which files I needed to get. These partitions remained fully in tact and separate from the crashed/OS partition.
2. Make backups and make regular backups. In this case, my most recent TimeShift backup was a couple months old. The TimeShift backup at the time of the 8/16 crash was incomplete, so I could not use it. Still, and fortunately, there were not many changes that I made during the interim.
3. With the restored disk up and running, all Manjaro updates completed successfully. And, the restore seems to run a bit more snappy. Go figure.

Going Forward…

1. I will get the restored disk back to where I need it with a few more little tweaks. Then I will probably return to the original disk using a CZ image.
2. Spend the money for a good backup drive, as you are able to.
3. Make backups and use them to your benefit.
4. Have a recovery plan or plan to have one.
5. I had hoped to find a way to do this all from the command-line. Just as well, however, this process went pretty smoothly once I came up with my plan and passed through the panic stage.

That is all. Happy *nixing.