[ARM Testing Update] 2023-08-19 - Kernel, ICU, Systemd, KDE, Phosh, LibreOffice, Firefox

The same happened to me even before this update and it seems to happen to other images as well - see Manjaro generic does not boot on Odroid N2+ - #6 by Firewave

@Firewave that appears to be a different issue. That only affects booting after initial install, whereas the issue with the pinephone pro affects both initial install (from updated images) and updated existing installs.

To avoid the issue on the ppp, please don’t update the kernel package (linux-pinephonepro) past version 6.3 for the time being.

If you need to fresh install, you’ll need to use an image that comes with 6.3 instead of 6.4.

@lectrode The initial issue, yes. I just posted mine there as it “seemed” related and I didn’t find anything else related in recent forum posts.

I also just have a PinePhone - not a Pro.

Is there way to fix a broken system? Like putting the 6.3 image back into the root partition it and pointing the boot partition to it?
As mentioned in the other post I have data in there and have not yet been able to access all of it yet because of permissions. /boot for instance is just empty in all the images.

And something off-topic: I also tried to restore the backup to the phone but for some stupid reason it says the file is too big to fit on the same MicroSD I pulled the backup from. Is there some block/sector size stupidity going on here because different filesystems are involved?

Apologies for the delayed response - my focus has been elsewhere of late.

Interesting that you ran into that on a regular Pinephone. I tested the update on mine, and didn’t run into any issues. Did you use the installation image for the Pinephone? Or did you try using a different image?

The /boot folder on both the Pinephone and Pinephone Pro images should be it’s own partition. Unless you’re specifically mounting that other partition at that location, it’s going to be empty. The contents for me are as follows:

boot partition files
BOOT_MNJRO]$ ls -la
total 29504
drwxr-xr-x  3 lectrode lectrode    16384 Dec 31  1969 .
drwxr-x---+ 3 root     root           60 Aug 30 03:38 ..
-rw-r--r--  1 lectrode lectrode     1513 Jul 18 22:04 boot.scr
-rw-r--r--  1 lectrode lectrode     1441 Jun  8 23:34 boot.txt
drwxr-xr-x  3 lectrode lectrode     8192 Jul 18 22:04 dtbs
-rw-r--r--  1 lectrode lectrode   174080 Jun  8 23:34 idbloader.img
-rw-r--r--  1 lectrode lectrode 21336072 Jul  2 03:34 Image
-rw-r--r--  1 lectrode lectrode  7724215 Jul 18 22:05 initramfs-linux.img
-rw-r--r--  1 lectrode lectrode   920064 Jun  8 23:34 u-boot.itb

If you have another device of the same architecture (not sure if a pinebook would work - it might), you might be able to mount the partitions of that install, then chroot into it using manjaro-chroot (part of the manjaro-tools-base-git package). You would then be able to run the following commands under the context of that install:

sudo pacman -U /var/cache/pacman/pkg/linux-pinephone-6.3.*.zst
sudo mkinitcpio -P
sudo update-grub

(Please note that while I’ve run similar commands on x86_64 systems, I don’t have any means of doing this on different architectures and these commands are an educated guess, but untested)

As far as the backup is concerned, how did you take the backup? Was it via dd? Are you using that to restore it as well? (it looks like dd should truncate “extra” image data by default when writing it to “smaller” physical storage)

My PinePhone Pro wouldn’t boot either after updating but I was able to recover doing as @lectrode suggested.
Specifically:

  1. with Tow-boot enter the USB Mass Storage mode by holding the volume up button at startup before and during the second vibration.
  2. Connect the phone to your PC (Manjaro) and you should have the phone available as an additional disk like the following
    $ lsblk 
    NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
    sdb           8:16   1 115.2G  0 disk 
    ├─sdb1        8:17   1 457.8M  0 part 
    └─sdb2        8:18   1 114.8G  0 part 
    
  3. $ mkdir pinephone
    
  4. mount partitions as appropriate
    $ sudo mount /dev/sdb2 pinephone/
    $ sudo mount /dev/sdb1 pinephone/boot
    
  5. install qemu-user-static on your PC
  6. $ sudo cp /usr/bin/qemu-aarch64-static pinephone/bin/
    
    You can remove this after or leave it for next time :stuck_out_tongue_winking_eye:
  7. Optional: If DNS doesn’t work you can hardcode some nameservers with
    $ echo 'nameserver 1.1.1.1' | sudo tee -a pinephone/etc/resolv.conf
    
    Might want to clean this up after.
  8. mount host devices (not really sure if theae are strictly necessary but I did it last time too, and truthfully I mounted these at step 4 but should be fine to do it here too)
    $ sudo mount -t proc proc pinephone/proc
    $ sudo mount -t sysfs sys pinephone/sys
    $ sudo mount -o bind /dev pinephone/dev
    
  9. chroot this will open a bash shell to execute the following commands in
    $ sudo chroot pinephone/ /bin/qemu-aarch64-static /bin/bash
    
  10. Downgrade the kernel, acording to @lectrode anything about 6.3 will work but I had available and chose the following.
    $ pacman -U /var/cache/pacman/pkg/linux-pinephone-6.3.11-1-aarch64.pkg.tar.zst
    
  11. exit the chroot
  12. clean up as necessary then
    $ sudo umount -fR pinephone
    
  13. disconnect the device and reboot.

This made my PinePhone Pro boot again but I seem to have laggy audio and my mic doesn’t seem to work in calls anymore…
Edit: I just needed to reinstall pipewire-pulse seems it was replaced with the not working pulseaudio and calls work again

1 Like

I pushed some new kernel which should fix that issue with PPP.

1 Like

linux-pinephonepro 6.4.12-0 boots successfully on the ppp. Unfortunately, the wifi adapter is not detected on that kernel (modem seems to be detected - running some tests). Downgrading to the 6.3 series works-around the issue.

@lectrode can you check if this kernel fixes your wifi issue? Pipeline · manjaro-arm / packages / core / linux-pinephonepro · GitLab

Thank you - linux-pinepohonepro 6.4.14-1 from unstable has confirmed working wifi and modem. I haven’t come across any further issues with that kernel.

Found a few packages that need a rebuild in both testing and unstable:

1 Like

Thanks a lot @lectrode and @G-M0N3Y-2503 for the outstanding feedback. Sorry for the late reply, but…stuff.

I did all the steps but unfortunately running pacman fails with bash: /usr/bin/pacman: cannot execute binary file: Exec format error.

I guess the problem is that the machine I am connecting from is on a x86 architecture.

Update: I was lacking the qemu-user-static-binfmt package

Update 2:

Okay, that didn’t help.

I still had the 6.3.11-1 kernel installed (I knew there was’t an update) so I must have hit another issue.

I tried 6.3.8-1, 6.3.7-1 and 6.4.14-1 but to no avail.

It is showing the WARNING: No modules were added to the image. This is probably not what you want. when it builds the initfs images though.

The latest beta14 image which works still has a 6.1 kernel. But that also stops working when I update it to the latest versions.

It appears it is the missing kernel modules. I used the beta14 image and updated the kernel only and it is working. It also does not complain about missing modules when updating the kernel and lsmod shows a truck load of modules loaded. I cannot figure out where the module configuration is though.

Update: I found and checked countless configuration files, steps and log files - everything is identical. The broken system just doesn’t add the module dependencies when building the kernel.

Update 2: It is not an obvious issue with mkinitcpio either as I “downgraded” the script (only the script not the package) with the same result. I need to collect the set -x output on both systems and figure out what is the difference.

Update 3: It is caused by mkinitcpio. I updated it (only the script not the package - from version 33 to 36) on the system which was working and it no longer finds the modules. Unfortunately there were lots of changes in the scripts, which makes the diff extremely noisy.

Update 4: Ah - I see. That script is not standalone and has other scripts it depends on. So I have to up/downgrade the whole package. I don’t have the older one available (it’s a pain/impossible to somehow get the hold of older packages) and I didn’t have working internet on the chroot for some reason although it worked before.

Update 5: I updated the package to version 36 and the issue persists. So it appears to be caused by mkinitcpio. Unfortunately I don’t have the version 33 package to see if a downgrade fixes it.

Update 6: For some reason it is not adding any of the modules. It seems like it is not getting the modules at all!? I cannot figure out where it goes wrong. There’s just too many differences.

Does sudo depmod -a and reboot do anything?

No. It seems it is not the modules. I was able to boot the system even with a kernel which complained about no modules being included.

Interestingly I am getting the warning when I am installing from a local package with pacman -U but not when I install the same package from the repo via pacman -S.

Also the kernel is built twice as there is linux-pinephone.hook and linux.hook.

I see now. The Warning was with building the initrd image. On some devices that is normal as modules are not needed to set things up to boot since they are built into the kernel. I would say if it boots and lsmod shows modules it is probably ok.

I do not know about your device but does not sound quite right. Sounds like parts of 2 different kernels.

It turns out I was on the completely wrong track. My issues were not related to the kernel at all. The filesystem was corrupt! After I ran fsck it started working again… I came across this by accident because updating a package complained about not being able to write a file.

But now it doesn’t start the UI. I just see some debugfs boot messages and no logs were written at all. I had the same earlier when I updated the beta14 image to the latest packages. So I ran into a different issue now…

Update: It appears to be the ICU issue. startplasma-wayland fails with not finding libicui18n.so.72 and icu is 73.2-2.

Sounds like yo have the discontinued Qt5-ES2 packages installed.

sudo pacman -S qt5-base qt5-declarative qt5-multimedia pyside2

I just came across that post and resolved the conflicts. It’s working again!

Thanks for everybody helping out even though it was mainly the wrong track. I (re-)learned a lot of things that will hopefully be helpful in the future.