freezes on rockpro64

Until now, I figured the freezing on my rockpro64 in Manjaro was from kernel bugs, since it's still pretty young in mainline. The freezes are frequent. I haven't been able to get past one full round of updates.

So this last time, I set it up, installing just the kernel, mesa, and keyring updates, figuring I could get them in before it froze again. I was able to install them and reboot. On the next boot, there was severe graphical corruption. KDE was unusable.

I hit ctrl+alt+f2 and killed the ssd task, bringing down KDE. Then I ran Pacman -Syu. I managed to get as far as "195/252 upgrading kdelibs4support 65%" before it froze again. It's once again dead.

The board isn't defective. It was running openmediavault for months just fine. I tried armbian and it was fine too.

I suspect the mainline isn't stable enough for this board yet. Has anyone gotten this working stable yet?

ssdm, not ssd. Sorry, but I can't edit the typo.

I tried switching to SD card instead of MMC and putting everything on USB2 ports instead of using USB3. The freeze occurred again anyway.

Next, I tried disconnecting all USB devices except for the keyboard and mouse, which was connected via receiver directly to the board. No hubs. Still froze.

It's also never exceeded 50 degrees so it's not overheating.

It sounds really odd to me.

@dodgejcr tests RockPro64 images regularly and has not experienced any issues as far as I know.
I assume you are using the 19.06 KDE image?

Yes. I am trying it without the PCIE SATA card now. Its definitely not a power draw issue, but I'm wondering if the PCIE was tested. I don't imagine many people use it.

We haven't been able to test the PCIe yet, so no, it's not tested and probably not working.

If this goes through, it confirms its PCIe. I think I read somewhere that mainline 5.3 has a fix for this.

Interesting result! I plugged in everything as normal (except the MMC. It's toast.). The only difference from the original setup is I'm using an SD card instead of MMC and and the PCIE slot is now empty.

And it still froze! It's not hardware-related. It doesn't happen with either Ayufan or mrfixit's kernels on openmediavault. It doesn't happen on Armbian with their ancient kernel. So it's either happening because I'm on mainline or because I'm on Manjaro.

To narrow this down, is it possible to build Manjaro with one of the three kernels I've mentioned above?

It's not, since we don't have those kernels in the repo.

I am wondering why @dodgejcr has not encountered these issues on his RockPro64 though.
If it was software related, he should have seen it too.

Maybe it's a hardware revision issue. He might be testing against newer hardware than I have. Were there any known major revisions of this board?

Not that I know of.

I can spin you a test image with kernel 5.3-rc3 if you like?

I never disconnected the ethernet cord when I was ruling things out, but I find it incredibly unlikely it was tested without that.

This is now beyond my skills. Any logs or maybe hardware serial I could watch while it crashes?

Yes, please spin that up.

This might be helpful. Do you have a UART serial console?

I could rig something up from the USB serial adaptors I have from working with ESP chips. If the connectors are on GPIO, I won't even have to tear down the build to get at it.

However, it is 4AM, and I am tired. I'll check this again tomorrow. Perhaps we'll come up with something after sleeping on it.

No problem.
I will have a test image ready for you by then.

Will update this post with the link when it's done.

EDIT: Minimal Test image for RockPro64 with kernel 5.3-rc3.

There's a very long boot time, but I remember reading that's a kernel bug.

I haven't been able to crash or freeze it so far. I had it install KDE so I could try and replicate the update process it froze on last time. Seems good.

Is it possible to get a build where the only difference from release is the updated kernel?

It should only take like 30-40 seconds in that image.
A long boot time would be around 3-5 minutes.

I can build you a KDE image with the new kernel if you like. I can't make a new KDE image, where only the kernel is updated, as the image gets created from packages in the repo.

It sits at a black screen for a while, doing nothing. After I get up and am out to pull the plug, it finally shows a blinking cursor and starts booting.

I'm trying to figure out the build process for myself but this is already outside my scope. I'd like to see if I can build a version of Manjaro KDE with the 5.3 rc3 kernel and another with mrfixit's kernel so I can do a side by side. See what it take to kill one and see if that kills the other.

Ayufan also has a patched mainline kernel, but I'm not exactly sure what that means.