Process Hanging System - how to find?

Also, if you are running xfce, there is Task Manager (xfce4-taskmanager)

If my computer is already hanging -then I can’t open up a terminal window - or if it is open I can’t swap over to it… any ideas on how TOP can help me after the fact?

Unfortunately not…

You have already tried to see whether there is something in the system log?
journalctl -r
for example,
lists it, most recent first

you can also filter for priority with the -p flag
man journalctl

and/or have a terminal open with top continuously running in it …

there is also the file ~/.xsession-errors if your session runs under xorg

If the entire system is hanging for 3-5 minutes, then that sounds like the OOM (out of memory) killer deciding to kill something. If so, then that should be in dmesg. You could try doing sudo dmesg -T | grep -Ei 'killed process' and you may find some info there.

But it’s also true that if it is the OOM killer, it may be worth having a system monitor open and sorting by RAM usage. I’d expect you to be able to see something gradually using up all the memory before it happens (unless you’re doing your own software development, where bugs can sneak up on you whenever).

I have been finding some OOM messages after my system is hanging…

Different processes are being killed (makes sense as it does not appear to be a particular app causing the issue)

My system only has 8gb ram (fixed) so I can’t expand it)

Is there any system settings I can adjust (caching? Swap space???) Any way to set priorities of what gets killed?

I guess I can just be careful with what I’m launching and when.

System information would help …

inxi -Farzy

Sure

CPU:
  Info: model: Intel Core i7-6500U bits: 64 type: MT MCP arch: Skylake
    gen: core 6 level: v3 note: check built: 2015 process: Intel 14nm family: 6
    model-id: 0x4E (78) stepping: 3 microcode: 0xF0
  Topology: cpus: 1x cores: 2 tpc: 2 threads: 4 smt: enabled cache:
    L1: 128 KiB desc: d-2x32 KiB; i-2x32 KiB L2: 512 KiB desc: 2x256 KiB
    L3: 4 MiB desc: 1x4 MiB
  Speed (MHz): avg: 1700 min/max: 400/3100 scaling: driver: intel_pstate
    governor: powersave cores: 1: 1700 2: 1700 3: 1700 4: 1700 bogomips: 20810
  Flags: 3dnowprefetch abm acpi adx aes aperfmperf apic arat
    arch_capabilities arch_perfmon art avx avx2 bmi1 bmi2 bts clflush
    clflushopt cmov constant_tsc cpuid cpuid_fault cx16 cx8 de ds_cpl dtes64
    dtherm dts epb erms est f16c flush_l1d fma fpu fsgsbase fxsr ht hwp
    hwp_act_window hwp_epp hwp_notify ibpb ibrs ida intel_pt invpcid lahf_lm
    lm mca mce md_clear mmx monitor movbe mpx msr mtrr nonstop_tsc nopl nx pae
    pat pbe pcid pclmulqdq pdcm pdpe1gb pebs pge pln pni popcnt pse pse36 pti
    pts rdrand rdseed rdtscp rep_good sdbg sep smap smep ss ssbd sse sse2
    sse4_1 sse4_2 ssse3 stibp syscall tm tm2 tsc tsc_adjust tsc_deadline_timer
    vme x2apic xgetbv1 xsave xsavec xsaveopt xsaves xtopology xtpr
  Vulnerabilities:
  Type: gather_data_sampling status: Vulnerable: No microcode
  Type: itlb_multihit status: KVM: VMX unsupported
  Type: l1tf mitigation: PTE Inversion
  Type: mds mitigation: Clear CPU buffers; SMT vulnerable
  Type: meltdown mitigation: PTI
  Type: mmio_stale_data mitigation: Clear CPU buffers; SMT vulnerable
  Type: retbleed mitigation: IBRS
  Type: spec_rstack_overflow status: Not affected
  Type: spec_store_bypass mitigation: Speculative Store Bypass disabled via
    prctl
  Type: spectre_v1 mitigation: usercopy/swapgs barriers and __user pointer
    sanitization
  Type: spectre_v2 mitigation: IBRS, IBPB: conditional, STIBP: conditional,
    RSB filling, PBRSB-eIBRS: Not affected
  Type: srbds mitigation: Microcode
  Type: tsx_async_abort status: Not affected

Most of it is missing.
Please include the full output, as well as make use of the code format buttons.

CPU:
  Info: model: Intel Core i7-6500U bits: 64 type: MT MCP arch: Skylake
    gen: core 6 level: v3 note: check built: 2015 process: Intel 14nm family: 6
    model-id: 0x4E (78) stepping: 3 microcode: 0xF0
  Topology: cpus: 1x cores: 2 tpc: 2 threads: 4 smt: enabled cache:
    L1: 128 KiB desc: d-2x32 KiB; i-2x32 KiB L2: 512 KiB desc: 2x256 KiB
    L3: 4 MiB desc: 1x4 MiB
  Speed (MHz): avg: 627 high: 726 min/max: 400/3100 scaling:
    driver: intel_pstate governor: powersave cores: 1: 685 2: 726 3: 400 4: 700
    bogomips: 20810
  Flags: 3dnowprefetch abm acpi adx aes aperfmperf apic arat
    arch_capabilities arch_perfmon art avx avx2 bmi1 bmi2 bts clflush
    clflushopt cmov constant_tsc cpuid cpuid_fault cx16 cx8 de ds_cpl dtes64
    dtherm dts epb erms est f16c flush_l1d fma fpu fsgsbase fxsr ht hwp
    hwp_act_window hwp_epp hwp_notify ibpb ibrs ida intel_pt invpcid lahf_lm
    lm mca mce md_clear mmx monitor movbe mpx msr mtrr nonstop_tsc nopl nx pae
    pat pbe pcid pclmulqdq pdcm pdpe1gb pebs pge pln pni popcnt pse pse36 pti
    pts rdrand rdseed rdtscp rep_good sdbg sep smap smep ss ssbd sse sse2
    sse4_1 sse4_2 ssse3 stibp syscall tm tm2 tsc tsc_adjust tsc_deadline_timer
    vme x2apic xgetbv1 xsave xsavec xsaveopt xsaves xtopology xtpr
  Vulnerabilities:
  Type: gather_data_sampling status: Vulnerable: No microcode
  Type: itlb_multihit status: KVM: VMX unsupported
  Type: l1tf mitigation: PTE Inversion
  Type: mds mitigation: Clear CPU buffers; SMT vulnerable
  Type: meltdown mitigation: PTI
  Type: mmio_stale_data mitigation: Clear CPU buffers; SMT vulnerable
  Type: retbleed mitigation: IBRS
  Type: spec_rstack_overflow status: Not affected
  Type: spec_store_bypass mitigation: Speculative Store Bypass disabled via
    prctl
  Type: spectre_v1 mitigation: usercopy/swapgs barriers and __user pointer
    sanitization
  Type: spectre_v2 mitigation: IBRS, IBPB: conditional, STIBP: conditional,
    RSB filling, PBRSB-eIBRS: Not affected
  Type: srbds mitigation: Microcode
  Type: tsx_async_abort status: Not affected
Repos:
  Packages: 1848 pm: pacman pkgs: 1822 libs: 473 tools: pamac,yay pm: flatpak
    pkgs: 26
  Active pacman repo servers in: /etc/pacman.d/mirrorlist
    1: https://irltoolkit.mm.fcix.net/manjaro/stable/$repo/$arch
    2: https://mirrors.ocf.berkeley.edu/manjaro/stable/$repo/$arch
    3: https://coresite.mm.fcix.net/manjaro/stable/$repo/$arch
    4: https://southfront.mm.fcix.net/manjaro/stable/$repo/$arch
    5: https://uvermont.mm.fcix.net/manjaro/stable/$repo/$arch
    6: https://ohioix.mm.fcix.net/manjaro/stable/$repo/$arch
    7: https://mirror.math.princeton.edu/pub/manjaro/stable/$repo/$arch
    8: https://ridgewireless.mm.fcix.net/manjaro/stable/$repo/$arch

Thought I’d add the killed processes:

    ~  sudo dmesg -T | grep -Ei 'killed process'                                                                                                                                                                             ✔  3s  
[sudo] password for admin: 
[Fri Mar 22 17:12:29 2024] Out of memory: Killed process 6404 (pamac-manager) total-vm:3658456kB, anon-rss:1880440kB, file-rss:476kB, shmem-rss:0kB, UID:1000 pgtables:4220kB oom_score_adj:200
[Fri Mar 22 18:10:57 2024] Out of memory: Killed process 6543 (beeper) total-vm:1195346836kB, anon-rss:450824kB, file-rss:128kB, shmem-rss:14464kB, UID:1000 pgtables:2108kB oom_score_adj:300
[Fri Mar 22 18:52:51 2024] Out of memory: Killed process 21969 (beeper) total-vm:1195334036kB, anon-rss:382084kB, file-rss:256kB, shmem-rss:13052kB, UID:1000 pgtables:1932kB oom_score_adj:300

Still missing most of the inxi output … like swap, memory, kernel, etc.
Note: Your terminal emulator can scroll.
We have established its an out-of-memory condition.
Which likely is related to your memory and swap, which we would like to see.

But also … seems to be caused by pamac.
I would advise trying to clean it, and/or opting to use something else.
See also all the threads;
https://forum.manjaro.org/search?expanded=true&q=pamac%20memory%20order%3Alatest

Sorry - was running -farzy - not -Farzy :frowning:

System:
  Kernel: 6.6.19-1-MANJARO arch: x86_64 bits: 64 compiler: gcc v: 13.2.1
    clocksource: tsc avail: hpet,acpi_pm
    parameters: BOOT_IMAGE=/boot/vmlinuz-6.6-x86_64
    root=UUID=148e2495-7438-48f3-8529-ca67fcc5f4bf rw quiet splash
    udev.log_priority=3
  Desktop: KDE Plasma v: 5.27.11 tk: Qt v: 5.15.12 info: frameworks
    v: 5.115.0 wm: kwin_x11 vt: 2 dm: SDDM Distro: Manjaro base: Arch Linux
Machine:
  Type: Laptop System: HP product: HP Spectre x360 Convertible v: N/A
    serial: <superuser required> Chassis: type: 10 serial: <superuser required>
  Mobo: HP model: 804E v: 33.23 serial: <superuser required>
    part-nu: N5S04UAR#ABA uuid: <superuser required> UEFI: American Megatrends
    v: F.2D date: 10/23/2015
Battery:
  ID-1: BAT0 charge: 50.7 Wh (100.0%) condition: 50.7/50.7 Wh (100.0%)
    volts: 12.8 min: 11.4 model: Hewlett-Packard Primary type: Li-ion serial: N/A
    status: full
  Device-1: hidpp_battery_0 model: Logitech Marathon Mouse/Performance Plus
    M705 serial: <filter> charge: 100% (should be ignored) rechargeable: yes
    status: discharging
  Device-2: hidpp_battery_1 model: Logitech MK700 serial: <filter>
    charge: 70% (should be ignored) rechargeable: yes status: discharging
CPU:
  Info: model: Intel Core i7-6500U bits: 64 type: MT MCP arch: Skylake
    gen: core 6 level: v3 note: check built: 2015 process: Intel 14nm family: 6
    model-id: 0x4E (78) stepping: 3 microcode: 0xF0
  Topology: cpus: 1x cores: 2 tpc: 2 threads: 4 smt: enabled cache:
    L1: 128 KiB desc: d-2x32 KiB; i-2x32 KiB L2: 512 KiB desc: 2x256 KiB
    L3: 4 MiB desc: 1x4 MiB
  Speed (MHz): avg: 500 high: 800 min/max: 400/3100 scaling:
    driver: intel_pstate governor: powersave cores: 1: 400 2: 400 3: 400 4: 800
    bogomips: 20810
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3
  Vulnerabilities:
  Type: gather_data_sampling status: Vulnerable: No microcode
  Type: itlb_multihit status: KVM: VMX unsupported
  Type: l1tf mitigation: PTE Inversion
  Type: mds mitigation: Clear CPU buffers; SMT vulnerable
  Type: meltdown mitigation: PTI
  Type: mmio_stale_data mitigation: Clear CPU buffers; SMT vulnerable
  Type: retbleed mitigation: IBRS
  Type: spec_rstack_overflow status: Not affected
  Type: spec_store_bypass mitigation: Speculative Store Bypass disabled via
    prctl
  Type: spectre_v1 mitigation: usercopy/swapgs barriers and __user pointer
    sanitization
  Type: spectre_v2 mitigation: IBRS, IBPB: conditional, STIBP: conditional,
    RSB filling, PBRSB-eIBRS: Not affected
  Type: srbds mitigation: Microcode
  Type: tsx_async_abort status: Not affected
Graphics:
  Device-1: Intel Skylake GT2 [HD Graphics 520] vendor: Hewlett-Packard
    driver: i915 v: kernel arch: Gen-9 process: Intel 14n built: 2015-16 ports:
    active: HDMI-A-1 off: eDP-1 empty: DP-1,HDMI-A-2 bus-ID: 00:02.0
    chip-ID: 8086:1916 class-ID: 0300
  Device-2: Sunplus Innovation HP Truevision Full HD driver: uvcvideo
    type: USB rev: 2.0 speed: 480 Mb/s lanes: 1 mode: 2.0 bus-ID: 1-6:4
    chip-ID: 1bcf:2c7d class-ID: 0e02
  Display: x11 server: X.Org v: 21.1.11 compositor: kwin_x11 driver: X:
    loaded: modesetting alternate: fbdev,vesa dri: iris gpu: i915 display-ID: :0
    screens: 1
  Screen-1: 0 s-res: 1920x1080 s-dpi: 96 s-size: 508x285mm (20.00x11.22")
    s-diag: 582mm (22.93")
  Monitor-1: HDMI-A-1 mapped: HDMI-1 pos: primary
    model: LG (GoldStar) FULL HD built: 2015 res: 1920x1080 hz: 60 dpi: 102
    gamma: 1.2 size: 480x270mm (18.9x10.63") diag: 551mm (21.7") ratio: 16:9
    modes: max: 1920x1080 min: 720x400
  Monitor-2: eDP-1 note: disabled model: LG Display built: 2013
    res: 1920x1080 dpi: 222 gamma: 1.2 size: 293x165mm (11.54x6.5")
    diag: 336mm (13.2") ratio: 16:9 modes: 2560x1440
  API: EGL v: 1.5 hw: drv: intel iris platforms: device: 0 drv: iris
    device: 1 drv: swrast surfaceless: drv: iris x11: drv: iris
    inactive: gbm,wayland
  API: OpenGL v: 4.6 compat-v: 4.5 vendor: intel mesa v: 24.0.2-manjaro1.1
    glx-v: 1.4 direct-render: yes renderer: Mesa Intel HD Graphics 520 (SKL GT2)
    device-ID: 8086:1916 memory: 7.49 GiB unified: yes
  API: Vulkan v: 1.3.279 layers: N/A device: 0 type: integrated-gpu
    name: Intel HD Graphics 520 (SKL GT2) driver: mesa intel v: 24.0.2-manjaro1.1
    device-ID: 8086:1916 surfaces: xcb,xlib
Audio:
  Device-1: Intel Sunrise Point-LP HD Audio vendor: Hewlett-Packard
    driver: snd_hda_intel v: kernel alternate: snd_soc_skl,snd_soc_avs
    bus-ID: 00:1f.3 chip-ID: 8086:9d70 class-ID: 0403
  API: ALSA v: k6.6.19-1-MANJARO status: kernel-api with: aoss
    type: oss-emulator tools: alsactl,alsamixer,amixer
  Server-1: JACK v: 1.9.22 status: off tools: N/A
  Server-2: PipeWire v: 1.0.3 status: off with: wireplumber status: active
    tools: pw-cli,wpctl
  Server-3: PulseAudio v: 17.0 status: active with: pulseaudio-alsa
    type: plugin tools: pacat,pactl
Network:
  Device-1: Intel Wi-Fi 6E AX210/AX1675 2x2 [Typhoon Peak] driver: iwlwifi
    v: kernel pcie: gen: 2 speed: 5 GT/s lanes: 1 bus-ID: 02:00.0
    chip-ID: 8086:2725 class-ID: 0280
  IF: wlp2s0 state: up mac: <filter>
  Info: services: NetworkManager, systemd-timesyncd, wpa_supplicant
Bluetooth:
  Device-1: Intel AX210 Bluetooth driver: btusb v: 0.8 type: USB rev: 2.0
    speed: 12 Mb/s lanes: 1 mode: 1.1 bus-ID: 1-7:5 chip-ID: 8087:0032
    class-ID: e001
  Report: rfkill ID: hci0 rfk-id: 1 state: up address: see --recommends
Drives:
  Local Storage: total: 238.47 GiB used: 197.85 GiB (83.0%)
  SMART Message: Unable to run smartctl. Root privileges required.
  ID-1: /dev/sda maj-min: 8:0 vendor: Samsung model: MZNTE256HMHP-000H1
    size: 238.47 GiB block-size: physical: 512 B logical: 512 B speed: 6.0 Gb/s
    tech: SSD serial: <filter> fw-rev: 2H0Q scheme: GPT
Partition:
  ID-1: / raw-size: 238.17 GiB size: 233.38 GiB (97.99%)
    used: 197.85 GiB (84.8%) fs: ext4 dev: /dev/sda2 maj-min: 8:2
  ID-2: /boot/efi raw-size: 300 MiB size: 299.4 MiB (99.80%)
    used: 312 KiB (0.1%) fs: vfat dev: /dev/sda1 maj-min: 8:1
Swap:
  Alert: No swap data was found.
Sensors:
  System Temperatures: cpu: 35.0 C pch: 32.0 C mobo: N/A
  Fan Speeds (rpm): N/A
Repos:
  Packages: 1848 pm: pacman pkgs: 1822 libs: 473 tools: pamac,yay pm: appimage
    pkgs: 0 pm: flatpak pkgs: 26
  Active pacman repo servers in: /etc/pacman.d/mirrorlist
    1: https://irltoolkit.mm.fcix.net/manjaro/stable/$repo/$arch
    2: https://mirrors.ocf.berkeley.edu/manjaro/stable/$repo/$arch
    3: https://coresite.mm.fcix.net/manjaro/stable/$repo/$arch
    4: https://southfront.mm.fcix.net/manjaro/stable/$repo/$arch
    5: https://uvermont.mm.fcix.net/manjaro/stable/$repo/$arch
    6: https://ohioix.mm.fcix.net/manjaro/stable/$repo/$arch
    7: https://mirror.math.princeton.edu/pub/manjaro/stable/$repo/$arch
    8: https://ridgewireless.mm.fcix.net/manjaro/stable/$repo/$arch
Info:
  Memory: total: 8 GiB available: 7.67 GiB used: 5.39 GiB (70.3%)
  Processes: 240 Power: uptime: 5h 53m states: freeze,mem,disk suspend: deep
    avail: s2idle wakeups: 0 hibernate: platform avail: shutdown, reboot,
    suspend, test_resume image: 3.06 GiB services: org_kde_powerdevil,upowerd
    Init: systemd v: 255 default: graphical tool: systemctl
  Compilers: clang: 16.0.6 gcc: 13.2.1 Shell: Zsh v: 5.9 default: Bash
    v: 5.2.26 running-in: konsole inxi: 3.3.33

Yup, no swap.
Which means … aside from oom-killer being able to get rid of things, your system would otherwise be expected to lock up when it reaches the end of its memory.

Theres lots of options for SWAP.

https://wiki.archlinux.org/title/Swap

I usually suggest zram these days unless hibernation is required.

Heres an example that would create zram for swap equal to the size of your memory;

Continuing the discussion from Awful performance since install:

Though that still wont ‘fix’ the runaway memory usage.

For that … please see related threads on pamac as linked above.

I’d point out that while Pamac isn’t helping things, the fact that beeper is getting OOM killed substantially later on to me suggest that @LuigiCalamar might just be using too many memory hungry applications for 8GB of RAM. Open up the system monitor and see what’s using your memory; it’s entirely possible that the actual cause is lots of tabs in your web browser, for example, depending on how the OOM killer ranks the processes on your system (in particular, the way that beeper allocates its memory might make it particularly attractive).

I’d definitely second setting of some form of swap for the system. zram is a pretty good choice, but you may actually benefit from a swap file as RAM appears to be very tight. See Arch Wiki on Swap, and in particular Section 3: Swap File for more details.

I concur that too many tabs in my browser seems to be an issue… system will hang if the only thing I’m doing is using my librewolf browser…

Pretty impressive if you can get nothing but the browser to eat 8G of ram though.
I cant manage to do so even with hundreds of tabs.

Regardless … You definitely want swap.

I guess ‘we will see’ if you still get runaway memory usage.

(I expect you will … because pamac seems to cause it in general right now)

(( and because I notice the link above is still not clicked … heres another: Bizarre Pamac & Pacman behavior: used 62GiB memory, took 10min to refresh DBs - #29 by aZero ))

If LibreWolf looks to be the problem, then I would recommend going to about:processes in the browser (edit: tried to make a link, but it looks like the link won’t work in this forum software). That will give you a detailed breakdown of everything that LibreWolf is doing, including in terms of memory (works on all Firefox based browsers). That will include if there’s a problem with extensions using up lots of memory.

However, it’s also potentially worth looking at System Monitor and sorting by memory usage, or perhaps posting the output of ps aux --sort -rss | head.

Yeah - protonmail is sucking up almost .5gb ram per tab… and I often have 4-5 tabs open to protonmail… so I’ll bring that back to 1 tab. Also noticed that when PM is loading - it uses over 100% CPU… so Protonmail is pretty resource intensive…

The Beeper desktop app is taking over 1.1gb by itself also.

However I have implemented a disk swap - so we’ll see how that performs now!

Thanks to all for the help!

If you find any problematic process, you can use kill command to terminate the process. First find the process PID by using the top and then use the kill PID command to terminate the process.