Journal zstd Problem

Everything is up to date. I ran the three pacman commands and still get the same result.

inxi -Fza

System:
  Kernel: 5.17.6-1-MANJARO arch: x86_64 bits: 64 compiler: gcc v: 11.2.0
    parameters: BOOT_IMAGE=/boot/vmlinuz-5.17-x86_64
    root=UUID=dc4f25fb-8562-4340-920a-bbf219aac5cc ro quiet splash apparmor=1
    security=apparmor udev.log_priority=3
  Desktop: GNOME v: 42.1 tk: GTK v: 3.24.33 wm: gnome-shell dm: GDM v: 42.0
    Distro: Manjaro Linux base: Arch Linux
Machine:
  Type: Laptop System: HP product: HP Pavilion x360 Convertible 11-ad0xx
    v: Type1ProductConfigId serial: <superuser required> Chassis: type: 10
    serial: <superuser required>
  Mobo: HP model: 830D v: 30.21 serial: <superuser required>
    UEFI-[Legacy]: Insyde v: F.04 date: 03/22/2017
Battery:
  ID-1: BAT0 charge: 33.8 Wh (100.0%) condition: 33.8/33.8 Wh (100.0%)
    volts: 8.6 min: 7.7 model: 333-54-2C KN02037XL type: Li-ion
    serial: <filter> status: full
  Device-1: hid-0018:04F3:24B0.0004-battery model: ELAN0732:00 04F3:24B0
    serial: N/A charge: N/A status: N/A
CPU:
  Info: model: Intel Pentium N4200 bits: 64 type: MCP arch: Goldmont
    family: 6 model-id: 0x5C (92) stepping: 9 microcode: 0x46
  Topology: cpus: 1x cores: 4 smt: <unsupported> cache: L1: 224 KiB
    desc: d-4x24 KiB; i-4x32 KiB L2: 2 MiB desc: 2x1024 KiB
  Speed (MHz): avg: 2355 high: 2476 min/max: 800/2500 scaling:
    driver: intel_cpufreq governor: schedutil cores: 1: 2171 2: 2385 3: 2388
    4: 2476 bogomips: 8756
  Flags: ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3
  Vulnerabilities:
  Type: itlb_multihit status: Not affected
  Type: l1tf status: Not affected
  Type: mds status: Not affected
  Type: meltdown status: Not affected
  Type: spec_store_bypass status: Not affected
  Type: spectre_v1
    mitigation: usercopy/swapgs barriers and __user pointer sanitization
  Type: spectre_v2 mitigation: Retpolines, IBPB: conditional, IBRS_FW,
    STIBP: disabled, RSB filling
  Type: srbds status: Not affected
  Type: tsx_async_abort status: Not affected
Graphics:
  Device-1: Intel Celeron N3350/Pentium N4200/Atom E3900 Series Integrated
  Graphics
    vendor: Hewlett-Packard driver: i915 v: kernel ports: active: eDP-1
    empty: HDMI-A-1 bus-ID: 00:02.0 chip-ID: 8086:5a84 class-ID: 0300
  Device-2: Chicony HP Wide Vision HD Camera type: USB driver: uvcvideo
    bus-ID: 1-4:3 chip-ID: 04f2:b5d6 class-ID: 0e02
  Display: wayland server: X.org v: 1.21.1.3 with: Xwayland v: 22.1.1
    compositor: gnome-shell driver: gpu: i915 display-ID: 0
  Monitor-1: eDP-1 model: LG Display 0x0497 built: 2015 res: 1366x768
    dpi: 44 gamma: 1.2 size: 256x144mm (10.08x5.67") diag: 790mm (31.1")
    modes: 1366x768
  OpenGL: renderer: Mesa Intel HD Graphics 505 (APL 3) v: 4.6 Mesa 22.0.3
    direct render: Yes
Audio:
  Device-1: Intel Celeron N3350/Pentium N4200/Atom E3900 Series Audio Cluster
    vendor: Hewlett-Packard driver: snd_hda_intel v: kernel
    alternate: snd_soc_skl,snd_sof_pci_intel_apl bus-ID: 00:0e.0
    chip-ID: 8086:5a98 class-ID: 0403
  Sound Server-1: ALSA v: k5.17.6-1-MANJARO running: yes
  Sound Server-2: sndio v: N/A running: no
  Sound Server-3: JACK v: 1.9.21 running: no
  Sound Server-4: PulseAudio v: 15.0 running: yes
  Sound Server-5: PipeWire v: 0.3.51 running: yes
Network:
  Device-1: Intel Dual Band Wireless-AC 3168NGW [Stone Peak] driver: iwlwifi
    v: kernel pcie: gen: 1 speed: 2.5 GT/s lanes: 1 bus-ID: 02:00.0
    chip-ID: 8086:24fb class-ID: 0280
  IF: wlo1 state: up mac: <filter>
Bluetooth:
  Device-1: Intel Wireless-AC 3168 Bluetooth type: USB driver: btusb v: 0.8
    bus-ID: 1-6:4 chip-ID: 8087:0aa7 class-ID: e001
  Report: rfkill ID: hci0 rfk-id: 1 state: up address: see --recommends
Drives:
  Local Storage: total: 465.76 GiB used: 256.33 GiB (55.0%)
  SMART Message: Required tool smartctl not installed. Check --recommends
  ID-1: /dev/sda maj-min: 8:0 vendor: Western Digital
    model: WD5000LPCX-60VHAT0 size: 465.76 GiB block-size: physical: 4096 B
    logical: 512 B speed: 6.0 Gb/s type: HDD rpm: 5400 serial: <filter>
    rev: 1A01 scheme: GPT
Partition:
  ID-1: / raw-size: 188.56 GiB size: 184.54 GiB (97.87%)
    used: 106.06 GiB (57.5%) fs: ext4 dev: /dev/sda6 maj-min: 8:6
Swap:
  Kernel: swappiness: 60 (default) cache-pressure: 100 (default)
  ID-1: swap-1 type: file size: 512 MiB used: 407.4 MiB (79.6%)
    priority: 50 file: /var/lib/systemd-swap/swapfc/1
  ID-2: swap-2 type: file size: 512 MiB used: 139.6 MiB (27.3%)
    priority: 49 file: /var/lib/systemd-swap/swapfc/2
Sensors:
  System Temperatures: cpu: 48.0 C mobo: 44.0 C
  Fan Speeds (RPM): N/A
Info:
  Processes: 238 Uptime: 2h 56m wakeups: 3 Memory: 3.67 GiB
  used: 2.03 GiB (55.4%) Init: systemd v: 250 tool: systemctl Compilers:
  gcc: 11.2.0 clang: 13.0.1 Packages: pacman: 2240 lib: 554 flatpak: 0
  Shell: Zsh v: 5.8.1 running-in: gnome-terminal inxi: 3.3.16

mhwd -li

> Installed PCI configs:

              NAME               VERSION          FREEDRIVER           TYPE

       video-linux            2018.05.04                true            PCI

mhwd-kernel -li

Currently running: 5.17.6-1-MANJARO (linux517)
The following kernels are installed in your system:
   * linux514
   * linux517
   * linux59

pacman -Qdt

afpfs-ng 0.8.2-2
amf-headers 1.4.24-2
android-udev 20220102-1
ant 1.10.12-1
asciidoc 10.1.4-1
avrdude 1:7.0-1
bison 3.8.2-4
bomutils 0.2-2
boost 1.78.0-2
brave-browser 1.38.119-1
celt 0.11.3-4
chrpath 0.16-3
cmake 3.23.1-1
cowpatty 4.8-2
cython2 0.29.28-1
dhcp 4.4.3-1
dos2unix 7.4.2-1
dotnet-sdk 6.0.2.sdk102-1
doxygen 1.9.3-1
dsniff 2.4b1-28
dvd+rw-tools 7.1-9
dvdauthor 0.7.2-11
ffnvcodec-headers 11.1.5.1-2
flake8 1:4.0.1-5
flex 2.6.4-3
game-devices-udev 0.16-1
gdb 11.2-1
gdl 3.40.0-1
gendesk 1.0.9-3
gestures 0.3.0-1
ghostpcl 9.56.1-1
ghostxps 9.56.1-1
glew1.10 1.10.0-3
gnome-bluetooth 3.34.5-3
gnome-common 3.18.0-4
gnome-shell-extension-gamemode 7-1
gnome-shell-extension-pop-shell 1.2.0+220+gcb76a4c-1
gnome-shell-extension-unite 64-1
gnu-netcat 0.7.1-8
gobject-introspection 1.72.0-1
haskell-entropy 0.4.1.7-2
haskell-hslogger 1.3.1.0-167
haskell-random 1.2.1-91
haskell-regex-tdfa 1.3.1.2-14
haskell-sandi 0.5-303
haskell-sha 1.6.4.4-19
haskell-utf8-string 1.0.2-91
help2man 1.49.2-1
hexer-hobu 1.4.0-7
hostapd 2.10-1
http-parser 2.9.4-1
java-openjfx 18.0.1.u2-1
jbigkit 2.1-6
jq 1.6-4
jupyter-nbclassic 0.3.7-2
jupyter_console 6.4.3-1
kvantum-theme-matchama 20191118-1
laszip 3.4.3-1
laz-perf 2:3.0.0-1
lib32-alsa-oss 1.1.8-3
lib32-db 5.3.28-5
lib32-dconf 0.40.0-2
lib32-faudio 22.05-1
lib32-gettext 0.21-1
lib32-glew1.10 1.10.0-5
lib32-gst-plugins-base-libs 1.20.2-1
lib32-json-c 0.16-1
lib32-ladspa 1.17-1
lib32-libappindicator-gtk2 12.10.0-13
lib32-libavc1394 0.5.4-2
lib32-libcaca 0.99.beta20-1
lib32-libcurl-compat 7.83.0-1
lib32-libcurl-gnutls 7.83.0-1
lib32-libgcrypt15 1.5.6-7
lib32-libgusb 0.3.10-2
lib32-libidn11 1.33-2
lib32-libiec61883 1.2.0-2
lib32-libjpeg6-turbo 1.5.3-2
lib32-libmng 2.0.3-2
lib32-libnm 1.36.4-1
lib32-libnm-glib 1.18.5dev+12+ga8746f48ca-1
lib32-libpcap 1.10.1-1
lib32-libpng12 1.2.59-2
lib32-librtmp0 2.4-4
lib32-libtheora 1.1.1-13
lib32-libtiff4 3.9.7-5
lib32-libudev0-shim 1-4
lib32-libvpx 1.11.0-2
lib32-libvpx1.3 1.3.0-2
lib32-libwrap 7.6.31-3
lib32-libxss 1.2.3-1
lib32-lz4 1.9.3-1
lib32-openal 1.22.0-1
lib32-openssl-1.0 1.0.2.u-2
lib32-polkit 0.120-5
lib32-sdl2_image 2.0.5-2
lib32-sdl2_mixer 2.0.4-1
lib32-sdl2_ttf 2.0.18-1
lib32-sdl_image 1.2.12-7
lib32-sdl_mixer 1.2.12-3
lib32-sdl_ttf 2.0.11-5
lib32-speex 1.2.0-2
lib32-v4l-utils 1.22.1-1
lib32-vkd3d 1.3-1
libcec 6.0.2-3
libcmis 0.5.2-9
libcroco 0.6.13-2
libcurl-compat 7.83.0-1
libcurl-gnutls 7.83.0-1
libdazzle 3.44.0-1
libgcrypt15 1.5.6-6
libgweather 40.0+r87+g80e5a652-2
libhandy0 0.0.13+12+gaa6b6f4-3
libhdhomerun 20220203-1
libidn11 1.33-2
libirecovery 1.0.0-1
libjpeg6-turbo 1.5.3-2
libmusicbrainz5 5.1.0-4
libofa 0.9.3-9
libopenaptx 0.2.0-1
libpackagekit-glib 1.2.5-1
libpng12 1.2.59-2
libquvi 0.9.4-6
libretro-core-info 1.10.3-1
librtmp0 2.4-5
libtiff4 3.9.7-5
libtvcontrol 0.0.1-3
libvpx1.3 1.3.0-3
lighttpd 1.4.64-1
live-media 2022.02.07-1
lldb 13.0.1-1
lsof 4.95.0-1
lttng-ust 2.13.2-1
lua51 5.1.5-9
manjaro-gnome-extension-settings 20220517-1
matcha-gtk-theme 20211225-1
mdk4 4.2-1
mercurial 6.1.2-1
meson 0.62.1-1
mingw-w64-gcc 11.2.0-1
mono-msbuild 16.10.1.xamarinxplat.2021.05.26.14.00-2
nasm 2.15.05-1
ncurses5-compat-libs 6.3-1
nmap 7.92-1
opencl-headers 2:2022.01.04-1
openscenegraph 3.6.5-11
pepper-flash 32.0.0.465-1
perl-date-manip 6.86-1
perl-dbd-mysql 4.050-7
perl-image-size 3.300-4
perl-io-socket-inet6 2.73-1
perl-json 4.05-1
perl-net-upnp 1.4.6-1
perl-soap-lite 1.27-6
perl-xml-simple 2.25-6
perl-xml-xpath 1.44-4
php-cgi 8.1.5-2
physfs 3.0.2-2
pixiewps 1.4.2-2
podofo 0.9.8-1
python-aiohttp 3.8.1-4
python-aiosqlite 0.17.0-3
python-asn1crypto 1.5.1-1
python-behave 1.2.7.dev2-1
python-build 0.7.0-4
python-cairosvg 2.5.2-3
python-cherrypy 18.6.1-3
python-click-plugins 1.1.1-9
python-coverage 6.1.1-3
python-flit-core 3.5.1-3
python-installer 0.5.1-1
python-mock 3.0.5-9
python-monotonic 1.6-3
python-mutagen 1.45.1-6
python-netaddr 0.8.0-5
python-networkx 2.7.1-1
python-nose 1.3.7-14
python-oauthlib 3.2.0-1
python-phonenumbers 8.12.47-1
python-poetry 1.1.13-3
python-pybluez 0.23-4
python-pycryptodomex 3.12.0-1
python-pynput 1.7.6-2
python-pytest 7.1.2-1
python-pytz-deprecation-shim 0.1.0.post0-3
python-responses 0.20.0-1
python-setuptools-scm 6.4.2-1
python-simplejson 3.17.6-3
python-tenacity 8.0.1-3
python-texttable 1.6.4-3
python-uvloop 0.16.0-3
python-vdf 3.4-2
python2-gobject 3.36.1-5
python2-pexpect 4.8.0-1
python2-pytest 4.6.11-1
python2-pyxdg 0.27-1
python2-setuptools-scm 5.0.2-4
qt5-script 5.15.9-1
qt5-webengine 5.15.9-2
qt5ct 1.5-2
qt6ct 0.5-4
quazip-qt5 1.3-1
rtl-sdr 1:0.8.0-4
sdl2_net 1:2.0.1-3
sdl2_ttf 2.0.18-1
setconf 0.7.7-2
shairplay 20180824.096b61a-3
swig 4.0.2-5
tepl 6.0.1-1
tinyxml 2.6.2-9
vala 0.56.1-1
vkd3d 1.3-1
waylandpp 1.0.0-1
wxgtk2-light 3.0.5.1-3
xine-lib 1.2.12-2
xorg-util-macros 1.19.3-1
xorg-xmessage 1.0.5-2
yasm 1.3.0-5
zita-alsa-pcmi 0.4.0-1
zita-resampler 1.8.0-1

you have installed EOL kernels, uninstall them:
sudo mhwd-kernel -r linux59
sudo mhwd-kernel -r linux514
reboot
post output from:
journalctl --boot=0 --priority=3 --no-pager

I Removed the EOL kernels then rebooted.

journalctl --boot=0 --priority=3 --no-pager
Journal file /var/log/journal/2822d22470664181983908c471ed7fbd/system@5c6ef8853c1447db8a426c8329129953-000000000011693d-0005db7161a43c2a.journal uses an unsupported feature, ignoring file. Use SYSTEMD_LOG_LEVEL=debug journalctl --file=/var/log/journal/2822d22470664181983908c471ed7fbd/system@5c6ef8853c1447db8a426c8329129953-000000000011693d-0005db7161a43c2a.journal to see the details.

SYSTEMD_LOG_LEVEL=debug journalctl --file=/var/log/journal/2822d22470664181983908c471ed7fbd/system@5c6ef8853c1447db8a426c8329129953-000000000011693d-0005db7161a43c2a.journal
Journal effective settings seal=no keyed_hash=yes compress=no compress_threshold_bytes=8B Journal file /var/log/journal/2822d22470664181983908c471ed7fbd/system@5c6ef8853c1447db8a426c8329129953-000000000011693d-0005db7161a43c2a.journal uses incompatible flag zstd-compressed disabled at compilation time. Failed to open journal file /var/log/journal/2822d22470664181983908c471ed7fbd/system@5c6ef8853c1447db8a426c8329129953-000000000011693d-0005db7161a43c2a.journal: Protocol not supported mmap cache statistics: 0 hit, 1 miss

Can you check the compiled options of journalctl?

journalctl --version

An alternative would be to simply delete the problematic journal file.

can you report

sudo journalctl --disk-usage

also this point

find /etc -type f -name "*.pacnew"

Welcome to Manjaro! :smiling_face_with_three_hearts:

  1. Please read the information behind this link. It will help you to post necessary information. [HowTo] Provide System Information
  2. Please press the three dots below your post and then press the :pencil2:
  • Presenting the information in this way allows everyone to be familiar with the format and quickly find the items they need without missing anything.
  • To make it more readable, add 3 backticks ``` on an extra line before and after the pasted text.

journalctl --version

systemd 246 (246)
-PAM -AUDIT -SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP -LIBCRYPTSETUP -GCRYPT -GNUTLS -ACL +XZ +LZ4 -ZSTD -SECCOMP +BLKID -ELFUTILS -KMOD -IDN2 -IDN -PCRE2 default-hierarchy=unified

sudo journalctl --disk-usage

Journal file /var/log/journal/2822d22470664181983908c471ed7fbd/system@5c6ef8853c1447db8a426c8329129953-000000000011693d-0005db7161a43c2a.journal uses an unsupported feature, ignoring file.
Use SYSTEMD_LOG_LEVEL=debug journalctl --file=/var/log/journal/2822d22470664181983908c471ed7fbd/system@5c6ef8853c1447db8a426c8329129953-000000000011693d-0005db7161a43c2a.journal to see the details.
Archived and active journals take up 0B in the file system.

find /etc -type f -name “*.pacnew”



/etc/default/grub.pacnew
/etc/sane.d/canon_lide70.conf.pacnew
/etc/sane.d/escl.conf.pacnew
/etc/pacman.conf.pacnew
/etc/plymouth/plymouthd.conf.pacnew
/etc/sudoers.pacnew
/etc/locale.gen.pacnew
/etc/pipewire/media-session.d/media-session.conf.pacnew
/etc/pipewire/media-session.d/alsa-monitor.conf.pacnew

Older systemd version? ZSTD support disabled?

Are you sure you fully updated everything?


Outputs of the following:

pacman -F /usr/bin/journalctl
pacman -Q systemd
pacman -Q systemd-libs

I’ve run sudo pacman -Syyu a few times now as well as reinstalling systemd

pacman -F /usr/bin/journalctl

usr/bin/journalctl is owned by core/systemd 250.5-1

pacman -Q systemd

systemd 250.5-1

pacman -Q systemd-libs

systemd-libs 250.5-1

Thanks everyone for trying to help me with this

Your journalctl --version output is odd.

For comparison, here’s mine (on a fully updated Manjaro system, on the Stable Updates train):

journalctl --version                                                                                                                                                                                                        

systemd 250 (250.5-1-manjaro)
+PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified

Notice not only the version, but also ZSTD support?

Can you try force re-installing systemd-libs ?

Done and restarted

journalctl --version still looks weird
systemd 246 (246) -PAM -AUDIT -SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP -LIBCRYPTSETUP -GCRYPT -GNUTLS -ACL +XZ +LZ4 -ZSTD -SECCOMP +BLKID -ELFUTILS -KMOD -IDN2 -IDN -PCRE2 default-hierarchy=unified

What about this:

/usr/bin/systemctl --version

You can repeat the same command with /usr/bin/journalctl

Those look better

/usr/bin/systemctl --version

systemd 250 (250.5-1-manjaro)
+PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified

/usr/bin/journalctl --version

systemd 250 (250.5-1-manjaro)
+PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified

AHA! :scream:

You’ve likely got an errant package that is superseding the proper packages for systemd.

pacman -Qs systemd

And for kicks, try to read your logs, like previously, but this time explicitly use /usr/bin/journalctl as the command.

:wink:

yay. Thanks so much. running /usr/bin/journalctl explicitly works

pacman -Qs systemd

local/lib32-systemd 250.5-1
    system and service manager (32-bit)
local/netctl 1.28-1
    Profile based systemd network management
local/python-systemd 234-11
    Python bindings for systemd
local/qopenvpn 2.0.0-6
    Simple OpenVPN GUI written in PyQt for systemd based distributions
local/systemd 250.5-1
    system and service manager
local/systemd-fsck-silent 239-1
    File system checks for SystemD (silent)
local/systemd-libs 250.5-1
    systemd client libraries
local/systemd-swap 4.4.0-2
    Script for creating hybrid swap space from zram swaps, swap files and swap partitions.
local/systemd-sysvcompat 250.5-1
    sysvinit compat for systemd
1 Like

That’s one problem down, but still another looms.

Shotgun approach to try and find the errant executable:

sudo find /bin /usr/bin /opt /etc /lib /lib64 /sbin -type f -name journalctl

For comparison, it only finds one file on my system:

sudo find /bin /usr/bin /opt /etc /lib /lib64 /sbin -type f -name journalctl

/usr/bin/journalctl

sudo find /bin /usr/bin /opt /etc /lib /lib64 /sbin -type f -name journalctl

/usr/bin/journalctl

Weird considering the problem I would have expected to find more than that one

This should not find anything, but…

find $HOME -type f -name journalctl