[BUG] rspamd 2.5-2 gives a core dump on start

Hi!

Today the problem reported on the old forum - [BUG] rspamd 2.5-2 gives a core dump on start - Bug Reports - Manjaro Linux Forum - received a fix from one of the developers.

[BUG] Cannot start rspamd on a Raspberry Pi 2 due to lpeg_allocate_mem_low segmentation fault 路 Issue #3459 路 rspamd/rspamd 路 GitHub

:grinning:

I built a new package using the latest git tag and I am not having any luck starting rspamd. I used the arch-arm aarch64 PKGBUILD which will not build as is as they had the pkgver changing down in the PKGBUILD trying to use dashes adding the git version.

[ray@pi4 ~]$ systemctl status rspamd
鈼 rspamd.service - rapid spam filtering system
     Loaded: loaded (/usr/lib/systemd/system/rspamd.service; disabled; vendor preset: disabled)
     Active: failed (Result: core-dump) since Thu 2020-09-17 19:18:38 CDT; 50s ago
       Docs: https://rspamd.com/doc/
    Process: 13190 ExecStart=/usr/bin/rspamd -c /etc/rspamd/rspamd.conf -f (code=dumped, signal=SEGV)
   Main PID: 13190 (code=dumped, signal=SEGV)

Sep 17 19:18:38 pi4 systemd[1]: rspamd.service: Scheduled restart job, restart counter is at 5.
Sep 17 19:18:38 pi4 systemd[1]: Stopped rapid spam filtering system.
Sep 17 19:18:38 pi4 systemd[1]: rspamd.service: Start request repeated too quickly.
Sep 17 19:18:38 pi4 systemd[1]: rspamd.service: Failed with result 'core-dump'.
Sep 17 19:18:38 pi4 systemd[1]: Failed to start rapid spam filtering system.

On GitHub - rspamd/rspamd: Rapid spam filtering system. - vstakhov [Fix] Do not use GC64 workaround on 32bit platforms, omg - fd396a6 - 2 hours ago

I don鈥檛 fully understand the development cycle, maybe we need to wait Arch Linux get this fix to get it into Manjaro鈥

There were 2 patches added but the one you show seems to fixing something involving 32 bit platforms.

From [BUG] Cannot start rspamd on a Raspberry Pi 2 due to lpeg_allocate_mem_low segmentation fault 路 Issue #3459 路 rspamd/rspamd 路 GitHub

@peracchi I read your post and I鈥檝e just tried to built rspamd using the official Arch Linux Arm repo PKGBUILD (1 and 2). The only changes I applied to the PKGBUILD are

source=(
 #git+https://github.com/rspamd/rspamd.git#tag=61cbcd0483d271f71632e700f52fd8a19351109b
 # actual latest commit
 git+https://github.com/rspamd/rspamd.git#tag=37f19ff44db4601bea40fef01db776c312e48ea1
 # [...]
)
pkgver(){
 #cd rspamd
 #git describe --tags
 # was giving me errors with the above tag
 echo 2.6
}

This is the successful makepkg log. As you can see the program runs fine.

@vstakhov if you have some spare time can you give a rapid look to the compilation part (lines 402 to 762) to ensure those warnings are ok?

@Darksky can you please look at latest info from AlphaJack?

Looks like they made another commit since I tested.

Added:

Still giving a core dump. Going to compile it on a 32 bit os and see if it loads. That guy was using a pi2.

Ok, thank you very much for your support!

The developer vstakhov said: 鈥淚t just works on my aarch64 vm that I鈥檝e rented from Amazon EC2.鈥. I don鈥檛 know how this relate to RPI4.

I do not know either and pretty much do not want to know. lol

It is appearing like I said above that commit was for a 32 bit OS. It works fine on my pi4 32bit OS.

[alarm@alarmpi rspamd]$ sudo systemctl start rspamd
[alarm@alarmpi rspamd]$ systemctl status rspamd
* rspamd.service - rapid spam filtering system
     Loaded: loaded (/usr/lib/systemd/system/rspamd.service; disabled; vendor preset: disabled)
     Active: active (running) since Sun 2020-09-20 11:16:07 CDT; 4s ago
       Docs: https://rspamd.com/doc/
   Main PID: 6524 (rspamd)
      Tasks: 5 (limit: 4915)
     CGroup: /system.slice/rspamd.service
             |-6524 rspamd: main process
             |-6528 rspamd: rspamd_proxy process (localhost:11332)
             |-6529 rspamd: controller process (localhost:11334)
             |-6530 rspamd: normal process (localhost:11333)
             `-6531 rspamd: normal process (localhost:11333)

Sep 20 11:16:07 alarmpi systemd[1]: Started rapid spam filtering system.
[alarm@alarmpi rspamd]$  
[alarm@alarmpi rspamd]$ uname -r
5.4.58-udlv7l+
[alarm@alarmpi rspamd]$

I tried all over again, from scratch, using Arch Linux ARM aarch64.

I followed this guide - How to Install Arch Linux on Raspberry Pi [Step-by-step] - only replacing http://os.archlinuxarm.org/os/ArchLinuxARM-rpi-4-latest.tar.gz for http://os.archlinuxarm.org/os/ArchLinuxARM-rpi-aarch64-latest.tar.gz and also doing sed -i 's/mmcblk0/mmcblk1/g' root/etc/fstab before unmounting the partitions.

Rspamd 2.5-2 worked without any modification鈥

Doing the same steps using Manjaro, Rspamd gives the core dump.

We have a mistery here! :crazy_face:

Below some output from my RPI4.

                   -`                    peracchi@ninja 
                  .o+`                   -------------- 
                 `ooo/                   OS: Arch Linux ARM aarch64 
                `+oooo:                  Host: Raspberry Pi 4 Model B 
               `+oooooo:                 Kernel: 5.8.9-2-ARCH 
               -+oooooo+:                Uptime: 1 min 
             `/:-:++oooo+:               Packages: 249 (pacman) 
            `/++++/+++++++:              Shell: bash 5.0.18 
           `/++++++++++++++:             Terminal: /dev/pts/0 
          `/+++ooooooooooooo/`           CPU: (4) @ 1.500GHz 
         ./ooosssso++osssssso+`          Memory: 83MiB / 3788MiB 
        .oossssso-````/ossssss+`
       -osssssso.      :ssssssso.                                
      :osssssss/        osssso+++.                               
     /ossssssss/        +ssssooo/-
   `/ossssso+/:-        -:/+osssso+-
  `+sso+:-`                 `.-/+oso:
 `++:.                           `-/+/
 .`                                 `/

[peracchi@ninja ~]$ 
[peracchi@ninja ~]$ hostnamectl status
   Static hostname: ninja
         Icon name: computer
        Machine ID: 89d56f26ab5a4352b9063956ee7fb6af
           Boot ID: 3ec5d1c5a28047f0be00921660caafc7
  Operating System: Arch Linux ARM
            Kernel: Linux 5.8.9-2-ARCH
      Architecture: arm64
[peracchi@ninja ~]$ 
[peracchi@ninja ~]$ sudo systemctl status rspamd
* rspamd.service - rapid spam filtering system
     Loaded: loaded (/usr/lib/systemd/system/rspamd.service; disabled; vendor preset: disabled)
     Active: inactive (dead)
       Docs: https://rspamd.com/doc/
[peracchi@ninja ~]$ 
[peracchi@ninja ~]$ sudo systemctl start rspamd
[peracchi@ninja ~]$ 
[peracchi@ninja ~]$ sudo systemctl status rspamd
* rspamd.service - rapid spam filtering system
     Loaded: loaded (/usr/lib/systemd/system/rspamd.service; disabled; vendor preset: disabled)
     Active: active (running) since Sun 2020-09-20 15:25:04 -03; 8s ago
       Docs: https://rspamd.com/doc/
   Main PID: 566 (rspamd)
      Tasks: 5 (limit: 4453)
     Memory: 54.6M
     CGroup: /system.slice/rspamd.service
             |-566 rspamd: main process
             |-572 rspamd: rspamd_proxy process (localhost:11332)
             |-573 rspamd: controller process (localhost:11334)
             |-574 rspamd: normal process (localhost:11333)
             `-575 rspamd: normal process (localhost:11333)

Sep 20 15:25:04 ninja systemd[1]: Started rapid spam filtering system.
[peracchi@ninja ~]$ 
[peracchi@ninja ~]$ sudo pacman -Si rspamd
Repository      : community
Name            : rspamd
Version         : 2.5-2
Description     : Fast, free and open-source spam filtering system
Architecture    : aarch64
URL             : https://rspamd.com
Licenses        : BSD
Groups          : None
Provides        : None
Depends On      : file  glib2  glibc  icu  jemalloc  libevent  libsodium  luajit  openblas  openssl
                  pcre  sqlite  zlib
Optional Deps   : redis: Redis cache and data storage
Conflicts With  : None
Replaces        : None
Download Size   : 2.52 MiB
Installed Size  : 9.64 MiB
Packager        : Arch Linux ARM Build System <builder+n1@archlinuxarm.org>
Build Date      : Fri May 1 14:54:45 2020
Validated By    : MD5 Sum  SHA-256 Sum  Signature

[peracchi@ninja ~]$ uname -a
Linux ninja 5.8.9-2-ARCH #1 SMP Tue Sep 15 00:48:59 UTC 2020 aarch64 GNU/Linux

The only thing I can think of is maybe a module needs to be enabled in the kernel.

Well, maybe we can discard the problem as being a matter of 32 x 64 bits and even the need of that commit that the guy of RPI2 asked for.

Probably so.

Same steps repeated on SD card prepared with manjaro-arm-installer (running in Manjaro KDE 20.1).

鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻  鈻堚枅鈻堚枅鈻堚枅鈻堚枅   peracchi@rainbow 
鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅  鈻堚枅鈻堚枅鈻堚枅鈻堚枅   ---------------- 
鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅  鈻堚枅鈻堚枅鈻堚枅鈻堚枅   OS: Manjaro ARM Linux aarch64 
鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅  鈻堚枅鈻堚枅鈻堚枅鈻堚枅   Host: Raspberry Pi 4 Model B Rev 1.1 
鈻堚枅鈻堚枅鈻堚枅鈻堚枅            鈻堚枅鈻堚枅鈻堚枅鈻堚枅   Kernel: 5.4.61-1-MANJARO-ARM 
鈻堚枅鈻堚枅鈻堚枅鈻堚枅  鈻堚枅鈻堚枅鈻堚枅鈻堚枅  鈻堚枅鈻堚枅鈻堚枅鈻堚枅   Uptime: 12 mins 
鈻堚枅鈻堚枅鈻堚枅鈻堚枅  鈻堚枅鈻堚枅鈻堚枅鈻堚枅  鈻堚枅鈻堚枅鈻堚枅鈻堚枅   Packages: 269 (pacman) 
鈻堚枅鈻堚枅鈻堚枅鈻堚枅  鈻堚枅鈻堚枅鈻堚枅鈻堚枅  鈻堚枅鈻堚枅鈻堚枅鈻堚枅   Shell: bash 5.0.18 
鈻堚枅鈻堚枅鈻堚枅鈻堚枅  鈻堚枅鈻堚枅鈻堚枅鈻堚枅  鈻堚枅鈻堚枅鈻堚枅鈻堚枅   Resolution: 1920x1080 
鈻堚枅鈻堚枅鈻堚枅鈻堚枅  鈻堚枅鈻堚枅鈻堚枅鈻堚枅  鈻堚枅鈻堚枅鈻堚枅鈻堚枅   Terminal: /dev/pts/0 
鈻堚枅鈻堚枅鈻堚枅鈻堚枅  鈻堚枅鈻堚枅鈻堚枅鈻堚枅  鈻堚枅鈻堚枅鈻堚枅鈻堚枅   CPU: BCM2835 (4) @ 1.500GHz 
鈻堚枅鈻堚枅鈻堚枅鈻堚枅  鈻堚枅鈻堚枅鈻堚枅鈻堚枅  鈻堚枅鈻堚枅鈻堚枅鈻堚枅   Memory: 128MiB / 3809MiB 
鈻堚枅鈻堚枅鈻堚枅鈻堚枅  鈻堚枅鈻堚枅鈻堚枅鈻堚枅  鈻堚枅鈻堚枅鈻堚枅鈻堚枅
鈻堚枅鈻堚枅鈻堚枅鈻堚枅  鈻堚枅鈻堚枅鈻堚枅鈻堚枅  鈻堚枅鈻堚枅鈻堚枅鈻堚枅                           
                                                       


[peracchi@rainbow ~]$ 
[peracchi@rainbow ~]$ hostnamectl status
   Static hostname: rainbow
         Icon name: computer
        Machine ID: 93ac5338fb3a4fc2983471f814e15b39
           Boot ID: 10155704dc71467e934b0a7b84bea244
  Operating System: Manjaro ARM
            Kernel: Linux 5.4.61-1-MANJARO-ARM
      Architecture: arm64
[peracchi@rainbow ~]$ 
[peracchi@rainbow ~]$ sudo systemctl status rspamd
鈼 rspamd.service - rapid spam filtering system
     Loaded: loaded (/usr/lib/systemd/system/rspamd.service; disabled; vendor preset: disabled)
     Active: inactive (dead)
       Docs: https://rspamd.com/doc/
[peracchi@rainbow ~]$ 
[peracchi@rainbow ~]$ sudo systemctl start rspamd
[peracchi@rainbow ~]$ 
[peracchi@rainbow ~]$ sudo systemctl status rspamd
鈼 rspamd.service - rapid spam filtering system
     Loaded: loaded (/usr/lib/systemd/system/rspamd.service; disabled; vendor preset: disabled)
     Active: failed (Result: core-dump) since Sun 2020-09-20 16:50:37 -03; 29s ago
       Docs: https://rspamd.com/doc/
    Process: 1303 ExecStart=/usr/bin/rspamd -c /etc/rspamd/rspamd.conf -f (code=dumped, signal=SEGV)
   Main PID: 1303 (code=dumped, signal=SEGV)

Sep 20 16:50:37 rainbow systemd[1]: rspamd.service: Scheduled restart job, restart counter is at 5.
Sep 20 16:50:37 rainbow systemd[1]: Stopped rapid spam filtering system.
Sep 20 16:50:37 rainbow systemd[1]: rspamd.service: Start request repeated too quickly.
Sep 20 16:50:37 rainbow systemd[1]: rspamd.service: Failed with result 'core-dump'.
Sep 20 16:50:37 rainbow systemd[1]: Failed to start rapid spam filtering system.
[peracchi@rainbow ~]$ 
[peracchi@rainbow ~]$ sudo pacman -Si rspamd
Repository      : community
Name            : rspamd
Version         : 2.5-2
Description     : Fast, free and open-source spam filtering system
Architecture    : aarch64
URL             : https://rspamd.com
Licenses        : BSD
Groups          : None
Provides        : None
Depends On      : file  glib2  glibc  icu  jemalloc  libevent  libsodium  luajit  openblas  openssl
                  pcre  sqlite  zlib
Optional Deps   : redis: Redis cache and data storage
Conflicts With  : None
Replaces        : None
Download Size   : 2.52 MiB
Installed Size  : 9.64 MiB
Packager        : Arch Linux ARM Build System <builder+n1@archlinuxarm.org>
Build Date      : Fri 01 May 2020 02:54:45 PM -03
Validated By    : MD5 Sum  SHA-256 Sum  Signature

[peracchi@rainbow ~]$ 
[peracchi@rainbow ~]$ uname -a
Linux rainbow 5.4.61-1-MANJARO-ARM #1 SMP PREEMPT Tue Sep 1 13:29:23 CDT 2020 aarch64 GNU/Linux

:pensive:

I made some progress today. I installed the arch-arm image for aarch64 today and as you say raspamd loaded.

I have been working on trying to get the needed pi4 configs into our linux-rc kernel in our repo so it would boot with uboot. I loaded my first attempt with the kernel on the same arch-arm 64 bit OS and rspamd would not load. I then used meld to compare the arch-arm and manjaro鈥檚 linux-rc and enabled some more modules really trying to get manjaro鈥檚 linux-rc to boot with uboot. When I booted up in my latest manjaro鈥檚 linux-rc kernel I decided to try rspamd again and it worked! The only thing is I have no clue which new module it needed as I enabled quite a few.

Any way there is some module/modules that needs to be enabled in manjaro-arm kernel鈥檚 that raspamd needs. One would think that they would have what all is needed for rspamd to work on their site some where.

[alarm@alarm ~]$ sudo systemctl status rspamd
* rspamd.service - rapid spam filtering system
     Loaded: loaded (/usr/lib/systemd/system/rspamd.service; disabled; vendor preset: disabled)
     Active: active (running) since Mon 2020-09-21 20:31:08 UTC; 16s ago
       Docs: https://rspamd.com/doc/
   Main PID: 198 (rspamd)
      Tasks: 5 (limit: 9295)
     Memory: 65.4M
     CGroup: /system.slice/rspamd.service
             |-198 rspamd: main process
             |-200 rspamd: rspamd_proxy process (localhost:11332)
             |-201 rspamd: controller process (localhost:11334)
             |-202 rspamd: normal process (localhost:11333)
             `-203 rspamd: normal process (localhost:11333)

Sep 21 20:31:08 alarm systemd[1]: Started rapid spam filtering system.
[alarm@alarm ~]$  
[alarm@alarm ~]$ uname -r
5.9.0-rc5-3-MANJARO-ARM

@peracchi It looks like I now have our kernel linux-rpi4 kernel config where raspamd will now load. I am attaching a link for you to test further other than just loading the the systemd service before I upload to our git as I have no clue to setting it up and testing.

The tarball includs the linux-rpi4 kernel/headers and the latest rspamd package with their latest commit today. I have no idea what the commit is about but you might. All they said was 鈥淸Feature] Fuzzy_check: Add weight_threshold option for fuzzy rules鈥 in the commit message.

[ray@pi4 ~]$ sudo systemctl status rspamd
鈼 rspamd.service - rapid spam filtering system
     Loaded: loaded (/usr/lib/systemd/system/rspamd.service; enabled; vendor preset: disabled)
     Active: active (running) since Tue 2020-09-22 19:02:58 CDT; 47s ago
       Docs: https://rspamd.com/doc/
   Main PID: 473 (rspamd)
      Tasks: 5 (limit: 9073)
     CGroup: /system.slice/rspamd.service
             鈹溾攢473 rspamd: main process
             鈹溾攢556 rspamd: rspamd_proxy process (localhost:11332)
             鈹溾攢557 rspamd: controller process (localhost:11334)
             鈹溾攢558 rspamd: normal process (localhost:11333)
             鈹斺攢559 rspamd: normal process (localhost:11333)

Sep 22 19:02:58 pi4 systemd[1]: Started rapid spam filtering system.
[ray@pi4 ~]$ uname -a
Linux pi4 5.4.65-2-MANJARO-ARM #1 SMP PREEMPT Tue Sep 22 18:00:30 CDT 2020 aarch64 GNU/Linux

md5sum:
68c74d718358f3ab4646c6d9bf72a52b pi4-kernel_rspamd.tar

https://drive.google.com/file/d/1U54tke8pYsBaKjV2nm3tvKoBeyVk-cKc/view?usp=sharing

The environment is ready, now I will setup the mail server following the Setting up a mail server with OpenSMTPD, Dovecot and Rspamd and will report back later.

[peracchi@rainbow ~]$ timedatectl status
               Local time: Wed 2020-09-23 09:32:53 -03   
           Universal time: Wed 2020-09-23 12:32:53 UTC   
                 RTC time: n/a                           
                Time zone: America/Sao_Paulo (-03, -0300)
System clock synchronized: yes                           
              NTP service: active                        
          RTC in local TZ: no                            
[peracchi@rainbow ~]$ 
[peracchi@rainbow ~]$ hostnamectl status
   Static hostname: rainbow
         Icon name: computer
        Machine ID: 93ac5338fb3a4fc2983471f814e15b39
           Boot ID: 3add084b75ca4f189b5b336572ec62ce
  Operating System: Manjaro ARM
            Kernel: Linux 5.4.65-2-MANJARO-ARM
      Architecture: arm64
[peracchi@rainbow ~]$ 
[peracchi@rainbow ~]$ uname -a
Linux rainbow 5.4.65-2-MANJARO-ARM #1 SMP PREEMPT Tue Sep 22 18:00:30 CDT 2020 aarch64 GNU/Linux
[peracchi@rainbow ~]$ 
[peracchi@rainbow ~]$ sudo pacman -Si rspamd
Repository      : community
Name            : rspamd
Version         : 2.5-2
Description     : Fast, free and open-source spam filtering system
Architecture    : aarch64
URL             : https://rspamd.com
Licenses        : BSD
Groups          : None
Provides        : None
Depends On      : file  glib2  glibc  icu  jemalloc  libevent  libsodium  luajit  openblas  openssl
                  pcre  sqlite  zlib
Optional Deps   : redis: Redis cache and data storage
Conflicts With  : None
Replaces        : None
Download Size   : 2.52 MiB
Installed Size  : 9.64 MiB
Packager        : Arch Linux ARM Build System <builder+n1@archlinuxarm.org>
Build Date      : Fri 01 May 2020 02:54:45 PM -03
Validated By    : MD5 Sum  SHA-256 Sum  Signature

[peracchi@rainbow ~]$ 
[peracchi@rainbow ~]$ sudo systemctl start rspamd
[peracchi@rainbow ~]$ 
[peracchi@rainbow ~]$ sudo systemctl status rspamd
鈼 rspamd.service - rapid spam filtering system
     Loaded: loaded (/usr/lib/systemd/system/rspamd.service; disabled; vendor preset: disabled)
     Active: active (running) since Wed 2020-09-23 09:33:55 -03; 25s ago
       Docs: https://rspamd.com/doc/
   Main PID: 397 (rspamd)
      Tasks: 5 (limit: 4253)
     CGroup: /system.slice/rspamd.service
             鈹溾攢397 rspamd: main process
             鈹溾攢398 rspamd: rspamd_proxy process (localhost:11332)
             鈹溾攢399 rspamd: controller process (localhost:11334)
             鈹溾攢400 rspamd: normal process (localhost:11333)
             鈹斺攢401 rspamd: normal process (localhost:11333)

Sep 23 09:33:55 rainbow systemd[1]: Started rapid spam filtering system.

@Darksky I finished my tests and everything went fine.

Rspamd is doing DKIM signing as intended.

I suspect that the cause was something related to the kernel config/headers鈥

:smile:

1 Like

I was messing with this today and was getting strange things which indicated it was not completely loaded after some reboots. I could load it fine while booted up but after a reboot was getting a strange output with 鈥渟ystemctl status rspamd鈥 and 鈥渏ournalctl | grep rspamd鈥 was indicating it was not loaded.

I removed this line in /usr/lib/systemd/system/rspamd.service and the strange situation went away on reboots.

After=nss-lookup.target network-online.target

Hi @Darksky!

Today I needed to rebuild my RPI4 server from zero.

And I discovered that, to rspamd work I only needed to install the linux-rpi4-5.4.65-2-aarch64.pkg.tar.xz and linux-rpi4-headers-5.4.65-2-aarch64.pkg.tar.xz that you provided. No need to install rspamd-2.6.1-2-aarch64.pkg.tar.xz, the default package from repositories, for rspamd version 2.5-2, worked.