High RAM & SWAP usage (zSWAP & ZFS) - apps seems to be using just small portion

Hello, please how to discover what is eating my RAM and SWAP so i can kill it and regain more space?

The GUI tools like Task manager or Usage shows apps has minimal RAM usage, but the total RAM and SWAP usage indicators are both nearly 100% and computer start lagging:

$ free;sudo ps_mem|tail;sudo systemd-swap

 38.6 MiB +  39.3 MiB =  77.9 MiB	netdata (2)
 50.0 MiB +  53.0 MiB = 102.9 MiB	app3
 64.2 MiB + 106.4 MiB = 170.6 MiB	Xorg
106.1 MiB + 121.6 MiB = 227.7 MiB	QtWebEngineProcess (8)
204.6 MiB + 222.9 MiB = 427.4 MiB	thunderbird
676.8 MiB + 684.3 MiB =   1.3 GiB	app2 (2)
864.2 MiB +   1.1 GiB =   1.9 GiB	firefox (12)
              4.8 GiB

      total        used        free      shared  buff/cache   available
    Mem:          15918       12912        2110         237         895        2352
    Swap:          6911        3952        2958

    .  accept_threshold_percent   90
    .  compressor                 zstd
    .  enabled                    Y
    .  max_pool_percent           80
    .  same_filled_pages_enabled  Y
    .  zpool                      z3fold
    Zswap:
    .  .  duplicate_entry         0                      
    .  .  pool_limit_hit          0                      
    .  .  pool_total_size         1791172608             
    .  .  reject_alloc_fail       0                      
    .  .  reject_compress_poor    0                      
    .  .  reject_kmemcache_fail   0                      
    .  .  reject_reclaim_fail     0                      
    .  .  same_filled_pages       139521                 
    .  .  stored_pages            992033                 
    .  .  written_back_pages      0                      
    .  .  compress_ratio          44%                    
    .  .  zswap_store/swap_store  4063367168/4144939008  98%
    swapD:
    .  NAME  TYPE  SIZE  USED  PRIO
    swapFC:
    .  NAME                             TYPE  SIZE  USED    PRIO
    .  /var/lib/systemd-swap/swapfc/1   file  256M  215,7M  50
    .  /var/lib/systemd-swap/swapfc/2   file  256M  175M    49
    .  /var/lib/systemd-swap/swapfc/3   file  256M  186,3M  48
    .  /var/lib/systemd-swap/swapfc/4   file  256M  174,2M  47
    .  /var/lib/systemd-swap/swapfc/5   file  256M  183M    46
    .  /var/lib/systemd-swap/swapfc/6   file  256M  177,7M  45
    .  /var/lib/systemd-swap/swapfc/7   file  256M  161,9M  44
    .  /var/lib/systemd-swap/swapfc/8   file  256M  171,3M  43
    .  /var/lib/systemd-swap/swapfc/9   file  256M  174,5M  42
    .  /var/lib/systemd-swap/swapfc/10  file  256M  146,6M  41
    .  /var/lib/systemd-swap/swapfc/11  file  256M  166,2M  40
    .  /var/lib/systemd-swap/swapfc/12  file  256M  160,2M  39
    .  /var/lib/systemd-swap/swapfc/13  file  256M  158,5M  38
    .  /var/lib/systemd-swap/swapfc/14  file  256M  184,1M  37
    .  /var/lib/systemd-swap/swapfc/15  file  256M  188,6M  36
    .  /var/lib/systemd-swap/swapfc/16  file  256M  197,7M  35
    .  /var/lib/systemd-swap/swapfc/17  file  256M  172,6M  34
    .  /var/lib/systemd-swap/swapfc/18  file  256M  143,4M  33
    .  /var/lib/systemd-swap/swapfc/19  file  256M  129,2M  32
    .  /var/lib/systemd-swap/swapfc/20  file  256M  110,9M  31
    .  /var/lib/systemd-swap/swapfc/21  file  256M  100,1M  30
    .  /var/lib/systemd-swap/swapfc/22  file  256M  94,4M   29
    .  /var/lib/systemd-swap/swapfc/23  file  256M  94,6M   28
    .  /var/lib/systemd-swap/swapfc/24  file  256M  80M     27
    .  /var/lib/systemd-swap/swapfc/25  file  256M  84,8M   26
    .  /var/lib/systemd-swap/swapfc/26  file  256M  66,7M   25
    .  /var/lib/systemd-swap/swapfc/27  file  256M  54,5M   24

Netdata shows that the ZFS ARC size (zfs.arc_size) is:

type, GBytes
arcsz, 7.17
target, 7.17
min (hard limit), 0.49
max (high water)7.77
( full ARC stats here )

In:
ls /sys/module/zfs/parameters/|grep arc|grep -E "max|limit"
i have found some files that may be relevant:
zfs_arc_max
0
Description of these mem. parameters:

  1. Module Parameters — OpenZFS documentation : “zfs_arc_max - Maximum size of ARC in bytes. If set to 0 then the maximum ARC size is set to 1/2 of system RAM”
  2. ZFS Memory Management Parameters - Oracle® Solaris 11.3 Tunable Parameters Reference Manual

Here an user is using 500MB ARC cache max on 3GB computer
but here is said “for every TB of data, you’ll want at least 1GB of RAM for caching ZFS metadata”
Here suggested that the value may need to be changed in /etc/modprobe.d/zfs.conf instead

/proc/spl/kmem/slab output here

I am wondering how to discover what is using RAM and SWAP so i can kill it, because my system becomes slow sometimes and restarting is not ideal sometimes.

Regarding ZF$, how would you tune it, so it is using less memory or how to make it to give more priority to other apps than to itself regarding memory consumption? Any command ideas how to shed more light on this?
Obviously i am looking to get more physical RAM, but i want to know what is using RAM and how to reduce its RAM usage (obviously at a cost of more disk operations).

Btw. ontopic quote:

To free pagecache, dentries and inodes:
echo 3 > /proc/sys/vm/drop_caches
This is a non-destructive operation and will only free things that are completely unused.

RE: earlyoom, i have tried to install it just now, not so easy, unsure if will help. nohang-git is in AUR, i have not tried it, not sure if is better. thx btw: here the list of tools that may help in OOM conditions.

Wow, 27 swap files! I don’t think I’ve seen so many at once. That’s close to the default limit of 32. If this happens a lot, you could do by increasing swap file size from 256mb to something bigger…

BTW, have you tried tools like nohang or earlyoom? They should do what you are trying to accomplish, but automatically. Closing the more offending apps in the memory department before it’s too late.

About what is using RAM, I mean, you already have and are using the tools. About what is swapping, here there is an interesting related tutorial by @Fabby

1 Like

as you’ve redacted your original question, I won’t be able to add anything meaningful to @Signalrunner 's comment :point_up_2: but if you have 27 swap files: delete them all and create one (=1) that is your total amount of RAM + square root of your RAM if you use hibernation…

If you don’t use hibernation, SQRT(RAM) is sufficient)…

:+1:

Nah, he’s using systemd-swap, which has this neat feature of dinamic, on-demand, chunked swap files. In like, the more swap space you need, the more chunked files are created, and when it isn’t needed anymore, it deletes them. Before swap is needed, it won’t even create the file yet.

That’s more convenient than a traditional swap file, with a lot of GB pre-allocated, when you might never or rarely need it. It’s just that with the default 256Mb chunk size, if you swap a lot, you can end with too many files, like it was OP’s case.

1 Like

Call me old-fashioned, but I still use a swap partition, though nowadays a swap file is as good as a partition.

Allocating swap files as the need arises is more of a Windows piece of |@#¼ not something I would advise to use as it’s inefficient…

:man_shrugging:

1 Like