Disconnecting from SSH kills all the processes even though I have KillMode=process set!

I’ve been looking into why I’m losing the running processes, like tmux, when I disconnect from ssh. And I found section 5.9 of the manual:

5.9 tmux/screen session killed when disconnecting from SSH

If your processes get killed at the end of the session, it is possible that you are using socket activation and it gets killed by systemd when it notices that the SSH session process exited. In that case there are two solutions. One is to avoid using socket activation by using ssh.service instead of ssh.socket. The other is to set KillMode=process in the Service section of ssh@.service.

The KillMode=process setting may also be useful with the classic ssh.service, as it avoids killing the SSH session process or the screen or tmux processes when the server gets stopped or restarted.

When searching and trying to find my sshd.service, I find it in the following locations:

❯ sudo find / -name "*sshd.service"
/etc/systemd/system/multi-user.target.wants/sshd.service
/run/systemd/units/invocation:sshd.service
find: ‘/run/user/3000/doc’: Permission denied
find: ‘/run/user/3000/gvfs’: Permission denied
/sys/fs/cgroup/system.slice/sshd.service
/usr/lib/systemd/system/sshd.service

I’m not sure which of the sshd.service files is in effect so I checked all of them. Interesting enough, I can see KillMode=process in all of the files. Still, when I disconnect from the ssh, the tmux session is lost.

Does anyone know how I can keep my tmux process when disconnecting from ssh?

For good measure, here’s the content of /usr/lib/systemd/system/sshd.service (which I have a hunch is the correct file):

[Unit]
Description=OpenSSH Daemon
Wants=sshdgenkeys.service
After=sshdgenkeys.service
After=network.target

[Service]
ExecStart=/usr/bin/sshd -D
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=always

[Install]
WantedBy=multi-user.target

Thanks for the reply. I didn’t know about that. Even though it didn’t help, it might be my fault.

First of all, I don’t have the /etc/systemd/logind.conf.d/20-kill-user-processes.conf file. But what I do have is /etc/systemd/logind.conf. And in there, I could find the KillUserProcesses=no commented out. Which I uncommented and then reboot. But still, when I disconnect the ssh, I lose tmux process. Any other idea?

Not really since all my instances of tmux on various distros work without any additional config ever.

So probably something you or some package you installed modified your system and you’ll just have to go over various search results.

Example: https://stackoverflow.com/questions/62605288/tmux-windows-closing-due-to-auto-logout

Do you maybe need to enable linger on your server?

loginctl enable-linger $USER

Thanks again. Here are other things I checked and the problem is still there:

In my ~/.tmux.conf file I had:

❯ cat ~/.tmux.conf           
set-option -g default-shell /usr/bin/zsh

Which I changed to:

❯ cat ~/.tmux.conf           
set-option -g default-command bash

No help.

Then:

❯ tmux showenv -g | grep TMOUT

As it might not be obvious, there was not TMOUT present in the tmux showenv -g.

I’m out of things to check. Any help is appreciates.

Thanks for the suggestion. But it didn’t help. Here’s the long version.

I didn’t have my user enabled to linger:

❯ ls /var/lib/systemd/linger

By which I mean the list was empty. Then I enabled it for my user per suggestion:

❯ loginctl enable-linger $USER

Now:

❯ ls /var/lib/systemd/linger
mehran

For good measure:

❯ sudo reboot now

But still, I lose the tmux process when disconnecting the ssh!

A new observation. If I log into the server locally, then ssh into it remotely using the same user, tmux will not be killed if I disconnect the ssh.

I’m not sure what exactly this means. But I was hoping this could give someone a clue so they can help me with the problem.

I have tmux running on a remote system. I am always reconnected to my session. I never changed anything related to systemd.

I cannot remember where I got the tip but it is fairly simple.

Added this to my ~/.bashrc

if [[ $- =~ i ]] && [[ -z "$TMUX" ]] && [[ -n "$SSH_TTY" ]]; then
  tmux attach-session -t ssh_tmux || tmux new-session -s ssh_tmux
fi

Technically - if all conditions returns true - the snippet attach to the session ssh_tmux or creates a new ssh_tmux session.

1 Like

I use something similar in my .bash_profile or on systems with ZSH .zprofile . The if part looks like

if [ "$TMUX" = "" ] && [ "$SSH_CONNECTION" != "" ]; then

which does the same as the example form @linux-aarhus . The tmux commands are the same. I never needed to change systemd configs for this to work. Just make sure you don’t kill the tmux process, simply detach form it and exit, or just within tmux

tmux detach -P 

This will detach and close the SSH connection.

1 Like

That edit didn’t actually have any effect because
#KillUserProcesses=no
is the default anyway
as you can read in the text at the top of the file:

# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file, or by creating "drop-ins" in
# the logind.conf.d/ subdirectory. The latter is generally recommended.
# Defaults can be restored by simply deleting this file and all drop-ins.

I just wanted to say that while I’m grateful for the new suggestions, they didn’t help.

Also, I tested with a test VM and it worked without any effort. I’m sure some config on my machine is wrong. It’s just that I don’t know what it is. It could be the fact that I’m using VPN, or that I’m keys instead of password. But whatever it is, I still lose my processes when I disconnect the SSH. And again, this only happens if I’m not logged in to my account on the server. If I log in to the server first and then SSH remotely, the processes inside tmux will remain.

Still open to any other suggestions.

Well, they have nothing to do with your problem, so that’s why. :stuck_out_tongue:

Create a new user, ssh into new user, test tmux.

I just added a new user and this time, I used the password for authentication instead of the key. And the same problem. When I disconnect the SSH, the tmux process is lost.

I’m not sure if adding a new user caused it or something else changed, but now even logging in to server locally does not help. No matter what, when I disconnect the SSH, the processes are lost!

Also, I matched the /etc/ssh/sshd_config line by line on my working VM with my broken machine. I could not break the VM. VM is still working just fine (I do not lose my process after disconnecting SSH from my VM). This means that my problem is not related to SSH config. Unless there’s some config some other place that I missed.

New finding, the process in tmux survives an SSH disconnect if the tmux session was initiated on the local machine and then attached to by the remote SSH. In this case, I won’t lose the process if I reconnect to the SSH.

Newer finding. I started a new session in tmux using the local machine:

local❯ tmux 

local❯ tmux ls 
0: 1 windows (created Sun Nov 19 12:39:53 2023)

Then the same using the remote:

remote❯ tmux

remote❯ tmux ls 
0: 1 windows (created Sun Nov 19 12:39:53 2023)
1: 1 windows (created Sun Nov 19 12:41:09 2023)

Both these session will survive the disconnect. Even though one of them was initiated through SSH.