I have a bit of a weird issue that started just recently. When I ssh into another machine from my manjaro system (called beast) the ssh session hangs and I can only end it using ssh escape sequence ~. It does not matter which system I ssh into: at around 30 seconds (but not exactly 30 seconds) it just stops. As beast is my primary desktop, I am like a carpenter with a broken hammer
What I have tried so far:
Searched the internet and this forum
Installed kitty next to konsole: same issue.
Running journalctl -f in a separate window, to see if a message coincides with the hangup: no hints there
ssh into a set of different systems: all show the same problem, hangup in about 30 seconds
Used another system to ssh to the same set of systems : no issues, ssh still working after 5 minutes
ssh from beast to beast: no issues, ssh still working after 5 minutes
Boot into kernel 6.6.19 and 6.1.80 : same issue hangup in about 30 seconds
Although I donât see myself as a linux (or manjaro) noob I am at loss here. Hope someone here can help me solve this (or give me pointers for where to look.
As you can see here the script stops at 29 seconds⌠and at that point I have to stop ssh, as it will not respond any more. Log is added below
Another example: not using the script just an interactive shell, it stops responding after about 30 seconds, so I stop it using ssh escape codes.
I donât understand the routing remark, if routing is a problem can it still mess up an already established ssh session? I am not using a VPN , this is all local on one subnet.
Yes I can reconnect for another 30 seconds immediately after I close the âhung connectionâ.
When I connect from my laptop to this same server: no issues what so ever.
When I connect from my steamdeck to this same server: no issues what so ever
The ssh.log from the first example:
OpenSSH_9.6p1, OpenSSL 3.2.1 30 Jan 2024
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 2: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: Connecting to pve.familie-dokter.lan [192.168.3.10] port 22.
debug1: Connection established.
debug1: identity file /home/dries/.ssh/id_rsa type 0
debug1: identity file /home/dries/.ssh/id_rsa-cert type -1
debug1: identity file /home/dries/.ssh/id_ecdsa type -1
debug1: identity file /home/dries/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/dries/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/dries/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/dries/.ssh/id_ed25519 type 3
debug1: identity file /home/dries/.ssh/id_ed25519-cert type -1
debug1: identity file /home/dries/.ssh/id_ed25519_sk type -1
debug1: identity file /home/dries/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/dries/.ssh/id_xmss type -1
debug1: identity file /home/dries/.ssh/id_xmss-cert type -1
debug1: identity file /home/dries/.ssh/id_dsa type -1
debug1: identity file /home/dries/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_9.2p1 Debian-2+deb12u2
debug1: compat_banner: match: OpenSSH_9.2p1 Debian-2+deb12u2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to pve.familie-dokter.lan:22 as 'root'
debug1: load_hostkeys: fopen /home/dries/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: sntrup761x25519-sha512@openssh.com
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:/AKVmsRxSoX9rlEgdKe9uLALVnm8RbEfpO6XB4C8fis
debug1: load_hostkeys: fopen /home/dries/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'pve.familie-dokter.lan' is known and matches the ED25519 host key.
debug1: Found key in /home/dries/.ssh/known_hosts:85
debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: ssh_packet_read_poll2: resetting read seqnr 3
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_ext_info_client_parse: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com,ssh-dss,ssh-rsa,rsa-sha2-256,rsa-sha2-512>
debug1: kex_ext_info_check_ver: publickey-hostbound@openssh.com=<0>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: get_agent_identities: bound agent to hostkey
debug1: get_agent_identities: agent returned 2 keys
debug1: Will attempt key: /home/dries/.ssh/id_rsa RSA SHA256:P5vMmO7lz0hHXGwu9xPw9hlWPW6kAIAmCSMRVpiE8pg agent
debug1: Will attempt key: /home/dries/.ssh/id_ed25519 ED25519 SHA256:hEMLwKWjnLUANVaTiTM3miJCBahNBuaSUNp5BnfPkls agent
debug1: Will attempt key: /home/dries/.ssh/id_ecdsa
debug1: Will attempt key: /home/dries/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/dries/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/dries/.ssh/id_xmss
debug1: Will attempt key: /home/dries/.ssh/id_dsa
debug1: Offering public key: /home/dries/.ssh/id_rsa RSA SHA256:P5vMmO7lz0hHXGwu9xPw9hlWPW6kAIAmCSMRVpiE8pg agent
debug1: Server accepts key: /home/dries/.ssh/id_rsa RSA SHA256:P5vMmO7lz0hHXGwu9xPw9hlWPW6kAIAmCSMRVpiE8pg agent
Authenticated to pve.familie-dokter.lan ([192.168.3.10]:22) using "publickey".
debug1: channel 0: new session [client-session] (inactive timeout: 0)
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: client_input_hostkeys: searching /home/dries/.ssh/known_hosts for pve.familie-dokter.lan / (none)
debug1: client_input_hostkeys: searching /home/dries/.ssh/known_hosts2 for pve.familie-dokter.lan / (none)
debug1: client_input_hostkeys: hostkeys file /home/dries/.ssh/known_hosts2 does not exist
debug1: client_input_hostkeys: no new or deprecated keys from server
debug1: Remote: /root/.ssh/authorized_keys:6: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: /root/.ssh/authorized_keys:6: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Sending command:
start=$(date +%s)
while true; do
time="$(($(date +%s) - $start))"
printf "%s\\r" "$(date -u -d "@$time" +%H:%M:%S)"
done
debug1: pledge: fork
debug1: channel 0: free: client-session, nchannels 1
Killed by signal 2.
I created the script to be able to debug the problem, so if I do not run the script the same thing happens (as you can see in the linked example in my previous reply)
I donât have an issue with outgoing ssh connections - but for the sake of testing - I just did what you do - testing with the timer and the timer is not the issue - if that is of any comfort and I am sure you knew that.
My ssh connection to an internal device is running a steady clock.
With relation to the routing issue - it can happen if the local network is reset - a flaky connection perhaps - thus it would point to - perhaps an unstable network driver or the MTU is too high - perhaps you have jumbo frames enabled which not all nic support - most likely with older nics - the MTU is usually set to Automatic.
This would also - partly - explain why you donât see this behaviour on other systems.
If you have root access to one of the target machines, one trick Iâve found useful for debugging SSH problems over the years is to shut down the SSHD daemon, then from a terminal run /usr/bin/sshd -ddd
which will give you very verbose output when you try to connect from another machine. Itâs got me out of trouble many times.
Donât forget to restart the daemon afterwards.
I have booted into a live manjaro disk on my desktop system. When I do that, everything appears to be working as it should, so I think that means ânetwork environmentâ like cable, switch, etc. can be ruled out. This is a problem in my manjaro desktop configuration. So now need to find out when all this started and when I changed something to cause thatâŚ
I am still not sure what caused it but I removed my network definition and added a new one for the wired interface and it appears that did the trick. so: problem solved.
Thanks @anon51566685 , @linux-aarhus , @mithrial , @andreas85 , @beermad for spending some time on your Sunday to help me out.