Cannot ssh out of manjaro server

ARM
, ,
1

I’m running 5.10.88-1-MANJARO-ARM-RPI aarch64 on a Raspberry Pi 4 Model B Rev 1.4. I have a server setup… no desktop environment. I’ve been using this setup for a couple of weeks with no significant issues. Just the learning curve of this being my first manjaro system.

Until today I’ve had no issues using ssh and sftp on this machine. Today I found that I could not ssh/sftp out. Each time I get a message similar to this:

ct@tanker:~ $ ssh curtis@frigate
ssh: connect to host frigate port 22: Connection timed out

Note: host “frigate” is on my home network. I get the same result when using it’s IP address.

When I check “systemctl status sshd” I get the following:

ct@tanker:~ $ sudo systemctl status sshd
[sudo] password for ct: 
● sshd.service - OpenSSH Daemon
     Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: disabled)
     Active: active (running) since Fri 2022-01-07 19:45:25 EST; 27min ago
   Main PID: 508 (sshd)
      Tasks: 1 (limit: 8975)
        CPU: 6.043s
     CGroup: /system.slice/sshd.service
             └─508 "sshd: /usr/bin/sshd -D [listener] 0 of 10-100 startups"

Jan 07 20:10:11 tanker sshd[1502]: Disconnected from invalid user sybase123 45.124.144.116 port 39120 [preauth]
Jan 07 20:10:24 tanker sshd[1491]: fatal: Timeout before authentication for 112.85.42.151 port 61528
Jan 07 20:11:21 tanker sshd[1516]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=45.124.144.116  user=root
Jan 07 20:11:23 tanker sshd[1516]: Failed password for root from 45.124.144.116 port 58284 ssh2
Jan 07 20:11:23 tanker sshd[1516]: Received disconnect from 45.124.144.116 port 58284:11: Bye Bye [preauth]
Jan 07 20:11:23 tanker sshd[1516]: Disconnected from authenticating user root 45.124.144.116 port 58284 [preauth]
Jan 07 20:12:34 tanker sshd[1520]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=45.124.144.116  user=root
Jan 07 20:12:36 tanker sshd[1520]: Failed password for root from 45.124.144.116 port 49210 ssh2
Jan 07 20:12:38 tanker sshd[1520]: Received disconnect from 45.124.144.116 port 49210:11: Bye Bye [preauth]
Jan 07 20:12:38 tanker sshd[1520]: Disconnected from authenticating user root 45.124.144.116 port 49210 [preauth]

When I check “journalctl -xeu sshd” I get the following:

ct@tanker:~ $ journalctl -xeu sshd
Jan 07 20:06:23 tanker sshd[1470]: Received disconnect from 45.124.144.116 port 38102:11: Bye Bye [preauth]
Jan 07 20:06:23 tanker sshd[1470]: Disconnected from authenticating user root 45.124.144.116 port 38102 [preauth]
Jan 07 20:07:35 tanker sshd[1477]: Invalid user lst from 45.124.144.116 port 57266
Jan 07 20:07:35 tanker sshd[1477]: pam_faillock(sshd:auth): User unknown
Jan 07 20:07:35 tanker sshd[1477]: pam_unix(sshd:auth): check pass; user unknown
Jan 07 20:07:35 tanker sshd[1477]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=45.124.144.116
Jan 07 20:07:35 tanker sshd[1477]: pam_systemd_home(sshd:auth): systemd-homed is not available: Unit dbus-org.freedesktop.home1.service not found.
Jan 07 20:07:35 tanker sshd[1477]: pam_faillock(sshd:auth): User unknown
Jan 07 20:07:36 tanker sshd[1477]: Failed password for invalid user lst from 45.124.144.116 port 57266 ssh2
Jan 07 20:07:37 tanker sshd[1477]: Received disconnect from 45.124.144.116 port 57266:11: Bye Bye [preauth]
Jan 07 20:07:37 tanker sshd[1477]: Disconnected from invalid user lst 45.124.144.116 port 57266 [preauth]
Jan 07 20:07:43 tanker sshd[1452]: fatal: Timeout before authentication for 221.181.185.111 port 55666
Jan 07 20:08:26 tanker sshd[1492]: error: kex_exchange_identification: Connection closed by remote host
Jan 07 20:08:26 tanker sshd[1492]: Connection closed by 141.98.11.27 port 35886
Jan 07 20:08:44 tanker sshd[1493]: Unable to negotiate with 141.98.11.27 port 43542: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie-hellman-group-e>
Jan 07 20:08:51 tanker sshd[1495]: Invalid user bmw from 45.124.144.116 port 48198
Jan 07 20:08:51 tanker sshd[1495]: pam_faillock(sshd:auth): User unknown
Jan 07 20:08:51 tanker sshd[1495]: pam_unix(sshd:auth): check pass; user unknown
Jan 07 20:08:51 tanker sshd[1495]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=45.124.144.116
Jan 07 20:08:51 tanker sshd[1495]: pam_systemd_home(sshd:auth): systemd-homed is not available: Unit dbus-org.freedesktop.home1.service not found.
Jan 07 20:08:51 tanker sshd[1495]: pam_faillock(sshd:auth): User unknown

I don’t know how to interpret what I’m seeing here. What action should I take to correct the issue based on this output? What else should I be looking at to troubleshoot the issue?

I can successfully ssh into the machine (host: tanker) from at least two other computers on my home network.

2

I take it that tanker is open to the internet, as it appears script kiddies are banging on it.

Are you able to capture the errors from your login attempt as user curtis?

You can also use ssh -vvv curtis@frigate to show details of the ssh handshake from the client side.

On a related note… do you have fail2ban installed? If not, you might want to consider it to reduce the chances of the ssh hacking scripts.

3

Thanks for the reply. Here is the output:

ct@tanker:~ $ ssh -vvv curtis@frigate 
OpenSSH_8.8p1, OpenSSL 1.1.1l  24 Aug 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/ct/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/ct/.ssh/known_hosts2'
debug2: resolving "frigate" port 22
debug3: resolve_host: lookup frigate:22
debug3: ssh_connect_direct: entering
debug1: Connecting to frigate [192.168.7.101] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug1: connect to address 192.168.7.101 port 22: Connection timed out
ssh: connect to host frigate port 22: Connection timed out
4

And you can successfully ssh to frigate from other computers on your network?

So lets start from the beginning. Can you ping frigate from tanker?

5

Yes, I can ping and ssh to frigate from my Mac mini.

Here’s ping:

curtis@mac-mini ~> ping -a frigate

PING frigate (192.168.7.101): 56 data bytes
64 bytes from 192.168.7.101: icmp_seq=0 ttl=64 time=116.537 ms
64 bytes from 192.168.7.101: icmp_seq=1 ttl=64 time=8.086 ms
64 bytes from 192.168.7.101: icmp_seq=2 ttl=64 time=14.821 ms
64 bytes from 192.168.7.101: icmp_seq=3 ttl=64 time=19.244 ms
64 bytes from 192.168.7.101: icmp_seq=4 ttl=64 time=14.554 ms
64 bytes from 192.168.7.101: icmp_seq=5 ttl=64 time=14.904 ms
64 bytes from 192.168.7.101: icmp_seq=6 ttl=64 time=18.365 ms
64 bytes from 192.168.7.101: icmp_seq=7 ttl=64 time=7.874 ms
64 bytes from 192.168.7.101: icmp_seq=8 ttl=64 time=14.387 ms
64 bytes from 192.168.7.101: icmp_seq=9 ttl=64 time=8.032 ms
64 bytes from 192.168.7.101: icmp_seq=10 ttl=64 time=16.505 ms
^C
--- frigate ping statistics ---
11 packets transmitted, 11 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 7.874/23.028/116.537/29.818 ms

Here’s ssh:

curtis@mac-mini ~> ssh curtis@frigate

Linux frigate 5.10.63-v7l+ #1488 SMP Thu Nov 18 16:15:28 GMT 2021 armv7l
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

Last login: Fri Jan 7 22:06:25 2022 from 192.168.7.193
curtis@frigate:~ $
6

I’ve also removed the port forwarding from my router. The manjaro server (tanker) is no longer open to the internet.

7

If tanker can ping frigate, then we need to look deeper.

Are you familiar with tcpdump?

8

No… I’ve never used it.

9

It is a network sniffer, it can log all network traffic on a network port. If you would like to try it, install it on frigate with pacman -S tcpdump.

You can run it like: tcpdump -i eth0
That will connect to eth0 and start logging all ethernet traffic to and from frigate’s network port, press Ctrl + c to stop.

While you have it running, try to ssh from tanker to frigate.

This is what it looks like for my setup:

21:41:11.402422 IP abbynormal.56648 > mrfixit.ssh: Flags [P.], seq 4026:4062, ack 3974, win 501, options [nop,nop,TS val 354037357 ecr 2437210856], length 36
21:41:11.403160 IP mrfixit.ssh > abbynormal.56648: Flags [P.], seq 3974:4010, ack 4062, win 501, options [nop,nop,TS val 2437217620 ecr 354037357], length 36
21:41:11.403235 IP abbynormal.56648 > mrfixit.ssh: Flags [.], ack 4010, win 501, options [nop,nop,TS val 354037357 ecr 2437217620], length 0
21:41:11.403301 IP mrfixit.ssh > abbynormal.56648: Flags [P.], seq 4010:4054, ack 4062, win 501, options [nop,nop,TS val 2437217620 ecr 354037357], length 44
21:41:11.403338 IP abbynormal.56648 > mrfixit.ssh: Flags [.], ack 4054, win 501, options [nop,nop,TS val 354037358 ecr 2437217620], length 0
21:41:11.432411 IP mrfixit.47898 > abbynormal.ssh: Flags [S], seq 1564094979, win 64240, options [mss 1460,sackOK,TS val 2437217649 ecr 0,nop,wscale 7], length 0
21:41:11.432572 IP abbynormal.ssh > mrfixit.47898: Flags [S.], seq 660708421, ack 1564094980, win 65160, options [mss 1460,sackOK,TS val 354037387 ecr 2437217649,nop,wscale 7], length 0
21:41:11.432682 IP mrfixit.47898 > abbynormal.ssh: Flags [.], ack 1, win 502, options [nop,nop,TS val 2437217650 ecr 354037387], length 0
21:41:11.434774 IP mrfixit.47898 > abbynormal.ssh: Flags [P.], seq 1:22, ack 1, win 502, options [nop,nop,TS val 2437217652 ecr 354037387], length 21: SSH: SSH-2.0-OpenSSH_8.8
21:41:11.434891 IP abbynormal.ssh > mrfixit.47898: Flags [.], ack 22, win 509, options [nop,nop,TS val 354037389 ecr 2437217652], length 0
21:41:11.497804 IP abbynormal.ssh > mrfixit.47898: Flags [P.], seq 1:22, ack 22, win 509, options [nop,nop,TS val 354037452 ecr 2437217652], length 21: SSH: SSH-2.0-OpenSSH_8.8
21:41:11.497896 IP mrfixit.47898 > abbynormal.ssh: Flags [.], ack 22, win 502, options [nop,nop,TS val 2437217715 ecr 354037452], length 0
10

This is what I found and installed:

ct@tanker:~ $ sudo pacman -Ss tcpdump
extra/tcpdump 4.99.1-1
    Powerful command-line packet analyzer

I’ve already looked at some of the examples in the tcpdump manpage. Trying to figure out what may work to shed some light on my problem.

Thanks for your help so far on this issue. I’ll have to start again on it in the morning… it’s been a long day and I need to sleep now.

11

If you assign static ip addresses, make sure the netmask is correct. It is easy to forget to add it when using nmtui. And if you do forget, nmtui assigns /8 without complaint.

12

My home router is an eero6. I’ve never explicitly set static IP addresses on my home network. However, the Raspberry Pi the IP address seems to be linked to the hardware. No matter which SDHC card I use its IP address is always the same.

In other words, if I boot this machine with an SDHC setup for Manjaro its IP address is 192.168.7.226. If I replace the SDHC card and boot into Ubuntu 20.04 on the same machine, the IP address will be the same.

13

What tcpdump will help with, is determine if this is a network issue or a ssh issue.
tanker attempts to connect to frigate on port 22 and tanker indicates a timeout occurs. This is usually caused by sshd not running on frigate, or a firewall is running and is blocking/dropping packets.

In the example output above, you see abbynormal on port 56648 attempting to connect to mrfixit on port ssh (22). Next is a response from mrfixit from port ssh (22). So ssh communication is working between these two machines.

14

This is a function of the dhcpd server, running on your router. Each ethernet port has a mac address. This is a unique identifier that the dhcpd server remembers. So each time that network port connects, it will get the same address. The dhcpd server will reserve the IP for that mac address. This will be true for all of your computers, printers, phones, etc, not just the rpi.

15

It wasn’t a firewall blocking packets, it was the WireGuard VPN server. Apparently, the iptables rule I used to allow ssh traffic through the tunnel is not bi-directional. When I deactivated the tunnel interface ssh connections originating from host tanker started working again.

I should have caught it sooner but, thanks for your help!

16

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.