Docker container can't send outbound traffic

magnunm · 12 December 2022 16:42

Hey! None of my docker containers are able to send outbound internet traffic.

I don’t think the issue is DNS related, using an IP address directly does not work either:

> docker run --rm -it joffotron/docker-net-tools
/ # ping -c 1 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes

--- 8.8.8.8 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

Thought the issue might be firewall related, so I reset iptables completely as described in the “Iptables#Resetting_rules” page of the Arch Wiki before restarting docker, but with no luck.

I am out of troubleshooting ideas, any suggestions?

Details:

> uname -srvom
Linux 5.12.19-1-MANJARO #1 SMP PREEMPT Tue Jul 20 20:57:37 UTC 2021 x86_64 GNU/Linux
> docker --version
Docker version 20.10.21, build baeda1f82a

In container:

/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
755: eth0@if756: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

/ # ip route
default via 172.17.0.1 dev eth0
172.17.0.0/16 dev eth0  src 172.17.0.2

/ # traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 46 byte packets
 1  172.17.0.1 (172.17.0.1)  0.008 ms  0.013 ms  0.004 ms
 2  *  *  *
 3  *  *  *
 4  *  *  *
 5  *  *  *
...

Host:

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s31f6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 38:f3:ab:c3:aa:dd brd ff:ff:ff:ff:ff:ff
3: wlp9s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 2c:8d:b1:f5:24:63 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.116/24 brd 192.168.10.255 scope global dynamic noprefixroute wlp9s0
       valid_lft 83160sec preferred_lft 83160sec
    inet6 2a01:799:1ba1:2f00:4aec:164a:6f95:3089/64 scope global dynamic noprefixroute
       valid_lft 3597sec preferred_lft 3597sec
    inet6 fd15:da7:edf3:4e2a:ba04:89b9:a708:bf2b/64 scope global dynamic noprefixroute
       valid_lft 1682sec preferred_lft 183sec
    inet6 fe80::a3af:a0fd:9995:78ca/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:70:e0:d7:64 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:70ff:fee0:d764/64 scope link
       valid_lft forever preferred_lft forever
5: br-404647890c89: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:2b:f9:d5:3b brd ff:ff:ff:ff:ff:ff
    inet 172.200.1.1/24 brd 172.200.1.255 scope global br-404647890c89
       valid_lft forever preferred_lft forever
    inet6 fe80::42:2bff:fef9:d53b/64 scope link
       valid_lft forever preferred_lft forever
6: docker_gwbridge: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:67:46:d5:29 brd ff:ff:ff:ff:ff:ff
    inet 172.19.0.1/16 brd 172.19.255.255 scope global docker_gwbridge
       valid_lft forever preferred_lft forever

# ip route
default via 192.168.10.1 dev wlp9s0 proto dhcp src 192.168.10.116 metric 600
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.19.0.0/16 dev docker_gwbridge proto kernel scope link src 172.19.0.1 linkdown
172.200.1.0/24 dev br-404647890c89 proto kernel scope link src 172.200.1.1 linkdown
192.168.10.0/24 dev wlp9s0 proto kernel scope link src 192.168.10.116 metric 600

Iptables setup (without the complete reset described above):

# iptables -nL
Chain INPUT (policy DROP)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED

Chain FORWARD (policy DROP)
target     prot opt source               destination
DOCKER-USER  all  --  0.0.0.0/0            0.0.0.0/0
DOCKER-ISOLATION-STAGE-1  all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0

Chain DOCKER (3 references)
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            172.200.1.2          tcp dpt:10080
ACCEPT     tcp  --  0.0.0.0/0            172.18.0.2           tcp dpt:465
ACCEPT     tcp  --  0.0.0.0/0            172.18.0.2           tcp dpt:143
ACCEPT     tcp  --  0.0.0.0/0            172.200.1.3          tcp dpt:10081
ACCEPT     tcp  --  0.0.0.0/0            172.200.1.7          tcp dpt:5434
ACCEPT     tcp  --  0.0.0.0/0            172.200.1.8          tcp dpt:6380

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  0.0.0.0/0            0.0.0.0/0
DOCKER-ISOLATION-STAGE-2  all  --  0.0.0.0/0            0.0.0.0/0
DOCKER-ISOLATION-STAGE-2  all  --  0.0.0.0/0            0.0.0.0/0
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Chain DOCKER-ISOLATION-STAGE-2 (3 references)
target     prot opt source               destination
DROP       all  --  0.0.0.0/0            0.0.0.0/0
DROP       all  --  0.0.0.0/0            0.0.0.0/0
DROP       all  --  0.0.0.0/0            0.0.0.0/0
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

mithrial · 13 December 2022 07:32

Is this the default docker setup? The gwbridge seems off.
Your iptables IPs are all 0s? And your kernel is horribly outdated.

magnunm · 13 December 2022 09:54

Yes, I have not messed with the setup myself in any way. The iptables is also the way docker sets it up automatically.
Thanks for notifying about the kernel version! Tried with 5.15 and 6.0 now, but still the same issue sadly.

xabbu · 13 December 2022 10:57

So, ask the person how did mess with this systems setup. The outputs from your first post does not show a default docker setup. It also looks like you have a VM network setup. There might be a conflict. But at first, you should find out why you have a network interface called docker_gwbridge.

magnunm · 14 December 2022 11:00

Got it working again now. The hints that the docker_gwbridge was off were helpful, as removing that interface seems to have solved the problem. It was created when I tried to set up a docker swarm.

system · 17 December 2022 01:00

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.