What network service manager/daemon does Manjaro use?

fsck_and_pray · 10 February 2021 16:46

I have a batteries-included (non-Architect) Manjaro install. I think a recent kernel switch (to 5.11 and back to 5.4) broke my networking. On every boot, I have no Internet at all until I run sudo dhcpcd. Then Internet magically returns.

I assume some networking service got stopped or uninstalled, and I’m wondering which one that might be. I suspect systemd-networkd or systemd-resolved, both of which are disabled:

$ systemctl status systemd-networkd
● systemd-networkd.service - Network Service
     Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
TriggeredBy: ● systemd-networkd.socket
       Docs: man:systemd-networkd.service(8)

$ systemctl status systemd-resolved.service 
● systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: man:systemd-resolved.service(8)
             man:org.freedesktop.resolve1(5)
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients

This answer suggests it’s systemd-resolved, but I want to make sure.

xabbu · 10 February 2021 16:52

The default is NetworkManager.

Check

systemctl status NetworkManager.service

fsck_and_pray · 10 February 2021 16:53

That one seems enabled:

$ systemctl status NetworkManager.service 
● NetworkManager.service - Network Manager
     Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled; vendor preset: disabled)

xabbu · 10 February 2021 16:58

Then check the journal to find the problem.

To see all messages from the current boot, you can use

journalctl -b -u NetworkManager.service --no-hostname --no-pager

fsck_and_pray · 10 February 2021 17:29

Thanks! I’m having trouble finding anything obvious though. My interface is enp6s0, and all I see is:

Feb 10 12:10:57 NetworkManager[781]: <info>  [1612977057.5665] manager: (enp6s0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/3)
Feb 10 12:10:57 NetworkManager[781]: <info>  [1612977057.5672] device (enp6s0): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Feb 10 12:10:57 NetworkManager[781]: <info>  [1612977057.5981] ovsdb: Could not connect: No such file or directory
Feb 10 12:10:57 NetworkManager[781]: <info>  [1612977057.5991] modem-manager: ModemManager available
Feb 10 12:11:00 NetworkManager[781]: <info>  [1612977060.8264] device (enp6s0): carrier: link connected
Feb 10 12:11:00 NetworkManager[781]: <info>  [1612977060.8268] device (enp6s0): state change: unavailable -> disconnected (reason 'carrier-changed', sys-iface-state: 'managed')
Feb 10 12:11:00 NetworkManager[781]: <info>  [1612977060.8280] policy: auto-activating connection 'Wired connection 2' (f696a677-c384-3bd5-b197-efeae69d9dd5)
Feb 10 12:11:00 NetworkManager[781]: <info>  [1612977060.8285] device (enp6s0): Activation: starting connection 'Wired connection 2' (f696a677-c384-3bd5-b197-efeae69d9dd5)
Feb 10 12:11:00 NetworkManager[781]: <info>  [1612977060.8286] device (enp6s0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Feb 10 12:11:00 NetworkManager[781]: <info>  [1612977060.8289] manager: NetworkManager state is now CONNECTING
Feb 10 12:11:00 NetworkManager[781]: <info>  [1612977060.8290] device (enp6s0): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Feb 10 12:11:00 NetworkManager[781]: <info>  [1612977060.8296] device (enp6s0): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
Feb 10 12:11:00 NetworkManager[781]: <info>  [1612977060.8302] dhcp4 (enp6s0): activation: beginning transaction (timeout in 45 seconds)
Feb 10 12:11:00 NetworkManager[781]: <info>  [1612977060.8321] dhcp4 (enp6s0): dhclient started with pid 1629
Feb 10 12:11:00 dhclient[1629]: DHCPREQUEST for 176.11.93.142 on enp6s0 to 255.255.255.255 port 67
.
.
.
Feb 10 12:11:07 dhclient[1629]: DHCPREQUEST for 176.11.93.142 on enp6s0 to 255.255.255.255 port 67
Feb 10 12:11:07 dhclient[1629]: DHCPACK of 176.11.93.142 from 176.11.93.1
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9073] dhcp4 (enp6s0):   address 176.11.93.142
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9074] dhcp4 (enp6s0):   plen 24 (255.255.255.0)
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9074] dhcp4 (enp6s0):   gateway 176.11.93.1
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9074] dhcp4 (enp6s0):   lease time 28800
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9074] dhcp4 (enp6s0):   nameserver '8.8.8.8'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9074] dhcp4 (enp6s0):   nameserver '1.0.0.1'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9074] dhcp4 (enp6s0): option broadcast_address    => '176.11.93.255'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9074] dhcp4 (enp6s0): option dad_wait_time        => '0'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9074] dhcp4 (enp6s0): option dhcp_lease_time      => '28800'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9074] dhcp4 (enp6s0): option dhcp_message_type    => '5'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9074] dhcp4 (enp6s0): option dhcp_server_identifier => '176.11.93.1'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9075] dhcp4 (enp6s0): option domain_name_servers  => '8.8.8.8 1.0.0.1'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9075] dhcp4 (enp6s0): option expiry               => '1613005867'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9075] dhcp4 (enp6s0): option ip_address           => '176.11.93.142'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9075] dhcp4 (enp6s0): option network_number       => '176.11.93.0'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9075] dhcp4 (enp6s0): option next_server          => '176.11.93.1'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9075] dhcp4 (enp6s0): option ntp_servers          => '205.4.124.163 131.216.128.37'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9075] dhcp4 (enp6s0): option requested_broadcast_address => '1'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9075] dhcp4 (enp6s0): option requested_domain_name => '1'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9075] dhcp4 (enp6s0): option requested_domain_name_servers => '1'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9075] dhcp4 (enp6s0): option requested_host_name  => '1'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9075] dhcp4 (enp6s0): option requested_ms_classless_static_routes => '1'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9075] dhcp4 (enp6s0): option requested_ntp_servers => '1'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9075] dhcp4 (enp6s0): option requested_rfc3442_classless_static_routes => '1'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9075] dhcp4 (enp6s0): option requested_root_path  => '1'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9076] dhcp4 (enp6s0): option requested_routers    => '1'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9076] dhcp4 (enp6s0): option requested_static_routes => '1'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9076] dhcp4 (enp6s0): option requested_subnet_mask => '1'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9076] dhcp4 (enp6s0): option requested_time_offset => '1'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9076] dhcp4 (enp6s0): option requested_wpad       => '1'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9076] dhcp4 (enp6s0): option routers              => '176.11.93.1'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9076] dhcp4 (enp6s0): option subnet_mask          => '255.255.255.0'
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9076] dhcp4 (enp6s0): state changed unknown -> extended
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9086] device (enp6s0): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9102] device (enp6s0): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9104] device (enp6s0): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9110] manager: NetworkManager state is now CONNECTED_LOCAL
Feb 10 12:11:07 dhclient[1629]: bound to 176.11.93.142 -- renewal in 11599 seconds.
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9187] device (enp6s0): Activation: successful, device activated.
Feb 10 12:11:07 NetworkManager[781]: <info>  [1612977067.9192] manager: startup complete

It seems mostly happy stuff. All of that was while Internet was unavailable:

$ ping 8.8.8.8
ping: connect: Network is unreachable

Then, after running sudo dhcpcd and Internet comes back,

$ sudo dhcpcd
dhcpcd-9.4.0 starting
...
forked to background, child pid 5257

$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=121 time=4.22 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=121 time=4.10 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=121 time=4.15 ms
^C

all I see is:

Feb 10 12:16:34 NetworkManager[781]: <info>  [1612977394.9330] manager: NetworkManager state is now CONNECTED_SITE
Feb 10 12:16:34 NetworkManager[781]: <info>  [1612977394.9331] policy: set 'Wired connection 2' (enp6s0) as default for IPv4 routing and DNS

xabbu · 10 February 2021 17:44

There is indeed no real error.

It would be interesting to see ip a s after you see in the journal NetworkManager state is now CONNECTED_LOCAL . And maybe a difference after you started dhcpcd.

It should show the requested IP.

Also check if you have a config in /etc/NetworkManager/conf.d/ which sets a special dhcp client. Of course the dhclient works sometimes good with problematic DHCP Servers but you can try the normal build in dhcp client.

fsck_and_pray · 10 February 2021 17:57

It would be interesting to see ip a s after you see in the journal NetworkManager state is now CONNECTED_LOCAL . And maybe a difference after you started dhcpcd.

Unfortunately, it only seems to differ in the link’s valid_lft counting down.

ip a s before starting dhcpcd (at the CONNECTED_LOCAL stage):

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp5s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 04:d9:f5:38:db:4a brd ff:ff:ff:ff:ff:ff
3: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 04:d9:f5:38:db:49 brd ff:ff:ff:ff:ff:ff
    inet 176.11.93.142/24 brd 176.11.93.255 scope global dynamic noprefixroute enp6s0
       valid_lft 28715sec preferred_lft 28715sec
    inet6 fe80::6d9:f5ff:fe38:db49/64 scope link 
       valid_lft forever preferred_lft forever

And after starting dhcpcd (with Internet resuming):

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp5s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 04:d9:f5:38:db:4a brd ff:ff:ff:ff:ff:ff
3: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 04:d9:f5:38:db:49 brd ff:ff:ff:ff:ff:ff
    inet 176.11.93.142/24 brd 176.11.93.255 scope global dynamic noprefixroute enp6s0
       valid_lft 28642sec preferred_lft 28642sec
    inet6 fe80::6d9:f5ff:fe38:db49/64 scope link 
       valid_lft forever preferred_lft forever

Also check if you have a config in /etc/NetworkManager/conf.d/ which sets a special dhcp client.

I have a file named dhclient.conf in there with the content:

[main]
dhcp=dhclient

xabbu · 10 February 2021 18:49

Indeed I also don’t see any difference.

Is there any difference between ip r s, if you run it before and after the dhcpd command?

This is really weird. You can also try to remove dhclient.conf in a different test (copy the file to a different place).

fsck_and_pray · 10 February 2021 23:32

Before:

$ ip r s
172.16.93.0/24 dev enp6s0 proto kernel scope link src 172.16.93.142 metric 100

After sudo dhcpcd and regaining Internet:

$ ip r s
default via 172.16.93.1 dev enp6s0 proto dhcp src 172.16.93.142 metric 1003 
172.16.93.0/24 dev enp6s0 proto dhcp scope link src 172.16.93.142 metric 1003

You can also try to remove dhclient.conf in a different test (copy the file to a different place).

Also, this had no effect.

xabbu · 11 February 2021 11:32

NetworManager doesn’t set a default route.

Check the connection settings and make sure you don’t use “Use this connection only for resources on its network”.

In Xfce, it on the IPv4 Settings tab, you see it after you click on Routes. Since I don’t use KDE, I don’t know the place in the KDE GUI. But it should be similar.

fsck_and_pray · 11 February 2021 17:40

Check the connection settings and make sure you don’t use “Use this connection only for resources on its network”.

Good detective work, xabbu!

I did that, and now I have a connection at boot without running sudo dhcpcd.

In Xfce, it on the IPv4 Settings tab, you see it after you click on Routes. Since I don’t use KDE, I don’t know the place in the KDE GUI. But it should be similar.

It’s the same in KDE:

However, the plot thickens slightly. There was a reason that checkbox was ticked. I ticked it earlier to solve the net issue I was having just before this one. Ever since switching kernels, I have no Internet when my VPN is connected to any of its servers. Well, sort of. I can still ping:

$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=11.6 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=119 time=11.7 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=119 time=11.6 ms
^C

But an nmap fails:

nmap google.com
Starting Nmap 7.91 ( https://nmap.org ) at 2021-02-11 12:31 EST
Failed to resolve "google.com".
WARNING: No targets were specified, so 0 hosts scanned.
Nmap done: 0 IP addresses (0 hosts up) scanned in 0.02 seconds

I also can’t connect to any websites.
After disconnecting from my VPN, both work:

$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=121 time=4.38 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=121 time=4.41 ms
^C

$ nmap google.com
Starting Nmap 7.91 ( https://nmap.org ) at 2021-02-11 12:31 EST
Nmap scan report for google.com (172.217.9.206)
Host is up (0.0046s latency).
Other addresses for google.com (not scanned): 2607:f8b0:4004:806::200e
rDNS record for 172.217.9.206: iad30s14-in-f14.1e100.net
Not shown: 998 filtered ports
PORT    STATE SERVICE
80/tcp  open  http
443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 4.59 seconds

Someone somewhere suggested that ticking the “Use this connection only for resources on its network” option would fix that, and I forgot to untick it, causing the problem you just helped me fix. But it did fix it – it just caused me to have no networking at all until I ran sudo dhcpcd.

ACTUALLY, it seems the checkbox wasn’t necessary at all; running sudo dhcpcd gets me connectivity (DNS resolution, I’m guessing?) on my VPN!

I think my VPN uses its own DNS, but I’m not sure how that’d work into things. Does any of that indicate anything to you that might obviously be wrong?

xabbu · 11 February 2021 18:14

This is now just a simple DNS problem.

Since you don’t use systemd-resolved, you just need to check /etc/resolv.conf

Do it before and after you connect to the VPN. It needs to contain at least one working DNS Server.

How easy it is to fix depends highly on what kind of VPN Software you use.

fsck_and_pray · 11 February 2021 18:31

/etc/resolv.conf before connecting to VPN:

# Generated by resolvconf
nameserver 8.8.8.8
nameserver 1.0.0.1

After:

# Generated by resolvconf
nameserver 10.64.0.1
nameserver 8.8.8.8
nameserver 1.0.0.1

xabbu · 11 February 2021 18:40

So is 10.64.0.1 a working DNS server after you connected to the VPN?

You might want to try it with dig after you connected to the VPN.

dig A google.com @10.64.0.1

Btw. it is not exactly what I would have expected. Do you use NetworkManager with the rc-manager option somewhere in /etc/NetworkManager/conf.d/ ?

After you connected to the VPN check the /etc/resolv.conf to make sure you use the correct IP, it might change.

fsck_and_pray · 11 February 2021 18:47

When connected to the VPN (and 10.64.0.1, which is still the same after checking, is in /etc/resolv.conf):

$ dig A google.com @10.64.0.1

; <<>> DiG 9.16.11 <<>> A google.com @10.64.0.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52217
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 15c89e9a9dec2169dcc4cb9560257a70c16e5a9e741624f6 (good)
;; QUESTION SECTION:
;google.com.                    IN      A

;; ANSWER SECTION:
google.com.             247     IN      A       108.177.122.113
google.com.             247     IN      A       108.177.122.101
google.com.             247     IN      A       108.177.122.100
google.com.             247     IN      A       108.177.122.139
google.com.             247     IN      A       108.177.122.102
google.com.             247     IN      A       108.177.122.138

;; AUTHORITY SECTION:
google.com.             145707  IN      NS      ns3.google.com.
google.com.             145707  IN      NS      ns1.google.com.
google.com.             145707  IN      NS      ns2.google.com.
google.com.             145707  IN      NS      ns4.google.com.

;; Query time: 23 msec
;; SERVER: 10.64.0.1#53(10.64.0.1)
;; WHEN: Thu Feb 11 13:41:52 EST 2021
;; MSG SIZE  rcvd: 235

When disconnected from the VPN (and 10.64.0.1 is no longer in /etc/resolv.conf), it doesn’t work:

$ dig A google.com @10.64.0.1

; <<>> DiG 9.16.11 <<>> A google.com @10.64.0.1
;; global options: +cmd
;; connection timed out; no servers could be reached

I don’t think so; the only file in my /etc/NetworkManager/conf.d/ is dhclient.conf, which is still just:

[main]
dhcp=dhclient

xabbu · 11 February 2021 18:53

Hm, DNS should work after you connected to the VPN. Maybe your VPN software does something more.

Which VPN Software do you use?

fsck_and_pray · 11 February 2021 18:53

Mullvad

Also, I’m pretty confident that this happened as soon as I switched to kernel 5.11 (and has remained a problem ever since, after switching back to 5.10 and further back to 5.4, which I’m on now). Could a kernel switch cause something like this?

fsck_and_pray · 11 February 2021 19:04

WAIT!!! Discovery!!!

BEFORE running `sudo dhcpcd`

/etc/resolv.conf before connecting to VPN

# Generated by resolvconf
nameserver 8.8.8.8
nameserver 1.0.0.1

After:

# Generated by resolvconf
nameserver 8.8.8.8
nameserver 1.0.0.1

And I can’t connect to sites after connecting to my VPN!

AFTER running `sudo dhcpcd`

/etc/resolv.conf before connecting to VPN

# Generated by resolvconf
nameserver 8.8.8.8
nameserver 1.0.0.1

After:

# Generated by resolvconf
nameserver 10.64.0.1
nameserver 8.8.8.8
nameserver 1.0.0.1

And now I can connect to sites after connecting to my VPN.

xabbu · 11 February 2021 19:04

No, the kernel doesn’t care about DNS resolution. The kernel does care about network packages and getting it to the correct destination.

But I would not think a kernel update can do this, maybe if you use Mullvad with wireguard, but even then I would not suspect the kernel.

xabbu · 11 February 2021 19:06

So it is a problem with your VPN software. It does not set a DNS server.

But blocks Mullvad 8.8.8.8 and 1.0.0.1 ?

What network service manager/daemon does Manjaro use?

BEFORE running sudo dhcpcd

AFTER running sudo dhcpcd

BEFORE running `sudo dhcpcd`

AFTER running `sudo dhcpcd`