Cannot run docker on Manjaro ARM 19.12 - overlay2 missing

Hello,

I've been using Manjaro ARM on Raspberry Pi 4 to run several projects, all of them requiring docker. While on 19.10 all was fine.

I've recently flashed a new card with a fresh 19.12, updated everything and then installed docker. While it installed fine, it failed to run the docker daemon.

Running dockerd gives these errors:

ERRO[2020-01-13T15:38:05.905937659+03:00] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded.  storage-driver=overlay2
ERRO[2020-01-13T15:38:05.914551633+03:00] AUFS was not found in /proc/filesystems       storage-driver=aufs
ERRO[2020-01-13T15:38:05.922784317+03:00] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded.  storage-driver=overlay
WARN[2020-01-13T15:38:11.676072588+03:00] Your kernel does not support swap memory limit
WARN[2020-01-13T15:38:11.676216011+03:00] Your kernel does not support cgroup cfs period
WARN[2020-01-13T15:38:11.676277936+03:00] Your kernel does not support cgroup cfs quotas
WARN[2020-01-13T15:38:11.676334417+03:00] Your kernel does not support cgroup rt period
WARN[2020-01-13T15:38:11.676389472+03:00] Your kernel does not support cgroup rt runtime
INFO[2020-01-13T15:38:11.677311364+03:00] Loading containers: start.
WARN[2020-01-13T15:38:11.689866906+03:00] Running modprobe bridge br_netfilter failed with message: modprobe: WARNING: Module bridge not found in directory /lib/modules/4.19.85-1-MANJARO-ARM
modprobe: WARNING: Module br_netfilter not found in directory /lib/modules/4.19.85-1-MANJARO-ARM
, error: exit status 1
WARN[2020-01-13T15:38:11.702716442+03:00] Running iptables --wait -t nat -L -n failed with message: `iptables v1.8.4 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.`, error: exit status 3
INFO[2020-01-13T15:38:12.039567495+03:00] stopping event stream following graceful shutdown  error="<nil>" module=libcontainerd namespace=moby
INFO[2020-01-13T15:38:12.040324446+03:00] stopping healthcheck following graceful shutdown  module=libcontainerd
INFO[2020-01-13T15:38:12.040374445+03:00] stopping event stream following graceful shutdown  error="context canceled" module=libcontainerd namespace=plugins.moby
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.4 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
 (exit status 3)

sudo modprobe overlay2 returns

modprobe: FATAL: Module overlay2 not found in directory /lib/modules/4.19.85-1-MANJARO-ARM

Previously I used to install docker through the convenience script listed at docker.com however they've decided to be a**holes and return "OS not supported" now. However, the docker version pacman installs is the latest version of docker.

From what I can see the overlay module should be baked into the kernel since 4.0?

I'm at a loss on how to proceed or what to look at. (Also a linux noob, maybe i'm not looking in the right places?)

Just so we are on the same page, please make sure that your system is up to date.
sudo pacman -Syu

The current rpi4 kernel is at 4.19.93.

Yes done that but kernel is 4.19.85-1

Did you also reboot?

rebooted, reflashed, reinstalled, reupdated many times. On two different pi4 devices (one 4 GB, one 2 GB)

Something is wrong with how you update then.

Please do:
sudo pacman-mirrors -g && sudo pacman -Syu
And post the output for me.

EDIT: Also seems the raspberry pi kernel might not even have the needed functions yet.
Bu if it worked for you with 19.10, why not just update that one?

This is interesting. While you were writing, i managed to mess up the sd card I was playing with, so i reflashed it to the exact same image, reinstalled and reupdated. But today, unlike yesterday, it updated to 4.19.89-1. And docker worked....

I'm not the only one tackling this problem. We're a couple of enthusiasts putting together an install script for a software we like so non-linux users can install it with minimal guidance. The script was put together while 19.08 and 19.10 was out (19.08 was a bit of a showstopper, lacking ssh on first install and RPi4 requiring an unconventional mini-HDMI cable) and stopped working due to Docker issues with 19.12. The script basically assumes the user downloaded the latest Manjaro Arm image (so sticking with a particular release may lead to misunderstandings) and wrote it onto an SD Card and hasn't done anything else, so it already included update commands - yet it was still failing.

We're using Manjaro Arm because Raspbian is garbage and Manjaro has been providing the best support for docker. We even got a few long time Debian users saying "hey this is cool!" :slight_smile:

While the problem is currently solved, it wasn't just me running into this problem (pacman -Syu on a fresh install, then install docker and start docker service and have it fail). If we run into it again, what would be good places to start poking about before coming here?

1 Like
:: Querying servers, this may take some time...
Australia
-> 0.966 http://notmaindomain.me/stable/$arch/$repo
Denmark
-> 0.856 https://www.uex.dk/public/manjaro-arm/stable/$arch/$repo
-> 0.485 https://mirrors.dotsrc.org/manjaro-arm/stable/$arch/$repo
Germany
-> 0.477 https://manjaro-arm.moson.eu/stable/$arch/$repo
-> 0.612 https://smail.net.pl/manjaro-arm/stable/$arch/$repo
Switzerland
-> 1.162 https://mirror.scachemaille.ch/manjaro-arm/stable/$arch/$repo
United_Kingdom
United_States
-> 0.612 http://ftp-osl.osuosl.org/pub/manjaro-arm/repo/stable/$arch/$repo
:: Generated and saved '/etc/pacman.d/mirrorlist' mirrorlist.
:: Synchronising package databases...
 core                                                                         218.3 KiB   317 KiB/s 00:01 [###############################################################] 100%
 extra                                                                          2.4 MiB  2.63 MiB/s 00:01 [###############################################################] 100%
 community                                                                      5.3 MiB  7.51 MiB/s 00:01 [###############################################################] 100%
:: Starting full system upgrade...
 there is nothing to do

Here's the output from that command. I'm still at 4.19.89, not 4.19.93 btw.

We ant to help you out. :slight_smile:
But we would need to know exactly what kernel modules would be needed.
You can always check what modules are built with the current kernel in the repo, by scrolling through our Gitlab, where the config is located.

Actually the config looks pretty good as is. The aufs will not be there as upstream dropped it a while back and has to be patched in to have it.

[ray@ray-pc Desktop]$ ./check-kernel-config.sh 
info: reading kernel config from ./config ...
 
Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_NF_NAT_IPV4: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled
 
Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: missing
- CONFIG_MEMCG_SWAP_ENABLED: missing
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_IOSCHED_CFQ: enabled
- CONFIG_CFQ_GROUP_IOSCHED: enabled
- CONFIG_CGROUP_PERF: missing
- CONFIG_CGROUP_HUGETLB: missing
- CONFIG_NET_CLS_CGROUP: enabled (as module)
- CONFIG_CGROUP_NET_PRIO: missing
- CONFIG_CFS_BANDWIDTH: missing
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
    - CONFIG_BRIDGE_VLAN_FILTERING: missing
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled (as module)
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: enabled (as module)
      - CONFIG_CRYPTO_GHASH: enabled (as module)
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled (as module)
  - "ipvlan":
    - CONFIG_IPVLAN: enabled (as module)
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled (as module)
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled (as module)
    - CONFIG_DM_THIN_PROVISIONING: enabled (as module)
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing
 
Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

This topic was automatically closed after 90 days. New replies are no longer allowed.

Forum kindly sponsored by