I have an RBI where I run several containers with docker-compose.
What I have done today is:
#docker system prune
#pacman -Syyu
#reboot
After reboot I have 6 docker containers running but one is not running. I am getting this message after docker-compose up -d --build:
Error response from daemon: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: chown /sys/fs/cgroup/system.slice/docker-72b8aa786475a12505bb14602a8453c1eb49148125e0811cd65628aa5bf6b664.scope/memory.oom.group: no such file or directory: unknown
More in details running dockerd --debug I am getting:
DEBU[2022-01-28T11:46:45.686091144+01:00] Calling HEAD /_ping
DEBU[2022-01-28T11:46:45.952842537+01:00] Calling HEAD /_ping
DEBU[2022-01-28T11:46:45.990377849+01:00] Calling GET /v1.41/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.project%3Dmarket_console%22%3Atrue%7D%7D&limit=0
DEBU[2022-01-28T11:46:46.005217136+01:00] Calling GET /v1.41/images/market_console_app/json
DEBU[2022-01-28T11:46:46.006184564+01:00] Calling GET /v1.41/images/market_console_influxdb/json
DEBU[2022-01-28T11:46:46.006383264+01:00] Calling GET /v1.41/images/grafana/grafana/json
DEBU[2022-01-28T11:46:46.015598721+01:00] Calling GET /v1.41/info
DEBU[2022-01-28T11:46:46.066461534+01:00] Calling GET /v1.41/networks/market_console_default
DEBU[2022-01-28T11:46:46.080195118+01:00] Calling GET /v1.41/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.oneoff%3DFalse%22%3Atrue%2C%22com.docker.compose.project%3Dmarket_console%22%3Atrue%2C%22com.docker.compose.service%3Dinfluxdb%22%3Atrue%7D%7D&limit=0
DEBU[2022-01-28T11:46:46.080195340+01:00] Calling GET /v1.41/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.oneoff%3DFalse%22%3Atrue%2C%22com.docker.compose.project%3Dmarket_console%22%3Atrue%2C%22com.docker.compose.service%3Dapp%22%3Atrue%7D%7D&limit=0
DEBU[2022-01-28T11:46:46.086039501+01:00] Calling GET /v1.41/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.oneoff%3DFalse%22%3Atrue%2C%22com.docker.compose.project%3Dmarket_console%22%3Atrue%2C%22com.docker.compose.service%3Dgrafana%22%3Atrue%7D%7D&limit=0
DEBU[2022-01-28T11:46:46.093000921+01:00] Calling POST /v1.41/containers/0613615cff4c8dd13c1525c8bbda0b9f368734d91bbe207ce386ecd19e2b5fa3/start
DEBU[2022-01-28T11:46:46.098024726+01:00] container mounted via layerStore: &{/var/lib/docker/overlay2/42412201c72d1fd9cbc2818c24022c38b5306e2fdb61d0d3f804d1f9aa4dfcc6/merged 0xaaaac54fbf80 0xaaaac54fbf80} container=0613615cff4c8dd13c1525c8bbda0b9f368734d91bbe207ce386ecd19e2b5fa3
DEBU[2022-01-28T11:46:46.100128080+01:00] Assigning addresses for endpoint market-console's interface on network market_console_default
DEBU[2022-01-28T11:46:46.100293910+01:00] RequestAddress(LocalDefault/172.20.0.0/16, <nil>, map[])
DEBU[2022-01-28T11:46:46.100485648+01:00] Request address PoolID:172.20.0.0/16 App: ipam/default/data, ID: LocalDefault/172.20.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65531, Sequence: (0xf0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:4 Serial:false PrefAddress:<nil>
DEBU[2022-01-28T11:46:46.136409897+01:00] Assigning addresses for endpoint market-console's interface on network market_console_default
DEBU[2022-01-28T11:46:46.136833056+01:00] a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054 (cc259f7).addSvcRecords(market-console, 172.20.0.4, <nil>, true) updateSvcRecord sid:a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054
DEBU[2022-01-28T11:46:46.137230938+01:00] a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054 (cc259f7).addSvcRecords(market-console, 172.20.0.4, <nil>, false) updateSvcRecord sid:a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054
DEBU[2022-01-28T11:46:46.137421880+01:00] a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054 (cc259f7).addSvcRecords(app, 172.20.0.4, <nil>, false) updateSvcRecord sid:a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054
DEBU[2022-01-28T11:46:46.137786429+01:00] a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054 (cc259f7).addSvcRecords(0613615cff4c, 172.20.0.4, <nil>, false) updateSvcRecord sid:a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054
DEBU[2022-01-28T11:46:46.170207700+01:00] a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054 (cc259f7).addSvcRecords(market-console, 172.20.0.4, <nil>, true) updateSvcRecord sid:a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054
DEBU[2022-01-28T11:46:46.170596230+01:00] a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054 (cc259f7).addSvcRecords(market-console, 172.20.0.4, <nil>, false) updateSvcRecord sid:a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054
DEBU[2022-01-28T11:46:46.170792060+01:00] a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054 (cc259f7).addSvcRecords(app, 172.20.0.4, <nil>, false) updateSvcRecord sid:a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054
DEBU[2022-01-28T11:46:46.170969335+01:00] a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054 (cc259f7).addSvcRecords(0613615cff4c, 172.20.0.4, <nil>, false) updateSvcRecord sid:a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054
DEBU[2022-01-28T11:46:46.186922401+01:00] Programming external connectivity on endpoint market-console (a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054)
DEBU[2022-01-28T11:46:46.198020788+01:00] EnableService 0613615cff4c8dd13c1525c8bbda0b9f368734d91bbe207ce386ecd19e2b5fa3 START
DEBU[2022-01-28T11:46:46.198172619+01:00] EnableService 0613615cff4c8dd13c1525c8bbda0b9f368734d91bbe207ce386ecd19e2b5fa3 DONE
DEBU[2022-01-28T11:46:46.217122097+01:00] createSpec: cgroupsPath: system.slice:docker:0613615cff4c8dd13c1525c8bbda0b9f368734d91bbe207ce386ecd19e2b5fa3
DEBU[2022-01-28T11:46:46.234536749+01:00] bundle dir created bundle=/var/run/docker/containerd/0613615cff4c8dd13c1525c8bbda0b9f368734d91bbe207ce386ecd19e2b5fa3 module=libcontainerd namespace=moby root=/var/lib/docker/overlay2/42412201c72d1fd9cbc2818c24022c38b5306e2fdb61d0d3f804d1f9aa4dfcc6/merged
DEBU[2022-01-28T11:46:46.576978076+01:00] event published ns=moby topic=/containers/create type=containerd.events.ContainerCreate
time="2022-01-28T11:46:46.644053968+01:00" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/0613615cff4c8dd13c1525c8bbda0b9f368734d91bbe207ce386ecd19e2b5fa3 pid=6127
DEBU[2022-01-28T11:46:46.729791140+01:00] failed to delete task error="rpc error: code = NotFound desc = container not created: not found" id=0613615cff4c8dd13c1525c8bbda0b9f368734d91bbe207ce386ecd19e2b5fa3
INFO[2022-01-28T11:46:46.730757661+01:00] shim disconnected id=0613615cff4c8dd13c1525c8bbda0b9f368734d91bbe207ce386ecd19e2b5fa3
WARN[2022-01-28T11:46:46.730952325+01:00] cleaning up after shim disconnected id=0613615cff4c8dd13c1525c8bbda0b9f368734d91bbe207ce386ecd19e2b5fa3 namespace=moby
INFO[2022-01-28T11:46:46.730994361+01:00] cleaning up dead shim
WARN[2022-01-28T11:46:46.757367992+01:00] cleanup warnings time="2022-01-28T11:46:46+01:00" level=info msg="starting signal loop" namespace=moby pid=6147
ERRO[2022-01-28T11:46:46.758590324+01:00] copy shim log error="read /proc/self/fd/37: file already closed"
ERRO[2022-01-28T11:46:46.760136224+01:00] stream copy error: reading from a closed fifo
ERRO[2022-01-28T11:46:46.761628513+01:00] stream copy error: reading from a closed fifo
DEBU[2022-01-28T11:46:46.801748062+01:00] event published ns=moby topic=/containers/delete type=containerd.events.ContainerDelete
DEBU[2022-01-28T11:46:46.853864484+01:00] Revoking external connectivity on endpoint market-console (a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054)
DEBU[2022-01-28T11:46:46.869845493+01:00] DeleteConntrackEntries purged ipv4:0, ipv6:0
DEBU[2022-01-28T11:46:46.947617854+01:00] a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054 (cc259f7).deleteSvcRecords(market-console, 172.20.0.4, <nil>, true) updateSvcRecord sid:a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054
DEBU[2022-01-28T11:46:46.947844869+01:00] a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054 (cc259f7).deleteSvcRecords(market-console, 172.20.0.4, <nil>, false) updateSvcRecord sid:a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054
DEBU[2022-01-28T11:46:46.947947997+01:00] a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054 (cc259f7).deleteSvcRecords(app, 172.20.0.4, <nil>, false) updateSvcRecord sid:a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054
DEBU[2022-01-28T11:46:46.948060643+01:00] a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054 (cc259f7).deleteSvcRecords(0613615cff4c, 172.20.0.4, <nil>, false) updateSvcRecord sid:a72998b7a595243558764ef04f65bed724e39254372131baf6ff93aaf32be054
DEBU[2022-01-28T11:46:47.008847586+01:00] Releasing addresses for endpoint market-console's interface on network market_console_default
DEBU[2022-01-28T11:46:47.009008602+01:00] ReleaseAddress(LocalDefault/172.20.0.0/16, 172.20.0.4)
DEBU[2022-01-28T11:46:47.009147470+01:00] Released address PoolID:LocalDefault/172.20.0.0/16, Address:172.20.0.4 Sequence:App: ipam/default/data, ID: LocalDefault/172.20.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65530, Sequence: (0xf8000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:5
ERRO[2022-01-28T11:46:47.513577185+01:00] 0613615cff4c8dd13c1525c8bbda0b9f368734d91bbe207ce386ecd19e2b5fa3 cleanup: failed to delete container from containerd: no such container
DEBU[2022-01-28T11:46:47.546139678+01:00] garbage collected d=8.211326ms
Any good hint?