Docker tmpfs storage issue on a clean install (through replacement of partition)

Hi all!

It’s my first time posting here, let me know if I missed something.

I just installed manjaro by replacing my old debian luks partition by the manjaro luks thanks to the installer (great work, it was super smooth).

However, I encounter issues with docker.

The installation of docker works perfectly (docker run hello world works) through the AUR or Snap but both cases seem to have issues with storage, each time I need to have shared volumes or download a larger docker it breaks with “No storage left”.

This issue is known on Arch: hxxps://wiki.archlinux.org/title/Docker#Failed_to_create_some/path/to/file:_No_space_left_on_device

However, I am not sure how to debug this. My computer has enough space and there seems to have tmpfs partition:

TARGET                                       SOURCE                                                FSTYPE          OPTIONS
/                                            /dev/mapper/luks-dcfb534e-d8ae-4692-af4c-1fab61b01056 ext4            rw,noatime
├─/proc                                      proc                                                  proc            rw,nosuid,nodev,noexec,relatime
│ └─/proc/sys/fs/binfmt_misc                 systemd-1                                             autofs          rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18434
├─/sys                                       sys                                                   sysfs           rw,nosuid,nodev,noexec,relatime
│ ├─/sys/firmware/efi/efivars                efivarfs                                              efivarfs        rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/security                     securityfs                                            securityfs      rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup                           cgroup2                                               cgroup2         rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot
│ ├─/sys/fs/pstore                           pstore                                                pstore          rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/bpf                              none                                                  bpf             rw,nosuid,nodev,noexec,relatime,mode=700
│ ├─/sys/kernel/debug                        debugfs                                               debugfs         rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/tracing                      tracefs                                               tracefs         rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/config                       configfs                                              configfs        rw,nosuid,nodev,noexec,relatime
│ └─/sys/fs/fuse/connections                 fusectl                                               fusectl         rw,nosuid,nodev,noexec,relatime
├─/dev                                       dev                                                   devtmpfs        rw,nosuid,relatime,size=12178116k,nr_inodes=3044529,mode=755,inode64
│ ├─/dev/shm                                 tmpfs                                                 tmpfs           rw,nosuid,nodev,inode64
│ ├─/dev/pts                                 devpts                                                devpts          rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
│ ├─/dev/hugepages                           hugetlbfs                                             hugetlbfs       rw,relatime,pagesize=2M
│ └─/dev/mqueue                              mqueue                                                mqueue          rw,nosuid,nodev,noexec,relatime
├─/run                                       run                                                   tmpfs           rw,nosuid,nodev,relatime,mode=755,inode64
│ └─/run/user/1000                           tmpfs                                                 tmpfs           rw,nosuid,nodev,relatime,size=2438100k,nr_inodes=609525,mode=700,uid=1000,gid=1000,inode64
│   └─/run/user/1000/gvfs                    gvfsd-fuse                                            fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=1000
├─/tmp                                       tmpfs                                                 tmpfs           rw,nosuid,nodev,nr_inodes=409600,inode64
├─/boot/efi                                  /dev/nvme0n1p1                                        vfat            rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro
├─/var/lib/snapd/snap/gtk-common-themes/1515 /dev/loop0                                            squashfs        ro,nodev,relatime
├─/var/lib/snapd/snap/core18/2074            /dev/loop2                                            squashfs        ro,nodev,relatime
├─/var/lib/snapd/snap/snap-store/547         /dev/loop1                                            squashfs        ro,nodev,relatime
├─/var/lib/snapd/snap/snapd/12704            /dev/loop3                                            squashfs        ro,nodev,relatime
└─/var/lib/snapd/snap/gnome-3-34-1804/72     /dev/loop4                                            squashfs        ro,nodev,relatime

I have another computer where I installed Manjaro on a clean disk and didn’t encounter this issue.

Have you already encountered such an issue?

uname: Linux xyz 5.10.53-1-MANJARO #1 SMP PREEMPT Mon Jul 26 07:18:28 UTC 2021 x86_64 GNU/Linux

Docker images are stored in /var/lib/docker/. Did you create a separate varpartition or mounted it on ram?
Please post the output of cat /etc/fstab to confirm.

Why did you install docker through AUR and/or (?) snap? It is the community repository.

The wiki post you linked, did you check the bullet points? (Disabled tmpfs or XFS)

1 Like

Hi!

Thank you for your answer!

So,

I didn’t changed anything in the partitioning. I used the default for encrypted partitioning.

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a device; this may
# be used with UUID= as a more robust way to name devices that works even if
# disks are added and removed. See fstab(5).
#
# <file system>             <mount point>  <type>  <options>  <dump>  <pass>
UUID=E3BA-CDF6                            /boot/efi      vfat    umask=0077 0 2
/dev/mapper/luks-dcfb534e-d8ae-4692-af4c-1fab61b01056 /              ext4    defaults,noatime 0 1

Currently, it’s from the community repo. I tried various sources, but now I am using the one from the community repo.

I don’t know how to check this. I don’t believe I have disabled tmpfs, and I don’t know how to check if I am using XFS. I don’t think I did anything specific on that side when I installed the OS.

Thank you! Hope this helps.

So update:

Here are the partitions:

~ >>> df -Th                                                                                                                                                                                                        
Filesystem     Type      Size  Used Avail Use% Mounted on
dev            devtmpfs   12G     0   12G   0% /dev
run            tmpfs      12G  1,9M   12G   1% /run
/dev/dm-0      ext4      468G   35G  410G   8% /
tmpfs          tmpfs      12G   41M   12G   1% /dev/shm
tmpfs          tmpfs      12G   19M   12G   1% /tmp
/dev/nvme0n1p1 vfat      511M  3,9M  508M   1% /boot/efi
/dev/loop1     squashfs  219M  219M     0 100% /var/lib/snapd/snap/gnome-3-34-1804/72
/dev/loop0     squashfs   33M   33M     0 100% /var/lib/snapd/snap/snapd/12704
/dev/loop4     squashfs   56M   56M     0 100% /var/lib/snapd/snap/core18/2074
/dev/loop2     squashfs   66M   66M     0 100% /var/lib/snapd/snap/gtk-common-themes/1515
/dev/loop3     squashfs   51M   51M     0 100% /var/lib/snapd/snap/snap-store/547
tmpfs          tmpfs     2,4G   88K  2,4G   1% /run/user/1000

I tried to switch to /dev/shm and /tmp for the data-root in /etc/docker/daemon.json but that didn’t work (I probably shouldn’t have done that). I’ll keep looking…

For investigating, could you pull a large docker image that leads do this error and watch the disk sizes?
In one terminal, do docker pull ... and in the other run watch df -Th.
The watch command will run the given command (df) every 2 seconds and give you the output.
You should be able to see which partition will increase.

Hi!

That is a really cool tool I didn’t know about!

I tried it and here are the findings:

  • Docker installs images and extracts them in /var/lib/docker/devicemapper/mnt/HASH which is etx4 and not tmpfs
  • this repo is owned by root
  • running the docker pull as root doesn’t fix the issue
  • I tried to changed the destination repo in /etc/docker/daemon.json but it doesn’t seem to be seen by the daemon even after reboot.
  • tmpfs runs on my machine, for example I can see Teams using it

I am tempted to chown /var/lib/docker/ to the normal user, but I am unsure about the “goodness” of that approach. What do you think?

Thank you!

No, that does not help.

The docker daemon always runs as root. It doesn’t matter with which user you’re running the docker command.

Could you follow which of the folders increase in size up to 100% when running a docker pull?

Sooo, found the culprit and the solution.

Indeed the docker images were downloaded into /var/lib/docker/devicemapper/mnt/ and by default the size was too short.

According to this answer here, it is possible to resize the base device size, which I did with:

systemctl stop docker.service
dockerd --storage-opt dm.basesize=12G
(you should terminate the command with Ctrl+C)
reboot now
(for good measure)

After that, the download worked seamlessly!

Thank you for your assistance mithrial!

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.