Update removes symlink

[vivasteve]

Even if I use du to show dir sizes in /, the conclusion is still that the total space used by system files (excluding /home) is about 19G (ext4 fs)

$ sudo du -h --exclude=mnt --exclude=home --exclude=proc --max-depth=1 /
16K     /srv
16K     /lost+found
0       /dev
1.3G    /var
24K     /tmp
13M     /boot
0       /sys
40M     /etc
969M    /opt
15G     /usr
1.9G    /root
1.6M    /run
19G     /

Even more proof - using du on individual directories in /
the only directories of any real size are /usr, /var, /opt, /root

$ du -sh /usr
15G     /usr

$ du -sh /opt
969M    /opt

$ sudo du -sh /var
1.3G    /var

$ sudo du -sh /root
1.9G    /root

Still adds to 19G
The mystery remains (at least in my mind)

Seem to have strayed a bit from the original topic.
Altho not really because if my system is only using 19G the original problem was never there in the first place {:

[Aragorn]

Part of your filesystem is also being used by the ext4 journal. I’m not sure of the percentage, but I believe the maximum size for the journal on an ext4 filesystem with 4K block size to be 2 GiB.

Another thing that’s possible — I should have thought about this earlier — could be that you have at one point dumped a whole lot of stuff into /home without that /home was mounted. As a result, all of those files would still be there and consuming space on the root filesystem, but they would be invisible for as long as your /home partition is mounted.

You have to keep in mind that in and of itself, /home is only a directory on the root filesystem, and if nothing is mounted there, then whatever gets written to /home will be taking up space on the root filesystem itself. If you then mount another filesystem on /home later without having cleaned out the directory first, then that content is simply hidden by the new mount, but that doesn’t mean it’s no longer there.

The best way of checking this is to boot up from the live USB — without chrooting! — and mount your root filesystem (/dev/sdb3) to /mnt without also mounting your /home filesystem (/dev/sdb2), and then checking whether there is anything under /mnt/home. I’m willing to bet that there’s a lot of stuff there.

[vivasteve]

OK I did as you suggested - I booted up with a debian usb and mounted /dev/sdb3 to /mnt/manjaro and /dev/sdb2 to /mnt/manjaro-home.
When I looked into /mnt/manjaro/home there was a dir with my username, but the dir is empty - no contents.
This separate home partition was set up when I installed manjaro a few years back. It’s unlikely that /home has ever been unmounted

[Aragorn]

@vivasteve

When I looked into /mnt/manjaro/home there was a dir with my username, but the dir is empty - no contents.
This separate home partition was set up when I installed manjaro a few years back. It’s unlikely that /home has ever been unmounted

If there is a directory with your username under the unmounted /home, then it must have been used unmounted at some point, because if /home was always a mountpoint, then the /home on the root filesystem would be empty — no directory with your username.

And just because the directory with your username appears empty doesn’t mean that it is. There could be hidden directories and hidden files in there.

And what about swap files? Do you have any of those on your root partition?

[vivasteve]

OK, that makes sense …
I have another (fake) user registered … I noticed that there was only one user name under /home on the debian mount - the other user name was not there

What if I remount sdb3 in the debian system and delete the dir with my username under /home?

I have a separate swap partitiion

[dmt]

@vivasteve

sudo du -h --exclude=mnt --exclude=home --exclude=proc --max-depth=1 /

This is better and much shorter.

# -x skip directories on different filesystems
# -d1 max depth = 1
sudo du -hxd1 /

# I like to add in a sort
sudo du -hxd1 / | sort -h

@vivasteve

my home partition is 20G and root partition is 19G
why does df show root partition size as 36G? This 36G also includes /home

You may want to recheck your maths: 19 + 20 != 36

To see what’s actually there: Load a live USB, mount sdb3 and run du on it.

sudo mount /dev/sdb3 /mnt
sudo du -xhd1 /mnt | sort -h

[Aragorn]

There’s even more to check… In your df output, you have these two…

/dev/sda1       484G  443G   16G  97% /mnt/touro
/dev/sda2       215G  199G   16G  93% /mnt/data

What if there is a bunch of files and directories under /mnt because you copied/moved that stuff there without that /dev/sda1 and/or /dev/sda2 were mounted?

So, considering that nothing under /mnt is needed by the system, I would suggest that you temporarily unmount /dev/sda1 and /dev/sda2 and then take a look at what still lives under /mnt/data and /mnt/touro.

[vivasteve]

This is better and much shorter

Very nice tip, thanks - it’s in my regular toolbox now (:

I mounted sdb3 on a live USB and ran du
It shows a total usage of 19G
As @Aragorn suspected above, the home folder did have some hidden folders dated 2018 which I deleted. Only amounted to 1.5M
But wait, there’s more …

[vivasteve]

There were indeed a bunch of files inside /mnt/touro and /mnt/data, when I unmounted those partitions … some pacman packages (2.7G) and some old duplicity backup files (15G). I have removed those now. How they got there I can’t say - at some point when duplicity or pacman were writing to disc but those partitions were not mounted - it’s hard to see when a situation like that could have arisen seeing that those are mounted by fstab …

Anyhoo … look at the shiny new output from df

$ df -h
/dev/sdb3        39G   19G   18G  52% /
/dev/sdb1        39G   19G   18G  51% /mnt/moredata
/dev/sda1       484G  434G   25G  95% /mnt/touro
/dev/sda2       215G  199G   17G  93% /mnt/data
/dev/sdb2        24G   23G  290M  99% /home

Shows / at 52% which agrees exactly with the output from du

I can’t thank you all enough … this clears up an issue I have had for a long time, and the world makes sense again, at least for a while.

PS how to prevent future files from sneaking into /home or /mnt when the occupants are not there?

[Aragorn]

@vivasteve

There were indeed a bunch of files inside /mnt/touro and /mnt/data, when I unmounted those partitions … some pacman packages (2.7G) and some old duplicity backup files (15G).

I told ya, didn’t I?

@vivasteve

PS how to prevent future files from sneaking into /home or /mnt when the occupants are not there?

The best way is to ensure that all filesystems are mounted at boot, either by way of /etc/fstab or by way of a systemd mount, instead of relying on automounting on login. It’s really not hard to do, and it can save you a lot of trouble later.

I’ve written a tutorial about it, too.

[Tutorial] Working with additional 'drives'

Difficulty: ★★☆☆☆ 1. INTRODUCTION Microsoft Windows started its life as a graphical user interface on top of MS-DOS, a 16-bit single-user, single-tasking operating system that in turn originated as 86DOS, an unauthorized 16-bit rewrite (by Tim Paterson of Seattle Computer) of Digital Research’s originally 8-bit CP/M operating system. Both CP/M and MS-DOS were at the time developed for computers that did not support any other storage media than floppy disks. Considering this legacy, Microsof…

[dmt]

@vivasteve

It shows a total usage of 19G

Ah, so you found the extra files and deleted them first.

[vivasteve]

Yes, I deleted the main offending files under /mnt inside my usual machine, before scanning sdb3 with du under the debian usb. The offending files under /home were only 1.5M so didn’t really register in the du output

[vivasteve]

Thanks for the excellent tutorials - you have gone to a lot of trouble to write everything out so clearly in so much detail - nice.

They inspired me to tidy up my fstab and refresh my mind about all those tricky mount options.
One issue came up - I assigned the ssd option to /home which cause boot to stall with the message - unrecognized option ssd - home not mounted … not sure why that would happen … still investigating that one

But otherwise all running very smoothly

[cscs]

ssd does not appear to be a valid option according to man mount.

[vivasteve]

I agree - I’ve been searching around for any uses of ssd as an option
Strangely, in the same boot that /home would not mount, / mounted fine and both had the ssd option
Think I’ll remove that option
It is a whole new topic but the intricacies of the various mount options is probably poorly understood by many …

[Aragorn]

@cscs

ssd does not appear to be a valid option according to man mount.

It’s filesystem-specific. Not all filesystems support it. btrfs does support it, and possibly xfs too, but I don’t think ext4 does.

[cscs]

Its not listed as an option for any filesystem.
The only ssd mentioned anywhere in the man page(s) is while explaining the use of the trim option.

[Aragorn]

man 5 btrfs

MOUNT OPTIONS
   BTRFS SPECIFIC MOUNT OPTIONS
       This section describes mount options specific to BTRFS.  For the generic mount options please refer to mount(8) manual page. The options are sorted alphabetically (discarding the no prefix).

[...]

ssd, ssd_spread, nossd, nossd_spread
              (default: SSD autodetected)

              Options to control SSD allocation schemes.  By default, BTRFS will enable or disable SSD optimizations depending on status of a device with respect to rotational or non-rotational type. This is
              determined by the contents of /sys/block/DEV/queue/rotational). If it is 0, the ssd option is turned on.  The option nossd will disable the autodetection.

[cscs]

Ah, so in man 5 btrfs, which I neglected to check.