[HowTo] bulletproof your personal effects (save your stuff!)

Difficulty: ★★☆☆☆

Content table

I am experimenting with including this for longer articles. Provide feedback if it was useful!

Preface
Preparation
     Assumptions
     Dual-boot
     File systems
     Media configuration
Execution
     Enabling the common space
     Using the common space
Why even?

Preface

Beyond the simple act of making another partition for /home during installation, there may be instances where a user may wish to have files accessible elsewhere, on another operating system with no direct support of the ext4 format as a common space used by multiple machines and systems. Whether you rock a legacy board with BIOS or a modern board with UEFI, these tips can help to save your precious work from being destroyed, and be a bit more organized.

How partitions mounted at startup are handled may differ between desktop environements / shells. As I use MATE for my desktop environment, I generally have no issues or extra provisions to worry about for the tricks below to be functional.

Preparation

:information_source: Ideally, you should consider this before installing the system as retrofitting it into an existing deployment will take much, much longer depending on length of use and files involved.

:lock: Encryption won’t be mentioned here, but if you want to give it a try, good luck! Most of these configurations may not be so encryption-friendly.

Assumptions

For brevity, I am going to assume this: You were already going to create a dual-partition setup, or you have an existing Windows system you want to access the same files from as on your Linux system, and you want to limit operations to /home as much as possible for productivity.

Linux system configuration
Additionally, I am going to assume the following partition sizes if they do not exist already, as these haven’t failed me yet with strict organization discipline:

  • 32GB for / (root, system)
  • 16GB for /home (your stuff)
Learn why these figures were used

:house: Why so small? Because the whole point of this is to not keep your files in /home. Rather, directories will be referenced from $HOME into another mounted partition. Multiple users? I would allocate another 4GB per extra user in a household for insurance. Config files aren’t that big and so long everybody agrees on disciplined file organization there shouldn’t be any problems with this!

:hash: Some people expressed concerns about the small system size. In a worst-case scenario there may be less than 100GB to spare for someone to trial and use long-term an open-source system, and personally I found so long conventional packages and shared libraries are used, 32GB is good enough, but if it is not then users are free to make their root partition larger to suit their needs.

:arrow_right: Another major advantage to using smaller system partitions, so long they are not encrypted is the idea of making an installation portable. Open-source systems can operate without fatal issues via USB 2.0 (but faster is always better) and with the above figures, a duplicate installation can be stored on a 64GB USB-attached media, with plenty of space for important effects as needed!

Dual-boot

If you are a current Windows user, expanding this may be of interest to you.

Moving effects
If using Microsoft Windows, moving your personal effects off of the disk is preferable for shrinking Windows as small as possible (with space for updates), but considering the difficulty of separating effects from the system compared to most open-source solutions, the best you can do is copy the “Big five” library directories from %userprofile% to somewhere else.

To see how this is possible, see Execution / Using the common space.

If it isn’t tenable to resize the partition Microsoft Windows is on for inclusion of parts to install an open-source system, then provision an external media to store both the open-source system and personal effects. This way, you can still access your stuff and use the open-source system on any machine you have use rights for, so long library files for hardware compatibility are installed in advance.

:arrow_right: If installing on an external media, don’t forget to place the bootloader / EFI partition there too or else the system will be inaccessible without another device used as a boot media!

File systems

Whatever your configuration is, it should be suited to the systems you intend to access with. ext4 is a given, and supports everything you need but doesn’t mount on Microsoft Windows, Apple or Android. Apple’s apfs is proprietary, and hfs+ has a reputation to incur bit rot over time (of which, the validity of such claims won’t be examined here). There are two safe options (courtesy of Microsoft) you can use as a common space for your personal effects:

  1. ntfs :lock: : Microsoft’s standard partition format for Windows which most mainstream Linux systems can read. Uses file ID system rather than a hard limit of file index nodes. While typically only the domain of Windows, this format supports symbolic links with ext4 and in both directions.
  2. exfat :bangbang:: Microsoft’s extended file allocation table format which allows for more than 4GB per file, versus fat32 which requires splitting files past this hard limit. Practically usable across modern systems, but does not support symbolic links coming from it.

If you need to access the same data across Android, OS X, Windows and open-source systems, then exfat becomes your only option. If sharing data in a common space between your open-source OS and Microsoft Windows alone, then you can use ntfs without any issues.

:lock: Addendum for ntfs: I forgot to mention this, but if power down was disgraceful, once in awhile the partition will be in a read-only state. If this happens, perform the following:

  • Reboot into Windows
  • Run chkdsk on affected partition
  • Reboot into Windows once more
  • Power down completely
  • Boot into open-source instance
  • Run sudo ntfsfix -d /sev/sdXN
    :pencil2: Replace /dev/sdXN with device and partition as shown with lsblk -f.

Eventually you’ll clear the read-only state and new content can be saved onto the ntfs partition.

:bangbang: Addendum for exfat: This partition format is not nearly as safe compared to ntfs and it is also much harder to resize this partition format without the use of third-party tools on Windows. To prevent excessive data corruption, perform a scan with chkdsk in a Windows session occasionally.

:pencil2: Previous versions of this article had a section dedicated to hardware, but this information was not practical to include due to additional, non-related questions it may generate and unnecessary length when this topic is primarily about software.

Media configuration

About limited partitions
Most recent storage media devices are able to support more than four partitions, however if your options for partitions are limited, open-source systems can operate comfortably inside of an extended partition space, where an additional one megabyte is used per partition.

If there are recovery or diagnostic partitions for your particular system, don’t delete those. Try to work around them, and if it isn’t possible to create new partitions due to media limitations then duplicating the EFI partition to another device and making said duplicate bootable can allow for deleting the original EFI partition to create an extended partition. This is an advanced procedure which will not be covered in this topic with any depth beyond this text.

Regardless whether or not other partitions existed prior to installation, the whole point of this is to not use an ext4 partition for your stuff. Without going into any degree of depth about the installation process, it is assumed from hereon all partitions you need for system and storage of personal effects had already been configured in gparted / kparted or gnome-disks.

Needless to say, none of the automated options in Calamares supports any of what this topic covers, so you should be comfortable as an end-user to exercise more agency over how your media is arranged and manually configure when prompted for which installation method to use.

Execution

After installation of the system is finished, or if retrofitting this advice by shoehorning other partitions into an existing setup, there are a few operations which need to be performed for use of this common space established to store your personal effects.

:apple: I have no experience with OS X, so I have nothing for those users. If using an OS X system in tandem as a hackintosh or compatible PowerPC system, how to resolve this should come naturally using the other advice below.

Enabling the common space

Windows

Microsoft Windows will, by default mount all connected devices and enable all partitions on attached media by default. There is no further action necessary for this to work.

Linux

This assumes gnome-disk-utility package is installed.

For ease of use, there is no better software I can think of than GNOME Disks. Upon execution, you’ll notice the left pane with all attached media, and the right pane shows details about the selected attached device.

Select the device your intended common space (and probably also, system) is on and in the graph bar, the partition where your effects will be / are at. Click on the gears symbol beneath the bar graph and select Edit Mount Options…

From this dialogue, disable User Session Defaults and toggle on Mount at system startup. Optionally, change via the Identity dropdown how the partition is identified according to /etc/fstab. That is all which is necessary. If for some reason you don’t want to use gnome-disks, here is a generic /etc/fstab entry you modify to suit:

:spiral_notepad: insert with correct device and partition in /etc/fstab:

<IDENTITY> <MOUNTPOINT> auto nosuid,nodev,nofail 0 0

:book: There are a variety of ways to handle this based upon available information, so it is best to read the manual for fstab in a terminal emulator before using the generic example above.

When finished, click on the play button below the bar graph and the partition will be mounted. WIth subsequent reboots, this partition will be mounted automatically.

Using the common space

Method of access and partition used are mutually exclusive. While method of access may be favourable depending on partition format, both can be used for each, which is why using one method for a particular format will only be advised rather than strictly for that format.

:fast_forward: For brevity, everything past this will assume use of the the big five directories:

  • Documents
  • Downloads
  • Music
  • Pictures
  • Videos

Also, we’ll assume the partition (when mentioned) is labelled foo and this is already in /etc/fstab:

In a hypothetical, example /etc/fstab:

LABEL=foo /mnt/foo auto nosuid,nodev,nofail 0 0

For anything involving Windows, we’ll assume what is known as /mnt/foo/ is F: and the directory for two-level links is known as foostuff since dot-hidden directories are not a thing which is possible.

Expand the sections below depending on what you wish to use:

Windows
Registry

To do this in more modern builds of Windows (Vista+), simply right-click on a library directory in Explorer and see its Properties. There should be a tab labelled Target which allows you to define where the files are. If you have another USB-attached media which can accommodate space for all directories, then Explorer will put forth the effort on your behalf to move the files to the new location.

Once finished, resize the ntfs partition Microsoft Windows is on using diskmgmt.msc within the current session (as Windows allows this) and create the new common space which will become the target location for these library directories.

The benefit in doing it like this is that these directories will already be defined in the Windows registry so continued operations in an open-source system can take place, with less rebooting necessary.

Limited disk space
If resizing the Windows partition substantially is impossible — even after use of chkdsk and defragmenting the media:warning: — then you may just want to keep your effects on whatever separate device you have. So long there is enough space for the open-source system partitions, the open-source system can still be installed.

:warning: Do not defragment a solid state media! It will only cause more wear for no gain!

:left_right_arrow: If using mklink rather than defining library targets, don’t forget that utility does things backward compared to GNU coreutils’ ln provided by most open-source systems.

 
Symbolic links

If you would rather not muck with registry settings or use Explorer to define directory paths for library locations, then this can be done instead to make use of common space in Windows possible.

Use of two-level links
If for some reason the partition location or drive letter changes, the operations shown after this will have to be performed again. To prevent the need of recreating every link, the mountpoint could be linked instead as shown:

The first of two levels

# Define links
mklink /S %userprofile%\foostuff F:

If incorporating this, all future links should begin with %userprofile%\foostuff as this link directs to the same path as F:. If this changes, re-creation with the new path will re-establish all links:

This time with G:

unlink %userprofile%\foostuff
mklink /S %userprofile%\foostuff G:

To utilize the directory paths from /mnt/foo/ as symbolic links, the procedure below will enable access to the directories from there in $HOME.

# Copy because move could cause file corruption if interrupted.
xcopy /E %userprofile%\Documents\ F:
xcopy /E %userprofile%\Downloads\ F:
xcopy /E %userprofile%\Music\ F:
xcopy /E %userprofile%\Pictures\ F:
xcopy /E %userprofile%\Videos\ F:
# Delete originals pre-link
rmdir /Q /S %userprofile%\Documents
rmdir /Q /S %userprofile%\Downloads
rmdir /Q /S %userprofile%\Music
rmdir /Q /S %userprofile%\Pictures
rmdir /Q /S %userprofile%\Videos
# Define links
mklink /S %userprofile%\Documents F:\Documents
mklink /S %userprofile%\Downloads F:\Downloads
mklink /S %userprofile%\Music F:\Music
mklink /S %userprofile%\Pictures F:\Pictures
mklink /S %userprofile%\Videos F:\Videos
Linux
XDG directories

While exfat doesn’t allow symbolic links from it, ext4 (and presumably, ntfs) allows symbolic links to directories in an exfat partition. Regardless, if using exfat it may be advisable to redefine XDG directories instead for avoiding trouble with links.

The following will do just that, changing the paths XDG uses for the big five directly:

Sample script for populating an exfat part and using XDG for home directories:

# Copy because move could cause file corruption if interrupted.
cp -R $HOME/Documents/ /mnt/foo/
cp -R $HOME/Downloads/ /mnt/foo/
cp -R $HOME/Music/ /mnt/foo/
cp -R $HOME/Pictures/ /mnt/foo/
cp -R $HOME/Videos/ /mnt/foo/
# Define paths
xdg-user-dirs-update --set DOCUMENTS /mnt/foo/Documents/
xdg-user-dirs-update --set DOWNLOAD /mnt/foo/Downloads/
xdg-user-dirs-update --set MUSIC /mnt/foo/Music/
xdg-user-dirs-update --set PICTURES /mnt/foo/Pictures/
xdg-user-dirs-update --set VIDEOS /mnt/foo/Videos/
# Delete originals post-copy
rm -rf $HOME/Documents/ $HOME/Downloads/ $HOME/Music/ $HOME/Pictures/ $HOME/Videos/

:mag_right: Yes, you read that right, DOWNLOAD. Not plural.

 
Symbolic links

If you would rather not configure using XDG and find it easier to perform symbolic links regardless whatever partition format is in use, then below is the only other way to make this work.

Use of two-level links
If for some reason the partition location or identity changes, the operations shown after this will have to be performed again. To prevent the need of recreating every link, the mountpoint could be linked instead as shown:

The first of two levels

# Define links
ln -s /mnt/foo/ $HOME/.foo

If incorporating this, all future links should begin with $HOME/.foo/ as this link directs to the same path as /mnt/foo/. If this changes, re-creation with the new path will re-establish all links:

This time with /mnt/bar/

unlink $HOME/.foo
ln -s /mnt/bar/ $HOME/.foo

To utilize the directory paths from /mnt/foo/ as symbolic links, the procedure below will enable access to the directories from there in $HOME.

Sample script for populating an ntfs part and using symlinks for home directories:

# Copy because move could cause file corruption if interrupted.
cp -R $HOME/Documents/ /mnt/foo/
cp -R $HOME/Downloads/ /mnt/foo/
cp -R $HOME/Music/ /mnt/foo/
cp -R $HOME/Pictures/ /mnt/foo/
cp -R $HOME/Videos/ /mnt/foo/
# Delete originals pre-link
rm -rf $HOME/Documents/ $HOME/Downloads/ $HOME/Music/ $HOME/Pictures/ $HOME/Videos/
# Define links
ln -s /mnt/foo/Documents/ $HOME/Documents
ln -s /mnt/foo/Downloads/ $HOME/Downloads
ln -s /mnt/foo/Music/ $HOME/Music
ln -s /mnt/foo/Pictures/ $HOME/Pictures
ln -s /mnt/foo/Videos/ $HOME/Videos

Links to common space
The use of symbolic links to common space is discouraged, though if for some reason you would like to perform this, understand that any links coming from the ext4 partition will be invalid if using Microsoft Windows as the physical system in session. Strict discipline to minimize use of links coming from ext4 will ensure a happier time if needing to use Microsoft Windows.

Why even?

If you scrolled to the bottom of this post for a tl;dr to the entire thread, put simply it’s this: Placing your stuff elsewhere is nice, and for sake of productivity, using paths you define for where software expects certain directories while improving organization can help to perform tasks more quickly, and make it stupid simple to back up everything for later use when the original media quits working.

That’s right — This isn’t a cop-out for backing up, but rather this is a way of getting stuff done faster. With all of your stuff in a single place without any system files attached you have no excuse — and you’d be an idiot — to not image whole partition to another device for safe keeping when the reaper takes your original copies of effects to the great recycle bin in the sky.

You should also back up your /home partition. If that fails for any reason, a quick restore later and access to your stuff would be returned. If the system happens to quit working, having a backup of that isn’t strictly necessary but preferred.

If you keep everything on a single disk, then you should still clone the entire media. If you’ve done everything correctly, then these provisions to assert more agency and responsibility over your data can help to save it against corruption of ext4 partitions due to disgraceful dismounts and the warring conflict between your open-source system and Microsoft Windows’ updates.

:heart: Special thanks

@nam1962 for highlighting the deep linking feature of Discourse which allows for a ToC.
@bogdancovaciu for making me a damned fool with pointing out how forgetful I was of how HTML anchors worked. The table works now because of it.

11 Likes

This thread is a mess I get it. But there’s a lot of information here, and is effectively a dump of all basic knowledge I’ve gained from playing with partitions and getting to grips about what works for which instances. It’s worth it to just read through it once — everything will begin to make sense with subsequent viewings, so you can use this information in a practical application.

(Thread moved to #contributions:Tutorials.)

Cleaned up terminology because I wrote most of this in a semi-conscious state. So for my sake, I am writing about data terminology to compare against and also to enhance people’s understanding of digital storage, regardless if it were used in the OP:

Media: Any device used to communicate with a medium via a transport.
Medium: The host controller, with attached transport to media.
Transport: Any method of transfer, wireless or cabled.
Disk: Generally short for hard disk, as most people store data on those (if not using a solid-state media).
Partition: A defined range of space on any media to keep a finite amount of data.
Symbolic link: A reference by file ID, index node or absolute path to an existing file.

As I am also a Windows user, I was prone to using the word “Drive” a whole lot. I should have grown up in my eleven years and ventured to kick that habit real fast because drive could mean just about anything and mixing this in with disks and media wasn’t a good look, but I guess as with many other things I do that’s bad regarding Linux, old habits die hard.

Most recent edits were basically to clarify between “Disk”, “Media” and “Partition”, and to excise any instance of “Drive” as unnecessary and useless verbiage. The intent of this article is to explain how a common storage space is a great idea, but somehow I avoided mention of that at all in a post that’s about creating a common space for cross-platform access and ease of backup by excluding system files, which was a failure of my ability to communicate this idea directly.

I also waffle about between using one device and multiple, even though I mentioned I were assuming the worst and discussing about use of only a single media device. There are multiple scenario resolutions which accommodate both instances, leaving readers to resolve their data woes with what they have already. For novices, this article may overload them with too much information, which in future may be wholly re-written to further clarify the intent of this post.

Apologies for any confusion this may have caused.

Thank you, I was not sure where to put this information.

1 Like

This is great info, but it’s also “a wall of text” which won’t be read unless you split the commands, logic and explanations by using the [details=""] section much more than you’ve done now.

Have a look here. That one is also a “wall of text” when you expand all sections, and this one too but the point is: you don’t need to expand all sections: you just expand the ones you need to do what you want to do…

E.G. You can have detail sections within detail sections:

Headline
  • Lorem ipsum dolor sit amet,

    consectetur

    adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

  • Ut enim ad minim veniam,

    quis

    nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.

by doing this:

[details="Headline"]
* Lorem ipsum dolor sit amet,
   [details="consectetur"]
   adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
   [/details]
* Ut enim ad minim veniam, 
   
   [details="quis"]
   nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
   [/details]
[/details]

(Yeah, it’s a bitch getting the spacing correct)

:innocent: :+1:

1 Like

I am use to dealing with that. Especially painful when writing about code, but if one already deals in code, one also is probably dealing in spacing of code and thus is use to the pain. (That, or one uses a text editor and does /n[spaces] performing find and replace operations.)

I understand the wall of text issue. I understand it too well. Manuals often suffer from this, so I suppose you could say as a compromise to accessibility, the way I write is more consistent with software manuals.

That is done by @admins not moderators.

1 Like

This is fantastic! I haven’t updated my desktop yet; I am going to follow this guide before I do. I really appreciate the step by step and the explanations. Thank you for taking your time to share this!

4 posts were split to a new topic: Correct size for root partition

The only reason why they do that is because of FlatPak being sandboxed. You’ll have 10+ copies of the same library across multiple applications, which is stupid in comparison to a shared library setup which takes up much less space.

As much as I love FlatPak and Snap in concept and understand how much dep hell it can prevent, I would rather just different version packages for different software. If something needs to be that specific to require its own libs, I wouldn’t mind it. And reallly, that is this topic’s target — the user who installs conventional packages, and maybe the occasional sandboxed software.

Not to say you’re wrong for what you’re doing, testing is great! And not everybody is conventional, nor should follow my advice. @Persephone should evaluate what software on the system is necessary and one’s own use habits before committing to the above advice. But I believe I provided sane defaults for most people trying any Linux system out.

Per my understanding, this configuration makes backing up more tenable for the average user than what OEMs provide for Windows by default because less thinking and configuration is involved. It can better integrate into a typical user lifestyle so when stuff goes wrong, it can become right faster.

Doing the dishes? Back up home.
Mowing the lawn? Back up root.
Working AFK or socializing? Back up effects.

Nice Article! Well done.

There are methods to access ext4 on windows – some of them are awkward, require programs or are read only. It’s been years since i’ve had to dual-boot and havent had windows in a long while, but I used to do it quite regularly.

think this is the one I used to use: https:// www.ext2fsd. com/

Other links a quick search found
https:// www.thewindowsclub. com/how-to-read-ext4-in-windows-10
https:// www.howtogeek. com/112888/3-ways-to-access-your-linux-partitions-from-windows/

(remove spaces from links to make them work)

I should had said, by default. Everything about data accessibility in the OP is about accommodating what your firmware / system allows normally rather than pulling out tricks to make systems do things they don’t perform by design.

The best part about this is every example provided doesn’t require the installation of additional software. A common space for effects which all systems respect is infinitely better than a configuration where you happen to need that one file from your home part, so you have to spin up your Ext2+ hack in Windows to grab it. Rather, this common space is what Windows and open-source systems can tap into without odd pieces of kit.

OS X, I am not sure about — Paragon still provides a utility for reading NTFS partitions but since I lack experience there, and I am not sure if Apple allows this by default now, I’d recommend exfat for Apple systems since in Apple’s own documentation they mention it as something which can be used in their Disk Utility application.

1 Like

Yeah,this is incorrect. @admins please give me additional time to edit so I can fix this. ntfs on OS X only applies to users of Paragon NTFS for OS X (or some other free solution) whereas exfat is suited for OS X as a common space.

So for practically everything, exfat, for Windows only, ntfs.

Yes… If they stick to vanilla (I never expect a new user to stick to vanilla) and if you consider today’s norm.
My two cents are there to allow for long run installs. Several years as I see on my 50+ converted friends.

:wink:

I’ve been using Ubuntu for years with my figures. Though, if someone isn’t sure and feel they need more space for the time being 48GB or even 64GB is a good bet.

Another thing I never mentioned (but should had) is the idea of making a backup that you can boot from to continue work. If it can fit on a USB stick, then you can continue to perform work if for some reason the primary media no longer functions and you kept work off of it. That’s actually a really good point I would love to append — the capability to become portable with use of a monolithic kernel.

That is why in scenarios where a RAID setup isn’t in anyone’s future, at the very least doing work on a USB stick can save the trouble of trying to recover most recent work if using a common area as a vault after work is no longer necessary to persist on the portable media.

1 Like

I will make a proper post on this later but it might be worth mentioning a project with a related but different aim:

I was finally given the capability to edit this post (and made an effort to remove my pleas from this topic), so I’ve tightened everything up and excised the fat I added previous so that the message conveyed comes across more clearly with correct information and additional reasoning for performing as described in the OP.

Half of the OP is a complete re-write, so it might behoove previous readers to look through it again.

1 Like

I added a ToC but the ToC is broken? Does it even work here?

Code:

> <big><a href='heading--one'>Preface</a></big>
> <big><a href='heading--two'>Preparation</a></big>
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<a href='heading--twoA'>Assumptions</a>
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<a href='heading--twoB'>Dual-boot</a>
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<a href='heading--twoC'>File systems</a>
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<a href='heading--twoD'>Media configuration</a>
> <big><a href='heading--three'>Execution</a></big>
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<a href='heading--threeA'>Enabling the common space</a>
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<a href='heading--threeB'>Using the common space</a>
> <big><a href='heading--four'>Why even?</a></big>

Sample heading

<H1 id='heading--one'>Preface</H1>

Very complete edition !
For unknown reason, here, I see the table of content, my cursor “sees” the links, but nothing happens : doesn’t move to the anchors.

Edited last post.

For anyone wondering what advice I followed, it’s here: Deep Linking to Headings (Anchors) - faq - Discourse Meta

1 Like