Erase windows partition, allocate free space to LUKS encrypted Linux partition

fractal_sounds · 8 July 2023 20:37

Hello,

I am running Manjaro (Kernel 6.3.5-2) and KDE desktop on a dual-boot system. The Linux partition is encrypted using LUKS. I never use the Windows partition, and would like to remove it and allocate the space taken up by it for my Linux system. I would like this reallocated space to also be encrypted, just like my current Linux partition.

Is there a way to do this without needing to reinstall Linux from scratch? If so, could you please advise the best method to create an entire system snapshot, so that if anything goes wrong during the attempt, I can revert to my current state if needed? My usual backup regiment is a little haphazard, where I keep backups of important files, but I have never done a full system backup/restore.

Or is it best to simply start from scratch with a fresh install of Linux on a single-boot setup? If so, are there ways I can backup/restore as much of my Linux system customisations as possible (things such as packages, shortcuts, KDE activity/desktop customisations, etc), in order to minimise the amount of work I need to do in getting the Linux system looking/feeling like how I currently have it running?

In terms of my level of knowledge for performing this task, I am quite proficient and comfortable on the command line and with Bash scripting in general, but am not super knowledgeable on the finer details of the “under the hood” of the Linux system, and partition/encryption technicalities.

For instance, I completed the Architect install of Manjaro when I was first setting up the dual-boot system a few years ago, including the LUKS encryption. I was using a guide provided on these forums to complete this, and didn’t run into any issues, but many of the terms and tasks were new to me, in particular regarding boot systems, grub, lvm, etc.

Details of my current partition scheme are provided below.

Thank you very much in advance for any advice you might be able to provide.

Best,
fractal_sounds

~ plasmashell --version
plasmashell 5.27.5

~ kf5-config --version
Qt: 5.15.9
KDE Frameworks: 5.106.0
kf5-config: 1.0



**************************************************************



~ sudo fdisk -l

Disk /dev/nvme0n1: 476.94 GiB, 512110190592 bytes, 1000215216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt

Device             Start        End   Sectors   Size Type
/dev/nvme0n1p1      2048   31336447  31334400  14.9G Windows recovery environment
/dev/nvme0n1p2  31336448   31541247    204800   100M EFI System
/dev/nvme0n1p3  31541248   31574015     32768    16M Microsoft reserved
/dev/nvme0n1p4  31574016  515821567 484247552 230.9G Microsoft basic data
/dev/nvme0n1p5 998166528 1000214527   2048000  1000M Windows recovery environment
/dev/nvme0n1p6 515821568  516870143   1048576   512M EFI System
/dev/nvme0n1p7 516870144  998166527 481296384 229.5G Linux filesystem



**************************************************************



~ sudo parted -l

Number  Start   End     Size    File system  Name                          Flags
 1      1049kB  16.0GB  16.0GB  ntfs         Basic data partition          diag
 2      16.0GB  16.1GB  105MB   fat32        EFI system partition          boot, esp
 3      16.1GB  16.2GB  16.8MB               Microsoft reserved partition  msftres
 4      16.2GB  264GB   248GB   ntfs         Basic data partition          msftdata
 6      264GB   265GB   537MB   fat32                                      boot, esp
 7      265GB   511GB   246GB
 5      511GB   512GB   1049MB  ntfs         Basic data partition          hidden, diag



**************************************************************



~ lsblk -a -o NAME,FSTYPE,FSVER,SIZE,MOUNTPOINTS

NAME                FSTYPE      FSVER      SIZE MOUNTPOINTS
nvme0n1                                  476.9G
├─nvme0n1p1         ntfs                  14.9G
├─nvme0n1p2         vfat        FAT32      100M
├─nvme0n1p3                                 16M
├─nvme0n1p4         ntfs                 230.9G
├─nvme0n1p5         ntfs                  1000M
├─nvme0n1p6         vfat        FAT32      512M /boot/efi
└─nvme0n1p7         crypto_LUKS 1        229.5G
  └─cryptroot       LVM2_member LVM2 001 229.5G
    ├─vg-lvol--root ext4        1.0         50G /
    ├─vg-lvol--home ext4        1.0        174G /home
    └─vg-lvol--swap swap        1          5.5G [SWAP]

cscs · 8 July 2023 20:43

Hi there,
Resizing a partition with data on it always has a greater-than-0 chance of being at least partially destructive.
I also noticed that during your initial install you opted to create a second ESP (EFI System Partition) instead of using the existing one (nvme0n1p2).
With both of these things in mind I would probably opt for a clean install if I were you.
For backups … there are a lot of options.

Synchronization and backup programs - ArchWiki

rsync - ArchWiki

fractal_sounds · 8 July 2023 21:50

@cscs Thank you very much for the response.

I think from memory the decision to create a second ESP was because I vaguely remembered having read somewhere that the 100MB existing partition may potentially have been too small if I had wanted to install multiple kernels in the future. But yeah, I wasn’t really sure whether I was doing the right thing to be honest.

If I go ahead with a fresh install, do you think the accepted solution for this question would be applicable for me as a way of gaining back all my settings/customisations? In other words, does my scheme of having / and /home as separate logical volumes on nvme0n1p7 quality as them being on separate partitions, as specified in the linked solution?

Or do you think, even if it is possible to do what is suggested in that solution, that this would defeat the purpose of starting with a clean install? In other words, should I look at manually backing up and restoring all my customisations?

Thank you so much for your assistance.

Best,
fractal_sounds

bedna · 8 July 2023 22:02

If you have a bit of extra space somewhere, why not make a clone of your whole drive so you can restore if something happens (with clonezilla f ex).
Then just try removing the windows partitions, reboot and see what happens. since you boot from your separate boot on nvme0n1p6 anyway right?
None of your windows partitions seems to be mounted on your linux anyway, so why would removing them change anything? (except maybe you should also run an update-grub so your windows os is removed from grub)
If that works, THEN try merging them.
Or are you afraid a resize or restore with clonezilla (if it fails) would remove your access to your encrypted drives?
I have never run encryption so I’m just spitballing here.

cscs · 8 July 2023 22:04

Yes that would work. Though you are either still going to resize the populated partition or if you dont touch it then obviously all of your extra space will have to be allocated elsewhere, such as /.

If you mainly want to preserve your plasma configuration then there are some specific tools for saving and applying them to a new install:

Plasma Customization Saver - KDE Store

cscs / Transfuse · GitLab

(if you look at the script take note that it probably still works ok, but it could use an update/refresh)

fractal_sounds · 9 July 2023 11:08

Hmmm that’s true, about the full clone being a safety net to allow for experimenting. Yep, nvme0n1p6 is where I boot my Linux system from, separate to the Windows install.

Yeah, a combination of that (worries about encryption access issues) and just what you commonly hear about resizing drives/partitions that have needed data on them as being potentially risky. I mean, if I do a full clone as you suggested, even if things go haywire there is still a path back to restoring to the current state. But yeah, I guess having never done something like it before and not feeling super knowledgable about the nitty gritty of luks, grub, boot partitions, repartitioning/merging etc, I’m a little anxious I’ll end up in a deep hole! One of those situations where you know there’s no logical reason to worry if you have a backup, but the thought still stresses you out, you know?

fractal_sounds · 9 July 2023 11:09

Oh great, thank you very much, I’ll look into those for sure.

bedna · 9 July 2023 14:06

I can’t speak for an encrypted drive.
But personally I have resized my root (witch is in btrfs with subvolumes) multiple times with KDE partition manager in a live environment, never bumped into problems. But that doesn’t mean it’s without risk.

Alfy · 9 July 2023 15:05

This may be outside of your usecase but why not just create a 2nd luks partition and use it for data?

keveinp56 · 10 July 2023 06:17

Back up data, understand partitioning and LUKS encryption, then proceed carefully to erase Windows partition and allocate space to encrypted Linux partition for enhanced security and storage efficiency.

mithrial · 10 July 2023 06:38

Instead of resizing the partition, I propose an easier approach: use a large enough USB drive, format it using encryption, and rsync all encrypted data to the USB drive.
Then, delete all partitions you don’t need anymore, recreate the LUKS partition, then rsync everything back from your USB drive.

Few notes:

The UUID will change, you have to edit this in /etc/fstab, /etc/crypttab, etc.
Maybe you create a new key, remember to replace the old key.

bedna · 10 July 2023 16:06

Well, to “just rsync” probably would not work without a bunch of instructions to rsync and a bunch of work afterwards, like you mention.

A dd would though (if done from live env) or by using ddrescue (that can do backups even on mounted block devices that are being used,like root) and would also transfer uuid, flags etc.
This is not recommended as stated in the documentation but I have done it multiple times without problems, BUT IT IS NOT GUARANTEED!

Never try to rescue a r/w mounted partition. The resulting copy may be useless. It is best that the device or partition to be rescued is not mounted at all, not even read-only.

Just make sure you don’t have both partitions available when booting because the same uuid will probably confuse your system.

I would go for the backup method I mentioned earlier, seems to be the absolute safest rout imho.

fractal_sounds · 12 July 2023 19:31

Thank you for the suggestion. I did think about that initially, and would probably have done this precisely if I didn’t also wish to allocate some of the free space to the root partition. I was frequently finding myself on the verge of running out of space, even with maintenance and clean-up of old packages etc, as there are a few software packages I require (or would prefer to have) that eat up a lot of storage.

Knowing how flexible Linux is, I’m guessing there probably is a way to achieve this task too, like having my original root partition and a new secondary partition act in unison as one larger root partition. But I thought I’d use this opportunity to tidy up my install and opt for a simpler partitioning scheme.

bedna · 12 July 2023 19:38

I still stand by my recommendation. Create a backup OF THE WHOLE DISK to somewhere. Remove the windows stuff and resize/move the partitions as needed in a live env.
If you have the extra space, why not also make backups on the partitions you want to use, then wipe the whole disk and try to restore that partition to a new partition you make at the start of the disk.
Don’t forget your boot!
2 different methods right there to try.

Just make DAMN SURE, the backups are correct, at least verify them with whatever backup tool you use. Partition backups CAN be tricky and not working correctly, that is why I promptly insist on a FULL DISK BACKUP and partitions as a bonus if you have the space.
A backup is never really a backup until you KNOW it is working (ie when you restore it back somewhere and can see it works)

If it fails, restore the backup of the disk you made and you are back to square one.

fractal_sounds · 12 July 2023 20:53

Your suggestion was definitely the path I would have felt most comfortable with had I chosen to persist with resizing partitions and keeping my old install. Thank you for all your explanations and advice.

However, there were a few other factors that made me decide to choose a fresh install in the end. Chief among these were some problems I had been experiencing with my Python installation (I suspect from having used both conda and pip to install packages and caused some conflicts somewhere). I know this, in and of itself, could have been resolved with some tinkering, or at worst with a purge and fresh install of Python. But it’s just one more factor that made me favour a fresh Manjaro install in the end.

As fate would have it though, things got complicated unexpectedly.

When I installed Manjaro a few years ago, I used the Architect installer. But I soon discovered this is no longer maintained and is not provided as one of the official ISOs.

Trying the official ISO for the KDE desktop environment, I was having difficulties trying to achieve my desired LVM on LUKS scheme. The installer even crashed at one point as I was trying to tinker with creating logical volumes. A search revealed others had been experiencing crashes with the installer too when attempting logical volume creation, and advice seemed to be to use the Architect installer for doing non-typical installs like LVM on LUKS.

I found that it was still possible to download and build the Architect install script from the repositories within the Manjaro live USB environment. But that too was giving me issues, throwing an error when I was trying to create the LUKS partition. I wasn’t sure whether I was doing something wrong or if it was a result of the Architect installer no longer being maintained.

When I searched for the error being thrown by the attempted LUKS creation, results suggested that this error can occur when you upgrade a kernel without rebooting. So I thought perhaps when I tried building the Architect installer, it was maybe also downloading a newer kernel at the same time, causing that error. And of course any attempt at reboot would bring me back to not having the Architect installer at all as I was in a live USB environment.

In hindsight, I perhaps could have tried to find an old Architect installer ISO somewhere, and made a live USB of that, but I didn’t think of that at the time.

Now, @bedna, I should say that I was only armed with a backup of my home directory, not a full disk backup, as I was intending a fresh install anyway. And now having already created a new partition table in attempting the fresh install, the option of going back to square one was lost to me. Lesson learnt: always arm yourself with a full disk backup as insurance when making big system changes, even if you don’t need to use it in the pathway you’ve chosen.

The option still remained of doing a plain LUKS install, with no LVM. But I really preferred to have separate root and home partitions with a single encryption key, and if I’m not mistaken, LVM on LUKS is the only way to achieve this?

It was at this point that I decided to bite the bullet and do something I had been curious to experiment with anyway: Try installing Arch Linux! Well it was certainly a challenge, and much link-following and reading needed to be completed throughout, but I’m glad to report it all went well in the end and I was able to install the system as desired. Best of all, despite my fears that I would mess something up with GRUB or something else and not be able to boot, I got everything working first time round, which I felt very proud of!

Lot of work still ahead though… I haven’t installed a desktop environment yet, and I’m sure that there will be countless occassions where I’ll be like, “Wait that’s not included either???” But I’m looking forward to the learning curve.

I feel kinda sad that my Manjaro journey has stopped, at least for the time being, particularly as it happened unexpectedly. I have an old Macbook Pro that has been on its last legs for a while. Maybe when it finally goes to Mac heaven I’ll convert it to a Manjaro machine for nostalgia. It really is a great distribution, and being my very first foray into Linux, I’ll always have a soft spot for it.

But for now, wish me luck as I inevitability have many hair-pulling moments of frustration!

bedna · 12 July 2023 21:09

Glad you got it working out for you!

Thank you for the lengthy explanation, I enjoyed reading that.

I will probably also walk down that road in the future (going pure arch) but I am not even near the comfort zone for doing that jump yet, I will most likely chicken out and first try to run it separately while keeping manjaro as my main, like I am doing with windows 11 right now, allthough Manjaro is my main and win 11 is only there to provide comfort for me in case of worst case scenario. xD

Best wishes in your future adventures!

fractal_sounds · 12 July 2023 21:49

Thank you so much!

Hehe glad my tendency for verbose replies wasn’t annoying and was enjoyed!

Well I can’t speak for the as-yet-undicovered world of pain I might be in for, haha… but in terms of the install process, if I could do it, you’ll be just fine given how much more experience you’ve had tinkering with systems. The guides on the arch wiki were super useful, and well linked for anything you might need to read up more on. But yeah, running it separately whenever you feel like experimenting with it sounds like a good option, to not have the added stress of it being your main system.

Thank you again for all your help. Best wishes too, and hope to cross paths again.

ps. I’m a bit unsure what I should do for the “Solution” choosing seeing as my eventual path was so completely different from the initial question. Is it best to leave it without a solution, or choose the response that best matched my initial intended path?