Unable to mount ext4 partition after shrinking it

Aarni501 · October 18, 2020, 6:43pm

Yes, I should have had backups and what I did was stupid. I couldn’t find threads regarding my situation and I was too afraid to try random tips found on other threads regarding corruption/unmountable partitions, to not make the situation worse.

Situation:
I shrinked an ext4 partition from my secondary HDD with KDE Partition Manager to make space, the partition was unmounted and it was mostly empty. The operation ended with a “failed check”. I then tried mounting the partition, but Dolphin simply gave this error: Error mounting /dev/sdc1 at /run/media/[USER]/d4840cc7-b60f-4fcf-a759-566b2e276a6e: wrong fs type, bad option, bad superblock on /dev/sdc1, missing codepage or helper program, or other error

I ran fsck, which gave the following:

$ LANG=C sudo fsck /dev/sdc1
fsck from util-linux 2.36
e2fsck 1.45.6 (20-Mar-2020)
The filesystem size (according to the superblock) is 488378368 blocks
The physical size of the device is 435949568 blocks
Either the superblock or the partition table is likely to be corrupt!
Abort<y>? yes

So for some reason the superblock claims the partition to be the size it was before resizing it despite the partition now being smaller than that. I don’t know what that means nor where to continue in troubleshooting this.

What I’ve tried:

I tried making KDE Partition Manager to “check and repair” the partition several times but it ended with the same error as above.
I also tried following this advice, but it didn’t work (i didn’t test all of the superblocks, only 4-5).

How can I recover my data or troubleshoot this issue?

megavolt · October 15, 2020, 11:17pm

What were your options for fsck?

Maybe try this (dry-run, remove n for real action)

sudo fsck.ext4 -nycf /dev/sdc1

Maybe you have to use testdisk to recover the old partition table: TestDisk Step By Step - CGSecurity

Just in case… make a backup:

mksquashfs image-dir squash.img -p 'sdc1_backup.img f 444 root root dd if=/dev/sdc1 bs=4M'

Aarni501 · October 16, 2020, 1:22pm

The command I used in the op was simply $ sudo fsck /dev/sdc1

Here’s the output of the commands you gave:

$ LANG=C sudo fsck.ext4 -nycf /dev/sdc1
e2fsck: Only one of the options -p/-a, -n or -y may be specified.

$ LANG=C sudo fsck.ext4 -ycf /dev/sdc1  
e2fsck 1.45.6 (20-Mar-2020)
The filesystem size (according to the superblock) is 488378368 blocks
The physical size of the device is 435949568 blocks
Either the superblock or the partition table is likely to be corrupt!
Abort? yes

Aarni501 · October 16, 2020, 1:43pm

This is what KDE Partition Manager looks like when trying to “check” the partition.

freggel.doe · October 16, 2020, 6:11pm

This looks like the filesystem is bigger than the partition(device) holding it.

Aarni501 · October 17, 2020, 7:21pm

If this is the case, what should I do? Would it be possible to somehow re-extend the partition? I’d just need to be able to mount/read the partition so I could get my files to safety.

Or is it possible to backup the partition even in its current state without backing up the whole 1,62 TiB of mostly empty space, that I have nowhere to put.

Aarni501 · October 18, 2020, 7:34pm

All the guides or threads I find online (like this Ubuntu DataRecovery) are dealing with partitions that disappeared from the partition manager. But mine is still there, it’s just unmountable. Are those guides still applicable for me? Would it be useful to try parted, testdisk or gpart like mentioned in the Ubuntu guide?

@megavolt did mention TestDisk, but I didn’t really understand what “recovering the old partition table” means in my case. I haven’t lost the partition table as I can still see the partition? Could the partition table still be corrupted? This is all way above my head.

On a side note I couldn’t use the mksquashfs command. I tried to search about it but I couldn’t understand how it’s supposed to be used. And in the end, if the backup is the size of the partition (1,6 TiB) I have nowhere to store it.

megavolt · October 19, 2020, 4:37am

Hey @Aarni501

The superblock says that the file system has 488378368 blocks, but the the physical size is 435949568 block. You need to complete the resize and set the correct size in the superblock:

resize2fs /dev/sdc1 435949568

and then again:

fsck -f /dev/sdc1

I think that is the solution…

Aarni501 · October 19, 2020, 2:43pm

Thank you @megavolt, but I’m having trouble running the command. When I run it, it complains that I have to run e2fsck first:

$ LANG=C sudo resize2fs /dev/sdc1 435949568
resize2fs 1.45.6 (20-Mar-2020)
Please run 'e2fsck -f /dev/sdc1' first.

I’ve ran that command before, but I always aborted it. Now I tried not aborting it, but ran into a new problem:

$ LANG=C sudo e2fsck -f /dev/sdc1
e2fsck 1.45.6 (20-Mar-2020)
The filesystem size (according to the superblock) is 488378368 blocks
The physical size of the device is 435949568 blocks
Either the superblock or the partition table is likely to be corrupt!
Abort<y>? no
Pass 1: Checking inodes, blocks, and sizes
Error reading block 436207648 (Invalid argument) while getting next inode from scan.  Ignore error<y>? yes
Force rewrite<y>? no
Error reading block 436207649 (Invalid argument) while getting next inode from scan.  Ignore error<y>? yes
Force rewrite<y>? no
Error reading block 436207650 (Invalid argument) while getting next inode from scan.  Ignore error<y>? yes
Force rewrite<y>? no
Error reading block 436207651 (Invalid argument) while getting next inode from scan.  Ignore error<y>? yes
Force rewrite<y>? no
Error reading block 436207652 (Invalid argument) while getting next inode from scan.  Ignore error<y>? yes
Force rewrite<y>? no
Error reading block 436207653 (Invalid argument) while getting next inode from scan.  Ignore error<y>? no
Error while scanning inodes (109051904): Can't read next inode
e2fsck: aborted

When the check reaches the portion of the disc that was left out after the shrinking, it complains it cannot read the block. I tell it to ignore the error. It asks whether to force rewrite, I say no just in case, because I don’t understand what that means. Same happens for the next block, and the next, and next… same would probably continue for the remaining 52 million blocks. If I try to use the -y argument to make fsck automatically continue with the ignores and rewrites, it aborts immediately:

$ LANG=C sudo e2fsck -yf /dev/sdc1  
e2fsck 1.45.6 (20-Mar-2020)
The filesystem size (according to the superblock) is 488378368 blocks
The physical size of the device is 435949568 blocks
Either the superblock or the partition table is likely to be corrupt!
Abort? yes

So… how do I run the command?

megavolt · October 19, 2020, 3:04pm

Then please force to do that:

sudo resize2fs -f /dev/sdc1 435949568

Normally resize2fs will enforce you to check the fs first, but because of the failure of fsck, you have to force it to do so. The block size in the superblock and the physical block have to be the same.
But just in case, create a undo file before:

sudo resize2fs -z /home/$USER/sdc1.e2undo /dev/sdc1

Aarni501 · October 19, 2020, 3:43pm

@megavolt thank you, not sure how I managed to miss that. Apparently it still refuses to continue, however.

$ sudo resize2fs -f /dev/sdc1 435949568
resize2fs 1.45.6 (20-Mar-2020)
Resizing the filesystem on /dev/sdc1 to 435949568 (4k) blocks.
resize2fs: Block bitmap checksum does not match bitmap while trying to resize /dev/sdc1
Please run 'e2fsck -fy /dev/sdc1' to fix the filesystem
after the aborted resize operation.

megavolt · October 19, 2020, 4:43pm

@Aarni501 I guess now there is now only one possible solution: Reinitialize superblock and group descriptors without touching the inodes and try to recover the data. That would be the last thing i would advice.

sudo mkfs.ext4 -S /dev/sdc1

then

sudo fsck.ext4 -fy /dev/sdc1

The recovered data will be then in the lost+found folder.

But recover the old partition size would be better using testdisk. That means the physical size of the partition have to be again 488378368 which is now 435949568.

Aarni501 · October 20, 2020, 6:55pm

I was able to recover the partition with @megavolt’s help, but all data in it was lost beyond recovery.

I restored the partition by resizing it with sudo cfdisk /dev/sdc to its original size, then rewrote the superblock with sudo mkfs.ext4 -S /dev/sdc1 and after that ran sudo fsck.ext4 -fy /dev/sdc1.

After this the partition was once again mountable, and any data was moved to lost+found folder. However in my case there was very little data in there and all of it was unusable.

Lesson learned: take multiple backups.