Ubuntu: Fixing a Readonly Filesystem.

Just in case anyone else does this or has a similar problem here’s how I managed to end up with a readonly filesystem and how I (eventually) fixed it.

Basically I fouled up a backup and as a result my hard-drive ended up as read-only. This incidentally turned out to be (I think) a rather cunning defence mechanism mounted by Linux to protect my file-system from any further stupidity before I fixed what I had done.

I was actually being a good boy for once. I had cloned my hard drive onto a USB attached HDD using Clonezilla.

 

Then <just to check> the USB drive was OK (here is the stupid bit) I tried booting from the USB drive without detaching the original HD first.

As both of main filesystem partitions on each disk (sda5 – main, sdb5 – USB) had the same UUID, Linux got upset. It decided (quite sensibly) to remove the mount for the sda5 from the fstab file in sda5 and then boot up with a readonly filesystem.

I realised all was not well about ½ way through booting the USB clone. I aborted the boot. But by then the damage was done.

When I tried booting the laptop without the USB drive attached it still managed to get up to <I think> run level 3 but the file-system was mounted read only. So I was left at the command prompt with a read only file system. After the despair at my own stupidity ebbed away this is what I did to fix the problem.

First “fix” the file system, (just in case).

NOTE: In my case the fstab was NOT corrupted – just missing information (deleted by the previous aborted duff boot) so this file system check/fix may have been ineffectual.

But if the file system has been mounted read-only because there is some other corruption to it then this is definitely worth doing. So it is most certainly not a waste of time.

To do this file system check and fix, I booted from the Clonezilla DVD. Any live Linux medium would have done. Then I dropped into a command prompt from the menu.

I went to superuser with:

sudo su -

Now I was running as root I ran a file system check/fix with e2fsck on /dev/sda5 (if you are doing this then your primary drive and relevant partition may be different)

e2fsck -f -p -C 0 /dev/sda5

Flags are:

-f       force check even if it thinks its OK at start.
-p      “gently” fix any errors
-C 0    show a status bar while its running

After doing this the system still showed the same problem. (why wouldn’t it? – I now know that the fstab was not corrupted. Just incorrect!)

I quit the Clonezilla live system and rebooted. I ended up at a command prompt on a read only system. But at least I knew the file-system was pristine.

I found the problem with the fstab file by inspection. It wouldn’t boot as rw so something must be stopping it. It seemed reasonable that fstab was somehow involved. That’s why I looked. And there it was (or rather wasn’t) the missing line required to boot from sda5 using a UUID.

Now I needed to fix the fstab file that had lost the line that told the system which was the primary filesystem partition and drive. First go to root.

sudo su -

then remount the filesystem as rw

mount -o remount,rw /

Then check out the fstab

less /etc/fstab

this is what it should have been.

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda5 during installation
UUID=********-****-****-****-*******caca0 / ext4 errors=remount-ro 0 1
# swap was on /dev/sda6 during installation
UUID=********-****-****-****-*******87a2a none  swap sw   0       0

But the line defining the entry for sda5 (the line with the UUID ending “caca0”) was missing. I asume that Linux must have erased this somehow when confronted with the contentious boot from two drives with the same UUID. (NOTE: The UUIDs here have been overwritten with “*”s out of security paranioa.)

This is what my broken fstab looked like

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda5 during installation
# swap was on /dev/sda6 during installation
UUID=********-****-****-****-*******87a2a none  swap sw   0        0

Then when subsequently booting properly with only the /dev/sda5 connected the necessary definition for sda5 in fstab was missing.

It figures that this as far as the system is concerned is a significant error. So while the system could see /dev/sda5 (as directed to by grub on boot) it did not know that it was what it should see. That is why (I believe) I ended up with a readonly filesystem even after the cloned USB drive was removed.

As by this time I was totally paranoid so I backed up the duff fstab

cp fstab fstab.broke

To rebuild the missing line I needed to re-find the lost UUID of the sda5 partition. I used blkid as so:

blkid /dev/sda5

I noted the UUID and copied it down carefully onto a piece of paper then edited fstab.

At this point there is no gui so use vi or nano.

cd /etc
vi fstab

Edit the file so it looks like the good one displayed above (i.e. add the missing line)

Obviously CHANGE the UUID and partition to whatever you are using.

Save and reboot

Mine worked as if nothing had ever been wrong with it.

My stress levels fell.

I hope some of that above might help someone else.

But whatever always remember:

Never, never NEVER try booting off a cloned HDD before you have disconnected the original! Otherwise you’ll end up with a readonly filesystem