How to perform a fsck (File System Check) on a system disk

Sometimes errors occur on the filesystem of your server's system disk that prevent it from working properly.

In this case, the only solution is to perform a filesystem check to (hopefully) fix the errors.

The fsck on a disk must be done while it is unmounted or in read-only mode. If not, the risk of losing data is very high! Always check the state of the disk before beginning the check and carefully read any warnings produced by the program.

There are two ways of proceeding with the check of the system disk depending on whether you have shell access or not.

If you can access a shell to execute commands:

If you can log in on your server and get to a shell, then you will be able to proceed to the checking on the disk directly from your server.

Switch the disk to Read-Only

First of all, you must switch the system disk into read-only mode with the following command:

# mount -o remount,ro /

If you get the message mount: / is busy in return, it means the disk is in use by some programs. You will have to stop or kill the process who have w/u on /. The command lsof / will display that information, as in the example below:

# lsof /
COMMAND   PID   USER  FD  TYPE DEVICE SIZE/OFF  NODE NAME
rsyslogd  1237  root  3u  REG  202,1  3762377   3544 /var/log/syslog
rsyslogd  1237  root  5w  REG  202,1    29569  25879 /var/log/messages
rsyslogd  1237  root  6w  REG  202,1  2217748  25872 /var/log/auth.log
rsyslogd  1237  root  7w  REG  202,1 36729452  17350 /var/log/daemon.log
rsyslogd  1237  root  9w  REG  202,1   978650  25878 /var/log/user.log
rsyslogd  1237  root 10w  REG  202,1     8108  12417 /var/log/kern.log

In the case above, we can see that that rsyslogd is using the system disk in read-write. Therefore, you just have to stop or kill rsyslogd, as well as all other processes having w/u on /.

Next, check that the disk has been mounted correctly in read-only mode (ro) with the mount command:

# mount
/dev/xvda1 on / type ext4 (ro,noatime,errors=remount-ro)

Proceed to the fsck

To begin the fsck, execute the following command:

# fsck -f /dev/xvda1

The output might look something like this:

fsck from util-linux 2.20.1
e2fsck 1.42.5 (29-Jul-2012)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
DEBIAN7_64: 32869/196608 files (0.2% non-contiguous), 274868/786432 blocks

If the shell is unavailable on your server

If the shell is not available on your server, you will have to check the disk from another Gandi server. Indeed, this is one of the advantages of server virtualization, which allows you to attach a disk to another server.

To create a new server, you must have sufficient resources (at least one share) available in your Gandi account.

When you buy a share, be sure not to select the “Obligation” checkbox. That way, once the filesystem has been repaired, you can delete the new server and release the share for a prorated refund.

Once you get the share, create a new server (in expert mode) using the Linux distribution of your choice. The creation of the new server might take a few minutes.

Once the new server has been created, stop the server with errors on the filesystem. Then detach the system disk and attach it to the newly created server.

Unmount the system disk attached to the new server

Once the system disk is attached to the new server, it should be mounted automatically. You can verify that with the mount command :

# mount
/dev/xvda1 on / type ext4 (rw,noatime,errors=remount-ro)
/dev/xvdk on /srv/DEBIAN7_64 type ext4 (rw,nosuid,nodev,noatime)

In the case above, the label of the system disk to check is 'DEBIAN6_64'. To unmount the disk, execute this command:

# umount /srv/label_of_the_disk

Proceed to the fsck

Once the disk has been unmounted, you can proceed to the check of the system disk with the command fsck -f /dev/xvdX (where X is the disk to check):

# fsck -f /dev/xvdk
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
DEBIAN7_64: 32869/196608 files (0.2% non-contiguous), 274868/786432 blocks
Last modified: 05/28/2013 at 15:02 by Ryan A. (Gandi)