mdadm: growing a 3-disk RAID-0 array to a 4-disk RAID-5 one

As lots of other people, I strongly believe that a proper backup policy is a mandatory key-factor in every scenario.

Unfortunately –again, as lots of other people– I also know that a proper backup policy is hard to implement, expecially when involving serious budget constraints.

To be short: for one of my customers, I setup an HP-Microserver Gen-8 as a CentOS 7 box acting:

  • as an NFS server towards both a XEN-Server 6.2 and ESXi 5.5, to receive proper snapshots of running VMs (actually, thanks to two great pieces of open-source softwares: snapback and ghettoVCB) ;
  • as a BareOS server, relying on local storage for common/standard client-server backups.

As you might guess, I needed as much local storage as possible so…. I choosed to buy the largest HDD currently available: 8TB. I decided to buy them on Amazon as they were really cheap (0.027€ per GB!) but unfortunately you can buy only three HDD per order. So I said to myself: “Ok! I’m going to buy the first three and setup everything as a 24TB RAID-0 array. Then I’ll buy the 4th and will simply change the RAID-0 array to a RAID-5 one, while keeping everything intact: no reinstallation!“. At least, that was what I tought!

Here is the RAID-0 initially built:

that actually is also well used:

and here are the details of the first three 8TB HDD:

(Note: sdb and sdc are exactly the same with the only exception of the Serial Number)

After a couple of weeks, I ordered and received the fourth HDD so it was time to play some nice exercise. My two-item wondering-list was:

  1. I’ve a free slot in my MicroServer: even if it is explicitely reported to not include an hot-plug HDD controller, will it be able to recognize the fourth disk, without a reboot?
  2. Will I be able to reshape the currently-perfectly-working RAID-0 array to a RAID-5 one, without any reboot/recreate/annoiance?

The reason underlying item 1) was simply that I’d like NOT to interrupt BareOS services. Even if it was not running any backup (at such a time) I said to myself: “Ok! But what happens if backup jobs are running and you don’t want to stop them?“. So I simply pulled the HDD-guide of the empty fourth bay, properly screwed the new HDD and…. pushed it inside the microserver.

Good news! No kernel-panic; no burning flames; no problem at all! On my remote-console (I had a tail -f system.log  running on my SSH remote session) I saw:

So 1) item was solved: even if it is dangerous, an additional HDD S-ATA disk can be added to a running system even if it has no explicit hot-swap capabilities.

After cloning the partition table (GPT-based) from sda to sdd (with a sgdisk --replicate=/dev/sdd /dev/sda ) and ensure that new partitions were properly recognized by the running system (with a partprobe ), I ended up with this partition layout:

so, sdd5 were ready to be included into the RAID-Array, where such an array needed to be reshaped from the running RAID-0 to the new RAID-5.

After some (actually… not very useful/effective) on-line research, I decided to go with this command:

Very good news! It seems it worked!

Obviously I perfectly understand that in order to reshape a 24TB existing RAID-0 array to a RAID-5 format, some time (a LOT of time) is needed… So output below was no surprise, after more than 30 minutes of reshaping progress:

Just to further check that everything is properly in progress, I checked the details with:

As for the reshaping activity, it’s interesting to note that it’s (correctly…) involving:

  • existing three drives (sda, sdb, sdc) with READ activity;
  • the new drive (sdd) with WRITE activity;
  • the amount of WRITE is, more or less, the sum of the whole READs

It can be clearly seen with a iostat -d -x 1 100 like this:

That’s all!

Hope this will be useful for someone 😉

Leave a Reply

Your email address will not be published. Required fields are marked *