Thursday, 4 June 2015

RAID-1 Degraded - missing device

I have got two hard disks, both the same type/brand. They are two USB2.0 connected WesternDigital drives that work pretty well. I've setup a software-based RAID1 system using them both, by means of the MDADM Linux program. I find it works very well.

Thankfully I noticed something odd about the hard disks I used. It is probably due to Partial RAID1 mounting that I tried recently that I screwed something up.

The raid1 I setup contains two disks, as visible in the lsscsi command:
root@micemouse:~# lsscsi
[0:0:0:0]    disk    ATA      WDC WD5000AAKS-0 01.0  /dev/sda 
[4:0:1:0]    cd/dvd  SAMSUNG  DVD-ROM SDR-430  1.06  /dev/sr0 
[10:0:0:0]   disk    WD       Ext HDD 1021     2021  /dev/sdb 
[11:0:0:0]   disk    WD       Ext HDD 1021     2021  /dev/sdc
As you can see in the printout below, it is set to "Degraded", and we're missing one of the devices.
root@micemouse:~# mdadm --detail /dev/md127
/dev/md127:
        Version : 1.2
  Creation Time : Wed Mar  6 22:16:05 2013
     Raid Level : raid1
     Array Size : 1953380160 (1862.89 GiB 2000.26 GB)
  Used Dev Size : 1953380160 (1862.89 GiB 2000.26 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Sat May 16 15:50:47 2015
          State : clean, degraded 
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : micemouse:0  (local to host micemouse)
           UUID : ed4531c4:59c132b2:a6bfc3d1:6da3b928
         Events : 3748

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       33        1      active sync   /dev/sdc1
We are missing /dev/sdb1.

Rebuilding the array

Let's re-add the missing device.
root@micemouse:~# mdadm --remove /dev/md127 /dev/sdb1
mdadm: hot remove failed for /dev/sdb1: No such device or address
root@micemouse:~# mdadm --add /dev/md127 /dev/sdb1
mdadm: added /dev/sdb1
The array is rebuilding itself.
root@micemouse:~# mdadm -D /dev/md127
/dev/md127:
        Version : 1.2
  Creation Time : Wed Mar  6 22:16:05 2013
     Raid Level : raid1
     Array Size : 1953380160 (1862.89 GiB 2000.26 GB)
  Used Dev Size : 1953380160 (1862.89 GiB 2000.26 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Sat May 16 15:54:20 2015
          State : clean, degraded, recovering 
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1

 Rebuild Status : 0% complete

           Name : micemouse:0  (local to host micemouse)
           UUID : ed4531c4:59c132b2:a6bfc3d1:6da3b928
         Events : 3750

    Number   Major   Minor   RaidDevice State
       2       8       17        0      spare rebuilding   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
Trying out "watch -n 2 cat /proc/mdstat" to check on its progress (every two seconds in this case).
Every 2.0s: cat /proc/mdstat                            Sat May 16 16:00:38 2015

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [ra
id10]
md127 : active raid1 sdb1[2] sdc1[1]
      1953380160 blocks super 1.2 [2/1] [_U]
      [>....................]  recovery =  0.2% (5763072/1953380160) finish=6519
.5min speed=4978K/sec

unused devices: <none>
Recovery in progress, but it looks a long long way away yet.

Much later...

After a very long time, /proc/mdstat mentioned that things were finished:
Every 60.0s: cat /proc/mdstat                           Sun May 17 22:23:29 2015

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [ra
id10]
md127 : active raid1 sdb1[2] sdc1[1]
      1953380160 blocks super 1.2 [2/2] [UU]

unused devices: <none>
And a closer look at the raid device, shows:
root@micemouse:~# mdadm -D /dev/md127
/dev/md127:
        Version : 1.2
  Creation Time : Wed Mar  6 22:16:05 2013
     Raid Level : raid1
     Array Size : 1953380160 (1862.89 GiB 2000.26 GB)
  Used Dev Size : 1953380160 (1862.89 GiB 2000.26 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Sun May 17 20:46:50 2015
          State : clean 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : micemouse:0  (local to host micemouse)
           UUID : ed4531c4:59c132b2:a6bfc3d1:6da3b928
         Events : 5216

    Number   Major   Minor   RaidDevice State
       2       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
The important part being "State: clean".

References

RAID1 - State : clean, degraded
https://www.howtoforge.com/community/threads/raid1-state-clean-degraded.66744/
Mdadm - recover degraded Array
https://www.thomas-krenn.com/en/wiki/Mdadm_recover_degraded_Array

Thursday, 28 May 2015

Partial RAID-1 mounting

I recently (read: ages ago) needed to bring my RAID1 disks to a friend, and it seemed silly for a mirrored disk array to bring more than 1 harddisk. Surely a single harddisk would cover it? Mount it read-only and away we go.

Let's take /dev/sdb1 drive of an array {/dev/sdb1, /dev/sdc1}.

The simplest way is to just start up the raid array using mdadm with a single disk2. But for some reason I failed to get this to work.

One way to do it (according to [1]), is to assume that the raidarray uses ext4 as an underlying filesystem and then just fiddle with the offsets.

The command "mdadm --examine /dev/sdb1" shows "Data Offset : 262144 sectors".

Then create a loop device, starting from that offset multiplied by the mdadm sector size (512).
losetup --find --show --read-only --offset $((262144*512)) /dev/sdb1
Than, simply mount the loop device.
mount /dev/loop0 /mnt/raid1/

References

[1] StackExchange - How to mount/recover data on a disk that was part of a mdadm raid 1 on another machine?
http://unix.stackexchange.com/questions/64889/how-to-mount-recover-data-on-a-disk-that-was-part-of-a-mdadm-raid-1-on-another-m
[2] sleeplessbeastie's notes - How to mount software RAID1 member using mdadm
https://blog.sleeplessbeastie.eu/2012/05/08/how-to-mount-software-raid1-member-using-mdadm/