mdadm: hot add failed for /dev/sdb1: No space left on device

Asked by: milen
Published on: 14:47/13.01.2009
Status: OPEN

My system is RAID 1 with two hard drives each with two partitions. One of my hard drives failed and I have replaced it. Then I created the partitions to be with the same size like they are on the other disk. But when I tried to rebuild the mirror with

mdadm /dev/md0 -a /dev/sdb1

I received the following error:

mdadm: hot add failed for /dev/sdb1: No space left on device

What I've made wrong? I've done the same steps the other partition and there was no problem? How to fix this issue?

Answer by: xpert
Posted on: 13:43/13.01.2009
Rating: 3 from possible 5 with 2 votes

Even you created the partitions the same way as on /dev/sda with fdisk the size of partitions can differ from each other. First of all you should dump the disks with "fdisk -l" and observe the result:

> fdisk -l
Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1         262     2104514+  fd  Linux raid autodetect
/dev/sda2             263       60801   486279517+  fd  Linux raid autodetect

Disk /dev/sdb: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1         262     2104483+  fd  Linux raid autodetect
/dev/sdb2             263       60801   486279517+  fd  Linux raid autodetect

Please note that I give example with two disks with same geometry and size.
Note that sda1 size(2104514+) > sdb1 size(2104483+) which is like your problem.
In your case sda1 and sdb1 may have exactly the same start and end blocks but the size is different which is your case in fact.

That is why you receive this error "No space left". You should create partitions on /dev/sdb like this:

sfdisk -d /dev/sda | sfdisk /dev/sdb

and the result:

> sfdisk -d /dev/sda | sfdisk /dev/sdb
Checking that no-one is using this disk right now ...
OK

Disk /dev/sdb: 60801 cylinders, 255 heads, 63 sectors/track
Old situation:
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdb1          0       -       0          0    0  Empty
/dev/sdb2          0       -       0          0    0  Empty
/dev/sdb3          0       -       0          0    0  Empty
/dev/sdb4          0       -       0          0    0  Empty
New situation:
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/sdb1             1   4209029    4209029  fd  Linux raid autodetect
/dev/sdb2       4209030 976768064  972559035  fd  Linux raid autodetect
/dev/sdb3             0         -          0   0  Empty
/dev/sdb4             0         -          0   0  Empty
Warning: no primary partition is marked bootable (active)
This does not matter for LILO, but the DOS MBR will not boot this disk.
Successfully wrote the new partition table

Re-reading the partition table ...

If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)

And the above command will copy partitions from /dev/sda to /dev/sdb.

Then you can run again fdisk to list your partitions. Result should be like this:

> fdisk -l
...
Disk /dev/sda: 500.1 GB, 500107862016 bytes
...
/dev/sda1               1         262     2104514+  fd  Linux raid autodetect
/dev/sda2             263       60801   486279517+  fd  Linux raid autodetect

Disk /dev/sdb: 500.1 GB, 500107862016 bytes
...
/dev/sdb1               1         262     2104514+  fd  Linux raid autodetect
/dev/sdb2             263       60801   486279517+  fd  Linux raid autodetect
...

Now sda1 size(2104514+) is equal to sdb1 size(2104514+).
We can add the devices to the raid.

Answer by: xpert
Posted on: 13:47/13.01.2009
Rating: 3.7 from possible 5 with 3 votes

Now we add the devices like this:

> mdadm /dev/md0 -a /dev/sdb1
mdadm: added /dev/sdb1

Then watch the status of raid :

> mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90.03
  Creation Time : Thu Nov  6 14:44:55 2008
     Raid Level : raid1
     Array Size : 2104448 (2.01 GiB 2.15 GB)
    Device Size : 2104448 (2.01 GiB 2.15 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Mon Jan 12 16:24:42 2009
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : 4e7c5519:a88c8efb:0474a969:0541f941
         Events : 0.16

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1

Then you add the second partition:

> mdadm /dev/md1 -a /dev/sdb2
mdadm: re-added /dev/sdb2

and watch the status:

> mdadm --detail /dev/md1
/dev/md1:
        Version : 00.90.03
  Creation Time : Thu Nov  6 14:44:56 2008
     Raid Level : raid1
     Array Size : 486279424 (463.75 GiB 497.95 GB)
    Device Size : 486279424 (463.75 GiB 497.95 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Mon Jan 12 16:28:35 2009
          State : active, degraded, recovering
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1

 Rebuild Status : 1% complete

           UUID : 2b59d029:528fc9e3:1c106055:c8e88fd1
         Events : 0.502189

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       2       8       18        1      spare rebuilding   /dev/sdb2

And now you finish the work with copying the boot sector from your healthy drive to the new one. It is still not bootable. You should make it bootable if drive A fails the drive B should be able to boot the system, so we copy MBR from drive A to drive B.

dd if=/dev/sda of=/dev/sdb bs=512 count=1

But note you are only allowed to copy MBR if you have exactly the same partitions and same size harddrives.
Thats all.