Recently I've been working on a different type of cloud infrastructure with new storage options. One of the changes is to now use remote storage over iSCSI rather than FibreChannel directly attached storage.
On some of our CentOS 5.4 systems, we have the following layers above the iSCSI block devices:
- iSCSI block devices
/dev/sdc and /dev/sdd managed with iscsiadm
- dm-multipath devices
/dev/mapper/mpath1 for /dev/sdc and mpath2 for sdd
- Linux software RAID0 at
/dev/md0 over /dev/mapper/mpath1 and mpath2
- LVM2 VG
vgexample on /dev/md0
On one particular system, the setup had already been running once without the dm-multipath layer and was been working fine. When trying to enable the dm-multipath layer on this system (well, actually, when Puppet ran) it was unable to bring up the software RAID device /dev/md0.
Running the RAID assembly command gave the following output:
# mdadm -A /dev/md0 /dev/mapper/mpath[12]
mdadm: Cannot open /dev/mapper/mpath1: Device or resource busy
mdadm: create aborted
Strangely, the array could be assembled with the second device, but not the first (even though it wouldn't start):
# mdadm -A /dev/md0 /dev/mapper/mpath2
mdadm: /dev/md0 assembled from 1 drive - not enough to start the array.
Running mdadm under strace showed that it was unable to get an exclusive file handle on the device:
44 15176 open("/dev/mapper/mpath1", O_RDONLY|O_EXCL) = -1 EBUSY (Device or resource busy)
45 15176 write(2, "mdadm: cannot open device /dev/m"..., 70) = 70
46 15176 write(2, "mdadm: /dev/mapper/mpath1 has no"..., 63) = 63
47 15176 exit_group(1) = ?
Compared to the above mdadm assemble with the one working device:
44 15180 open("/dev/mapper/mpath2", O_RDONLY|O_EXCL) = 4
45 15180 fstat(4, {st_mode=S_IFBLK|0660, st_rdev=makedev(253, 1), ...}) = 0
46 15180 fstat(4, {st_mode=S_IFBLK|0660, st_rdev=makedev(253, 1), ...}) = 0
It turned out to be caused by a partition table that existed on the second disk (from a previous life where it had been partitioned). This could be seen by looking at the devices generated by dm-multipath and
fdisk -l:
# ls -l /dev/mapper/mpath*
total 0
brw-rw---- 1 root disk 253, 0 Jan 2 17:39 mpath1
brw-rw---- 1 root disk 253, 2 Jan 2 17:39 mpath1p1
brw-rw---- 1 root disk 253, 1 Jan 2 17:39 mpath2
By bad luck, the partition table on this disk was still intact underneath the RAID0 and so it was being read and a device for the partition was created, though multipath itself won't show you this:
# multipath -l
mpath2 (1494554000000000000000000010000006a09000011000000) dm-1 IET,VIRTUAL-DISK
[size=232G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
\_ 6:0:0:0 sdd 8:48 [active][undef]
mpath1 (1494554000000000000000000010000005e09000011000000) dm-0 IET,VIRTUAL-DISK
[size=232G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
\_ 5:0:0:0 sdc 8:32 [active][undef]
If a partition exists, dm-multipath creates a device and blocks exclusive access to the main device. The easiest way to fix this was to wipe the boot sector from the drive and to recreate the RAID on top. It may be possible to wipe the boot sector without recreating the RAID, but I wasn't going to risk this. To wipe the boot sector, simply wipe the first 512 bytes from the drive (at your own risk..):
# dd if=/dev/zero of=/dev/sdc bs=512 count=1