Software Raid
RAID 1 Installation Steps
I ignored the swap completely (you don’t do anything with swap)
CURRENT SITUATION ON TAV:
[root@tav ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_system-lv_root 50G 6.0G 41G 13% / tmpfs 3.9G 0 3.9G 0% /dev/shm /dev/sda1 146M 112M 27M 81% /boot /dev/mapper/vg_data-lv_share 860G 59G 758G 8% /opt @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ [root@tav ~]# cat /etc/fstab /dev/mapper/vg_system-lv_root / ext4 defaults 1 1 UUID=d20b31a8-2cd3-4aee-ab9f-9d6bea4c0647 /boot ext2 defaults 1 2 #/dev/mapper/vg_data-lv_share /srv/share ext4 defaults 1 2 /dev/mapper/vg_data-lv_share /opt ext4 defaults 1 2 UUID=b89df73b-90e2-49e2-bae0-7707d7840111 swap swap defaults 0 0 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[root@tav ~]# fdisk -l Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0002a4e1 Device Boot Start End Blocks Id System /dev/sdb1 * 1 20 153600 83 Linux Partition 1 does not end on cylinder boundary. /dev/sdb2 20 6547 52428800 8e Linux LVM /dev/sdb3 6547 7567 8192000 82 Linux swap / Solaris /dev/sdb4 7567 121602 915987160 5 Extended /dev/sdb5 7567 121602 915985408 8e Linux LVM Disk /dev/sda: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000ece4c Device Boot Start End Blocks Id System /dev/sda1 * 1 20 153600 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 20 6547 52428800 8e Linux LVM /dev/sda3 6547 7567 8192000 82 Linux swap / Solaris /dev/sda4 7567 121602 915987160 5 Extended /dev/sda5 7567 121602 915985408 8e Linux LVM Disk /dev/mapper/vg_system-lv_root: 53.7 GB, 53653536768 bytes 255 heads, 63 sectors/track, 6523 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/mapper/vg_data-lv_share: 937.9 GB, 937947037696 bytes 255 heads, 63 sectors/track, 114032 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ [root@tav ~]# pvdisplay --- Physical volume --- PV Name /dev/sda5 VG Name vg_data PV Size 873.55 GiB / not usable 21.00 MiB Allocatable yes (but full) PE Size 32.00 MiB Total PE 27953 Free PE 0 Allocated PE 27953 PV UUID pcZ4WO-DnMD-WzNn-yXzp-s8u8-19Mf-Ug4vWD
--- Physical volume --- PV Name /dev/sda2 VG Name vg_system PV Size 50.00 GiB / not usable 32.00 MiB Allocatable yes (but full) PE Size 32.00 MiB Total PE 1599 Free PE 0 Allocated PE 1599 PV UUID dP0qZ0-ILST-aEMp-GN3t-ONqP-Y8OG-0fQgKR
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
- The reason why the both drives have the same layout (see sfdisk -l below) is because I've
already copied the partition tables from sda to sdb on tav: sfdisk -d /dev/sda | sfdisk --force /dev/sdb ******
[root@tav ~]# sfdisk -l Disk /dev/sdb: 121601 cylinders, 255 heads, 63 sectors/track Warning: extended partition does not start at a cylinder boundary. DOS and Linux will interpret the contents differently. Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sdb1 * 0+ 19- 20- 153600 83 Linux /dev/sdb2 19+ 6546- 6528- 52428800 8e Linux LVM /dev/sdb3 6546+ 7566- 1020- 8192000 82 Linux swap / Solaris /dev/sdb4 7566+ 121601- 114036- 915987160 5 Extended /dev/sdb5 7566+ 121601- 114035- 915985408 8e Linux LVM Disk /dev/sda: 121601 cylinders, 255 heads, 63 sectors/track Warning: extended partition does not start at a cylinder boundary. DOS and Linux will interpret the contents differently. Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sda1 * 0+ 19- 20- 153600 83 Linux /dev/sda2 19+ 6546- 6528- 52428800 8e Linux LVM /dev/sda3 6546+ 7566- 1020- 8192000 82 Linux swap / Solaris /dev/sda4 7566+ 121601- 114036- 915987160 5 Extended /dev/sda5 7566+ 121601- 114035- 915985408 8e Linux LVM Disk /dev/mapper/vg_system-lv_root: 6523 cylinders, 255 heads, 63 sectors/track Disk /dev/mapper/vg_data-lv_share: 114032 cylinders, 255 heads, 63 sectors/track
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[root@tav ~]# vgdisplay -v Finding all volume groups Finding volume group "vg_data" --- Volume group --- VG Name vg_data System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 2 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 1 Max PV 0 Cur PV 1 Act PV 1 VG Size 873.53 GiB PE Size 32.00 MiB Total PE 27953 Alloc PE / Size 27953 / 873.53 GiB Free PE / Size 0 / 0 VG UUID mWLha7-je5U-9Pjf-wO0y-JgmT-X12l-YF9e0c --- Logical volume --- LV Path /dev/vg_data/lv_share LV Name lv_share VG Name vg_data LV UUID unYF5D-jvdd-LVXe-Ntuz-DbDU-FSE5-1mKJDM LV Write Access read/write LV Creation host, time tav.cluster.ucsf.bkslab.org, 2014-05-01 13:15:10 -0700 LV Status available # open 1 LV Size 873.53 GiB Current LE 27953 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:1 --- Physical volumes --- PV Name /dev/sda5 PV UUID pcZ4WO-DnMD-WzNn-yXzp-s8u8-19Mf-Ug4vWD PV Status allocatable Total PE / Free PE 27953 / 0 Finding volume group "vg_system" --- Volume group --- VG Name vg_system System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 2 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 1 Max PV 0 Cur PV 1 Act PV 1 VG Size 49.97 GiB PE Size 32.00 MiB Total PE 1599 Alloc PE / Size 1599 / 49.97 GiB Free PE / Size 0 / 0 VG UUID B09ehJ-3G3g-8TEB-HFie-vrjW-5bRM-QT3Rlb --- Logical volume --- LV Path /dev/vg_system/lv_root LV Name lv_root VG Name vg_system LV UUID TzNh0J-GWri-u2Or-YSjy-Uu9W-GnJm-X5JRR5 LV Write Access read/write LV Creation host, time tav.cluster.ucsf.bkslab.org, 2014-05-01 13:14:36 -0700 LV Status available # open 1 LV Size 49.97 GiB Current LE 1599 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:0 --- Physical volumes --- PV Name /dev/sda2 PV UUID dP0qZ0-ILST-aEMp-GN3t-ONqP-Y8OG-0fQgKR PV Status allocatable Total PE / Free PE 1599 / 0
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[root@tav ~]# iostat Linux 2.6.32-431.20.3.0.1.el6.centos.plus.x86_64 (tav.ucsf.bkslab.org) 08/08/2014 _x86_64_ (8 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 0.05 0.00 0.03 0.06 0.00 99.85 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sdb 0.00 0.04 0.00 11506 16 sda 0.85 4.47 14.99 1448876 4856642 dm-0 1.28 4.23 9.25 1370754 2996248 dm-1 0.72 0.15 5.74 48178 1860376
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[root@tav ~]# cat /proc/mdstat Personalities : unused devices: <none>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[root@tav ~]# grub-install -v grub-install (GNU GRUB 0.97)
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
SUMMARY: sda1 => boot sda2 => vg_system (or /) sda3 => swap sda4 => extended => Ignoring this partition sda5 => vg_data (or /opt)
THE FOLLOWING IS HOW TO SET UP A RAID 1 ARRAY WITH THE DRIVES SDA AND SDB, WITHOUT LOSING THE DATA ON SDA. # First you have to load these kernel modules to avoid reboot. modprobe linear modprobe raid0 modprobe raid1 # Typically you would do the following step to copy the partition tables, but I already did it: sfdisk -d /dev/sda | sfdisk --force /dev/sdb # The following is to format the system file type for the second drive sdb: fdisk /dev/sdb Command (m for help): t # Set partition type Command (m for help): 1 # Select partition number 1 Command (m for help): fd # Set file system type to Linux RAID type Command (m for help): t # Set partition type Command (m for help): 2 # Select partition number 2 Command (m for help): fd # Set file system type to Linux RAID type Command (m for help): t # Set partition type Command (m for help): 3 # Select partition number 3 Command (m for help): fd # Set file system type to Linux RAID type Command (m for help): t # Set partition type Command (m for help): 5 # Select partition number 5 Command (m for help): fd # Set file system type to Linux RAID type Command (m for help): w # Write the changes and exit frisk fdisk -l # Sanity check
# Creating RAID arrays /dev/md0 through /dev/md3. # /dev/sdb1 will be added to /dev/md0, /dev/sdb2 to /dev/md1, etc. # /dev/sda1, /dev/sda2, etc, can't be added right now because the system is currently running on them. # Therefore we use the placeholder "missing" for sda1-5. # You have to use the "--metadata=0.90" because the grub version is older than grub version 2. mdadm --create /dev/md0 --metadata=0.90 --level=1 --raid-disks=2 missing /dev/sdb1 mdadm --create /dev/md1 --metadata=0.90 --level=1 --raid-disks=2 missing /dev/sdb2 mdadm --create /dev/md2 --metadata=0.90 --level=1 --raid-disks=2 missing /dev/sdb3 mdadm --create /dev/md3 --metadata=0.90 --level=1 --raid-disks=2 missing /dev/sdb5 # Create the filesystems for the newly created RAID arrays: # md0 will be the boot partition, so ext2. mkfs.ext2 /dev/md0 # md1 will contain the vg_system, so lvm. pvcreate /dev/md1 vgextend vg_system /dev/md1 # md2 will contain the swap space. mkswap /dev/md2 # md4 will contain vg_data, so lvm. pvcreate /dev/md3 vgextend vg_data /dev/md3 mdadm --examine --scan > /etc/mdadm.conf cat /etc/mdadm.conf # Sanity Check vim /etc/fstab => Replace LABEL=/boot with /dev/md0 vim /etc/mtab => Replace /dev/sda1 with /dev/md0 vim /boot/grub/menu.lst add "fallback=1" right underneath "default=0" go to the bottom where you should find some kernel stanzas. Copy the first of them and paste the stanza before the first existing stanza; replace root (hd0,0) with root (hd1,0) Delete "rd_NO_MD" if it exists, so that md modules will be loaded at boot time. It should look something like this: [...] title CentOS (2.6.18-128.el5) root (hd1,0) kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/VolGroup00/LogVol00 initrd /initrd-2.6.18-128.el5.img title CentOS (2.6.18-128.el5) root (hd0,0) kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/VolGroup00/LogVol00 initrd /initrd-2.6.18-128.el5.img mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak mkinitrd /boot/initramfs-$(uname -r).img $(uname -r) # Moving the vg_system to RAID array md1 pvmove /dev/sda2 /dev/md1 vgreduce vg_system /dev/sda2 pvremove /dev/sda2 pvdisplay # Sanity check # Moving the vg_data to RAID array md3 pvmove /dev/sda5 /dev/md3 vgreduce vg_data /dev/sda5 pvremove /dev/sda5 pvdisplay # Sanity check swapoff /dev/sda3 # Deactivate previous swap space # Convert all the sda* partitions to Linux RAID types, except for boot partition (sda1) fdisk /dev/sda Command (m for help): t # Set partition type Command (m for help): 2 # Select partition number 2 Command (m for help): fd # Set file system type to Linux RAID type Command (m for help): t # Set partition type Command (m for help): 3 # Select partition number 3 Command (m for help): fd # Set file system type to Linux RAID type Command (m for help): t # Set partition type Command (m for help): 5 # Select partition number 5 Command (m for help): fd # Set file system type to Linux RAID type Command (m for help): w # Write the changes and exit frisk # Add the sda2-5 to the RAID mdadm --add /dev/md1 /dev/sda2 mdadm --add /dev/md2 /dev/sda3 mdadm --add /dev/md3 /dev/sda5 watch cat /proc/mdstat swapon /dev/md2 mkdir /mnt/raid mount /dev/md0 /mnt/raid cd /boot cp -dpRx . /mnt/raid # Copy the boot directory to the RAID array grub # Set the root device temporarily to the second disk: grub> root (hd0,0) grub> setup (hd0) # Repeat for the first disk: grub> root (hd1,0) grub> setup (hd1) grub> quit reboot # If all is well, format boot partition, sda1: fdisk /dev/sda Command (m for help): t # Set partition type Command (m for help): 1 # Select partition number 1 Command (m for help): fd # Set file system type to Linux RAID type Command (m for help): w # Write the changes and exit frisk # Add sda1 to the RAID array mdadm --add /dev/md0 /dev/sda1 cat /proc/mdstat mdadm --examine --scan > /etc/mdadm.conf cat /etc/mdadm.conf reboot
That should be it!
Debugging Raid
http://www.tldp.org/HOWTO/Software-RAID-HOWTO-6.html
http://www.techrepublic.com/blog/data-center/testing-your-software-raid-be-prepared/
About the md126 md127 thing how to fix it:
mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak1 mkinitrd /boot/initramfs-$(uname -r).img $(uname -r)
When I ran into problems with multiple pv’s (physical volumes) found, I did this:
mount -o remount / (If the root partition is on a lvm) rm /etc/lvm/cache/.cache vgscan pvdisplay => to make sure it’s right init 6
To see the version of grub
grub-install -v
References
These three were the best:
http://www.howtoforge.com/how-to-create-a-raid1-setup-on-an-existing-centos-redhat-6.0-system-p2
http://carlos-spitzer.com/2013/07/29/how-to-create-a-raid1-on-an-existing-centos-rhel-6-os-with-lvm/
The rest that I used:
http://community.spiceworks.com/how_to/show/340-lvm-single-drive-to-lvm-raid-1-mirror-migration
https://raid.wiki.kernel.org/index.php/RAID_setup
http://unix.stackexchange.com/questions/63928/can-i-create-a-software-raid-1-with-one-device
http://www.debian-administration.org/articles/238
http://feeding.cloud.geek.nz/posts/setting-up-raid-on-existing/
http://feeding.cloud.geek.nz/posts/setting-up-raid-on-existing/