Software Raid
Jump to navigation
Jump to search
RAID 1 Installation Steps
I ignored the swap completely (you don’t do anything with swap)
CURRENT SITUATION ON TAV:
[root@tav ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_system-lv_root
50G 6.0G 41G 13% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda1 146M 112M 27M 81% /boot
/dev/mapper/vg_data-lv_share
860G 59G 758G 8% /opt
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[root@tav ~]# cat /etc/fstab
/dev/mapper/vg_system-lv_root / ext4 defaults 1 1
UUID=d20b31a8-2cd3-4aee-ab9f-9d6bea4c0647 /boot ext2 defaults 1 2
#/dev/mapper/vg_data-lv_share /srv/share ext4 defaults 1 2
/dev/mapper/vg_data-lv_share /opt ext4 defaults 1 2
UUID=b89df73b-90e2-49e2-bae0-7707d7840111 swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[root@tav ~]# fdisk -l
Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0002a4e1
Device Boot Start End Blocks Id System
/dev/sdb1 * 1 20 153600 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sdb2 20 6547 52428800 8e Linux LVM
/dev/sdb3 6547 7567 8192000 82 Linux swap / Solaris
/dev/sdb4 7567 121602 915987160 5 Extended
/dev/sdb5 7567 121602 915985408 8e Linux LVM
Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000ece4c
Device Boot Start End Blocks Id System
/dev/sda1 * 1 20 153600 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 20 6547 52428800 8e Linux LVM
/dev/sda3 6547 7567 8192000 82 Linux swap / Solaris
/dev/sda4 7567 121602 915987160 5 Extended
/dev/sda5 7567 121602 915985408 8e Linux LVM
Disk /dev/mapper/vg_system-lv_root: 53.7 GB, 53653536768 bytes
255 heads, 63 sectors/track, 6523 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/mapper/vg_data-lv_share: 937.9 GB, 937947037696 bytes
255 heads, 63 sectors/track, 114032 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[root@tav ~]# pvdisplay
--- Physical volume ---
PV Name /dev/sda5
VG Name vg_data
PV Size 873.55 GiB / not usable 21.00 MiB
Allocatable yes (but full)
PE Size 32.00 MiB
Total PE 27953
Free PE 0
Allocated PE 27953
PV UUID pcZ4WO-DnMD-WzNn-yXzp-s8u8-19Mf-Ug4vWD
--- Physical volume ---
PV Name /dev/sda2
VG Name vg_system
PV Size 50.00 GiB / not usable 32.00 MiB
Allocatable yes (but full)
PE Size 32.00 MiB
Total PE 1599
Free PE 0
Allocated PE 1599
PV UUID dP0qZ0-ILST-aEMp-GN3t-ONqP-Y8OG-0fQgKR
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
- The reason why the both drives have the same layout (see sfdisk -l below) is because I've
already copied the partition tables from sda to sdb on tav: sfdisk -d /dev/sda | sfdisk --force /dev/sdb ******
[root@tav ~]# sfdisk -l Disk /dev/sdb: 121601 cylinders, 255 heads, 63 sectors/track Warning: extended partition does not start at a cylinder boundary. DOS and Linux will interpret the contents differently. Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sdb1 * 0+ 19- 20- 153600 83 Linux /dev/sdb2 19+ 6546- 6528- 52428800 8e Linux LVM /dev/sdb3 6546+ 7566- 1020- 8192000 82 Linux swap / Solaris /dev/sdb4 7566+ 121601- 114036- 915987160 5 Extended /dev/sdb5 7566+ 121601- 114035- 915985408 8e Linux LVM Disk /dev/sda: 121601 cylinders, 255 heads, 63 sectors/track Warning: extended partition does not start at a cylinder boundary. DOS and Linux will interpret the contents differently. Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sda1 * 0+ 19- 20- 153600 83 Linux /dev/sda2 19+ 6546- 6528- 52428800 8e Linux LVM /dev/sda3 6546+ 7566- 1020- 8192000 82 Linux swap / Solaris /dev/sda4 7566+ 121601- 114036- 915987160 5 Extended /dev/sda5 7566+ 121601- 114035- 915985408 8e Linux LVM Disk /dev/mapper/vg_system-lv_root: 6523 cylinders, 255 heads, 63 sectors/track Disk /dev/mapper/vg_data-lv_share: 114032 cylinders, 255 heads, 63 sectors/track
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[root@tav ~]# vgdisplay -v Finding all volume groups Finding volume group "vg_data" --- Volume group --- VG Name vg_data System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 2 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 1 Max PV 0 Cur PV 1 Act PV 1 VG Size 873.53 GiB PE Size 32.00 MiB Total PE 27953 Alloc PE / Size 27953 / 873.53 GiB Free PE / Size 0 / 0 VG UUID mWLha7-je5U-9Pjf-wO0y-JgmT-X12l-YF9e0c --- Logical volume --- LV Path /dev/vg_data/lv_share LV Name lv_share VG Name vg_data LV UUID unYF5D-jvdd-LVXe-Ntuz-DbDU-FSE5-1mKJDM LV Write Access read/write LV Creation host, time tav.cluster.ucsf.bkslab.org, 2014-05-01 13:15:10 -0700 LV Status available # open 1 LV Size 873.53 GiB Current LE 27953 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:1 --- Physical volumes --- PV Name /dev/sda5 PV UUID pcZ4WO-DnMD-WzNn-yXzp-s8u8-19Mf-Ug4vWD PV Status allocatable Total PE / Free PE 27953 / 0 Finding volume group "vg_system" --- Volume group --- VG Name vg_system System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 2 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 1 Max PV 0 Cur PV 1 Act PV 1 VG Size 49.97 GiB PE Size 32.00 MiB Total PE 1599 Alloc PE / Size 1599 / 49.97 GiB Free PE / Size 0 / 0 VG UUID B09ehJ-3G3g-8TEB-HFie-vrjW-5bRM-QT3Rlb --- Logical volume --- LV Path /dev/vg_system/lv_root LV Name lv_root VG Name vg_system LV UUID TzNh0J-GWri-u2Or-YSjy-Uu9W-GnJm-X5JRR5 LV Write Access read/write LV Creation host, time tav.cluster.ucsf.bkslab.org, 2014-05-01 13:14:36 -0700 LV Status available # open 1 LV Size 49.97 GiB Current LE 1599 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:0 --- Physical volumes --- PV Name /dev/sda2 PV UUID dP0qZ0-ILST-aEMp-GN3t-ONqP-Y8OG-0fQgKR PV Status allocatable Total PE / Free PE 1599 / 0
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[root@tav ~]# iostat
Linux 2.6.32-431.20.3.0.1.el6.centos.plus.x86_64 (tav.ucsf.bkslab.org) 08/08/2014 _x86_64_ (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.05 0.00 0.03 0.06 0.00 99.85
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sdb 0.00 0.04 0.00 11506 16
sda 0.85 4.47 14.99 1448876 4856642
dm-0 1.28 4.23 9.25 1370754 2996248
dm-1 0.72 0.15 5.74 48178 1860376
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[root@tav ~]# cat /proc/mdstat Personalities : unused devices: <none>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[root@tav ~]# grub-install -v grub-install (GNU GRUB 0.97)
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
SUMMARY: sda1 => boot sda2 => vg_system (or /) sda3 => swap sda4 => extended => Ignoring this partition sda5 => vg_data (or /opt)
THE FOLLOWING IS HOW TO SET UP A RAID 1 ARRAY WITH THE DRIVES SDA AND SDB, WITHOUT LOSING THE DATA ON SDA. # First you have to load these kernel modules to avoid reboot. modprobe linear modprobe raid0 modprobe raid1 # Typically you would do the following step to copy the partition tables, but I already did it: sfdisk -d /dev/sda | sfdisk --force /dev/sdb # The following is to format the system file type for the second drive sdb: fdisk /dev/sdb Command (m for help): t # Set partition type Command (m for help): 1 # Select partition number 1 Command (m for help): fd # Set file system type to Linux RAID type Command (m for help): t # Set partition type Command (m for help): 2 # Select partition number 2 Command (m for help): fd # Set file system type to Linux RAID type Command (m for help): t # Set partition type Command (m for help): 3 # Select partition number 3 Command (m for help): fd # Set file system type to Linux RAID type Command (m for help): t # Set partition type Command (m for help): 5 # Select partition number 5 Command (m for help): fd # Set file system type to Linux RAID type Command (m for help): w # Write the changes and exit frisk fdisk -l # Sanity check
# Creating RAID arrays /dev/md0 through /dev/md3.
# /dev/sdb1 will be added to /dev/md0, /dev/sdb2 to /dev/md1, etc.
# /dev/sda1, /dev/sda2, etc, can't be added right now because the system is currently running on them.
# Therefore we use the placeholder "missing" for sda1-5.
# You have to use the "--metadata=0.90" because the grub version is older than grub version 2.
mdadm --create /dev/md0 --metadata=0.90 --level=1 --raid-disks=2 missing /dev/sdb1
mdadm --create /dev/md1 --metadata=0.90 --level=1 --raid-disks=2 missing /dev/sdb2
mdadm --create /dev/md2 --metadata=0.90 --level=1 --raid-disks=2 missing /dev/sdb3
mdadm --create /dev/md3 --metadata=0.90 --level=1 --raid-disks=2 missing /dev/sdb5
# Create the filesystems for the newly created RAID arrays:
# md0 will be the boot partition, so ext2.
mkfs.ext2 /dev/md0
# md1 will contain the vg_system, so lvm.
pvcreate /dev/md1
vgextend vg_system /dev/md1
# md2 will contain the swap space.
mkswap /dev/md2
# md4 will contain vg_data, so lvm.
pvcreate /dev/md3
vgextend vg_data /dev/md3
mdadm --examine --scan > /etc/mdadm.conf
cat /etc/mdadm.conf # Sanity Check
vim /etc/fstab => Replace LABEL=/boot with /dev/md0
vim /etc/mtab => Replace /dev/sda1 with /dev/md0
vim /boot/grub/menu.lst
add "fallback=1" right underneath "default=0"
go to the bottom where you should find some kernel stanzas. Copy the first of them and paste the stanza before the first existing stanza; replace root (hd0,0) with root (hd1,0)
Delete "rd_NO_MD" if it exists, so that md modules will be loaded at boot time.
It should look something like this:
[...]
title CentOS (2.6.18-128.el5)
root (hd1,0)
kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/VolGroup00/LogVol00
initrd /initrd-2.6.18-128.el5.img
title CentOS (2.6.18-128.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/VolGroup00/LogVol00
initrd /initrd-2.6.18-128.el5.img
mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak
mkinitrd /boot/initramfs-$(uname -r).img $(uname -r)
# Moving the vg_system to RAID array md1
pvmove /dev/sda2 /dev/md1
vgreduce vg_system /dev/sda2
pvremove /dev/sda2
pvdisplay # Sanity check
# Moving the vg_data to RAID array md3
pvmove /dev/sda5 /dev/md3
vgreduce vg_data /dev/sda5
pvremove /dev/sda5
pvdisplay # Sanity check
swapoff /dev/sda3 # Deactivate previous swap space
# Convert all the sda* partitions to Linux RAID types, except for boot partition (sda1)
fdisk /dev/sda
Command (m for help): t # Set partition type
Command (m for help): 2 # Select partition number 2
Command (m for help): fd # Set file system type to Linux RAID type
Command (m for help): t # Set partition type
Command (m for help): 3 # Select partition number 3
Command (m for help): fd # Set file system type to Linux RAID type
Command (m for help): t # Set partition type
Command (m for help): 5 # Select partition number 5
Command (m for help): fd # Set file system type to Linux RAID type
Command (m for help): w # Write the changes and exit frisk
# Add the sda2-5 to the RAID
mdadm --add /dev/md1 /dev/sda2
mdadm --add /dev/md2 /dev/sda3
mdadm --add /dev/md3 /dev/sda5
watch cat /proc/mdstat
swapon /dev/md2
mkdir /mnt/raid
mount /dev/md0 /mnt/raid
cd /boot
cp -dpRx . /mnt/raid # Copy the boot directory to the RAID array
grub
# Set the root device temporarily to the second disk:
grub> root (hd0,0)
grub> setup (hd0)
# Repeat for the first disk:
grub> root (hd1,0)
grub> setup (hd1)
grub> quit
reboot
# If all is well, format boot partition, sda1:
fdisk /dev/sda
Command (m for help): t # Set partition type
Command (m for help): 1 # Select partition number 1
Command (m for help): fd # Set file system type to Linux RAID type
Command (m for help): w # Write the changes and exit frisk
# Add sda1 to the RAID array
mdadm --add /dev/md0 /dev/sda1
cat /proc/mdstat
mdadm --examine --scan > /etc/mdadm.conf
cat /etc/mdadm.conf
reboot
That should be it!
Debugging Raid
http://www.tldp.org/HOWTO/Software-RAID-HOWTO-6.html
http://www.techrepublic.com/blog/data-center/testing-your-software-raid-be-prepared/
About the md126 md127 thing how to fix it:
mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak1 mkinitrd /boot/initramfs-$(uname -r).img $(uname -r)
When I ran into problems with multiple pv’s (physical volumes) found, I did this:
mount -o remount / (If the root partition is on a lvm) rm /etc/lvm/cache/.cache vgscan pvdisplay => to make sure it’s right init 6
To see the version of grub
grub-install -v
References
These three were the best:
http://www.howtoforge.com/how-to-create-a-raid1-setup-on-an-existing-centos-redhat-6.0-system-p2
http://carlos-spitzer.com/2013/07/29/how-to-create-a-raid1-on-an-existing-centos-rhel-6-os-with-lvm/
The rest that I used:
http://community.spiceworks.com/how_to/show/340-lvm-single-drive-to-lvm-raid-1-mirror-migration
https://raid.wiki.kernel.org/index.php/RAID_setup
http://unix.stackexchange.com/questions/63928/can-i-create-a-software-raid-1-with-one-device
http://www.debian-administration.org/articles/238
http://feeding.cloud.geek.nz/posts/setting-up-raid-on-existing/
http://feeding.cloud.geek.nz/posts/setting-up-raid-on-existing/