LVM on RAID

From MythTV Official Wiki
Revision as of 16:02, 14 August 2005 by 66.56.208.117

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

The Problem

  • I want a lot of storage for my TV!
  • I don't want to lose data if a disk crashes
  • I want to add more disks later

The answers:

  • Buy disks - lots of them
  • Use raid
  • Use LVM

What's Raid?

Raid is about putting your data on 2 or more disks in such a way that if one of them crashes you don't lose your data (it's as if everything is stored twice - but cleverer!)

[comment: since I did this I've had one of my disks die and be RMA'd. No lost data :) OTOH, during the month that the drive was away I was 'unprotected' and had a minor read error which caused much trouble - but thankfully no data loss]

You always lose some space - but we don't want to lose half (which is what happens with simple 'mirroring') so we want to use raid5. That way we can buy 4 disks (or more!) and only 'lose' 1 of them.

Why not just use raid?

If we run out of room then we need to allocate more. You can't (currently) add disks to a raid5 array.


What's LVM?

What lvm lets you do is collect all your disks, raid arrays and what-not into a big 'box' of storage (called a volume group). You can then make logical volumes from this space (- think partitions). The good thing is that you can then simply add more disks into the big collection and allocate space to existing logical volumes - ie grow them. Essentially you can have a partition that starts on one disk and ends on another (and may include another in the middle!)


Why not just use LVM?

LVM only lets you mirror or stripe - we want resilience but mirroring is bad - we need to buy twice as much storage as we want (or more realistically - we only end up using half of what we can afford ;) ).

The Answer

Do both :)

Create a raid5 array and add it to our 'box'.

Then we make a filesystem from the space in the box.

In the future we can add another array and add that to the box too.

Then we can 'grow' the filesystem.

Oh yes - that's another thing.

You must use a filesystem that can grow - so either Reiser or XFS.

XFS can't shrink so (despite having worked for SGI myself) I'd use Reiser. (You need to set data=sorted as kernel parameter on pre 2.6.6 kernels so a power outage wont cause data loss). [Later edit: I changed my mind after having had a few corruptions - I now have pure XFS]

Doing it...

My Setup

I have 3 250Gb SATA disks (the fourth is on order)

They're called {{{/dev/sda /dev/sdb & /dev/sdc}}}.

Normal ide will be called things like {{{/dev/hdd /dev/hde & /dev/hdef}}}.

The Software

I'm using 2.6.6 with libata and SATA. I've compiled in the raid extensions. You also need {{{mdadm}}} - I'm using 1.5.0

Prepare The Disks

Use all the disks - but create a partition so you can specify the type. This lets the kernel pick up your raid array when it boots.

Format the disks : {{{fdisk /dev/sda}}} {{{ The number of cylinders for this disk is set to 30515. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs

  (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): n Command action

  e   extended
  p   primary partition (1-4)

p Partition number (1-4): 1 First cylinder (1-30515, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-30515, default 30515): Using default value 30515

Command (m for help): t Selected partition 1 Hex code (type L to list codes): fd Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): w The partition table has been altered!

Calling ioctl() to re-read partition table. Syncing disks. }}}

Make The Array

Wipe everything {{{ mdadm --stop /dev/md0 mdadm --zero-superblock /dev/sda1 mdadm --zero-superblock /dev/sdb1 mdadm --zero-superblock /dev/sdc1 }}}

Make the array : {{{mdadm -v --create /dev/md0 --chunk=128 --level=raid5 --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 missing}}} {{{ mdadm: layout defaults to left-symmetric mdadm: size set to 245111616K mdadm: array /dev/md0 started. }}}

What have we got? {{{mdadm --detail /dev/md0}}} {{{ u:~# mdadm --detail /dev/md0 /dev/md0:

       Version : 00.90.01
 Creation Time : Thu Jun  3 20:24:17 2004
    Raid Level : raid5
    Array Size : 735334656 (701.27 GiB 752.98 GB)
   Device Size : 245111552 (233.76 GiB 250.99 GB)
  Raid Devices : 4
 Total Devices : 3

Preferred Minor : 0

   Persistence : Superblock is persistent
   Update Time : Thu Jun  3 20:24:17 2004
         State : clean, no-errors
Active Devices : 3

Working Devices : 3

Failed Devices : 0
 Spare Devices : 0
        Layout : left-symmetric
    Chunk Size : 128K
   Number   Major   Minor   Raid Device State
      0       8        1        0      active sync   /dev/sda1
      1       8       17        1      active sync   /dev/sdb1
      2       8       33        2      active sync   /dev/sdc1
      3       0        0       -1      removed
          UUID : d6ac1605:db6659e1:6460b9c0:a451b7c8
        Events : 0.5078

}}}

Note:BR I've used 'missing' for one of my disks (it's in the post). This has created the array as if one of the disks is dead. When the new one arrives I'll hot-add it and watch the array rebuild itself.

Update the config file (do this if you ever change the config (eg add a spare, mark a disk as faulty etc)

{{{ echo 'DEVICE /dev/sd*' >/etc/mdadm/mdadm.conf echo 'PROGRAM /bin/echo' >>/etc/mdadm/mdadm.conf echo 'MAILADDR david@dgreaves.com' >>/etc/mdadm/mdadm.conf mdadm --detail --scan >>/etc/mdadm/mdadm.conf cat /etc/mdadm/mdadm.conf }}}


LVM2 : Logical Volumes

Now make the array device useable by LVM2 : {{{pvcreate /dev/md0}}}

{{{

 No physical volume label read from /dev/md0
 Physical volume "/dev/md0" successfully created

}}}

Create a volume group that we can use to make logical volumes from : {{{vgcreate -s 64M video_vg /dev/md0}}}BR {{{-s 64M}}} means that we're 'limited' to 4Tb - sounds mad but I can see us keeping all DVDs etc on a volume so WTH :) {{{

   Adding physical volume '/dev/md0' to volume group 'video_vg'
   Archiving volume group "video_vg" metadata.
   Creating volume group backup "/etc/lvm/backup/video_vg"
 Volume group "video_vg" successfully created

}}}

I only want to use 600Gb to start with...

Finally create a volume : {{{lvcreate -L600G -nvideo_lv video_vg}}} {{{

Logical volume "video_lv" created

}}}

Format it: {{{mkfs -treiserfs /dev/video_vg/video_lv}}} {{{ All data on /dev/video_vg/video_lv will be lost. Do you really want to create reiser filesystem (v3.6) (y/n) y Creating reiser filesystem (v3.6) with standard journal on /dev/video_vg/video_lv initializing skiped area: done initializing journal: done syncing...done }}}


and mount it: {{{mount /dev/video_vg/video_lv /huge}}}

and giggle : {{{df -k}}} {{{ cu:~# df -k Filesystem 1K-blocks Used Available Use% Mounted on /dev/hda2 19283776 697988 18585788 4% / /dev/hda1 97826 13003 79604 15% /boot /dev/mapper/video_vg-video_lv

                    629126396     32840 629093556   1% /huge

}}}


Category How To