Difference between revisions of "LVM on RAID"

From MythTV Official Wiki
Jump to: navigation, search
(Why not just use raid?)
Line 25: Line 25:
 
way we can buy 4 disks (or more!)  and only 'lose' 1 of them.
 
way we can buy 4 disks (or more!)  and only 'lose' 1 of them.
  
=== Why not just use raid? ===
+
=== Why not just use RAID? ===
 
If we run out of room then we need to allocate more.
 
If we run out of room then we need to allocate more.
You can't (currently) add disks to a raid5 array.
+
You can't (currently) add disks to a RAID5 array.
 
 
  
 
=== What's LVM? ===
 
=== What's LVM? ===

Revision as of 23:56, 26 January 2006

This article about LVM on RAID. You can also read about RAID itself.

The Problem

  • I want a lot of storage for my TV!
  • I don't want to lose data if a disk crashes
  • I want to add more disks later

The answers:

  • Buy disks - lots of them
  • Use RAID
  • Use LVM

What's RAID?

Raid is about putting your data on 2 or more disks in such a way that if one of them crashes you don't lose your data (it's as if everything is stored twice - but cleverer!)

[comment: since I did this I've had one of my disks die and be RMA'd. No lost data :) OTOH, during the month that the drive was away I was 'unprotected' and had a minor read error which caused much trouble - but thankfully no data loss]

You always lose some space - but we don't want to lose half (which is what happens with simple 'mirroring') so we want to use raid5. That way we can buy 4 disks (or more!) and only 'lose' 1 of them.

Why not just use RAID?

If we run out of room then we need to allocate more. You can't (currently) add disks to a RAID5 array.

What's LVM?

What LVM lets you do is collect all your disks, raid arrays and what-not into a big 'box' of storage (called a volume group). You can then make logical volumes from this space (- think partitions). The good thing is that you can then simply add more disks into the big collection and allocate space to existing logical volumes - ie grow them. Essentially you can have a partition that starts on one disk and ends on another (and may include another in the middle!)


Why not just use LVM?

LVM only lets you mirror or stripe - we want resilience but mirroring is bad - we need to buy twice as much storage as we want (or more realistically - we only end up using half of what we can afford ;) ).

The Answer

Do both :)

  • Create a raid5 array and add it to our 'box'.
  • Then we make a filesystem from the space in the box.
  • In the future we can add another array and add that to the box too.
  • Then we can 'grow' the filesystem.

Oh yes - that's another thing. You must use a filesystem that can grow - so either Reiser or XFS. XFS can't shrink so (despite having worked for SGI myself) I'd use Reiser. (You need to set data=sorted as kernel parameter on pre 2.6.6 kernels so a power outage wont cause data loss). [Later edit: I changed my mind after having had a few corruptions - I now have pure XFS]

Doing it...

My Setup

I have 3 250Gb SATA disks (the fourth is on order)

They're called /dev/sda /dev/sdb & /dev/sdc.

Normal ide will be called things like /dev/hdd /dev/hde & /dev/hdef.

The Software

I'm using 2.6.6 with libata and SATA. I've compiled in the raid extensions. You also need mdadm - I'm using 1.5.0

Prepare The Disks

Use all the disks - but create a partition so you can specify the type. This lets the kernel pick up your raid array when it boots.

Format the disks :

% fdisk /dev/sda

The number of cylinders for this disk is set to 30515.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-30515, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-30515, default 30515):
Using default value 30515

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Make The Array

  • Wipe everything:
% mdadm --stop /dev/md0
% mdadm --zero-superblock /dev/sda1
% mdadm --zero-superblock /dev/sdb1
% mdadm --zero-superblock /dev/sdc1
  • Make the array:
% mdadm -v --create /dev/md0 --chunk=128 --level=raid5 --raid-devices=4 /dev/sda1 \
             /dev/sdb1 /dev/sdc1 missing
mdadm: layout defaults to left-symmetric
mdadm: size set to 245111616K
mdadm: array /dev/md0 started.
  • What have we got?
% mdadm --detail /dev/md0

/dev/md0:
        Version : 00.90.01
  Creation Time : Thu Jun  3 20:24:17 2004
     Raid Level : raid5
     Array Size : 735334656 (701.27 GiB 752.98 GB)
    Device Size : 245111552 (233.76 GiB 250.99 GB)
   Raid Devices : 4
  Total Devices : 3
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Thu Jun  3 20:24:17 2004
          State : clean, no-errors
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 128K 

    Number   Major   Minor   Raid Device State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1
       2       8       33        2      active sync   /dev/sdc1
       3       0        0       -1      removed
           UUID : d6ac1605:db6659e1:6460b9c0:a451b7c8
         Events : 0.5078

Note:
I've used 'missing' for one of my disks (it's in the post). This has created the array as if one of the disks is dead. When the new one arrives I'll hot-add it and watch the array rebuild itself.

  • Update the config file (do this if you ever change the config (eg add a spare, mark a disk as faulty etc)
% echo 'DEVICE /dev/sd*' >/etc/mdadm/mdadm.conf
% echo 'PROGRAM /bin/echo' >>/etc/mdadm/mdadm.conf
% echo 'MAILADDR david@dgreaves.com' >>/etc/mdadm/mdadm.conf
% mdadm --detail --scan >>/etc/mdadm/mdadm.conf
% cat /etc/mdadm/mdadm.conf

LVM2 : Logical Volumes

  • Now make the array device useable by LVM2 : pvcreate /dev/md0
  No physical volume label read from /dev/md0
  Physical volume "/dev/md0" successfully created
  • Create a volume group that we can use to make logical volumes from:
% vgcreate -s 64M video_vg /dev/md0

-s 64M means that we're 'limited' to 4Tb - sounds mad but I can see us keeping all DVDs, etc. on a volume.

 Adding physical volume '/dev/md0' to volume group 'video_vg'
 Archiving volume group "video_vg" metadata.
 Creating volume group backup "/etc/lvm/backup/video_vg"
 Volume group "video_vg" successfully created

I only want to use 600Gb to start with...

  • Finally create a volume:
% lvcreate -L600G -nvideo_lv video_vg

Logical volume "video_lv" created
  • Format it:
% mkfs -treiserfs /dev/video_vg/video_lv

All data on /dev/video_vg/video_lv will be lost. Do you really want to create reiser filesystem
(v3.6) (y/n) y
Creating reiser filesystem (v3.6) with standard journal on /dev/video_vg/video_lv
initializing skiped area: done
initializing journal: done
syncing...done


  • Mount it:
% mount /dev/video_vg/video_lv /huge
  • See your results:
cu:~# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda2             19283776    697988  18585788   4% /
/dev/hda1                97826     13003     79604  15% /boot
/dev/mapper/video_vg-video_lv
                     629126396     32840 629093556   1% /huge