File storage

From MythTV Official Wiki
Revision as of 05:56, 4 August 2007 by Vossman (talk | contribs) (File systems: wiki link)

Jump to: navigation, search

File storage refers to the broad topic of hardware, software and the methodology behind keeping MythTV recordings on a computer (and computer network).

As everyone knows, storing copious amounts of video requires bucketloads of hard drive space. MythTV is exceptionally good at producing the requisite video, so we are going to need somewhere to store it — the question is how much, and for how long?

I do not watch a huge amount of TV, so my immediate storage requirements are not that colossal, and all of my MythTV space was cobbled together from whatever IDE drives I had lying around. However, I am a complete hoarder, and do not like deleting things unless I have to, or they are rubbish. As such, my storage requirements have started out basic and will eventually turn into multiple petabyte boxes over a fibre-channel SCSI SAN array. Well, maybe ;)

My chosen hardware (I will use my PVR-250 as an example, since it is probably the most common TV card in use that I have access to) records in MPEG2 format, and takes up about 1GB an hour. This can of course be increased or decreased by altering the quality, but we are still talking about a lot of space over time.

Pretty much any reasonably modern hard drive will be more than adequate both in space and speed for MythTV, but my advice to you is to buy the biggest hard drive(s) you can afford. Currently in the UK, the "sweet spot" is around the 200GB barrier; these drives offer the best GB/$ ratio.

Manufacturers

Important.png
Please note that this section will be full of hearsay, personal bias and much that is probably apocryphal, so take it with a pinch of salt, and always ask around. If you are adding anecdotal datapoints about a specific manufacturer, please indent one level and sign your comments.

The current [Mar-07] major manufacturers of consumer hard drives are Seagate, Western Digital, Maxtor (Now owned by Seagate), IBM/Hitachi and Samsung who all make IDE devices ranging up to 400GB, available in parallel (PATA) or the newer serial (SATA) interfaces, or as SCSI, though those have not caught up on the size front.

  • Seagate Barracudas are generally renowned to be quiet and reliable, having recently returned to offering a standard 5yr warranty
  • Western Digital drives are slightly better performers than the Seagates, at the expense of being noisier
  • All of the Maxtor drives I have used have been very noisy, and not particularly reliable
    • Note: All of the Maxtor drives I have used have been extremely reliable, and not very noisy at all. I have had only 2 fail in the six (?) years I have been using them in my computers and computers I build for customers. One was due to overheating (in a tiny amount of space in a hot area). The other -- I think was just a bad manufacture. Both times, Maxtor replaced the drives (and upgraded them, for free) for me with no hassle, and quite quickly, I might add. Just another personal opinion =) --Tyler Drake
    • My Maxtor Diamond Max 10 250GB is as quiet as a church mouse. --DavidC
    • I've had about 80% failure rate with Maxtors (40-250GB PATA) in the couple of dozen I've put in customer's boxes. They seem quiet though. My failure rate with Maxtors approaches 100% when they are attached to a TX2000 Promise RAID controller. They do normally last around a year so that they are out of warranty. I've had excellent luck with Seagates and you've got to love that 5yr warranty.
    • Since Seagate bought Maxtor and has turned them into their "lower" line of drives with a 3 year warranty instead of their usual 5 year. But it also has seemed to improve their quality.
  • IBM/Hitachi are recovering from a somewhat tarnished reputation from their "Deathstar" line of hard drives, and are producing some very good SATA drives, although I've never used any myself
  • Samsung Spinpoint drives are getting rave reviews (fast, quiet and reliable) from a lot of users here in the UK, although again I've never used them myself.

Based on my current bias, I can recommend the Seagate Barracuda drives for situations where you want quiet drives, although I prefer Western Digital for situations where noise isn't much of a problem.

Interfaces

There are four types of interfaces that are most commonly used these days. These interface are:

  • (P)ATA
  • SATA
  • SCSI
  • SAS

(P)ATA

As a much older standard, PATA is universally supported on most x86 hardware. This interface was originally called ATA but when Serial ATA (SATA) was introduced it was renamed Parallel ATA.

SATA

Most new hard drives and motherboards come with support for the newer SATA interface. Although SATA is a superior standard (it supports a lot of the SCSI subset, and features much smaller, thinner cables than PATA, amongst other improvements), it has been somewhat plagued in Linux by closed source SATA controller drivers. This has resulted in many Linux-based systems being unable to use SATA adequately due to poorly functioning controllers.

For the current status of SATA support under Linux you can check these two pages:

A quick gander through my kernel config shows me that the following controllers are supported under Linux 2.6.8:

  • Intel ICH5
  • nVidia SATA (nForce chipsets)
  • Promise SX4, TX2 and TX4
  • Silicon Image SATA controllers (very common - presumably both the 3112 and 3114 are supported now)
  • SiS 964/180
  • VIA SATA (VIA chipsets)
  • Vitesse VSC7174

Please bear in mind that the built-in software RAID functions on SATA chips will usually not work in Linux without extensive fooling around with the kernel (if at all), and that performance using the open source drivers may be less of that than the closed source proprietary drivers. If you wish for better SATA support under Linux, write to the device manufacturer and ask them to provide some open specs for the kernel hacking team!

SCSI

SCSI stands for Small Computer Systems Interface, and is/was a competing hard drive interface to IDE/ATA. However, back in the mists of time, SCSI was designated to the "high end hard drive" side of things, and is now much more expensive than IDE technology. Though, if you look at things like raw drive MTBF hours, you will see that cheaper IDE drives are only now barely catching up to the SCSI drive specs.

None but the highest end server and workstation motherboards come with inbuilt SCSI controllers, so these usually have to be added by means of a PCI card, which in themselves are not cheap. The cost of the hard drives themselves are very high indeed, and they offer much reduced storage capacity compared to a modern PATA or SATA drive. However, SCSI discs are incredibly fast and very reliable - but as we can see, it comes at a huge price. To be honest, there is very little chance of even an extensive MythTV setup requiring a SCSI system - SCSI excels in massive multi-user environments like databases and web/mail servers, but the advantages under a single user setup are hard to distinguish. With the recent addition of Western Digital's enterprise-class "Raptor" SATA drives, you can approach SCSI speeds without shelling out a kings ransom, although their size is limited to 74GB at the time of writing.

One thing of note is that SCSI drives are very very loud due to their very high rotation speed (10,000 or 15,000rpm) and so are going to be relegated to the backend under the stairs pretty quickly. Raptor drives are quieter, but still far louder than your average IDE drive.

[ Editorial comment: SCSI's not that bad a choice, particularly if you can get used drives cheaply on eBay, and you are building an Under The Stairs backend box -- instead of the 2 or 4 drives you can put on most IDE controllers, you can put 15 on a SCSI controller -- and multiple channel controllers are available. So it is a matter of scale and buying savvy as much as anything else. -- Bay Link (2004-10-01T18:06:44Z)

  • The problem comes that 15 drives are only useful for mass storage reasons, and the price/size ratio attainable through SATA is much better than with SCSI. For MythTV purposes SATA make the most sense technically and financially, with PATA a close second if you are not concerned with overall speed (i.e. as an archive array). That said I do have a 4xHDD U320 SCSI setup as my personal desktop... --Steve Adeff 16:23, 8 June 2006 (UTC)]

SAS

SAS, or Serial Attached SCSI, is a new technology that takes the best of SCSI and SATA and in many ways it compares to Fiber Channel (e.g. SAN technology) and USB as well. Most modern servers already ship with SAS instead of SCSI, and they're eventually expected to be the desktop standard. As of today it has a bus bandwidth if 3 Gbps, and is on target to increase to 12 Gbps by 2011. Individual SAS drives today have a transfer rate of 300 MB/sec, just under the SCSI rate of 320 MB/sec, but each drive gets the full 300 MB/sec to the host, instead of shared as with SCSI, SATA and PATA. Current benchmarks show comparable performance to the best 15K Ultra320 SCSI drives and in some areas SAS far surpasses SCSI performance. Some other cool features are:

  • The SAS interface is backwards compatible with all SATA drives.
  • It can support over 16,000 devices on a single bus, compared to 16 with SCSI and 1 with SATA
  • SAS Expanders provide the ability to hook up drives the same way we network computers using a switch, although over shorter distances (several meters).
  • 2.5" and 3.5" drives are available

External Links

Partitions

The Unix model of filesystems is much more flexible than that under windows, and Linux is no exception, allowing you to seamlessly integrate hard drives and partitions and different formats here, there and everywhere. Personally, I'm a big fan of multiple partition setups because, amongst other things, it allows you to tailor the filesystem (see below) to the files that are going to live on it. At the very least I would advocate at least three partitions:

  • /boot is where the kernel and bootloader things live. I usually format this ext2 since it is very rarely written to.
  • / is the root of your filesystem, pretty much the equivalent of "C:\" under windows; all of your programs and files will live on the root filesystem somewhere (such as /usr and /var/log), unless you specify a particular directory tree to live under a separate filesystem (like the /boot shown above).
  • The third partition would be where you store all of your MythTV files (as well as music and external video, if that is where you want it). If you want to see my partitioning setup for one of my backends, you can look at the "Advanced storage: example setup" section below.

Note that, particularly if you are prone to monkey with CVS Myth or advanced beta and alpha test drivers, you will be much happier if you put /var/log on its own partition.

File systems

As you probably know, Linux has a bewildering array of file systems available, most of which excel at a particular task. You are of course free to format your drives with whatever file system you choose, but here is some general info about the most popular file systems:

  • ext2 is the "old standard" file system. It is fairly speedy, but does not come with journalling to protect your data from corruption, and can take an age to run though a file system check (fsck), although it can be seamlessly upgraded to ext3
  • ext3 is an extension to the ext2 file system which introduced journalling as well as other improvements. It is a bit of a jack-of-all-trades of a file system, and does not excel at anything in particular, apart from very thorough testing!
  • ReiserFS is a high performance file system that is especially good at dealing with directories with lots of small files, which makes it a good choice for your system partitions, although it does not perform as well with large files
  • JFS was originally developed by IBM for their AIX operating system, and was later donated to Linux. JFS is incredibly good at dealing with the huge files that MythTV generates, and can delete pretty much any file in under a second (ext3 can take as long as 15 seconds to delete really big files). JFS is a very good file system to use for storing your videos on, and it is very conservative with CPU usage.
  • XFS is another "foreign" filesystem, developed by SGI for their IRIX operating system, and once again donated to Linux. Like JFS, it is exceptionally good at dealing with large files, and has the highest throughput of any Linux filesystem, albeit at a higher CPU loading. XFS also makes an excellent choice as storage for your movie files. (Note that XFS filesystems can be grown, but not shrunk, at the present time; this can occasionally be problematic. Note also that file system cleanings are forced using xfs_repair, not fsck; if you are going to use XFS, and Bay Link recommends that you do, read about it first.)

To use any of these file systems, you'll need support for them compiled into the kernel along with the relevant userland utilities. The file system driver(s) of your partitions should always be compiled statically into the kernel (which makes things much easier!) or into an initrd, and not as a module, otherwise your newly booted kernel won't be able to load the modules you need to understand the filesystem to load the module you need (made my head spin too, but if you re-read it enough, it makes sense).

Some distributions come with a choice of only one or two filesystems, although if you rebuild your kernel it is usually possible to enable all of them (including support for windows FAT32 and NTFS if you need it!). New, exotic and improved filesystems are cropping up all the time; hot on the horizon is Reiser4, which promises to be a very high performing and flexible system, although it is far from stable yet.

Many filesystems allow tweaking of the block size at format time - selecting a large block size will make more efficient use of your hard drive space when dealing with large files, whereas a small block size is better suited for your system partitions. If in doubt, read the manual thoroughly or just go with the defaults, since you cannot change the block size without reformatting the drive.

In short, a good choice is ext3/Reiser for your system partitions and JFS or XFS for your MythTV storage. Note that the XFS implementations on SuSE 9.0 and 9.1 were both a bit flaky, this can make installations and upgrades difficult if you do not know the magic. (I will put the magic here when I relocate it. --Bay Link)

Advanced storage

LVM

LVM stands for the Logical Volume Manager, and you can use it to make two or more separate hard drives (or partitions on those drives) appear as one huge hard drive to the operating system. LVM can stripe the partitions together, and you then make one or more big filesystems on the entire thing.

You will need to have LVM enabled in your kernel to do this, as well as having the userland LVM utilities installed. Users of 2.6 will be able to utilise the much improved LVM2.

The terminology of LVM is perhaps a little advanced to go into here, so if you want a good explanation of it you can read the LVM HOWTO. In short, you can dedicate either individual partitions or entire hard drives (the "physical volumes") for use by the LVM, which allows you to map them into one or more "volume groups", from which you then carve out "logical volumes" to install filesystems upon. Probably the best thing about LVM is that, as long as the filesystem you pick is capable of being resized, you can extend the volume group(s) and logical volume(s) over bigger and more hard drives without losing or having to copy any data, which makes it a great choice if you want to keep your expandability options open.

RAID

Originally, RAID stood for Redundant Array of Independant Discs, although now the word Independent has been substituted for Inexpensive (probably because most RAID setups use very expensive SCSI discs ;). What this basically means is that data is spread across multiple hard drives in such a way that if one of the hard drives explodes or is eaten by the cat, you will be able to reconstruct the lost data from the other hard drives. One of the lesser functions of RAID is to produce higher performance filesystems by spreading read/write load across multiple discs as well. For a very clear and concise RAID tutorial, you can read these pages http://www.acnc.com/04_00.html, but in the meantime here is a brief rundown of the most common RAID levels along with examples of storage capacity:

  • RAID0, also known as striping, is not true RAID, in that it offers no redundancy. If one of the discs in the array fails, all of the data in the array is lost. RAID0 scales linearly with every drive added; two 80GB drives will produce a single 160GB filesystem. Please note that RAID0 is distinctly different from LVM!
  • RAID1, also known as mirroring, involves copying data to two identical hard drives rather than just one. If one drive dies, the other will remain fully functional with all of your data intact. Two 80GB drives will produce a single 80GB filesystem.
  • RAID0+1 is a combination of mirroring and striping using four drives offering very fast read/write speeds. Four 80GB drives would combine to form a 160GB RAID0+1 filesystem.

Most of you will have seen these RAID levels advertised as being built into almost all modern motherboards; unfortunately this kind of RAID is achieved under proprietary drivers (all RAID calculations being done in software), typically available only for Windows. Fear not however, because the Linux kernel contains its own software RAID drivers, which (if the rumours I hear are true) perform even better than the proprietary software RAID drivers. Again, you can enable these in your kernel.

You will also note more exotic RAID levels such as RAID5 and RAID10. These are quite tricky to do in software (although it is possible), and are usually left to high-end dedicated RAID controllers. Previously, these were only available in high-end SCSI RAID setups, although recently 3ware have released an excellent series of cards that allow high end RAID features on much cheaper and larger capacity SATA discs. These cards are fully supported under Linux and offer excellent performance, and I use two of them at home. If you are looking for a relatively cheap and hassle-free IDE RAID setup under Linux, 3ware's are a very good choice. (P.S. thanks for the cheque, 3ware!)

  • RAID5 is a good compromise on RAID10, and spans data and parity data over three or more drives, giving redundancy and good read/write performance.
  • RAID10 is very like the high performing RAID0+1, but with better redundancy (it can survive up to two simultaneous drive failures, whereas 0+1 can sustain only one hard drive failure).

In the end, if you are not that worried about losing your data (or if you keep good backups), any kind of RAID is overkill. A good compromise can be reached if you place all your system directories on a RAID of some sort (which will protect all of your time consuming configuration — my workstation is in the process of being switched over to RAID1 on two Western Digital Raptors) whilst placing the TV storage on a single disc. But if you have enough money and inclination, you can RAID your whole setup — I am particularly paranoid, and plan to upgrade my backend to using a 3ware and four 250GB drives in RAID10 to (hopefully) put an end to my currently non-existent storage problems.

Network filesystems

As the name implies, these are mechanisms for accessing a files system (and therefore files) across a network.

Wireless LANs have come a long way, but if you're going to use remote files 100Mbit/s or better is probably the minimal requirement.

As with anything else, there are several network file system protocols. To confuse things a bit, the one most useful to MythTV users the long established and generic NFS (which, itself is an acronym for Network File System). I guess you could use Samba (the Windows compatible sharing system), or perhaps both in combination but unless you've got a very good reason too (you are using an Xbox Frontend for example) then stay away.

In a pure Linux environment NFS can be doubly useful as one can setup a diskless frontend (or indeed, a diskless backend or combination of the two). Removing all physical drives (aside from maybe DVD, etc) from will render the box much quieter.

Supermount

Storage Groups

Storage Groups are a feature introduced in late 2006-early 2007 in SVN trunk and intended for inclusion in version .21. The user designates a list of directories in mythtv-setup, which myth uses for storage. These directories may exist on the same or different drive partitions, drives, logical volumes, RAID arrays, or network storage locations. The coder, Chris Pinkham explains his thought process:

MythTV's Storage Groups will search for a recording in all directories referenced in the Storage Groups config. It first searches for the file in the Storage Group you put the file in originally, but it will search all directories in the Storage Groups table eventually if it does not find the file where it is supposed to be.

One of the things in the back of my mind when coding Storage Groups was archival. You could have an Archive Storage Group that points to a directory or directories that are used for archiving recordings. If you wanted to move a recording from the Default Storage Group into the Archive Group, just mv the file and Myth will find it. Since the Archive Group isn't referenced in any of your recording profiles, then no new recordings would go there, only recordings you moved there yourself. Since Myth falls back to searching all directories, it would find these recordings without you having to touch the database at all after moving the file. I started writing a builtin 'Migrate' job in the JobQueue that will be used for moving recordings between Storage Groups, but put it on the back burner a while back. Eventually you'll be able to do this mv within Myth from the JobQueue menu. The AutoExpirer might be able to take advantage of this eventually as well, so it could migrate a recording instead of deleting it or migrate recordings to keep free disk space equally spread out over the directories in a Storage Group. - Chris Pinkham (from mythtv-users list, April 25, 2007)

Example setup

Below I have detailed the partitioning setup for my own main backend, which has just been rejigged. My main rig consists of 80GB (hda) and 120GB (hdb) Seagate Barracuda's:

  • hda1 is used for the /boot partition, and is 25MB is size, ext2 formatted
  • hda2 is used for the / partition, and is 580MB in size, ReiserFS formatted
  • hda3 is the swap partition
  • hda5 is used for the /usr partition, is 6GB in size and ReiserFS formatted
  • hda6 is /var and is 2.5GB in size (way too big!), ReiserFS formatted
  • hda7 and hdb are joined into a JFS formatted LVM under /dev/mythvg/lvm0 of approximately 174GB, sitting on /home (TV data is stored in /home/mythtv/tvstore)
  • /home/mythtv/music and /home/mythtv/movies are read-only NFS mounts from my file server (which has everything stored on RAID1 on a 3ware 8506) which keep all my movies and music available to MythMusic and MythVideo
  • /usr/portage is also NFS mounted (read-write) to the file server, and used to distribute the portage cache and downloaded tarballs to my other Gentoo boxes, allowing for faster compiles and less rsyncing off the Gentoo servers, which will also allow me to use a smaller /usr partition.

It is probably a lot more complicated than is really necessary, but I felt like having a good experiment with things, and this is what I came up with.

There is also a walkthrough setting up an XFS on LVM on RAID5 system here: LVM on RAID.