Setup Storage Directories

From MythTV Official Wiki
(Redirected from Storage Groups Weighting)
Jump to: navigation, search


Software-update-available.png This page is up-to-date as of MythTV version 0.27.6, the current release is 34.0

This article describes the Storage Directories page of MythTV Setup.

This is a required setting. You have to set up at least one storage directory for the storage of recordings. There is no default directory, so you cannot record until you have performed this setup.

Setup Storage Groups

Storage Groups are lists of directories that are used to hold various MythTV files.

The start page shows these options for setting up Storage Groups:

  • Default
  • (Create LiveTV group)
  • (Create DB Backups group)
  • (Create Videos group)
  • (Create Trailers group)
  • (Create Coverart group)
  • (Create Fanart group)
  • (Create Screenshots group)
  • (Create Banners group)
  • (Create Music group)
  • (Create MusicArt group)
  • (Create new group)

Storage Groups are lists of directories that are used to hold MythTV recording files, giving you a flexible way to add capacity to your MythTV system without having to use exotic solutions such as LVM, filesystem expansion or RAID Online Capacity Expansion. You can also use Storage Groups to organize recordings and to put recordings of a certain type into one subdirectory.

Storage Groups do not offer redundancy in case of hard drive failure, but unlike LVM, if you lose a hard drive, you only lose the recordings that were on that drive. With LVM, if you lose a hard drive, you will most likely lose everything.

It is strongly recommended that you locate recordings in a separate file system from root. When you install Linux it will typically allocate the entire disk as one file system. You need to override that default and set it up with a large file system for recordings, separate from the root file system.

The Ubuntu or MythBuntu MythTV package creates empty subdirectories in /var/lib/mythtv for storage groups. You can mount your filesystem in those directories and then enter the directory names in setup.

An alternative place to mount file systems in Linux is /srv/. You can create subdirectories there to mount your recording file system.

You must create a directory for the Default group at least. If you do not create any on the other groups, all file types will be created in the Default group. To keep better control of your system, it is recommended that you create separate subdirectories for most of the other groups in the list (all except Live TV), even if they are just subdirectories off the same filesystem as your recordings. It is also recommended if possible that the DB Backups group be on a separate physical hard drive from your database. The database is normally in /var/lib/mysql which is normally in the root file system.

It is recommended that you name your directories in such a way as to make it easy and logical to add extra drives in future. So if you create a recordings1 directory, later you can add recordings2, recordings3, etc. if needed.

Groups are used as follows

Default Scheduled recordings are stored in the directories listed here unless another Storage Group is specified in the recording rule. All other content, as listed below, where there is no separate Storage Group defined for that type of data will be stored in the directories of the Default Storage Group.
LiveTV When you watch Live TV with MythTV it automatically creates a recording that is used for pausing, rewinding, etc. while watching Live TV.

Generally, there is no need to specify a directory list for Live TV recordings as MythTV will store Live TV in the directories specified for the Default Storage Group (which is the default location for scheduled recordings, and, therefore, should be a good location for Live TV recordings, too). Use this storage group only if you want live TV stored in a specific file system or somewhere other than the directories specified by Default Storage Group. Note that if you make a recording via the record button while watching live TV it will also end up here.

DB Backups MythTV includes a database backup program that will store database backups in the directories specified here. The Ubuntu MythTV package installs an anacron job that backs up weekly. Recommended that this be on a separate hard drive from your database.
Videos The "Watch Videos" feature of MythTV looks in this storage group for available videos. It is up to you to put any videos here that you want available to MythTV.
Trailers You can add movie trailers here and associate them with movies that are in the "Videos" storage group so that you can view the treiler from the video list in MythTV.
Coverart If you perform metadata lookup on Videos or TV shows, MythTV stores files here for displaying pictures when you select shows or videos in the frontend.
Fanart If you perform metadata lookup on Videos or TV shows, MythTV stores files here for displaying pictures when you select shows or videos in the frontend.
Screenshots The system stores screen shots from your videos here for showing when selecting a video. Note that screenshots for recorded shows are stored in the recording storage group where the recording is stored.
Banners If you perform metadata lookup on Videos or TV shows, MythTV stores files here for displaying pictures when you select shows or videos in the frontend.
Music If you rip music CDs with the inbuilt facility, or upload music files from other sources, the MythMusic plugin will store and search for the files in this group. (since 0.28)
new You can create any number of Additional Storage Groups of your own. Select them when setting up recordings to cause those recordings to be stored in different locations. Give them any name that is meaningful to you.

You can add entries to any group by selecting the group and selecting the option to add a directory.

To remove a storage group or a directory from the list, highlight it and press the 'D' key. This does not "delete" the files, it only removes the directory as a storage location. If the storage group you wish to remove is not empty (which is likely the case as MythTV balances the load), you can move these files from one storage group to another with a regular filesystem move. The next time MythTV tries to access these files, it will automatically search all available storage groups to locate them.

Adding multiple directories to a group will allow the system to spread the recordings over multiple locations. Whenever you add multiple directories to a group they should be in separate file systems.

For minimizing the Disk I/O load you can use file systems on multiple hard drives. The transfer rate of today's disks is such that you can record many HD shows at the same time on one disk. Spreading the load across disks can reduce fragmentation. If two shows are recording at the same time on the same filesystem, both files could be fragmented by being recorded across the same areas of free space.

If you have multiple backends, set up all of the directory locations on the master backend. Other backends will use the same locations on their own systems, so you should use the same structure for all backends. If you run MythTV setup on a remote backend you can override directory settings there. This is not recommended. It is better to add all possible directory locations to the master backend. If any directories do not exist on a particular backend, those will be ignored.

Directory Structure

If you put a mount point directory name into an SG directory list, MythTV cannot tell when the file system is not mounted (the mount point and the mounted file system directory are identical), so it will gladly write to the parent file system (and possibly fill it up--which is very bad if the parent file system is the system's root file system). Instead, always create one or more subdirectories within your file systems and put a subdirectory path into the SG directory list. Then, when the file system isn't mounted, the directory doesn't exist, so MythTV will ignore it.

Directory Ownership and Permissions

For *Ubuntu/Debian/mythbuntu ppa packaged versions of MythTV, each Storage Directory should be owned by the user running mythbackend, default is mythtv, and have permissions of 2775.

In a terminal session:

sudo chown mythtv:mythtv <directory>
sudo chmod 2775 <directory>

Monitoring

Usage information for all Storage Group directories is visible on the mythfrontend status screen as well as the mythbackend status webpage. MythTV is smart enough to determine which directories are on shared filesystems so it should not count free or used space multiple times if more than one directory is on the same filesystem.

Storage Usage

No matter how big your storage disks are, if you are like most users, you will run out of space, especially if you are recording HD. HD recordings take from 3GB to 8GB per hour. You need some strategy for freeing up space for more recordings once you have watched what you have recorded.

Auto Expire: If you set auto expire on your recording rules, then shows will automatically be deleted when space is needed for new ones, whether you have watched them or not. You can set auto expire on some rules and not on others.

Max to Keep: If you set a number in Max to Keep on a recording rule, only a fixed number of episodes of that show will be kept. You can choose between automatically deleting the oldest to make a new recording, or of not recording once the limit is reached. The automatic deletion is immediate and the show does not go into the "Deleted" group, so it cannot be undone. You can set Max to Keep on some recording rules and not on others and you can set different values on each rule.

Deleted Recordings: When you have watched a recording you can choose to delete it, thereby freeing up the space. There is a parameter, "Deleted Max Age" which defines how long to keep deleted recordings. If you set this to 7, then the deleted recordings are kept for 7 days in the "Deleted" group and during that time you can restore them again. You can set this to 0, which keeps them in the "Deleted" group for about 20 minutes, or to -1 which keep them forever or until the space is needed for new recordings.

You need to decide, based on your viewing habits whether the above listed options are appropriate for you.

Keeping recordings until the system needs the space requires that you make sure there is always enough space for new recordings. You will have to set "Extra Disk Space" (this is set in Frontend Setup) to a big enough value to hold some 15 minutes of recordings at the maximum rate with all recorders going.

Multiple File Systems

If you only have one file system for your recordings on each backend, then you do not need to be concerned with the methods of balancing the usage of file systems. However as soon as you add another drive this will become important.

The "global" setting "Extra Disk Space" (this is set in Frontend Setup) specifies the amount of extra space to keep available on each filesystem, not a total amount of extra space to keep available across all filesystems. Therefore, if the "Extra Disk Space" is set to 3GB, it should be keeping 3GB available on all filesystems (to which MythTV is recording at any particular time).

That parenthetical can actually be very important. If MythTV is not recording to a filesystem, it does not autoexpire programs to make room for the data that some other process is writing to that filesystem. Therefore, until a recording starts and is being written to the filesystem in question, no programs will be expired from that filesystem.

If you decide to keep recordings until the system needs the space you can have some unexpected problems. When a recording is started, the system uses an algorithm to decide which file system to use. Only after that does it start making space for the recording by expiring old recordings. It could have chosen a file system that has no expirable recordings or that has only very recent expirable recordings. The rule that says it should expire the oldest recording may seem to not be followed.

You could also run into failed recordings if all of your expired recordings are on one file system and it happens to select another file system for recording. You need to set up the Storage Group Disk Scheduler described next to help resolving some of these issues.

Storage Group Disk Scheduler

The Storage Group Disk Scheduler provides 4 different implementations, which can be selected with the mythtv-setup General setting "Storage Group Disk Scheduler". To change that go to MythTV-Setup, General, third page (Miscellaneous Settings).

Selection Description Recommendation
Balanced Free Space (default) Keeps the same number of gigabytes of free space on each file system. Use this if one file system is very under utilized and you want the system to correct that situation (e.g. you added a new drive).
Balanced Percent Free Space Keeps the same percentage of free space on each file system. You may want to use this instead of Balanced Free Space when the file systems are of different capacities (e.g you have a 1TB filesystem and a 2TB filesystem).
Balanced Disk I/O Records to file systems that are least busy. If all are equally busy records to the one with most free space. This is an improvement on the balanced free space options. It helps reduce fragmentation.
Combination Uses a more complex algorithm to find the best file system for each new recording. This is a further improvement on the prior systems, which takes into account the difference between local and network drives, to get the best balance. You can customize this to suit special situations. See the advanced notes below.

If you plan to keep recordings until the system needs the space, then you will have recording file systems that are always 99% full. In this case the Balanced Free Space and Balanced Percent Free Space schedulers will not do anything reasonable since there is always minimal free space. You should select one of the other schedulers.

Advanced Topics

Click Expand to see the Advanced Topics ...

Balanced Free Space

The only criteria considered here is free space, so if you have 5 tuners and 2 filesystems, and 5 things start recording at the same time, they'll all go to the filesystem with more free space.

Users with full or nearly-full filesystems (e.g. users who rely on auto-expiration to remove recordings) should almost definitely choose a Storage Group Disk Scheduler other than Balanced Free Space or Balanced Percent Free Space.

Balanced Percent Free Space

Like "Balanced Free Space", the only criteria considered here is free space, but in this case, preferring the file system with the greatest percentage of total file system size available. So if you have 5 tuners and 2 filesystems, and 5 things start recording at the same time, they'll all go to the filesystem with the greatest percentage unused.

Users with full or nearly-full filesystems (e.g. users who rely on auto-expiration to remove recordings) should almost definitely choose a Storage Group Disk Scheduler other than Balanced Percent Free Space or Balanced Free Space.

Balanced Disk I/O

The only criteria considered here is the 'weight' on the filesystem at the time. Items included in the weight are in-progress recordings, playback, commercial flagging, transcoding, etc.. The disk with the least I/O activity (weight) will get the recording. In this case, with the same 5 tuners and 2 filesystems mentioned above, you'd have 3 tuners recording to one filesystem and 2 tuners going to the other, but there is no preference as to which filesystem gets the 3 and which gets the 2.

As of 0.27 free space is used as a tiebreaker if, and only if, weights are equal. This means that if a system has five tuners and seven local filesystems with no recording playing and no jobs (meaning the filesystems' weights are equal), and all five tuners start recording at once, Balanced Disk I/O will record to the five filesystems with the most free space. For those with more than one or two tuners this behavior is probably more desirable than Balanced Free Space or Balanced Percent Free Space, which would put all five recordings onto the single filesystem with the most free space, possibly causing write buffer errors as mythbackend and the filesystem attempt to keep up with all five at once.

Combination

This is the original Storage Group balancing method. This method is no longer the default. This method uses a combination of free space and disk I/O to determine where to put the next recording.

Sorting is done using the following criteria:

If fs1 is local and fs2 is not
then if fs1 has less weight use fs1 else use fs2
else if fs1 and fs2 are either both local or both remote
then if fs1 or fs2 has a lower weight, use the one with the lowest weight
else if both have equal weights then use the one with the most free space
else if fs is not local and fs2 is
then if fs1 has less weight use fs1 else use fs2

Weighting

There is some limited ability to change the weighting, but there is no GUI to set them up.

MythTV uses weights to determine what filesystem/directory to record to. The following default values are used:

   SGweightPerRecording  = 10
   SGweightPerPlayback   =  5
   SGweightPerCommFlag   =  5
   SGweightPerTranscode  =  5
   SGweightLocalStarting = -1.99 * SGweightPerRecording
   SGmaxRecOverlapMins   =  3

The drive with the lowest weight is used first. In a tie, the drive with the highest amount of free disk space is used first.

Each Storage Group Directory has its own weight, but MythTV is smart enough to know if several directories are on the same shared filesystem. When a weight is applied to a directory that is in use, it is applied to all directories on that filesystem because we are trying to spread the load out across filesystems, not directories on the same filesystem.

The starting weight of all drives is 0. Local filesystems/directories are then offset by SGweightLocalStarting, so by default they are then at -19 because SGweightPerRecording defaults to 10. This makes it so that local drives are preferred over remote drives. A local drive has to have an effective weight of 20 before a remote drive will be used to store a new recording. If you do not have any playback going on, this would mean that you'd have to have 2 recordings going to a local drive before a remote drive would get used. If you have 2 local drives and 1 remote drive, you'd have to have 4 recordings going on locally before a remote drive would get used.

So, if you have less than 4 tuners and 2 local drives, then the default should already give you what you are looking for. If you have more than 4 tuners or want to guarantee that all recordings go to the local drives unless they fill up, you can do so using the undocumented SGweightPerDir setting.

In order to make MythTV not use a filesystem/directory, we need to artificially inflate the starting weight for that directory. We can do this by inserting a setting in the database.

The key for the setting is SGweightPerDir:HOSTNAME:DIRECTORY. The hostname is the hostname that sees the directory. So if the directory is actually on the fileserver which is server1 but is mounted via NFS on server2 which is running mythbackend, we'd use server2 here. The directory is the local path on HOSTNAME, so you'd put /mnt/video or whatever you use here for the remotely mounted directory.

   SGweightPerDir:server2:/mnt/video

The value that we put in this setting will be applied as an offset to the initial weight for this directory. You can play it safe by setting this to something large like 99 or 100 or even higher.

To set the setting, use wget from the command line or a script. For example:

BE=server2
SG_DIR=/mnt/video
VALUE=99
wget --post-data="HostName=${BE}&Key=SGweightPerDir:${BE}:${SG_DIR}&Value=${VALUE}" \
     http://${BE}:6544/Myth/PutSetting

The response should show a Result of true (curl may also be used with the --request POST option.)

So, now when the Storage Groups scheduling code runs, /mnt/video will start out with a weight of 0. Since it is a remote drive, the only offsets that will be initially applied will be the one we specified in the settings table with the SGweightPerDir setting. Your two local directories would each end up starting out at -19 and the remote directory would start out at +99. So unless you have a huge number of recordings and playbacks going on on each of your local drives, the remote directory will never be used unless the locals fill up.

If you run mythbackend with the "-v schedule,file" option, you can see the weights as they are applied and the logs will show you why MythTV chose one directory over another when determining where to put the next recording.

Multiple Backends

You should only define TV Storage Groups on the master backend (certainly for most users).

A Storage Group is simply a mapping between a user-friendly name ("Default", "Videos", "Your Own Group", ...) and a list of directory paths. This does not say anything about them containing information about the host that contains those directories.

Therefore, when you *define* a Storage Group (which can only be done on the master backend) and set up a list of directories, that Storage Group exists on *all* hosts. Note, also, that I did not say that all the directories in the directory list must exist on all the hosts--only that the Storage Group exists on all hosts (the /mapping/ between a user-friendly name and a list of directory paths exists on all hosts).

When you use mythtv-setup on any host other than the master backend to edit Storage Groups, all you're doing is overriding the directory list for an already-defined Storage Group--you're saying, "The list I gave on the master backend is wrong." (So, really, you should just fix the list on the master backend...)

This is true even in the situation where a remote backend has its own storage.

You do not have to run NFS or Samba on the network.

Example 1

Consider the situation where a master backend has 4 file systems it's using for recordings and a remote backend that has 3 file systems it's using for recordings.

The master backend has no access to the file systems on the remote backend and vice versa.

Regardless of this, *only* define Storage Groups on the master backend. In this example, the Default Storage Group has a directory list:

/srv/mythtv/tv/a/recordings
/srv/mythtv/tv/b/recordings
/srv/mythtv/tv/c/recordings
/srv/mythtv/tv/d/recordings

where a, b, c, d are mount points and actual recordings are put into subdirectories so that the backend(s) can tell if the file system is mounted or not (otherwise, if a, b, c, d were put in the directory list, it would write to the directory whether or not the file system is mounted--meaning it would fill up the parent file system, which is my root file system).

As mentioned before, in this example the master backend has 4 file systems and the remote backend has only 3 file systems. This isn't a problem. On master backend, a different file system is mounted at a, b, c, and d. On the remote backend, we not (yet) mount a file system at d. Therefore, the remote backend has directories:

/srv/mythtv/tv/a/recordings
/srv/mythtv/tv/b/recordings
/srv/mythtv/tv/c/recordings
/srv/mythtv/tv/d

Since there's no file system mounted at d, there's no /srv/mythtv/tv/d/recordings (which is the one the Default Storage Group uses), so MythTV sees that the directory doesn't exist and doesn't use it (it simply ignores it) on the remote backend.

This means that we have no Storage Groups/Storage Group directory entries defined for any host other than the master backend.

There is no reason to ever touch Storage Groups on any host other than the master backend.

If you prefer, you can do things slightly differently. Let's say you don't want to "re-use" directory names on your different backends because you may find it confusing to have 2 different file systems using the same directory path. In that case, the Storage Group definition on the master backend should contain the "union" of directory paths used by all backends. So, it may have something like:

/srv/mythtv/tv/master/a/recordings
/srv/mythtv/tv/master/b/recordings
/srv/mythtv/tv/remote1/c/recordings
/srv/mythtv/tv/remote1/d/recordings

(where picking directory names is the hard part, but the above gives you the idea). Again, a, b, c, d are all mount points and recordings is a directory existing within the file system mounted there. Then, since there's no /srv/mythtv/tv/master/a/recordings directory on the remote backend, and no /srv/mythtv/tv/remote/c/recordings directory on the master backend, MythTV will know to ignore those directories on the hosts in question.

Example 2

For the situation where a remote backend has no storage and you use NFS or Samba, you may have all your storage on the master backend or some NAS or whatever. If so, just list all the directories in the Storage Group definition on the master backend and other hosts will use the same list of directories. Since all recording is done to "local" file systems, the remote file systems must be mounted at the same absolute path on all hosts.

If you want the TV directories NFS mounted on each MythTV system, you can, but you need to choose an appropriate Storage Group Disk Scheduler. If you also mount the TV directories at the same absolute path on the frontend systems, then recordings will be played back directly via NFS--rather than having the backend read the recording via NFS and then stream the recording to the frontend (therefore requiring only one trip across the network instead of two).

There's no need for any backend to have access to any file system other than the file systems to which that backend records TV. All MythTV recordings access (recording, playback, etc.) is performed through the recording host's mythbackend. The only exception is when you enable the setting in General Setup:

If enabled, the master backend will stream and delete files if it finds them in the storage group. Useful if you are using a central storage location, like a NFS share, and your remote backend isn't running.

This is only meant to be used when you have a single file server containing all file systems and you want to be able to stream/playback recordings from remote backends even when the remote backends are shut down or when *all* of your storage exists on the master backend and remote backends record to it over the network--so without this setting, streaming from the recording backend would mean the recording backend reads a recording over NFS and then sends it across the network to the frontend. Instead, enabling the override allows the master backend to read it from local storage and send it across the network to the frontend (so, only one trip across the network).

Quick Rules

  • Only set Storage Group directory lists on the master backend. (Never create Storage Group directory list overrides on the remote backends.)
  • Once properly configured (with no SG overrides on the remote backends), only manage Storage Groups using mythtv-setup on the master backend
  • Make the master backend's Storage Group directory list contain the union of all directories on all hosts--whether those directories exist on all hosts or not.
  • Directories in a Storage Group directory list need not exist on any host (those that don't exist on a particular host are simply ignored).
  • If you don't want a directory used on a particular host make sure that directory doesn't exist on that host.
  • Never put a mount point into a Storage Group directory list--always put some subdirectory under the mount point so MythTV will know to ignore the (non-existent) directory entry if the file system fails to mount properly.
  • If you think you need to override a SG directory list on a remote backend or remote frontend, you're making things more difficult than they need to be.

Deleting Overrides

To delete a Storage Group override that has been set up on a remote backend (and revert to using the definition specified on the master backend), go to mythtv-setup on the remote backend host, then to Storage Directories. Highlight the Storage Group whose override you want to remove and hit the DELETE key. A pop-up appears that says, "Delete '<SG name>' Storage Group?" When you say, "Yes, delete group," mythtv-setup will delete the Storage Group directory list override for this host from your configuration.

To delete an override of the Default Storage Group for all hosts (making all hosts use the definition specified on the master backend), to to mythtv-setup on the master backend host, then to Storage Directories. Highlight the Default Storage Group and then hit your DELETE key. A pop-up appears that says, "Delete 'Default' Storage Group? (from remote hosts)". When you say, "Yes, delete group," mythtv-setup will delete the Default Storage Group directory list overrides for all hosts from your configuration. The Default Storage Group and the directory list on the master backend will remain and will now apply to all your remote backends. If you want to edit the Default Storage Group directory list on the master backend, you have to do so using mythtv-setup on the master backend.

Reorganizing Recordings

There may be situations where you want to move recordings between file systems. You may move any recording to any directory listed in any Storage Group, as long as that recording is still accessible by the recording host (the host that originally recorded the show)--whether it is accessible locally or via a network mount. No changes are required in the database when moving files as long as they are accessible from the original recording host.

Expanding Storage Groups

When you add a new directory to a storage group the "free space" will not be updated until after the first time mythtv writes a file to that directory.