Setup Storage Directories
Storage Groups are lists of directories that are used to hold MythTV recording files. By default, there is only one Storage Group, called "Default".
Additional Storage Groups can be created to store specific recordings in their own directories. Storage Groups are edited via the 'Storage Directories' section of mythtv-setup. It may be desirable to add multiple directories (which are mounted on different hard drives) to a Storage Group in order to spread out filesystem I/O onto multiple filesystems.
Multiple Storage Groups can also be created to group recordings together; recording schedules now have an option to specify which Storage Group to use.
MythTV will balance concurrent recordings across the available directories in a Storage Group in order to spread out the file I/O load. MythTV will prefer filesystems that are local to the backend over filesystems that are remote until the local filesystem has 2 concurrent recordings active or other equivalent I/O, then the next recording will go to the remote filesystem. The balancing method is based purely on I/O, Myth does not try to balance out disk space unless a filesystem is too low on free disk space in which case it will not be used except as a last resort.
Storage Groups are global, but can be overridden on a slave backend by creating a local Storage Group by running mythtv-setup on the slave. If a problem occurs and the slave backend is unable to use the desired Storage Group, it will fail back and try the directories defined in the master's Storage Group.
A special 'LiveTV' Storage Group may also be defined. If a LiveTV Storage Group directory exists, it will be used instead of putting LiveTV recordings in the Default Storage Group, allowing LiveTV recordings to be stored on their own filesystem. This is similar to the old MythTV method which used a RingBuffer for LiveTV.
To remove a storage group from the list, highlight it and press the 'D' key. This does not "delete" the files, it only removes the directory as a storage location. If the storage group you wish to remove is not empty (which is likely the case as MythTV balances the load), you can move these files from one storage group to another with a regular filesystem move. The next time MythTV tries to access these files, it will automatically search all available storage groups to locate them.
Usage information for all Storage Group directories is visible on the mythfrontend status screen as well as the mythbackend status webpage. MythTV is smart enough to determine which directories are on shared filesystems so it should not count free or used space multiple times if more than one directory is on the same filesystem.
- 1 Setting up Storage Groups
- 2 Default Storage Group
- 3 Directory Structure
- 4 Slave Backend Overrides
- 5 Expiration and Free Space settings
- 6 MythVideo
- 7 Storage Group Disk Scheduler
Setting up Storage Groups
You should only define Storage Groups on the Master Backend (ceryainly for 99.9% of users).
A Storage Group is simply a mapping between a user-friendly name ("Default", "Movies", "Archive," ...) and a list of directory paths. This does not say anything about them containing information about the host that contains those directories.
Therefore, when you *define* a Storage Group (which can only be done on the master backend) and set up a list of directories, that Storage Group exists on *all* hosts. Note, also, that I did not say that all the directories in the directory list must exist on all the hosts--only that the Storage Group exists on all hosts (the /mapping/ between a user-friendly name and a list of directory paths exists on all hosts).
When you use mythtv-setup on any host other than the master backend to edit Storage Groups, all you're doing is overriding the directory list for an already-defined Storage Group--you're saying, "The list I gave on the master backend is wrong." (So, really, you should just fix the list on the master backend...)
This is true even in the situation where a slave with it's own storage.
You do not have to run NFS or Samba on my network.
Consider the situation where a Master Backend has 4 file systems it's using for recordings and a Slave Backend that has 3 file systems it's using for recordings.
The Master Backend has no access to the file systems on the Slave Backend and vice versa.
Regardless of this, *only* define Storage Groups on the master backend. In this example, the Default Storage Group has a directory list:
/srv/mythtv/tv/a/recordings /srv/mythtv/tv/b/recordings /srv/mythtv/tv/c/recordings /srv/mythtv/tv/d/recordings
where a, b, c, d are mount points and actual recordings are put into subdirectories so that the backend(s) can tell if the file system is mounted or not (otherwise, if a, b, c, d were put in the directory list, it would write to the directory whether or not the file system is mounted--meaning it would fill up the parent file system, which is my root file system).
As mentioned before, in this example the Master Backend has 4 file systems and the Slave Backend has only 3 file systems. This isn't a problem. On Master Backend, a different file system is mounted at a, b, c, and d. On the Slave Backend, we not (yet) mount a file system at d. Therefore, the Slave Backend has directories:
/srv/mythtv/tv/a/recordings /srv/mythtv/tv/b/recordings /srv/mythtv/tv/c/recordings /srv/mythtv/tv/d
Since there's no file system mounted at d, there's no /srv/mythtv/tv/d/recordings (which is the one the Default Storage Group uses), so MythTV sees that the directory doesn't exist and doesn't use it (it simply ignores it) on the Slave Backend.
This means that we have no Storage Groups/Storage Group directory entries defined for any host other than the Master Backend.
There is no reason to ever touch Storage Groups on any host other than the master backend.
If you prefer, you can do things slightly differently. Let's say you don't want to "re-use" directory names on your different backends because you may find it confusing to have 2 different file systems using the same directory path. In that case, the Storage Group definition on the master backend should contain the "union" of directory paths used by all backends. So, it may have something like:
/srv/mythtv/tv/master/a/recordings /srv/mythtv/tv/master/b/recordings /srv/mythtv/tv/remote1/c/recordings /srv/mythtv/tv/remote1/d/recordings
(where picking directory names is the hard part, but the above gives you the idea). Again, a, b, c, d are all mount points and recordings is a directory existing within the file system mounted there. Then, since there's no /srv/mythtv/tv/master/a/recordings directory on the remote backend, and no /srv/mythtv/tv/remote/c/recordings directory on the master backend, MythTV will know to ignore those directories on the hosts in question.
For the situation where a Slave Backend has no storage and you use NFS or Samba, you may have all your storage on the master backend or some NAS or whatever. If so, just list all the directories in the Storage Group definition on the master backend and other hosts will use the same list of directories. Since all recording is done to "local" file systems, the remote file systems must be mounted at the same absolute path on all hosts.
If you feel you must have the TV directories NFS mounted on each MythTV system, you can, but you need to choose an appropriate Storage Groups scheduler.
There's no need for any backend to have access to any file system other than the file systems to which that backend records TV. All MythTV recordings access (recording, playback, etc.) is performed through the recording host's mythbackend. The only exception is when you enable the setting:
- Master Backend Override
If enabled, the master backend will stream and delete files if it finds them in the video directory. Useful if you are using a central storage location, like a NFS share, and your slave backend isn't running.
This is only meant to be used when you have a single file server containing all file systems and you want to be able to stream/playback recordings from remote backends even when the remote backends are shut down or when *all* of your storage exists on the master backend and remote backends record to it over the network--so without this setting, streaming from the recording backend would mean the recording backend reads a recording over NFS and then sends it across the network to the frontend. Instead, enabling the override allows the master backend to read it from local storage and send it across the network to the frontend (so, only one trip across the network).
- Only set Storage Group directory lists on the master backend. (Never create Storage Group directory list overrides on the remote backends.)
- Once properly configured (with no SG overrides on the remote backends), only manage Storage Groups using mythtv-setup on the master backend
- Make the master backend's Storage Group directory list contain the union of all directories on all hosts--whether those directories exist on all hosts or not.
- Directories in a Storage Group directory list need not exist on any host (those that don't exist on a particular host are simply ignored).
- If you don't want a directory used on a particular host make sure that directory doesn't exist on that host.
- Never put a mount point into a Storage Group directory list--always put some subdirectory under the mount point so MythTV will know to ignore the (non-existent) directory entry if the file system fails to mount properly.
- If you think you need to override a SG directory list on a remote backend or remote frontend, you're making things more difficult than they need to be.
Default Storage Group
The Default Storage Group is special. Users don't define it. It always exists because it's defined in code.
If you put a mount point directory name into an SG directory list, MythTV cannot tell when the file system is not mounted (the mount point and the mounted file system directory are identical), so it will gladly write to the parent file system (and possibly fill it up--which is very bad if the parent file system is the system's root file system). Instead, always create one or more subdirectories within your file systems and put a subdirectory path into the SG directory list. Then, when the file system isn't mounted, the directory doesn't exist, so MythTV will ignore it.
Slave Backend Overrides
While it is possible to override a Storage Group in a slave back end, it is not a good idea to do so. Only define your Storage Group once on the master backend and don't override the directory list on the remote backends.
To delete a Storage group Override that has been set up on a Slave backend, go to mythtv-setup, then in Storage Directories. Highlight the Default Storage Group and then hit your DELETE key. A pop-up appears that says, "Delete 'Default' Storage Group? (from remote hosts)". When you say, "Yes, delete group," it will delete the Default Storage Group directory list overrides from your configuration.
The Default Storage Group and the directory list on the master backend will remain. If you want to edit the Default Storage Group directory list on the Master Backend, you have to do it on the Master Backend.
Expiration and Free Space settings
The "global" setting "Extra Disk Space," specifies the amount of extra space to keep available on each filesystem, not a total amount of extra space to keep available across all filesystems. Therefore, if the "Extra Disk Space" is set to 3GB, it should be keeping 3GB available on all filesystems (to which Myth is recording at any particular time).
That parenthetical can actually be very important. If Myth is not recording to a filesystem, it does not autoexpire programs to make room for the data that some other process is writing to that filesystem. Therefore, until a recording starts and is being written to the filesystem in question, no programs will be expired from that filesystem.
Videos in Myth can be stored on the backend and streamed to the frontend without a NFS or Samba mount.
- Adding new frontends means zero configuration for videos, recordings, and metadata images.
- Can dynamically add space to your video library without using RAID or LVM. Loss of one drive does not mean loss of the entire library.
- Can spread video hosting across many/all backends.
- No need to set up network mounts of any kind.
- External Video Players (mplayer, xine, VLC) will not work with videos hosted on an SG.
Setting Up Video/Image Storage Groups
- On the backend to host the videos, stop the backend process and run mythtv-setup.
- In Storage Group configuration, set up directories for each of the following: Videos, Trailers, Fanart, Banners, Screenshots, and Coverart.
- Optional Step: If you would like to use a combination of Storage Group and locally hosted video, you can do the following. On the frontends, go to Utilities/Setup->Setup->Media Settings->Video Settings->General. Change "Directories that hold videos" to point at a directory that is not the same as the one the Storage Group points at. If the local video setting and the Storage Group setting point at the same path, MythVideo will prefer the Storage Group path and ignore the local one.
- Enter MythVideo. Press the "M" (MENU) key and choose "Scan For Changes."
Metadata Grabbing with Storage Group Files
Doing a metadata lookup when using Storage Groups will download the images to the Fanart/Banner/Screenshot/Cover Storage Groups on the backend that hosts the video in question, so they must be defined. If they are not defined, the images will be downloaded to the Video Storage Group instead.
Mass Metadata Grabbing
MythVideo includes Jamu, a script for automating mass metadata grabbing of your entire mythvideo library. It supports Storage Groups, TVDB, TMDB, and everything the Myth interface supports.
Storage Group Disk Scheduler
The Storage Group Disk Scheduler provides 3 different implementations, which can be selected with the mythtv-setup General setting "Storage Group Disk Scheduler".
Balanced Free Space
The only criteria considered here is free space, so if you have 5 tuners and 2 filesystems, one on brand new shiney empty disk drive, and 5 things are recording at the same time, they'll all go to the same disk since it has the most free space.
Balanced Disk I/O
The only criteria considered here is the 'weight' on the filesystem at the time. Items included in the weight are in-progress recordings, playback, commercial flagging, transcoding, etc.. The disk with the least I/O activity (weight) will get the recording. In this case, with the same 5 tuners and 2 filesystems mentioned above, you'd have 3 tuners recording to one filesystem and 2 tuners going to the other, but there is no preference as to which filesystem gets the 3 and which gets the 2.
This is the original Storage Group balancing method. This method is no longer the default. This method uses a combination of free space and disk I/O to determine where to put the next recording.
Sorting is done using the following criteria:
If fs1 is local and fs2 is not then if fs1 has less weight use fs1 else use fs2 else if fs1 and fs2 are either both local or both remote then if fs1 or fs2 has a lower weight, use the one with the lowest weight else if both have equal weights then use the one with the most free space else if fs is not local and fs2 is then if fs1 has less weight use fs1 else use fs2
Myth uses weights to determine what filesystem/directory to record to. The following default values are used:
SGweightPerRecording = 10 SGweightPerPlayback = 5 SGweightPerCommFlag = 5 SGweightPerTranscode = 5 SGweightLocalStarting = -1.99 * SGweightPerRecording SGmaxRecOverlapMins = 3
The drive with the lowest weight is used first. In a tie, the drive with the highest amount of free disk space is used first.
Each Storage Group Directory has its own weight, but Myth is smart enough to know if several directories are on the same shared filesystem. When a weight is applied to a directory that is in use, it is applied to all directories on that filesystem because we are trying to spread the load out across filesystems, not directories on the same filesystem.
The starting weight of all drives is 0. Local filesystems/directories are then offset by SGweightLocalStarting, so by default they are then at -19 because SGweightPerRecording defaults to 10. This makes it so that local drives are preferred over remote drives. A local drive has to have an effective weight of 20 before a remote drive will be used to store a new recording. If you do not have any playback going on, this would mean that you'd have to have 2 recordings going to a local drive before a remote drive would get used. If you have 2 local drives and 1 remote drive, you'd have to have 4 recordings going on locally before a remote drive would get used.
So, if you have less than 4 tuners and 2 local drives, then the default should already give you what you are looking for. If you have more than 4 tuners or want to guarantee that all recordings go to the local drives unless they fill up, you can do so using the undocumented SGweightPerDir setting.
In order to make Myth not use a filesystem/directory, we need to artificially inflate the starting weight for that directory. We can do this by insertting a setting in the database.
The key for the setting is SGweightPerDir:HOSTNAME:DIRECTORY. The hostname is the hostname that sees the directory. So if the directory is actually on the fileserver which is server1 but is mounted via NFS on server2 which is running mythbackend, we'd use server2 here. The directory is the local path on HOSTNAME, so you'd put /mnt/video or whatever you use here for the remotely mounted directory.
The value that we put in this setting will be applied as an offset to the initial weight for this directory. You can play it safe by setting this to something large like 99 or 100 or even higher. So in the example here, the actual setting key would be:
You need to run the following SQL to insert that into the settings table:
INSERT settings (value, data, hostname) VALUES ("SGweightPerDir:server2:/mnt/video", 99, "server2");
So, now when the Storage Groups scheduling code runs, /mnt/video will start out with a weight of 0. Since it is a remote drive, the only offsets that will be initially applied will be the one we specified in the settings table with the SGweightPerDir setting. Your two local directories would each end up starting out at -19 and the remote directory would start out at +99. So unless you have a huge number of recordings and playbacks going on on each of your local drives, the remote directory will never be used unless the locals fill up.
If you run mythbackend with the "-v schedule,file" option, you can see the weights as they are applied and the logs will show you why Myth chose one directory over another when determining where to put the next recording.