Storage Groups Weighting

I recently (early March 2007) asked on the mailing list if there was a way of playing with the weights given each of the storage groups so that local filesystems are preferred over remote file systems. Turns out that is already how it works!

Here is the response I received from Chris Pinkham. I'm pasting it here so that other's might find it useful as well.

* On Tue Mar 06, 2007 at 04:53:31PM +1030, Darryl Ross wrote:
> > Firstly, thanks heaps to the devs and testers who have got Storage
> > Groups working really well. It's simplified my setup hugely!

Glad it's working well for you.  :) 

> > Just wondering if there is any capability (or plans) or specify a
> > preference to how the storage groups are utilised?

There is limited ability for this via some undocumented settings, but
there is no GUI to set them up.

> > At the moment I believe Myth will record to the drive with the most
> > amount of free space if there's only a single recording and will spread
> > simultaneous the recordings across all available drives to reduce head
> > movements.

It's a bit more complicated than that.  Myth uses weights to determine
what filesystem/directory to record to.  The following default values are
used:

   SGweightPerRecording  = 10
   SGweightPerPlayback   =  5
   SGweightPerCommFlag   =  5
   SGweightPerTranscode  =  5
   SGweightLocalStarting = -1.99 * SGweightPerRecording
   SGmaxRecOverlapMins   =  3

The drive with the lowest weight is used first.  In a tie, the drive with
the highest amount of free disk space is used first.

Each Storage Group Directory has its own weight, but Myth is smart enough
to know if several directories are on the same shared filesystem.  When
a weight is applied to a directory that is in use, it is applied to all
directories on that filesystem because we are trying to spread the load
out across filesystems, not directories on the same filesystem.

The starting weight of all drives is 0.  Local filesystems/directories
are then offset by SGweightLocalStarting, so by default they are then
at -19 because SGweightPerRecording defaults to 10.  This makes it so
that local drives are preferred over remote drives.  A local drive has
to have an effective weight of 20 before a remote drive will be used to
store a new recording.  If you do not have any playback going on, this
would mean that you'd have to have 2 recordings going to a local drive
before a remote drive would get used.  If you have 2 local drives and
1 remote drive, you'd have to have 4 recordings going on locally before
a remote drive would get used.

So, if you have less than 4 tuners and 2 local drives, then the default
should already give you what you are looking for.  If you have more than
4 tuners or want to guarantee that all recordings go to the local drives
unless they fill up, you can do so using the undocumented SGweightPerDir
setting.

In order to make Myth not use a filesystem/directory, we need to
artificially inflate the starting weight for that directory.  We can do
this by insertting a setting in the database.

The key for the setting is SGweightPerDir:HOSTNAME:DIRECTORY.  The
hostname is the hostname that sees the directory.  So if the directory
is actually on the fileserver which is server1 but is mounted via NFS on
server2 which is running mythbackend, we'd use server2 here.  The
directory is the local path on HOSTNAME, so you'd put /mnt/video or
whatever you use here for the remotely mounted directory.

   SGweightPerDir:server2:/mnt/video

The value that we put in this setting will be applied as an offset to
the initial weight for this directory.  You can play it safe by setting
this to something large like 99 or 100 or even higher.  So in the
example here, the actual setting key would be:

You need to run the following SQL to insert that into the settings table:

   INSERT settings
      (value, data, hostname)
   VALUES
      ("SGweightPerDir:server2:/mnt/video", 99, "server2");

So, now when the Storage Groups scheduling code runs, /mnt/video will
start out with a weight of 0.  Since it is a remote drive, the only
offsets that will be initially applied will be the one we specified in
the settings table with the SGweightPerDir setting.  Your two local
directories would each end up starting out at -19 and the remote
directory would start out at +99.  So unless you have a huge number of
recordings and playbacks going on on each of your local drives, the
remote directory will never be used unless the locals fill up.

If you run mythbackend with the "-v schedule,file" option, you can
see the weights as they are applied and the logs will show you why
Myth chose one directory over another when determining where to put
the next recording.