[mythtv-users] Storage Groups, free space, (and auto-expire?)

[Michael T. Dean]

On 10/24/2007 01:33 PM, Brian Guilfoos wrote:
> Michael T. Dean wrote:
>   
>> Note, though, that if your recording directories are not accessible to
>> all backends, you /must/ modify the hostname when you move a recording
>> from a directory on one hostname to a directory on another.
>>     
> Yeah - I'm still mentally processing all the possible setups that would
> have to be accommodated by robust code.
>
> Aside: it seems that it would be possible to have a Storage Group
> directory that is only mounted on a slave backend - how does playback of
> those recordings happen on a frontend that doesn't mount that share?
>   

It's streamed from the backend.  That's exactly why you must update the
hostname if you move it to a directory that's not universally
accessible.  Otherwise, your frontend will ask the wrong backend for the
recording and it won't find it, so it will tell you the file doesn't exist.

This is exactly how I have my system setup (2 dedicated backends, each
with their own storage and no network shares).  I like this approach
because my Myth box can continue recording even in the event of a
network failure (though, if it's the backend for which the database is
remote, it does screw up the seektable, but that can be fixed) and I
don't have a single-point-of-failure in a file server.

>>> As I only have one Storage Group, I could easily automate what I'm doing
>>> manually every few days now.  If you've got multiple storage groups, I
>>> still think you'd need to query the database to make sure you only move
>>> recordings around within that SG.
>>>       
>> No.  If a recording is not found in the directories of the specified
>> storage group, Myth punts and looks at all (locally accessible) storage
>> groups (directories for the current backend).   So, there's no need to
>> change the storage group or to be limited to moving only within storage
>> groups.
>>     
> OK, so it seems that you'd take a tiny performance hit due to the lookup
> time.  That's probably the least-important issue, when compared to
> having all possible drives available for load-balancing and ensuring
> that auto-expires "make sense".
>   

Right.  I'd say if you're using an external script (and not using the
Perl bindings for that script), don't change the storage group.

If you're using the Perl bindings or, better, doing the mods to
autoexpire/mythfrontend, then go ahead and change the storage group as
appropriate--if you can.  It's possible that a recording directory may
appear in multiple storage groups, so in that case, don't change the
storage group.  I.e. if a user has a "Battlestar Galactica" episode in
the "SciFi" storage group (/video/SciFi directory) and you decide to
move it without user interaction to /video/extra (which appears in the
"Default" and "Archive" and "Favorites" storage groups), leave the
storage group set to "SciFi"--even though it doesn't contain the
/video/extra directory.  If you're doing it from mythfrontend
(interactively), ask the user to specify which of the storage groups to
use (once you've determined which include the location to which it's
being moved).

Note, also, that due to these "naming" issues (Is BSG really a
Favorite?  OK, bad example, but you get the idea. ;) and even some
system configuration issues (users may have set up storage groups for
performance according to their network shares/multiple
disk/backup/whatever plans), it may even make sense--if writing the
"auto-archive-instead-of-expire" functionality--to allow the user to
specify which directories can be used for moving around recordings.  One
simple approach for this would be to allow the user to set up an
"Archive" storage group which contains a list of all the directories
allowed for auto-archiving (and, this would also solve the problem of
changing the storage group as we could always just change it to
"Archive").  Just make sure if you do this that the display name is
translated, but the DB name is not translated.

Note, also, the user may override the storage group specified on the
master backend (i.e. the master backend has /video/recordings1 and
/video/recordings2 in "Default", but the slave backend has only
/video/recordings1--where all three are probably separate non-shared
filesystems).  At this point, I can't think of any issues this could
cause, but I'm mentioning it just in case you find one as you dig into
the code.

>>>   It seems you could still do that in a
>>> bash script with a "mysql --execute" command.  Again, I'm willing to
>>> work on something more robust if there is interest.
>>>       
>> If you write a script that you wish to be included in the MythTV source
>> distribution, please do so using the Perl bindings.  They provide all
>> the capability you need for finding information about the recordings and
>> the storage groups--you'd basically just have to provide the code to do
>> the actual moving of the file and the logic for when to do it.
>>     
> I had never looked at the source for those contrib scripts, and holy
> smokes are they good.  It never occurred to me that somebody might have
> made Perl bindings to provide reusable MythTV-specific code.  Oops.
>   

Yes.  I recently made myth_upcoming_recordings.pl and
myth_recent_recordings.pl and am now a huge believer in the bindings. 
I've been grumbling since the bindigns were introduced about how we need
to port the legacy contrib scripts to use them, but I said I would never
do it for the ones I don't use (myth.find_orphans.pl and
myth.rebuilddatabase.pl and mythname.pl, especially).  Now that I've
fallen in love with the bindings, though, I'm likely to rewrite at least
those three into a single much-more-capable than the sum of the
originals script.

>> See http://www.gossamer-threads.com/lists/mythtv/commits/239035#239035
>> for the best info on storage groups.  See, especially, "<spoiler, ie.
>> not in the current code>" and feel free to implement that, instead, for
>> a /much/ more capable solution.  Doing that /and/ adding GUI control to
>> the frontend for moving/archiving recordings would be a very nice
>> addition (and save someone--likely Chris Pinkham) a lot of time.  See,
>> also http://www.gossamer-threads.com/lists/mythtv/users/256679#256679
>> and http://www.gossamer-threads.com/lists/mythtv/users/269092#269092 for
>> a way to influence the choice of dirs used.
>>     
> I had read that spoiler before, and it had slipped my mind while
> thinking about this particular problem.  (I'm already using those
> per-drive adjustments to bias the scheduler against using the
> worst-performing of my network mounts.)
>
> So it seems to me that it'd be VeryUseful(tm) if a patch can be provided
> against trunk that would provide a "bool MoveThis(file, [destination])"
> functionality (and determine the "optimal" destination automatically if
> it's not provided).  That could allow someone to "fix" AutoExpire to try
> and move files first, and then delete them only if moving fails.
>   

Sounds good.  Also, remember that this may require having one backend
stream the file to another--i.e. if the user didn't set up network
shares.  This is much better than any "external" script approach due to
network permissions/authentication/authorization issues that would be
present (since mythbackend is running on both, we've got the network
permissions to do this).

> Second, a routine that could be run periodically that would attempt to
> balance free space across storage devices (that could reuse MoveThis())
> would be handy.
Probably true.  Might make sense to do this "early" rather than attempt
to do it when the space is needed (as it takes much longer to move a
file across a network than to delete the file, we could run out of space
before the move is complete).

>   Finally, adding a way for users to manually move files
> via the frontend GUI would be useful.
>   

Definitely.

> That specification still raises a few questions.  First, what should be
> the algorithm for determining the "optimal" destination for a recording?
> 1) Move the file to whichever storage device has the most free space,
> provided the file will fit, AND assuming the move won't make the
> destination device have less free space than the source device.  Ignore
> storage groups.
> 2) Move the file to whichever storage device in the Storage Group has
> the most space, provided the file will fit, AND assuming the move won't
> make the destination device have less free space than the source device.
>  If this fails, fall back to mode 1.
> 3) Some esoteric algorithm I haven't thought of that handles some use
> case I also haven't thought of.
>
> Second, where should "MoveThis()" or whatever it ends up being called be
> bolted on?  ProgramInfo?  StorageGroup?
>   

I'll let Chris Pinkham answer these questions.  Storage Groups (and,
incidentally, the housekeeper--which is where the "periodic" checks
would be performed) are both his babies.  I am liking the "Archive"
storage group idea a lot more, now that I've had a chance to think about
it.  That could even be the way "auto-archive" is enabled--with the
storage group it's enabled, but without it, it's not (though if doing
so, you may want the name to be something the user is less likely to
pick on his/her own--like "Auto Archive").

> I appreciate the feedback.  I think I'm convinced this is useful enough
> for me to bother with doing.  I wouldn't mind some additional input from
> some of the other "power users" utilizing Storage Groups.

Cool.  Definitely talk to Chris Pinkham.  He'll probably respond to
this/my other message and correct any lies I've told you :) and provide
his thoughts on the best approach.  He also hangs out on IRC sometimes,
so you could look him up there.

Thanks, and good luck/happy coding.
Mike