Difference between revisions of "Mythfs.pl"

From MythTV Official Wiki
Jump to: navigation, search
(mythtvfs-perl)
(mythfs-perl)
 
(20 intermediate revisions by one user not shown)
Line 7: Line 7:
 
|S26=yes}}
 
|S26=yes}}
  
=mythtvfs-perl=
+
=mythfs-perl=
  
This is a FUSE filesystem for MythTV (www.mythtv.org) which mounts the recordings on a Mythtv backend as files and directories on the same or different machine. It is distinguished from [[Mythtvfs]] and [[Mythfs.py]] in that it works via the Myth 0.25 REST API. Hence the recordings do not need to be on a shared
+
This is a FUSE filesystem for MythTV (www.mythtv.org).  It uses the
 +
Myth 0.25 API to mount the TV recordings known to a MythTV master backend onto a virtual filesystem on the client machine for convenient
 +
playback with XBMC or other video tools. Because it uses the
 +
MythTV network protocol, the recordings do not need to be on a shared
 
NFS-mounted disk, nor does the Myth database need to be accessible
 
NFS-mounted disk, nor does the Myth database need to be accessible
from the client. It also uses threading, providing good performance when accessing many files.
+
from the client. As new recordings are added (and old ones are deleted), the filesystem will automatically
 +
update to reflect the changes.
 +
 
 +
Backends earlier than 0.25 are '''not''' supported.
  
 
==Installation==
 
==Installation==
Line 18: Line 24:
 
distribution:
 
distribution:
  
  $ '''git clone git://github.com/lstein/mythfs-perl.git'''
+
  $ <b> git clone git://github.com/lstein/mythfs-perl.git</b>
  $ '''cd mythfs-perl'''
+
  $ <b>cd mythfs-perl</b>
  $ '''./Build.PL'''
+
  $ <b>./Build.PL</b>
  $ '''./Build test'''
+
  $ <b>./Build test</b>
  $ '''sudo ./Build install'''
+
  $ <b>sudo ./Build install</b>
  
 
If you get messages about missing dependencies, run:
 
If you get messages about missing dependencies, run:
  
  $ '''./Build installdeps'''
+
  $ <b>./Build installdeps</b>
  
and then "sudo ./Build install".
+
and then "sudo ./Build install". See [[#Fuse Notes]] if you get
 +
Fuse-related errors when attempting to install dependencies.
 +
 
 +
For best performance, Perl must have been compiled with support for
 +
IThreads. In addition, you will need at Fuse version 0.15 to run
 +
correctly under Perl versions >= 5.14.  Threading will be automatically
 +
disabled if not available. See [[#Fuse Notes]] for additional information.
  
 
==Usage==
 
==Usage==
Line 35: Line 47:
 
a local filesystem named "/tmp/mythfs" use this command:
 
a local filesystem named "/tmp/mythfs" use this command:
  
  $ '''mythfs.pl MyHost /tmp/mythfs'''
+
  $ <b>mkdir /tmp/mythfs</b>
 +
$ <b>mythfs.pl MyHost /tmp/mythfs</b>
  
The script will fork into the background and should be stopped with
+
After a pause to ensure that the backend is reachable, the script will fork into the background and operate silently. The mounted /tmp/mythfs directory will contain a series of
fusermount. The mounted /tmp/mythfs directory will contain a series of
+
 
human-readable recordings organized by title (directory) and subtitle
 
human-readable recordings organized by title (directory) and subtitle
(file). In the case that there is no subtitle, then the recording file
+
(file).  
will be placed into the top level of the directory.
+
  
To unmount:
+
To unmount the filesystem:
  
  $ '''fusermount -u /tmp/mythfs'''
+
  $ <b>fusermount -u /tmp/mythfs</b>
  
Note do NOT try to kill the mythfs.pl process. This will only cause a
+
NOTE: Do NOT try to kill the mythfs.pl process. This will only cause a
 
hung filesystem that needs to be unmounted with fusermount.
 
hung filesystem that needs to be unmounted with fusermount.
  
There are a number of options that you can pass to mythfs.pl, the most
+
There are a number of options that you can pass to mythfs.pl,
useful of which is "-o allow_other". If this is present, then others
+
including the ability to customize the filesystem layout and set the
on the system (including root) can see the mounted filesystem. Call
+
interval that the backend is checked for new and deleted
with the -h option for more help.
+
recordings. Call mythfs.pl with the -h option for the complete help
 +
text.
  
==Understanding the Directory Layout==
+
==Local Recordings==
 +
 
 +
The default behavior of this filesystem is to use the Myth API to
 +
stream recordings across the network when you attempt to read from
 +
them. This is done in an efficient way that fetches just the portion
 +
of the file you wish to read. However, if the underlying recording
 +
files are directly accessible (either in a regular director or via an
 +
NFS mount), you can get better performance by passing mythfs.pl the
 +
--mountpt option with the path to the directory in which the
 +
recordings can be found. The filesystem will then be set up as a set
 +
of symbolic links that point from a human readable file name to the
 +
recording file.
 +
 
 +
The main advantage of creating symbolic links is that NFSv4 can be
 +
significantly faster than the backend streaming protocol -- about 1.6X
 +
in my informal tests. The main limitation is that this mode does not
 +
understand storage groups, so all recordings need to be located in a
 +
single storage group in a locally-accessible directory. However if a
 +
recording file is not found in local directory, then mythfs.pl will
 +
fall back to the streaming protocol, so the recording is accessible one way or another.
 +
 
 +
==The Default Directory Layout==
  
 
Recordings that are part of a series usually have a title (the series
 
Recordings that are part of a series usually have a title (the series
Line 61: Line 94:
 
using a two-tier directory structure in which the top-level directory
 
using a two-tier directory structure in which the top-level directory
 
is the series name, and the contents are a series of recorded
 
is the series name, and the contents are a series of recorded
episodes.
+
episodes. The corresponding pattern (as described in the next section)
 +
is "%T/%S".
  
 
For recordings that do not have a subtitle, typically one-off movie
 
For recordings that do not have a subtitle, typically one-off movie
 
showings, the recording is placed at the top level.
 
showings, the recording is placed at the top level.
  
In all cases, the time the recorded was started is attached to the
+
If needed to ensure uniqueness, the channel number and the time the recorded was started is attached to the
filename, along with an extension indicating the recording type (.mpg
+
filename. In all cases, an extension is attached to indicate the recording type (.mpg
 
or .nuv). The file create and modification times correspond to the
 
or .nuv). The file create and modification times correspond to the
 
recording start time. For directories, the times are set to the most
 
recording start time. For directories, the times are set to the most
recent recording contained within the directgory.
+
recent recording contained within the directory, and the directory size is set to the number
 +
of recordings contained within it (recursively).
  
 
Here is an example directory listing:
 
Here is an example directory listing:
  
<pre>
 
 
  % <b>ls -lR  /tmp/mythfs</b>
 
  % <b>ls -lR  /tmp/mythfs</b>
 
  total 35
 
  total 35
  -r--r--r-- 1 lstein lstein 12298756208 Dec 30 00:00 A Funny Thing Happened on the Way to the Forum 2012-12-30-00:00.mpg
+
-r--r--r-- 1 lstein lstein 21262807708 Jan 11 13:10 007 Licence To Kill.mpg
  -r--r--r-- 1 lstein lstein 14172577964 Dec 25 16:00 A Heartland Christmas 2012-12-25-16:00.mpg
+
  -r--r--r-- 1 lstein lstein 12298756208 Jan 11 13:10 A Funny Thing Happened on the Way to the Forum.mpg
  dr-xr-xr-x 1 lstein lstein          5 Mar 11 03:00 Alfred Hitchcock Presents
+
  -r--r--r-- 1 lstein lstein 14172577964 Jan 11 13:10 A Heartland Christmas.mpg
  dr-xr-xr-x 1 lstein lstein          8 May  2 00:00 American Dad
+
-r--r--r-- 1 lstein lstein 17591877032 Mar 10 14:24 A Knight_s Tale.mpg
 +
  dr-xr-xr-x 1 lstein lstein          5 Mar 11 03:41 Alfred Hitchcock Presents
 +
  dr-xr-xr-x 1 lstein lstein          8 May  4 01:20 American Dad
 
  ...
 
  ...
 
+
 
  /home/lstein/Myth/Alfred Hitchcock Presents:
 
  /home/lstein/Myth/Alfred Hitchcock Presents:
 
  total 3
 
  total 3
  -r--r--r-- 1 lstein lstein 647625408 Dec 25 15:30 Back for Christmas 2012-12-25-15:30.mpg
+
  -r--r--r-- 1 lstein lstein 647625408 Dec 25 15:30 Back for Christmas.mpg
  -r--r--r-- 1 lstein lstein 647090360 Dec  7 00:00 Dead Weight 2012-12-07-00:00.mpg
+
  -r--r--r-- 1 lstein lstein 647090360 Dec  7 00:00 Dead Weight.mpg
  -r--r--r-- 1 lstein lstein 660841056 Mar 11 03:00 Rose Garden 2013-03-11-03:00.mpg
+
  -r--r--r-- 1 lstein lstein 660841056 Mar 11 03:00 Rose Garden.mpg
  -r--r--r-- 1 lstein lstein 647524452 Dec 25 00:00 Santa Claus and the 10th Ave. Kid 2012-12-25-00:00.mpg
+
  -r--r--r-- 1 lstein lstein 647524452 Dec 25 00:00 Santa Claus and the 10th Ave. Kid.mpg
  -r--r--r-- 1 lstein lstein 649819932 Dec 27 00:00 The Contest of Aaron Gold 2012-12-27-00:00.mpg
+
  -r--r--r-- 1 lstein lstein 649819932 Dec 27 00:00 The Contest of Aaron Gold.mpg
 
+
 
  /home/lstein/Myth/American Dad:
 
  /home/lstein/Myth/American Dad:
 
  total 4
 
  total 4
  -r--r--r-- 1 lstein lstein 3512038152 Apr 24 00:00 Flirting With Disaster 2013-04-24-00:00.mpg
+
  -r--r--r-- 1 lstein lstein 3512038152 Apr 24 00:00 Flirting With Disaster.mpg
</pre>
+
  
==Troubleshooting==
+
The size of directories corresponds to the number of recordings (not
 +
counting subdirectories) contained within it.The modification time of
 +
directories is the start time of the most recent recording contained
 +
within it.
  
 +
==Customizing the Directory Listing==
  
Your Perl must have been compiled with IThreads in order for this
+
You may customize the directory listing by providing a pattern for
script to work. To check if this is the case. you may run:
+
naming each recording using the -p option. For example:
  
  ''' $ perl -V | grep useithreads'''
+
  $ mythfs.pl -p '%C/%T:%S (%od-%ob-%oY)' mythbackend ~/Myth
    useithreads=define, usemultiplicity=define
+
  
 +
This will create filenames that look like this:
 +
 +
Sitcom/The Simpsons:The Food Wife (13-Nov-2011).mpg
 +
 +
Patterns contain a combination of constant strings plus substitution
 +
patterns consisting of the "%" sign plus 1 to three characters. A
 +
slash will be interpreted as a directory level: multiple levels are
 +
allowed.
 +
 +
Commonly-used substitution patterns are:
 +
 +
    %T  = Title (show name)
 +
    %S  = Subtitle (episode name)
 +
    %C  = Category
 +
    %cn  = Channel: channel number
 +
    %cN  = Channel: channel name
 +
    %y  = Recording start time:  year, 2 digits
 +
    %Y  = Recording start time:  year, 4 digits
 +
    %m  = Recording start time:  month, leading zero
 +
    %b  = Recording start time:  abbreviated month name
 +
    %B  = Recording start time:  full month name
 +
    %d  = Recording start time:  day of month, leading zero
 +
    %h  = Recording start time:  12-hour hour, with leading zero
 +
    %H  = Recording start time:  24-hour hour, with leading zero
 +
    %i  = Recording start time:  minutes
 +
    %s  = Recording start time:  seconds
 +
    %a  = Recording start time:  am/pm
 +
    %A  = Recording start time:  AM/PM
 +
 +
A full list of patterns can be obtained by running "mythfs.pl -p
 +
help".
 +
 +
Patterns are largely compatible with the excellent mythlink.pl
 +
(http://www.mythtv.org/wiki/Mythlink.pl) script, but there are a small
 +
number of enhancements, such as the ability to generate the month
 +
name. Also, the patterns that generate the month name without a
 +
leading zero are not supported.
 +
 +
You may wish to use a delimiter to separate fields of the recording
 +
name, for example "%T:%S" to generate "Title:Subtitle". Occasionally a
 +
recording field is empty, leading to names like "The Wild
 +
Ones:.mpg". To avoid this, pass the --trim option with the delimiter
 +
you use, and dangling/extra delimiters will be trimmed:
 +
 +
$ mythfs.pl -p '%T:%S' --trim=':' backend /tmp/myth
 +
 +
If after applying the pattern to a recording the resulting path is not
 +
unique, then this script will uniqueify the path by appending to it
 +
the channel number and recording start time, for example:
 +
 +
Masterpiece Classic/Downtown Abbey_17_1-2013-02-11T02:00.mpg
 +
Masterpiece Classic/Downtown Abbey_17_1-2013-03-10T06:00.mpg
 +
 +
==Caching==
 +
 +
New and updated recordings will appear in the filesystem after a
 +
slight delay due to the manner in which the script caches the
 +
recording list. By default the backend is only checked for updates
 +
every 10 minutes, but you can adjust this using the --cachetime
 +
option, which takes the interval in minutes at which the system
 +
checks for new and updated recordings.
 +
 +
For example, this command will reduce the update interval to 2
 +
minutes:
 +
 +
  $ <b>mythfs.pl MyHost --cachetime=2 /tmp/mythfs</b>
 +
 +
==Fuse Notes==
 +
 +
For best performance, you will need to run this filesystem using a
 +
version of Perl that supports IThreads. Otherwise it will fall back to
 +
non-threaded mode, which will introduce occasional delays during
 +
directory listings and have notably slower performance when reading
 +
from more than one file simultaneously.
 +
 +
If you are running Perl 5.14 or higher, you *MUST* use at least 0.15
 +
of the Perl Fuse module. At the time this was written, the version of
 +
Fuse 0.15 on CPAN was failing its regression tests on many
 +
platforms. I have found that the easiest way to get a fully
 +
operational Fuse module is to clone and compile a patched version of
 +
the source, following this recipe:
 +
 +
$ <b>git clone git://github.com/isync/perl-fuse.git</b>
 +
$ <b>cd perl-fuse</b>
 +
$ <b>perl Makefile.PL</b>
 +
$ <b>make test</b>  (optional)
 +
$ <b>sudo make install</b>
 +
 +
 +
==Troubleshooting==
  
 
This script has not yet undergone diligent testing. Try running with
 
This script has not yet undergone diligent testing. Try running with
Line 113: Line 241:
  
 
Lincoln Stein <lincoln.stein@gmail.com>
 
Lincoln Stein <lincoln.stein@gmail.com>
3 May 2013
+
8 May 2013
 +
 
 +
[[Category:Perl Scripts]]
 +
[[Category:Scripts]]

Latest revision as of 05:10, 9 May 2013

Author Lincoln Stein
Description This script maps TV recordings on a local or remote MythTV backend to a human-readable directory structure on the local machine.
Supports Version25.png  Version26.png 


mythfs-perl

This is a FUSE filesystem for MythTV (www.mythtv.org). It uses the Myth 0.25 API to mount the TV recordings known to a MythTV master backend onto a virtual filesystem on the client machine for convenient playback with XBMC or other video tools. Because it uses the MythTV network protocol, the recordings do not need to be on a shared NFS-mounted disk, nor does the Myth database need to be accessible from the client. As new recordings are added (and old ones are deleted), the filesystem will automatically update to reflect the changes.

Backends earlier than 0.25 are not supported.

Installation

Run the following commands from within the top-level directory of this distribution:

$  git clone git://github.com/lstein/mythfs-perl.git
$ cd mythfs-perl
$ ./Build.PL
$ ./Build test
$ sudo ./Build install

If you get messages about missing dependencies, run:

$ ./Build installdeps

and then "sudo ./Build install". See #Fuse Notes if you get Fuse-related errors when attempting to install dependencies.

For best performance, Perl must have been compiled with support for IThreads. In addition, you will need at Fuse version 0.15 to run correctly under Perl versions >= 5.14. Threading will be automatically disabled if not available. See #Fuse Notes for additional information.

Usage

To mount the recordings contained on the master backend "MyHost" onto a local filesystem named "/tmp/mythfs" use this command:

$ mkdir /tmp/mythfs
$ mythfs.pl MyHost /tmp/mythfs

After a pause to ensure that the backend is reachable, the script will fork into the background and operate silently. The mounted /tmp/mythfs directory will contain a series of human-readable recordings organized by title (directory) and subtitle (file).

To unmount the filesystem:

$ fusermount -u /tmp/mythfs

NOTE: Do NOT try to kill the mythfs.pl process. This will only cause a hung filesystem that needs to be unmounted with fusermount.

There are a number of options that you can pass to mythfs.pl, including the ability to customize the filesystem layout and set the interval that the backend is checked for new and deleted recordings. Call mythfs.pl with the -h option for the complete help text.

Local Recordings

The default behavior of this filesystem is to use the Myth API to stream recordings across the network when you attempt to read from them. This is done in an efficient way that fetches just the portion of the file you wish to read. However, if the underlying recording files are directly accessible (either in a regular director or via an NFS mount), you can get better performance by passing mythfs.pl the --mountpt option with the path to the directory in which the recordings can be found. The filesystem will then be set up as a set of symbolic links that point from a human readable file name to the recording file.

The main advantage of creating symbolic links is that NFSv4 can be significantly faster than the backend streaming protocol -- about 1.6X in my informal tests. The main limitation is that this mode does not understand storage groups, so all recordings need to be located in a single storage group in a locally-accessible directory. However if a recording file is not found in local directory, then mythfs.pl will fall back to the streaming protocol, so the recording is accessible one way or another.

The Default Directory Layout

Recordings that are part of a series usually have a title (the series name) and subtitle (the episode name). Such recordings are displayed using a two-tier directory structure in which the top-level directory is the series name, and the contents are a series of recorded episodes. The corresponding pattern (as described in the next section) is "%T/%S".

For recordings that do not have a subtitle, typically one-off movie showings, the recording is placed at the top level.

If needed to ensure uniqueness, the channel number and the time the recorded was started is attached to the filename. In all cases, an extension is attached to indicate the recording type (.mpg or .nuv). The file create and modification times correspond to the recording start time. For directories, the times are set to the most recent recording contained within the directory, and the directory size is set to the number of recordings contained within it (recursively).

Here is an example directory listing:

% ls -lR  /tmp/mythfs
total 35
-r--r--r-- 1 lstein lstein 21262807708 Jan 11 13:10 007 Licence To Kill.mpg
-r--r--r-- 1 lstein lstein 12298756208 Jan 11 13:10 A Funny Thing Happened on the Way to the Forum.mpg
-r--r--r-- 1 lstein lstein 14172577964 Jan 11 13:10 A Heartland Christmas.mpg
-r--r--r-- 1 lstein lstein 17591877032 Mar 10 14:24 A Knight_s Tale.mpg
dr-xr-xr-x 1 lstein lstein           5 Mar 11 03:41 Alfred Hitchcock Presents
dr-xr-xr-x 1 lstein lstein           8 May  4 01:20 American Dad
...

/home/lstein/Myth/Alfred Hitchcock Presents:
total 3
-r--r--r-- 1 lstein lstein 647625408 Dec 25 15:30 Back for Christmas.mpg
-r--r--r-- 1 lstein lstein 647090360 Dec  7 00:00 Dead Weight.mpg
-r--r--r-- 1 lstein lstein 660841056 Mar 11 03:00 Rose Garden.mpg
-r--r--r-- 1 lstein lstein 647524452 Dec 25 00:00 Santa Claus and the 10th Ave. Kid.mpg
-r--r--r-- 1 lstein lstein 649819932 Dec 27 00:00 The Contest of Aaron Gold.mpg

/home/lstein/Myth/American Dad:
total 4
-r--r--r-- 1 lstein lstein 3512038152 Apr 24 00:00 Flirting With Disaster.mpg

The size of directories corresponds to the number of recordings (not counting subdirectories) contained within it.The modification time of directories is the start time of the most recent recording contained within it.

Customizing the Directory Listing

You may customize the directory listing by providing a pattern for naming each recording using the -p option. For example:

$ mythfs.pl -p '%C/%T:%S (%od-%ob-%oY)' mythbackend ~/Myth

This will create filenames that look like this:

Sitcom/The Simpsons:The Food Wife (13-Nov-2011).mpg

Patterns contain a combination of constant strings plus substitution patterns consisting of the "%" sign plus 1 to three characters. A slash will be interpreted as a directory level: multiple levels are allowed.

Commonly-used substitution patterns are:

   %T   = Title (show name)
   %S   = Subtitle (episode name)
   %C   = Category
   %cn  = Channel: channel number
   %cN  = Channel: channel name
   %y   = Recording start time:  year, 2 digits
   %Y   = Recording start time:  year, 4 digits
   %m   = Recording start time:  month, leading zero
   %b   = Recording start time:  abbreviated month name
   %B   = Recording start time:  full month name
   %d   = Recording start time:  day of month, leading zero
   %h   = Recording start time:  12-hour hour, with leading zero
   %H   = Recording start time:  24-hour hour, with leading zero
   %i   = Recording start time:  minutes
   %s   = Recording start time:  seconds
   %a   = Recording start time:  am/pm
   %A   = Recording start time:  AM/PM

A full list of patterns can be obtained by running "mythfs.pl -p help".

Patterns are largely compatible with the excellent mythlink.pl (http://www.mythtv.org/wiki/Mythlink.pl) script, but there are a small number of enhancements, such as the ability to generate the month name. Also, the patterns that generate the month name without a leading zero are not supported.

You may wish to use a delimiter to separate fields of the recording name, for example "%T:%S" to generate "Title:Subtitle". Occasionally a recording field is empty, leading to names like "The Wild Ones:.mpg". To avoid this, pass the --trim option with the delimiter you use, and dangling/extra delimiters will be trimmed:

$ mythfs.pl -p '%T:%S' --trim=':' backend /tmp/myth

If after applying the pattern to a recording the resulting path is not unique, then this script will uniqueify the path by appending to it the channel number and recording start time, for example:

Masterpiece Classic/Downtown Abbey_17_1-2013-02-11T02:00.mpg
Masterpiece Classic/Downtown Abbey_17_1-2013-03-10T06:00.mpg

Caching

New and updated recordings will appear in the filesystem after a slight delay due to the manner in which the script caches the recording list. By default the backend is only checked for updates every 10 minutes, but you can adjust this using the --cachetime option, which takes the interval in minutes at which the system checks for new and updated recordings.

For example, this command will reduce the update interval to 2 minutes:

 $ mythfs.pl MyHost --cachetime=2 /tmp/mythfs

Fuse Notes

For best performance, you will need to run this filesystem using a version of Perl that supports IThreads. Otherwise it will fall back to non-threaded mode, which will introduce occasional delays during directory listings and have notably slower performance when reading from more than one file simultaneously.

If you are running Perl 5.14 or higher, you *MUST* use at least 0.15 of the Perl Fuse module. At the time this was written, the version of Fuse 0.15 on CPAN was failing its regression tests on many platforms. I have found that the easiest way to get a fully operational Fuse module is to clone and compile a patched version of the source, following this recipe:

$ git clone git://github.com/isync/perl-fuse.git
$ cd perl-fuse
$ perl Makefile.PL
$ make test   (optional)
$ sudo make install


Troubleshooting

This script has not yet undergone diligent testing. Try running with the -debug flag to see where the problems are occurring and report issues to https://github.com/lstein/mythfs-perl.

Author

Lincoln Stein <lincoln.stein@gmail.com> 8 May 2013