Difference between revisions of "MythNetvision Grabber Script Format"

From MythTV Official Wiki
Jump to: navigation, search
m (A MythNetvision Video Item)
m (MythNetvision Version Output)
 
(6 intermediate revisions by one user not shown)
Line 20: Line 20:
 
|/usr/share/mythtv/mythnetvision/scripts/youtube.py -v
 
|/usr/share/mythtv/mythnetvision/scripts/youtube.py -v
 
|
 
|
* This option is used by MythNetvision to identify the grabber's name and its capabilities, which can include search and/or tree view.
+
* This option is used by MythNetvision to identify the grabber's name and its capabilities, which can include search and/or tree view. See below for details.
* The format is grabber title then a "|" character followed by the grabber's supported options search and/or tree view.
+
* Example: "YouTube|ST" says that the "YouTube" grabber supports both MythNetvision search and tree view options.
+
 
|-
 
|-
 
| -p
 
| -p
Line 44: Line 42:
 
* Only required if a grabber supports a Tree View.
 
* Only required if a grabber supports a Tree View.
 
|}
 
|}
 +
 +
 +
= MythNetvision Version Output =
 +
 +
All Internet Content scripts must return their capabilities in the following format:
 +
 +
'''Command Line:'''
 +
 +
<pre>./youtube.py -v</pre>
 +
 +
'''Returned XML:'''
 +
 +
<pre>
 +
<grabber>
 +
  <name>YouTube</name>
 +
  <command>youtube.py</command>
 +
  <author>R.D. Vaughan</author>
 +
  <thumbnail>youtube.png</thumbnail>
 +
  <type>video</type>
 +
  <description>Share your videos with friends, family, and the world.</description>
 +
  <version>0.22</version>
 +
  <search>true</search>
 +
  <tree>true</tree>
 +
</grabber>
 +
</pre>
 +
 +
''Name'' is the "plain english" name of your script as displayed in the UI.  ''Author'' is your name or pseudonym.  ''Command'' is the basename command name for your grabber  ''Thumbnail'' is the image you distribute to represent your grabber or source.  ''Type'' can presently only be ''video''.  ''Description'' is a short explanation of the source.  ''Version'' is a ''double'' representing the version number (ie 0.11, 1.00, 2.10, etc).  ''Search'' and ''tree'' are booleans expressing whether or not the script is capable of searching, tree view, or both.
  
 
= MythNetvision Search Scripts =
 
= MythNetvision Search Scripts =
Line 51: Line 76:
 
'''Command Line:'''
 
'''Command Line:'''
  
<pre>/youtube.py -p 1 -S "Peep Show"</pre>
+
<pre>./youtube.py -p 1 -S "Peep Show"</pre>
  
 
'''Returned XML:'''
 
'''Returned XML:'''
Line 65: Line 90:
 
xmlns:atom="http://www.w3.org/2005/Atom"
 
xmlns:atom="http://www.w3.org/2005/Atom"
 
xmlns:amp="http://www.adobe.com/amp/1.0"
 
xmlns:amp="http://www.adobe.com/amp/1.0"
xmlns:dc="http://purl.org/dc/elements/1.1/">
+
xmlns:dc="http://purl.org/dc/elements/1.1/"
 +
xmlns:mythtv="http://www.mythtv.org/wiki/MythNetvision_Grabber_Script_Format">
 
     <channel>
 
     <channel>
 
         <title>YouTube</title>
 
         <title>YouTube</title>
Line 168: Line 194:
 
         <item>
 
         <item>
 
             <title>Peep Show S02E02 P01</title>
 
             <title>Peep Show S02E02 P01</title>
 +
            <mythtv:subtitle>Jeremy Makes It</mythtv:subtitle>
 
             <author>babyblue8five</author>
 
             <author>babyblue8five</author>
 
             <pubDate>Sun, 06 Jan 2008 21:44:36 GMT</pubDate>
 
             <pubDate>Sun, 06 Jan 2008 21:44:36 GMT</pubDate>
Line 194: Line 221:
 
|-
 
|-
 
| title
 
| title
| The description or plot text for the video.
+
| The title text for the video.
 +
|-
 +
| mythtv:subtitle
 +
| The subtitle text for the video.
 
|-
 
|-
 
| author
 
| author
Line 211: Line 241:
 
|  '''Note''': Do not populate this field if you wish to use MythTV's built-in player! The executable name to an external player application.  An example might be something like:
 
|  '''Note''': Do not populate this field if you wish to use MythTV's built-in player! The executable name to an external player application.  An example might be something like:
 
<pre><player>mplayer</player></pre>
 
<pre><player>mplayer</player></pre>
'''Not yet fully implemented.'''
 
 
|-
 
|-
 
| playerargs
 
| playerargs
 
| '''Note''': Do not populate this field if you wish to use MythTV's built-in player!  The arguments passed to the external binary in the <player> tags.  An example might be something like:
 
| '''Note''': Do not populate this field if you wish to use MythTV's built-in player!  The arguments passed to the external binary in the <player> tags.  An example might be something like:
 
<pre><playerargs>-fs -zoom mms://123.45.6.78/</playerargs></pre>
 
<pre><playerargs>-fs -zoom mms://123.45.6.78/</playerargs></pre>
'''Not yet fully implemented.'''
+
 
 +
You can insert variables here and MythNetvision will dynamically insert the correct values before playback. They are as follows:
 +
<pre>
 +
%URL% - The <link> value.
 +
%MEDIAURL% - The <media:content> url= value
 +
</pre>
 
|-
 
|-
 
| download
 
| download
Line 231: Line 265:
 
<pre>/home/someguy/download_123_456.mp4</pre>
 
<pre>/home/someguy/download_123_456.mp4</pre>
  
Note that you may output logging and errors to stderr, but only the output filename must be emitted to stdout.  The above example shows the use of userjob-style variables, which are discussed below in <playerargs>.
+
Note that you may output logging and errors to stderr, but only the output filename must be emitted to stdout.  The above example shows the use of userjob-style variables, which are discussed below in <downloadargs>.
 +
 
 +
'''Not yet fully implemented.'''
 
|-
 
|-
 
| downloadargs
 
| downloadargs
Line 242: Line 278:
  
 
'''%FILE%''' The output filename suggestion in MythNetvision format.
 
'''%FILE%''' The output filename suggestion in MythNetvision format.
 +
 +
'''Not yet fully implemented.'''
 
|-
 
|-
 
| media:group
 
| media:group
Line 268: Line 306:
 
|-
 
|-
 
| lang
 
| lang
| The ISO language code of the video. [http://en.wikipedia.org/wiki/List_of_ISO_639-1_codes Two character language codes]
+
| The ISO-639 language code of the video. [http://en.wikipedia.org/wiki/List_of_ISO_639-1_codes Two character language codes]
 
|-
 
|-
 
| rating
 
| rating
Line 291: Line 329:
  
 
For those unfamiliar with XML, RSS, or HTML, *all* tags must be closed or parsing the document will fail.
 
For those unfamiliar with XML, RSS, or HTML, *all* tags must be closed or parsing the document will fail.
 
  
 
= MythNetvision Tree Grabber Scripts =
 
= MythNetvision Tree Grabber Scripts =

Latest revision as of 21:02, 25 May 2010

Grabber Script Standards

  • All grabbers must support a search and/or a tree view option. Supporting both is optional.
  • Grabbers expect that the environment is set to UTF-8.
  • A search grabber by default should return a maximum of 20 video items per page. This usually keeps the response time to a reasonable duration.
  • A search grabber must support paging with a specific page number specified through the "-p" command line option. The grabber must interpret the page number into what the source api requires. Sometimes it is a page number and sometimes a video index (e.g. 21st video of the search results).
  • A grabber's file name must have a matching image file name for an icon to be displayed by the MythNetvision plugin.


Grabber Script Command Line Options

Command Line Option Arguments Example Description
-v None /usr/share/mythtv/mythnetvision/scripts/youtube.py -v
  • This option is used by MythNetvision to identify the grabber's name and its capabilities, which can include search and/or tree view. See below for details.
-p Unsigned Integer /usr/share/mythtv/mythnetvision/scripts/youtube.py -p 2 -S "A Search Term"
  • Page number. Always includes an integer and is only used with the search option (-S). When not provided the page number defaults to one.
-S Text String /usr/share/mythtv/mythnetvision/scripts/youtube.py -p 2 -S "A Search Term"
  • Search option. Requires a search term made of one or more words.
  • Only required if the grabber supports a Search view.
-T None /usr/share/mythtv/mythnetvision/scripts/youtube.py -T
  • Tree view. Returns a properly formatted tree view RSS feed.
  • Only required if a grabber supports a Tree View.


MythNetvision Version Output

All Internet Content scripts must return their capabilities in the following format:

Command Line:

./youtube.py -v

Returned XML:

<grabber>
  <name>YouTube</name>
  <command>youtube.py</command>
  <author>R.D. Vaughan</author>
  <thumbnail>youtube.png</thumbnail>
  <type>video</type>
  <description>Share your videos with friends, family, and the world.</description>
  <version>0.22</version>
  <search>true</search>
  <tree>true</tree>
</grabber>

Name is the "plain english" name of your script as displayed in the UI. Author is your name or pseudonym. Command is the basename command name for your grabber Thumbnail is the image you distribute to represent your grabber or source. Type can presently only be video. Description is a short explanation of the source. Version is a double representing the version number (ie 0.11, 1.00, 2.10, etc). Search and tree are booleans expressing whether or not the script is capable of searching, tree view, or both.

MythNetvision Search Scripts

Let's examine the following example of a search script:

Command Line:

./youtube.py -p 1 -S "Peep Show"

Returned XML:

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd"
xmlns:content="http://purl.org/rss/1.0/modules/content/"
xmlns:cnettv="http://cnettv.com/mrss/"
xmlns:creativeCommons="http://backend.userland.com/creativeCommonsRssModule"
xmlns:media="http://search.yahoo.com/mrss/"
xmlns:atom="http://www.w3.org/2005/Atom"
xmlns:amp="http://www.adobe.com/amp/1.0"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:mythtv="http://www.mythtv.org/wiki/MythNetvision_Grabber_Script_Format">
    <channel>
        <title>YouTube</title>
        <link>http://www.youtube.com/</link>
        <description>Share your videos with friends, family, and the world.</description>
        <numresults>2</numresults>
        <returned>2</returned>
        <startindex>1</startindex>
        <item>
            <title>Peep Show S02E02 P01</title>
            <author>babyblue8five</author>
            <pubDate>Sun, 06 Jan 2008 21:44:36 GMT</pubDate>
            <description>Peep Show season 2, episode 2, part 1/3</description>
            <link>http://www.youtube.com/watch?v=gCWmiXEkAGE&feature=youtube_gdata</link>
            <media:group>
                <media:thumbnail url='http://i.ytimg.com/vi/gCWmiXEkAGE/0.jpg'/>
                <media:content url='http://www.youtube.com/watch?v=gCWmiXEkAGE&feature=youtube_gdata' duration='' width='' height='' lang=''/>
            </media:group>
            <rating>4.988506</rating>
        </item>
        <item>
            <title>Peep Show S02E01 P01</title>
            <author>babyblue8five</author>
            <pubDate>Sun, 06 Jan 2008 20:13:10 GMT</pubDate>
            <description>Peep Show Season 2, episode 1, part 1/3</description>
            <link>http://www.youtube.com/watch?v=38EhKzJ0zS8&feature=youtube_gdata</link>
            <media:group>
                <media:thumbnail url='http://i.ytimg.com/vi/38EhKzJ0zS8/0.jpg'/>
                <media:content url='http://www.youtube.com/watch?v=38EhKzJ0zS8&feature=youtube_gdata' duration='' width='' height='' lang=''/>
            </media:group>
            <rating>4.931818</rating>
        </item>
    </channel>
</rss>

The above is a search command which returns two items. Let's break down the results so that you can see how it works.


Establishing The Namespace

 <?xml version="1.0" encoding="UTF-8"?>
 <rss version="2.0"
 xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd"
 xmlns:content="http://purl.org/rss/1.0/modules/content/"
 xmlns:cnettv="http://cnettv.com/mrss/"
 xmlns:creativeCommons="http://backend.userland.com/creativeCommonsRssModule"
 xmlns:media="http://search.yahoo.com/mrss/"
 xmlns:atom="http://www.w3.org/2005/Atom"
 xmlns:amp="http://www.adobe.com/amp/1.0"
 xmlns:dc="http://purl.org/dc/elements/1.1/"
 xmlns:mythtv="http://www.mythtv.org/wiki/MythNetvision_Grabber_Script_Format">

All MythNetvision returns (Search and Tree) start with the above. This is called setting the XML namespace. Namespaces are custom definitions of RSS/XML tags that can be used in your document. Presently the mythtv, media, amp, and itunes tags are used to some extent, but others may be supported in the future so the above should tentatively be used on all returns.

An RSS "Channel"

    <channel>
        <title>YouTube</title>
        <link>http://www.youtube.com/</link>
        <description>Share your videos with friends, family, and the world.</description>
        <numresults>2</numresults>
        <returned>2</returned>
        <startindex>1</startindex>

All RSS returns in MythNetvision are segmented into "channels." Although it's not generally necessary to have more than one channel in a return, it is possible and acceptable to do so. A channel is a logical grouping of results with a title, description, and some information to allow control of paging in search scripts. Let's take a look at what each of these tags means.

Tag name Description
channel Tag identifies the start of a channel.
title The title of the source of the metadata which will be displayed by the MythNetvision plugin.
link URL of the home page for the source of the metadata.
description Descriptive text about the metadata source.
numresults The number of results returned from a search. Usually this is total items or total pages of items. This tag is only used in search mode.
returned This number reflects the maximum number of items per page or the number returned when a search does not find a full page worth of video items. This tag is only used in search mode.
startindex The start index of the next page or next video item. If this number is greater than or equal to the number of results the MythNetvision plugin will not allow the user to request a next page. This tag is only used in search mode.


A MythNetvision Video Item

        <item>
            <title>Peep Show S02E02 P01</title>
            <mythtv:subtitle>Jeremy Makes It</mythtv:subtitle>
            <author>babyblue8five</author>
            <pubDate>Sun, 06 Jan 2008 21:44:36 GMT</pubDate>
            <description>Peep Show season 2, episode 2, part 1/3</description>
            <link>http://www.youtube.com/watch?v=gCWmiXEkAGE&feature=youtube_gdata</link>
            <media:group>
                <media:thumbnail url='http://i.ytimg.com/vi/gCWmiXEkAGE/0.jpg'/>
                <media:content url='http://www.youtube.com/watch?v=gCWmiXEkAGE&feature=youtube_gdata' length='' duration='' width='' height='' lang=''/>
            </media:group>
            <rating>4.988506</rating>
            <mythtv:country>us</mythtv:country>
            <mythtv:country>uk</mythtv:country>
            <mythtv:season>2</mythtv:season>
            <mythtv:episode>2</mythtv:episode>
        </item>

The above represents a single viewable video in MythNetvision. By manipulating the format of the information, you can express that the item is downloadable, web view only, and present different types of metadata.

Tag name Description
item Identifies the start of a video item.
title The title text for the video.
mythtv:subtitle The subtitle text for the video.
author The author who submitted the video to the source web site. Often this is submitter's user id or username.
pubDate The date that the video was submitted to the site. The format of "Thu, 17 Dec 2009 13:51:34 GMT" must always be used. If you cannot retrieve the timezone, simply specify the approximate timezone in which the source exists. If you do not know the time then set it to all zeros "00:00:00". pubDate follows the RFC-822 date-time format.
description The text description of the video. A description should not contain any HTML or line-feeds.
link A URL to the web page containing the video or directly to the video player. If this link is not identical to the "media:content url" tag then that indicates to MythNetvision that the "media:content url" file can be downloaded. Unless the source site TOS allows a video file to be downloaded then the "link" and "media:content url" URLs should be identical (Web View Only).
player Note: Do not populate this field if you wish to use MythTV's built-in player! The executable name to an external player application. An example might be something like:
<player>mplayer</player>
playerargs Note: Do not populate this field if you wish to use MythTV's built-in player! The arguments passed to the external binary in the <player> tags. An example might be something like:
<playerargs>-fs -zoom mms://123.45.6.78/</playerargs>

You can insert variables here and MythNetvision will dynamically insert the correct values before playback. They are as follows:

%URL% - The <link> value.
%MEDIAURL% - The <media:content> url= value
download Note: Do not populate this field if you wish to use MythNetvision's built-in download handling! The executable name to an external downloader application. An example might be something like:
<download>mydownloadscript</download>

An external download script must return a single line upon a successful result: The fully qualified path to the downloaded file. MythNetvision then reads in this value and uses it to play back the item with the internal player and (in the future) to perform any insertions into the database.

Example download script use:

./mydownloadscript -outpath %HOMEDIR% -outfile %FILE% http://my.ip.add.ress/file.html

Example script output:

/home/someguy/download_123_456.mp4

Note that you may output logging and errors to stderr, but only the output filename must be emitted to stdout. The above example shows the use of userjob-style variables, which are discussed below in <downloadargs>.

Not yet fully implemented.

downloadargs Note: Do not populate this field if you wish to use MythNetvision's built-in download handling! The arguments passed to the external binary in the <download> tags. An example might be something like:
<downloadargs>-outpath /home/someguy/ -outfile somefile.flv http://path.to.a/thing.html</downloadargs>

In addition to defining static locations, you can insert the following userjob-style variables in your downloader arguments, which MythNetvision will then replace with the local values:

%HOMEDIR% The frontend user's home directory.

%FILE% The output filename suggestion in MythNetvision format.

Not yet fully implemented.

media:group Indicates the start of a video's thumbnail and metadata. The item must eventually be closed with a "</media:group>"
media:thumbnail url A URL to the thumbnail image for the video file. The source site does not always have a thumbnail for a video. When there are multiple thumbnails choose the first one over 200 pixels wide as it will display well but keep download times reasonable.
media:content The main tag for the video link and video metadata.
url A URL to the video file/player/web page. This link should only differ from the URL "link" tag when the URL points to a downloadable video file.
length Unlike what it appears to be, length is actually the filesize in bytes. This is only used for downloadable media.
duration The duration of the video in seconds. Some sources use a format of "MM:SS". This value must be converted to seconds.
width The width in pixels of the video.
height The height in pixels of the video.
lang The ISO-639 language code of the video. Two character language codes
rating A floating point number representing a numerical rating for the video. There is no consistency for this value between the source sites. The most common values are X out or 5 or 10.
mythtv:country A country in which the video is playable. If the video is playable anywhere, return no country. If the video is only playable in certain locales, or the Terms of Service only allow playback in certain locales, return the country code for the playable countries here. You may add as many country items as you like.
mythtv:season An unsigned integer representing the season number of the item. Optional.
mythtv:episode An unsigned integer representing the episode number of the item. Optional.

Clean This Mess Up, Mister!

    </channel>
</rss>

For those unfamiliar with XML, RSS, or HTML, *all* tags must be closed or parsing the document will fail.

MythNetvision Tree Grabber Scripts

Let's examine the following example of a Site Tree building script:

Command Line:

/youtube.py -T

Returned XML:

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd"
xmlns:content="http://purl.org/rss/1.0/modules/content/"
xmlns:cnettv="http://cnettv.com/mrss/"
xmlns:creativeCommons="http://backend.userland.com/creativeCommonsRssModule"
xmlns:media="http://search.yahoo.com/mrss/"
xmlns:atom="http://www.w3.org/2005/Atom"
xmlns:amp="http://www.adobe.com/amp/1.0"
xmlns:dc="http://purl.org/dc/elements/1.1/">
    <channel>
        <title>YouTube</title>
        <link>http://www.youtube.com/</link>
        <description>Share your videos with friends, family, and the world.</description>
            <directory name="Feeds" thumbnail="/usr/share/mythtv/mythnetvision/icons/youtube.png">
                <directory name="Highest Rated" thumbnail="/usr/share/mythtv/mythnetvision/icons/directories/topics/rated.png">
                    <item>
                        <title>Project for Awesome - My Public Access Channel!</title>
                        <author>peron75</author>
                        <pubDate>Thu, 17 Dec 2009 13:51:34 GMT</pubDate>
                        <description>Please support all the Project for Awesome videos today with ratings/comments! Thank you! Thank you to Hank and John Green, Dan Brown and everyone 
involved!!!  I chose the public access station where I began What the Buck! They are fundraising to help with their new building project.  You can help buy simply signing up for th
is site and then when you shop, they get donations from that! Yay! Thanks if you can sign up! (its Free!) LOL xoxo Michael   Please sign up: http://igive.com/wpaa</description>
                        <link>http://www.youtube.com/v/tdBHzkoXB_8?f=standard&app=youtube_gdata&autoplay=1</link>
                        <media:group>
                            <media:thumbnail url='http://i.ytimg.com/vi/tdBHzkoXB_8/hqdefault.jpg'/>
                            <media:content url='http://www.youtube.com/v/tdBHzkoXB_8?f=standard&app=youtube_gdata&autoplay=1' duration='259' width='' height='' lang=''/>
                        </media:group>
                        <rating>4.9695706</rating>
                    </item>
                </directory>
            </directory>
    </channel>
</rss>

Wow, this is remarkably familiar, isn't it? The format of a site tree is very similar to the search view, and extremely similar to any old RSS feed. Let's only look at the items which differ from a search script.


Directories/Tree Hierarchy

            <directory name="Feeds" thumbnail="/usr/local/share/mythtv/mythnetvision/icons/youtube.png">
                <directory name="Highest Rated" thumbnail="/usr/local/share/mythtv/mythnetvision/icons/directories/topics/rated.png">

Shocking as it might seem, the only new element added in a tree grabber is the directory tag, which allows you to create a filesystem-like structure for your tree. The above example contains two nested directories, meaning the represented path would be:

YouTube/Feeds/Highest Rated/

Directories include a name, which is the directory name, and a thumbnail attribute, which is an option URL or absolute path to a folder icon.

Tag name Description
directory Identifies the start of a directory. The directory must eventually be closed with a "</directory>"
name Descriptive label for the directory. This description will be displayed by the MythNetvision plugin.
thumbnail An image URL or absolute file path for the image icon that will be displayed by the MythNetvision plugin.

Metadata standards

Consider the following "rules of thumb" when writing your grabber:

  • When a search returns no results then no RSS data is returned, exit with a return code of zero.
  • Do not include any video items or directories that do not have at least one video link.
  • Always include every tag even if the source does not have any data for that tag. Leave the content empty.
width=""
<description></description>
  • Whenever possible always use a video link that auto starts the video and displayed in full screen.
    • Sometimes the method to auto start or display in full screen is not included in the API documentation. The video's HTML Web page source can sometimes provide this information.
  • Replace any ampersand characters "&" with "&amp;".
  • Remove line feed characters and HTML tags from any text fields. This is usually an issue with the description and title text of a video.
  • Do not use the word "video" in any of your Tree view directory descriptions or grabber names as it is redundant (MythNetvision is a video plugin, after all).
  • Additional tags may by included by a grabber but they will be ignored by the MythNetvision plugin.


Tree view thumbnail icon images

  • Each directory tag can include a thumbnail image.
  • The thumbnail may be an absolute path or a URL.
  • The MythNetvision core icon set is found in "/usr{/local}/share/mythtv/mythnetvision/icons"
  • Included with MythNetvision are a set of generic directory icon images that can be used. See "/usr{/local}/share/mythtv/mythnetvision/icons/directories".


Error handling

  • Standard out (stdout) is reserved exclusively for valid RSS data read by the MythNetvision plugin.
  • Standard error (stderr) is reserved exclusively for error or grabber processing messages.
  • Logging is optional.
  • Return code is zero for any processing that completed without a script abort. This includes a search or treeview that did not return any results.
  • Return code of one if an error forced the premature termination of the grabber.
  • If a grabber script skips bad data during process an error about the bad data should be sent to stderr but a return code of zero would still be returned indicating the grabber completed its processing successfully.