Australian TV Listings

From MythTV Official Wiki
Revision as of 13:21, 17 October 2013 by GBee (Talk | contribs)

Jump to: navigation, search
Wikipedia-logo-en.png
Wikipedia has an article on:

Electronic Service Guide Information

Currently the best source of guide data is the EIT content being broadcast by the Free TV networks over DVB-T.

Free TV Australia Operational Practice OP-44

Implementation Guide for the DVB Event Information Table (EITp/f) provides good with interoperability with MythTV.

The status of the networks compliance is currently as follows: (Information based on Perth data)

Network event_name_char (Title) text_char (Episode) extended_event_descriptor (Description) content_descriptor (Categories)
ABC good contains description, gets truncated sometimes unused good
Seven good contains description, rpt info, closed captions info unused good
Nine good generally ok, contains extra flags (live, rpt, rating) at start, "Movie" if movie, sometimes a (truncated) duplicate of description ok, sometimes contains cast good
Ten ok (live events prefixed with LIVE:) contains description, some programs have just generic series description copyright declaration good
SBS good contains description unused good

Guide Data Copyright In Australia

Australia doesn't have a service like Zap2It to provide TV program guide listings. Over the years, several people have written grabbers to scrape the data from web sites such as Yahoo, Nine-MSN, YourTV etc. These grabbers have frequently stopped working because of efforts from companies such as Nine-MSN to obfuscate the data. Each time, the workaround is trivial, but the continual changes are draining on the volunteers maintaining the grabbers.

It is unknown what purpose these companies have in trying to prevent their TV guides being used by their audience, possibly some misguided sense of copyright.

Background:

Nine loses electronic program guide case

http://www.smh.com.au/news/home-theatre/nine-loses-program-guide-case/2007/08/09/1186530493565.html

Also loses High Court Appeal (22/04/09):

http://www.goodgearguide.com.au/article/300251/icetv_wins_landmark_high_court_decision_versus_nine_network

Free TV networks to broadcast industry electronic program guide

http://www.freetv.com.au/media/News-Media_Release/2007-0304_Free_TV_Networks_to_Broadcast_Industry_Electronic_Program_Guide_141107.pdf

Sources Of Guide Data

There are a number of sources of data available with different grabbers. One list is maintained here.

Also see XMLTV#Australia.

Nine-MSN, Yahoo And Other Site Scrappers

These sites have been the most detailed and accurate source of data but constantly break.

Ice TV

You can purchase the guide data from Ice TV with a AU$3 per week subscription. The #tv_grab_au_reg grabber will work with both IceTV and OzTiVo data sources.

OzTiVo

Alternatively, for those who want guide data for free, there is a source of data which is freely and openly available and designed to be downloaded for PVRs. This is provided by the nice folks at OzTiVo and was originally designed for Aussie owners of TiVo boxes. However, because it outputs guide data in XMLTV format it can also be used by MythTV users. In order to use this you will need to create a free account on their web site first. Users who go this route are requested to follow the tvguide.org.au mailing list to stay on top of any changes to the service and help with corrections. The #tv_grab_au_reg grabber will work with both IceTV and OzTiVo data sources, while #xmlguide.pl will only work for OzTiVo.

EPGStream

http://epgstream.net/ provides a simple source of data (one http request). Mainly designed for MediaCenter users, they also have an xmltv interface. I wrote a script to download the xmltv to a temporary file, then upload it to the database using the mythfilldatabase --file option. It's online here as EPGStreamUpdateTvGuide.

EIT Scanning

As of January 1st 2008 all commercial free to air networks will broadcast EPG data for up to 7 days (No minimum specified). Previously the only broadcasted data was for 'now' and 'next'. Therefore there may not be a need to use any of the grabbers/site scrappers mentioned within this page for free to air TV only.

See this article for more information: Aussie nets agree EPG start,Rapid TV News

Grabbers Of Guide Data

Site Scrapers

http://www.whuffy.com/tv_grab_au/

(10 Sept 2006) Rex 1.x grabber is broken due to changes in the web site.

(17 Sept 2006) Rex 1.x is broken but there is a later version at the link above which does work.

tv_grab_oztivo

Downloads guide data from OzTiVo

  • uses standard XMLTV grabber script options.
  • re-writing of XMLTV ids.
  • Changing timezones of channels data.
  • Filtering out specific channels.
  • Direct execution of mythfilldatabase on data.
  • Converting OzTiVo categories to DataDirect categories.
  • Setting HDTV flag on HD channels.

Authors site

tv_grab_au_reg

This python script grabs the #Ice TV and #OzTiVo data and does some clever things like:

  • store a username/password in a config file.
  • re-write XMLTV ids (useful if you're switching from a different data source).
  • force the addition of explicit timezones on timestamps (useful for people from some states and pay tv data).
  • filter out specific channels entirely.
  • grab some channel's data from tvguide.org.au and some from IceTV and merge the result into one XMLTV dataset (tvguide.org.au covers more channels than IceTV).
  • gives newbies who are looking for a 'tv_grab_au' script something to get hold of.
  • use standard XMLTV grabber script options.

You can find the script plus some other useful info about Aussie grabbers at William Uther's site tv_grab_au_reg.

xmlguide.pl

No longer available 22 Feb 2008.
You need to use one of the above programs to download guide data from OzTiVo


An alternative is to use the xmlguide.pl cgi script on the OzTiVo web site. This maybe a little more complicated setting up but hopefully if the API changes you should be covered automatically with no need to make any updates. Users who go this route are requested to follow the tvguide.org.au mailing list to stay on top of any changes to the service and help with corrections.

When accessed, the server script queries the preferences associated with your username and password to decide which channels to download. It then extracts up to 7 days worth of data for those channels into XMLTV format which you can then load into MythTV using mythfilldatabase. Full details available at XMLGuideAPI.

How To

Create an account on OzTiVo.

Create a file /home/mythtv/tv_grab_au_tvg.conf with one line like this where UserName and Password are the username and password for your tvguide.org.au account:

http://UserName:Password@http://www.oztivo.net/tivo-bin/tvguide.pl

Make this file rw------- by chmod 600 so that other users can't access your account logon details. The url for wget is specified in the --input-file param to hide your OzTiVo username and password from process viewers (eg. ps).

To load the guide data into the MythTV database with these two commands or create a file /usr/local/bin/mfd with:

wget --header="accept-encoding: gzip" --input-file=/home/mythtv/tv_grab_au_tvg.conf  --output-document=- | gzip -d > /tmp/xmltv.xml
mythfilldatabase --file 1 7 /tmp/xmltv.xml

You can change the output file name to suit after the ">".

An easy way to run this automatically is to run this from a daily cron job. That can be set up either in the crontab for a particular user, /etc/crontab, /etc/cron.d/ or /etc/cron.daily/. Whichever is best may depend on your linux distribution. One thing to note when running it from cron is that you should put a randomiser into the call so that it doesn't do it at exactly the same time each day. For example on Debian create a file /etc/cron.d/mfd with

0 1 * * * sleep $((RANDOM % 18000)) && /usr/local/bin/mfd

This helps to reduce the peak loads put on the OzTiVo servers by running the script sometime between 1am and 6am every day. Please note Vixie cron requires a '\' before '%' like this 'RANDOM \% 18000'.

The XMLTVIDs in this data probably won't match what you have in your MythTV channel setup, so there are a couple of options to get it to work.

  • Change the XMLTV IDs of your channels in Myth itself.
  • Use sed to translate XMLTVIDs
  • Use xslt to translate XMLTVIDs.
Use sed to translate XMLTVIDs

Change the /usr/local/bin/mfd file with the wget/sed commands all on one line. Make changes for your XMLTV IDs as required.

wget --header="accept-encoding: gzip" --input-file=/home/mythtv/tv_grab_au_tvg.conf 
--output-document=- | gzip -d | sed 's/channel="SBS-NEWS"/channel="news\.sbs\.com\.au"/g ; s/channel="Seven-Syd"/channel="sydney\.seven\.com\.au"/g ; 
s/channel="Ten-NSW"/channel="sydney\.ten\.com\.au"/g ; 
s/channel="SBS-NSW"/channel="sydney\.sbs\.com\.au"/g ; 
s/channel="ABC2"/channel="abc2\.abc\.gov\.au"/g ; 
s/channel="Nine-Syd"/channel="sydney\.nine\.com\.au"/g ; 
s/channel="ABC-NSW"/channel="nsw\.abc\.gov\.au"/g' > /tmp/xmltv.xml
mythfilldatabase --file 1 7 /tmp/xmltv.xml
Use xslt to translate XMLTVIDs

The trouble with the sed method is that you could substitute a string in a program description or elsewhere, rather than just in the channel ID. So to avoid that you can use xslt (extensible stylesheet language transformation) to ensure you only substitute that string when it's part of a channel setting.

Make sure you have the xsltproc binary. (On Fedora Core this is provided by the libxslt rpm).

Change the /usr/local/bin/mfd file with the wget/xsltproc commands all on one line.

wget --header="accept-encoding: gzip" --input-file=/home/mythtv/tv_grab_au_tvg.conf 
--output-document=- | gzip -d | xsltproc /home/mythtv/tv_grab_au_tvg.xsl - > /tmp/xmltv.xml
mythfilldatabase --file 1 7 /tmp/xmltv.xml

The file /home/mythtv/tv_grab_au_tvg.xsl looks like this (thanks go to Joshua King for providing this). If it all looks a bit scary to you, don't worry - all you have to do is modify the channel mappings near the beginning. (Note that this also has some untested code in it to adjust the time offset by 8 hours but the code is commented out).

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:output method="xml" indent="yes"/>

<xsl:template name="mod-channel">
    <xsl:choose>
        <!-- Add channel mappings here -->
        <xsl:when test="string()='ABC-NSW'">nsw.abc.gov.au</xsl:when>
        <xsl:when test="string()='ABC2'">abc2.abc.gov.au</xsl:when>
        <xsl:when test="string()='Seven-Syd'">sydney.seven.com.au</xsl:when>
        <xsl:when test="string()='Nine-Syd'">sydney.nine.com.au</xsl:when>
        <xsl:when test="string()='Ten-NSW'">sydney.ten.com.au</xsl:when>
        <xsl:when test="string()='SBS-NSW'">sydney.sbs.com.au</xsl:when>
        <xsl:when test="string()='SBS-NEWS'">news.sbs.com.au</xsl:when>

        <!-- This ensures any unmatched channels don't become empty -->
        <xsl:otherwise><xsl:value-of select="." /></xsl:otherwise>
    </xsl:choose>
</xsl:template>

<xsl:template name="add-timezone">
    <!-- This emulates force-explicit-timezone -->
    <xsl:value-of select="concat(.,' +0000')" />
</xsl:template>

<xsl:template name="mod-timezone">
    <!-- Change timezone in file by + 8 hours 2 etc lines down -->
    <xsl:choose>
        <xsl:when test="substring(.,9,2) + 8 >= 24">
            <!-- Need to increment day -->
            <xsl:choose>
                <xsl:when test="substring(.,7,2)=31 and substring(.,5,2)=12">
                    <!-- Need to increment year -->
                    <xsl:value-of select="substring(.,1,4) + 1" />0101<xsl:if test="substring(.,9,2) + 8 mod 24 < 10">0</xsl:if><xsl:value-of select="substring(.,9,2) + 8 mod 24" /><xsl:value-of select="substring(.,11)" /><xsl:text> +0800</xsl:text>
                </xsl:when>
                <xsl:when test="(substring(.,7,2)=31 and (substring(.,5,2)=1 or substring(.,5,2)=3 or substring(.,5,2)=5 or substring(.,5,2)=7 or substring(.,5,2)=8 or substring(.,5,2)=10)) or (substring(.,7,2)=30 and (substring(.,5,2)=4 or substring(.,5,2)=6 or substring(.,5,2)=9 or substring(.,5,2)=11))">
                    <!-- Need to increment month -->
                    <xsl:value-of select="substring(.,1,4)" /><xsl:if test="substring(.,5,2) + 1 < 10">0</xsl:if><xsl:value-of select="substring(.,5,2) + 1" />01<xsl:if test="substring(.,9,2) + 8 mod 24 < 10">0</xsl:if><xsl:value-of select="substring(.,9,2) + 8 mod 24" /><xsl:value-of select="substring(.,11)" /><xsl:text> +0800</xsl:text>
                </xsl:when>
                <xsl:when test="(substring(.,7,2)=29 and (substring(.,5,2)=2)) or (substring(.,7,2)=28 and substring(.,5,2)=2 and (substring(.,1,4) mod 4!=0 or substring(.,1,4) mod 400=0))">
                    <!-- Need to increment month -->
                    <xsl:value-of select="substring(.,1,4)" />0301<xsl:if test="substring(.,9,2) + 8 mod 24 < 10">0</xsl:if><xsl:value-of select="substring(.,9,2) + 8 mod 24" /><xsl:value-of select="substring(.,11)" /><xsl:text> +0800</xsl:text>
                </xsl:when>
                <xsl:otherwise>
                    <!-- Nothing to do but day and add -->
                    <xsl:value-of select="substring(.,1,6)" /><xsl:if test="substring(.,7,2) + 1 < 10">0</xsl:if><xsl:value-of select="substring(.,7,2) + 1"/><xsl:if test="substring(.,9,2) + 8 mod 24 < 10">0</xsl:if><xsl:value-of select="substring(.,9,2) + 8 mod 24" /><xsl:value-of select="substring(.,11)" /><xsl:text> +0800</xsl:text>
                </xsl:otherwise>
                     <xsl:value-of select="substring(.,1,6)" /><xsl:if test="substring(.,7,2) + 1 < 10">0</xsl:if><xsl:value-of select="substring(.,7,2) + 1"/><xsl:if test="substring(.,9,2) + 8 mod 24 < 10">0</xsl:if><xsl:value-of select="substring(.,9,2) + 8 mod 24" /><xsl:value-of select="substring(.,11)" /><xsl:text> +0800</xsl:text>
                </xsl:otherwise>
            </xsl:choose>
        </xsl:when>
        <xsl:otherwise>
            <!-- Nothing to do but add -->
            <xsl:value-of select="substring(.,1,8)" /><xsl:if test="substring(.,9,2) + 8 < 10">0</xsl:if><xsl:value-of select="substring(.,9,2) + 8" /><xsl:value-of select="substring(.,11)" /><xsl:text> +0800</xsl:text>
        </xsl:otherwise>
   </xsl:choose>
</xsl:template>

<xsl:template name="copy">
    <xsl:element name="{name()}">
    <xsl:for-each select="@*">
        <xsl:attribute name="{name()}">
        <xsl:choose>
            <xsl:when test="name(..)='programme' and name()='channel'"><xsl:call-template name="mod-channel" /></xsl:when>
            <xsl:when test="name(..)='channel' and name()='id'"><xsl:call-template name="mod-channel" /></xsl:when>

<!-- Uncomment here to try out the time filters
            <xsl:when test="name(..)='programme' and name()='start'"><xsl:call-template name="mod-timezone" /></xsl:when>
            <xsl:when test="name(..)='programme' and name()='stop'"><xsl:call-template name="mod-timezone" /></xsl:when>
-->
            <xsl:otherwise><xsl:value-of select="." /></xsl:otherwise>
        </xsl:choose>
        </xsl:attribute>
    </xsl:for-each>
    <xsl:for-each select="*|text()">
        <xsl:choose>
            <xsl:when test="name()">
                <xsl:call-template name="copy" />
            </xsl:when>
            <xsl:otherwise>
                <xsl:value-of select="string()" />
            </xsl:otherwise>
        </xsl:choose>
    </xsl:for-each>
</xsl:element>
</xsl:template>

<xsl:template match="/">
    <xsl:for-each select="*">
        <xsl:call-template name="copy" />
    </xsl:for-each>
</xsl:template>
</xsl:stylesheet>

Shepherd

Warning.png Warning: Please note that Shepherd is completely unsupported by the MythTV project. It is not an XMLTV compliant grabber and operates directly on your database. It does not allow MythTV to schedule listings updates like supported grabbers. This will only change if you lobby the developers to change their stance.

For anyone interested in a new tv_grab_au solution, the Shepherd project began in late 2006. As of Feb 2007 it's still in beta but it's a pretty advanced beta and seems to be working well.

Shepherd aims to solve the traditional problems of tv_grab_au scripts by:

  1. employing multiple TV guide grabbers and data sources and switching between them as necessary; and
  2. auto-updating to eliminate the need for users to manually address grabber breakages

There are details here: [1]http://svn.whuffy.com/wiki

Shepherd also includes the ability to update the channel icons displayed within mythTV. Examples of the icon-sets that can used are shown here.

TV Icons

Inspired by FrogBoy's most excellent icon set, a number of icons exist for the free2air channels:

An example of FrogBoy's work follows Au-nine.png Au-sbs-hd.png

Icons are available at: