Australian TV Listings

From MythTV Official Wiki
Revision as of 10:43, 20 March 2008 by WhyTey (Talk | contribs)

Jump to: navigation, search

Guide Data Copyright In Australia

Australia doesn't have a service like Zap2It to provide TV program guide listings. Over the years, several people have written grabbers to scrape the data from web sites such as Yahoo, Nine-MSN, YourTV etc. These grabbers have frequently stopped working because of efforts from companies such as Nine-MSN to obfuscate the data. Each time, the workaround is trivial, but the continual changes are draining on the volunteers maintaining the grabbers.

It is unknown what purpose these companies have in trying to prevent their TV guides being used by their audience, possibly some misguided sense of copyright.

Updates:

Nine loses electronic program guide case

http://www.smh.com.au/news/home-theatre/nine-loses-program-guide-case/2007/08/09/1186530493565.html

Free TV networks to broadcast industry electronic program guide

http://www.freetv.com.au/media/News-Media_Release/2007-0304_Free_TV_Networks_to_Broadcast_Industry_Electronic_Program_Guide_141107.pdf

Sources Of Guide Data

There are a number of sources of data available with different grabbers. One list is maintained here.

Also see XMLTV#Australia.

Nine-MSN, Yahoo And Other Site Scrappers

These sites have been the most detailed and accurate source of data but constantly break.

Ice TV

You can purchase the guide data from Ice TV with a AU$3 per week subscription. The #tv_grab_au_reg grabber will work with both IceTV and OzTiVo data sources.

OzTiVo

Alternatively, for those who want guide data for free, there is a source of data which is freely and openly available and designed to be downloaded for PVRs. This is provided by the nice folks at OzTiVo and was originally designed for Aussie owners of TiVo boxes. However, because it outputs guide data in XMLTV format it can also be used by MythTV users. In order to use this you will need to create a free account on their web site first. Users who go this route are requested to follow the tvguide.org.au mailing list to stay on top of any changes to the service and help with corrections. The #tv_grab_au_reg grabber will work with both IceTV and OzTiVo data sources, while #xmlguide.pl will only work for OzTiVo.

EIT Scanning

As of January 1st 2008 all commercial free to air networks will broadcast EPG data for up to 7 days (No minimum specified). Previously the only broadcasted data was for 'now' and 'next'. Therefore there may not be a need to use any of the grabbers/site scrappers mentioned within this page for free to air TV only.

See this article for more information: Aussie nets agree EPG start,Rapid TV News

Grabbers Of Guide Data

Site Scrapers

http://www.whuffy.com/tv_grab_au/

(10 Sept 2006) Rex 1.x grabber is broken due to changes in the web site.

(17 Sept 2006) Rex 1.x is broken but there is a later version at the link above which does work.

tv_grab_oztivo

  • this will convert the tv_grab_au_reg xml files

Authors site

tv_grab_au_reg

This python script grabs the #Ice TV and #OzTiVo data and does some clever things like:

  • store a username/password in a config file.
  • re-write XMLTV ids (useful if you're switching from a different data source).
  • force the addition of explicit timezones on timestamps (useful for people from some states and pay tv data).
  • filter out specific channels entirely.
  • grab some channel's data from tvguide.org.au and some from IceTV and merge the result into one XMLTV dataset (tvguide.org.au covers more channels than IceTV).
  • gives newbies who are looking for a 'tv_grab_au' script something to get hold of.
  • use standard XMLTV grabber script options.

You can find the script plus some other useful info about Aussie grabbers at William Uther's site tv_grab_au_reg.

xmlguide.pl

An alternative is to use the xmlguide.pl cgi script on the OzTiVo web site. This maybe a little more complicated setting up but hopefully if the API changes you should be covered automatically with no need to make any updates. Users who go this route are requested to follow the tvguide.org.au mailing list to stay on top of any changes to the service and help with corrections.

When accessed, the server script queries the preferences associated with your username and password to decide which channels to download. It then extracts up to 7 days worth of data for those channels into XMLTV format which you can then load into MythTV using mythfilldatabase. Full details available at XMLGuideAPI.

How To

Create an account on OzTiVo.

Create a file /home/mythtv/tv_grab_au_tvg.conf with one line like this where UserName and Password are the username and password for your tvguide.org.au account:

http://UserName:Password@minnie.tuhs.org/tivo-bin/xmlguide.pl

Make this file rw------- by chmod 600 so that other users can't access your account logon details. The url for wget is specified in the --input-file param to hide your OzTiVo username and password from process viewers (eg. ps).

To load the guide data into the MythTV database with these two commands or create a file /usr/local/bin/mfd with:

wget --header="accept-encoding: gzip" --input-file=/home/mythtv/tv_grab_au_tvg.conf  --output-document=- | gzip -d > /tmp/xmltv.xml
mythfilldatabase --file 1 7 /tmp/xmltv.xml

You can change the output file name to suit after the ">".

An easy way to run this automatically is to run this from a daily cron job. That can be set up either in the crontab for a particular user, /etc/crontab, /etc/cron.d/ or /etc/cron.daily/. Whichever is best may depend on your linux distribution. One thing to note when running it from cron is that you should put a randomiser into the call so that it doesn't do it at exactly the same time each day. For example on Debian create a file /etc/cron.d/mfd with

0 1 * * * sleep $((RANDOM % 18000)) && /usr/local/bin/mfd

This helps to reduce the peak loads put on the OzTiVo servers by running the script sometime between 1am and 6am every day. Please note Vixie cron requires a '\' before '%' like this 'RANDOM \% 18000'.

The XMLTVIDs in this data probably won't match what you have in your MythTV channel setup, so there are a couple of options to get it to work.

  • Change the XMLTV IDs of your channels in Myth itself.
  • Use sed to translate XMLTVIDs
  • Use xslt to translate XMLTVIDs.
Use sed to translate XMLTVIDs

Change the /usr/local/bin/mfd file with the wget/sed commands all on one line. Make changes for your XMLTV IDs as required.

wget --header="accept-encoding: gzip" --input-file=/home/mythtv/tv_grab_au_tvg.conf 
--output-document=- | gzip -d | sed 's/channel="SBS-NEWS"/channel="news\.sbs\.com\.au"/g ; s/channel="Seven-Syd"/channel="sydney\.seven\.com\.au"/g ; 
s/channel="Ten-NSW"/channel="sydney\.ten\.com\.au"/g ; 
s/channel="SBS-NSW"/channel="sydney\.sbs\.com\.au"/g ; 
s/channel="ABC2"/channel="abc2\.abc\.gov\.au"/g ; 
s/channel="Nine-Syd"/channel="sydney\.nine\.com\.au"/g ; 
s/channel="ABC-NSW"/channel="nsw\.abc\.gov\.au"/g' > /tmp/xmltv.xml
mythfilldatabase --file 1 7 /tmp/xmltv.xml
Use xslt to translate XMLTVIDs

The trouble with the sed method is that you could substitute a string in a program description or elsewhere, rather than just in the channel ID. So to avoid that you can use xslt (extensible stylesheet language transformation) to ensure you only substitute that string when it's part of a channel setting.

Make sure you have the xsltproc binary. (On Fedora Core this is provided by the libxslt rpm).

Change the /usr/local/bin/mfd file with the wget/xsltproc commands all on one line.

wget --header="accept-encoding: gzip" --input-file=/home/mythtv/tv_grab_au_tvg.conf 
--output-document=- | gzip -d | xsltproc /home/mythtv/tv_grab_au_tvg.xsl - > /tmp/xmltv.xml
mythfilldatabase --file 1 7 /tmp/xmltv.xml

The file /home/mythtv/tv_grab_au_tvg.xsl looks like this (thanks go to Joshua King for providing this). If it all looks a bit scary to you, don't worry - all you have to do is modify the channel mappings near the beginning. (Note that this also has some untested code in it to adjust the time offset by 8 hours but the code is commented out).

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:output method="xml" indent="yes"/>

<xsl:template name="mod-channel">
    <xsl:choose>
        <!-- Add channel mappings here -->
        <xsl:when test="string()='ABC-NSW'">nsw.abc.gov.au</xsl:when>
        <xsl:when test="string()='ABC2'">abc2.abc.gov.au</xsl:when>
        <xsl:when test="string()='Seven-Syd'">sydney.seven.com.au</xsl:when>
        <xsl:when test="string()='Nine-Syd'">sydney.nine.com.au</xsl:when>
        <xsl:when test="string()='Ten-NSW'">sydney.ten.com.au</xsl:when>
        <xsl:when test="string()='SBS-NSW'">sydney.sbs.com.au</xsl:when>
        <xsl:when test="string()='SBS-NEWS'">news.sbs.com.au</xsl:when>

        <!-- This ensures any unmatched channels don't become empty -->
        <xsl:otherwise><xsl:value-of select="." /></xsl:otherwise>
    </xsl:choose>
</xsl:template>

<xsl:template name="add-timezone">
    <!-- This emulates force-explicit-timezone -->
    <xsl:value-of select="concat(.,' +0000')" />
</xsl:template>

<xsl:template name="mod-timezone">
    <!-- Change timezone in file by + 8 hours 2 etc lines down -->
    <xsl:choose>
        <xsl:when test="substring(.,9,2) + 8 >= 24">
            <!-- Need to increment day -->
            <xsl:choose>
                <xsl:when test="substring(.,7,2)=31 and substring(.,5,2)=12">
                    <!-- Need to increment year -->
                    <xsl:value-of select="substring(.,1,4) + 1" />0101<xsl:if test="substring(.,9,2) + 8 mod 24 < 10">0</xsl:if><xsl:value-of select="substring(.,9,2) + 8 mod 24" /><xsl:value-of select="substring(.,11)" /><xsl:text> +0800</xsl:text>
                </xsl:when>
                <xsl:when test="(substring(.,7,2)=31 and (substring(.,5,2)=1 or substring(.,5,2)=3 or substring(.,5,2)=5 or substring(.,5,2)=7 or substring(.,5,2)=8 or substring(.,5,2)=10)) or (substring(.,7,2)=30 and (substring(.,5,2)=4 or substring(.,5,2)=6 or substring(.,5,2)=9 or substring(.,5,2)=11))">
                    <!-- Need to increment month -->
                    <xsl:value-of select="substring(.,1,4)" /><xsl:if test="substring(.,5,2) + 1 < 10">0</xsl:if><xsl:value-of select="substring(.,5,2) + 1" />01<xsl:if test="substring(.,9,2) + 8 mod 24 < 10">0</xsl:if><xsl:value-of select="substring(.,9,2) + 8 mod 24" /><xsl:value-of select="substring(.,11)" /><xsl:text> +0800</xsl:text>
                </xsl:when>
                <xsl:when test="(substring(.,7,2)=29 and (substring(.,5,2)=2)) or (substring(.,7,2)=28 and substring(.,5,2)=2 and (substring(.,1,4) mod 4!=0 or substring(.,1,4) mod 400=0))">
                    <!-- Need to increment month -->
                    <xsl:value-of select="substring(.,1,4)" />0301<xsl:if test="substring(.,9,2) + 8 mod 24 < 10">0</xsl:if><xsl:value-of select="substring(.,9,2) + 8 mod 24" /><xsl:value-of select="substring(.,11)" /><xsl:text> +0800</xsl:text>
                </xsl:when>
                <xsl:otherwise>
                    <!-- Nothing to do but day and add -->
                    <xsl:value-of select="substring(.,1,6)" /><xsl:if test="substring(.,7,2) + 1 < 10">0</xsl:if><xsl:value-of select="substring(.,7,2) + 1"/><xsl:if test="substring(.,9,2) + 8 mod 24 < 10">0</xsl:if><xsl:value-of select="substring(.,9,2) + 8 mod 24" /><xsl:value-of select="substring(.,11)" /><xsl:text> +0800</xsl:text>
                </xsl:otherwise>
                     <xsl:value-of select="substring(.,1,6)" /><xsl:if test="substring(.,7,2) + 1 < 10">0</xsl:if><xsl:value-of select="substring(.,7,2) + 1"/><xsl:if test="substring(.,9,2) + 8 mod 24 < 10">0</xsl:if><xsl:value-of select="substring(.,9,2) + 8 mod 24" /><xsl:value-of select="substring(.,11)" /><xsl:text> +0800</xsl:text>
                </xsl:otherwise>
            </xsl:choose>
        </xsl:when>
        <xsl:otherwise>
            <!-- Nothing to do but add -->
            <xsl:value-of select="substring(.,1,8)" /><xsl:if test="substring(.,9,2) + 8 < 10">0</xsl:if><xsl:value-of select="substring(.,9,2) + 8" /><xsl:value-of select="substring(.,11)" /><xsl:text> +0800</xsl:text>
        </xsl:otherwise>
   </xsl:choose>
</xsl:template>

<xsl:template name="copy">
    <xsl:element name="{name()}">
    <xsl:for-each select="@*">
        <xsl:attribute name="{name()}">
        <xsl:choose>
            <xsl:when test="name(..)='programme' and name()='channel'"><xsl:call-template name="mod-channel" /></xsl:when>
            <xsl:when test="name(..)='channel' and name()='id'"><xsl:call-template name="mod-channel" /></xsl:when>

<!-- Uncomment here to try out the time filters
            <xsl:when test="name(..)='programme' and name()='start'"><xsl:call-template name="mod-timezone" /></xsl:when>
            <xsl:when test="name(..)='programme' and name()='stop'"><xsl:call-template name="mod-timezone" /></xsl:when>
-->
            <xsl:otherwise><xsl:value-of select="." /></xsl:otherwise>
        </xsl:choose>
        </xsl:attribute>
    </xsl:for-each>
    <xsl:for-each select="*|text()">
        <xsl:choose>
            <xsl:when test="name()">
                <xsl:call-template name="copy" />
            </xsl:when>
            <xsl:otherwise>
                <xsl:value-of select="string()" />
            </xsl:otherwise>
        </xsl:choose>
    </xsl:for-each>
</xsl:element>
</xsl:template>

<xsl:template match="/">
    <xsl:for-each select="*">
        <xsl:call-template name="copy" />
    </xsl:for-each>
</xsl:template>
</xsl:stylesheet>

Shepherd

For anyone interested in a new tv_grab_au solution, the Shepherd project began in late 2006. As of Feb 2007 it's still in beta but it's a pretty advanced beta and seems to be working well.

Shepherd aims to solve the traditional problems of tv_grab_au scripts by:

  1. employing multiple TV guide grabbers and data sources and switching between them as necessary; and
  2. auto-updating to eliminate the need for users to manually address grabber breakages

There are details here: [1]http://svn.whuffy.com/wiki

Shepherd also includes the ability to update the channel icons displayed within mythTV. Examples of the icon-sets that can used are shown here.

Running Automatically

Put an entry in your crontab something like this:

0 1 * * * sleep $((RANDOM % 18000)) && <whatever the download command is>

This will ensure that your grabber runs sometime between 1am and 7am. The reason for doing it like this is because if everyone does it this way the load on the servers will get smoothed out between these hours. So please be a good netizen and do it this way!

TV Icons

Inspired by FrogBoy's most excellent icon set, a number of icons exist for the free2air channels:

An example of FrogBoy's work follows Au-nine.png Au-sbs-hd.png

Icons are available at: