Australian TV Listings
Contents
Guide Data Copyright In Australia
Australia doesn't have a service like Zap2It to provide TV program guide listings. Over the years, several people have written grabbers to scrape the data from web sites such as Yahoo, Nine-MSN, YourTV etc. These grabbers have frequently stopped working because of efforts from companies such as Nine-MSN to obfuscate the data. Each time, the workaround is trivial, but the continual changes are draining on the volunteers maintaining the grabbers.
It is unknown what purpose these companies have in trying to prevent their TV guides being used by their audience, possibly some misguided sense of copyright.
Sources Of Guide Data
There are a number of sources of data available with different grabbers. One list is maintained here.
Nine-MSN and Yahoo etc grabbers
http://www.whuffy.com/tv_grab_au/
Currently (10 Sept 2006) the Rex 1.x grabber is broken due to changes in the web site. It has previously been one of the best sources of data. While it doesn't get its data from NineMSN or Yahoo it is based on a grabber which used to. Rex 1.x is broken but there is a later version at the link above does work at time of writing (17-Sep-06).
Ice TV
You can purchase the guide data from Ice TV with a subscription. The tv_grab_au_reg grabber will work with both IceTV and OzTiVo data sources.
OzTiVo
Alternatively, for those who want guide data for free, there is a source of data which is freely and openly available and designed to be downloaded for PVRs. This is provided by the nice folks at OzTiVo. and was originally designed for Aussie owners of TiVo boxes. However, because it outputs guide data in XMLTV format it can also be used by MythTV users. Have a look at the Australian XMLTV grabber page to get more details. In order to use this you will need to create a free account on their web site first.
tv_grab_au_reg
This script written to grab the OzTiVo data is a python script which does some clever things like:
- store a username/password in a config file.
- re-write XMLTV ids (useful if you're switching from a different data source).
- force the addition of explicit timezones on timestamps (useful for people from some states).
- filter out specific channels entirely.
- grab some channel's data from tvguide.org.au and some from IceTV and merge the result into one XMLTV dataset (tvguide.org.au covers more channels than IceTV).
- gives newbies who are looking for a 'tv_grab_au' script something to get hold of.
- use standard XMLTV grabber script options.
You can find the script plus some other useful info about Aussie grabbers at William Uther's site.
xmlguide.pl
However, an alternative is to use the xmlguide.pl script which is a script built into the whole tvguide.org.au service. This script is good for those who want to get their hands dirty, and because it's built in to the service, if the API changes you should be covered automatically with no need to make any updates. Users who go this route are requested to follow the tvguide.org.au mailing list to stay on top of any changes to the service.
When accessed, the server script queries the preferences associated with your username and password to decide which channels to download. It then extracts 7 days worth of data for those channels into XMLTV format which you can then load into MythTV using mythfilldatabase. Full details available at XMLGuideAPI.
The XMLTVIDs in this data probably won't match what you have in your MythTV channel setup, so there are a couple of options to get it to work. Both of the options below use wget. The url for wget is specified in the --input-file param as it needs to contain your tvguide.org.au username and password. Make this file rw------- by chmod 600 so that others can't see your account logon details. The file should have one line like this where UserName and password are the username and password for your tvguide.org.au account:
http://UserName:password@minnie.tuhs.org/tivo-bin/xmlguide.pl
Use sed to translate XMLTVIDs
- Put the commands below in a shell script (NB - the wget/sed command should be all one line).
- Make changes for your XMLTV IDs if required. Either:
- Change the sed commands to be suitable for your own XMLTV IDs, or
- Change the XMLTV IDs of your channels in Myth itself.
- Change the output file name to suit after the ">"
wget --header="accept-encoding: gzip" --input-file=/usr/local/bin/mythtv/tv_grab_au_tvg.conf --output-document=- | sed 's/channel="SBS-NEWS"/channel="news\.sbs\.com\.au"/g ; s/channel="Seven-Syd"/channel="sydney\.seven\.com\.au"/g ; s/channel="Ten-NSW"/channel="sydney\.ten\.com\.au"/g ; s/channel="SBS-NSW"/channel="sydney\.sbs\.com\.au"/g ; s/channel="ABC2"/channel="abc2\.abc\.gov\.au"/g ; s/channel="Nine-Syd"/channel="sydney\.nine\.com\.au"/g ; s/channel="ABC-NSW"/channel="nsw\.abc\.gov\.au"/g' > /usr/local/bin/mythtv/xmltvSydney.xml
Use xslt to translate XMLTVIDs
The trouble with the sed method is that you _could_ substitute a string in a program description, for example, rather than just in the channel ID. So to avoid that you can use xslt (extensible stylesheet language transformation) to ensure you only substitute that string when it's part of a channel setting.
- Put the commands below in a shell script (NB - the wget/xsltptoc command should be all one line)
- Change the output file name to suit after the ">"
wget --header="accept-encoding: gzip" --input-file=/usr/local/bin/mythtv/tv_grab_au_tvg.conf --output-document=- | xsltproc /usr/local/bin/mythtv/tv_grab_au_tvg.xsl - > /usr/local/bin/mythtv/xmltvSydney.xml
The file tv_grab_au_tvg.xsl looks like this (thanks go to Joshua King for providing this). If it all looks a bit scary to you, don't worry - all you have to do is modify the channel mappings near the beginning. (Note that this also has some untested code in it to adjust the time offset by 8 hours but the code is commented out).
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:output method="xml" indent="yes"/>
<xsl:template name="mod-channel">
<xsl:choose>
<!-- Add channel mappings here -->
<xsl:when test="string()='ABC-NSW'">nsw.abc.gov.au</xsl:when>
<xsl:when test="string()='ABC2'">abc2.abc.gov.au</xsl:when>
<xsl:when test="string()='Seven-Syd'">sydney.seven.com.au</xsl:when>
<xsl:when test="string()='Nine-Syd'">sydney.nine.com.au</xsl:when>
<xsl:when test="string()='Ten-NSW'">sydney.ten.com.au</xsl:when>
<xsl:when test="string()='SBS-NSW'">sydney.sbs.com.au</xsl:when>
<xsl:when test="string()='SBS-NEWS'">news.sbs.com.au</xsl:when>
<!-- This ensures any unmatched channels don't become empty -->
<xsl:otherwise><xsl:value-of select="." /></xsl:otherwise>
</xsl:choose>
</xsl:template>
<xsl:template name="add-timezone">
<!-- This emulates force-explicit-timezone -->
<xsl:value-of select="concat(.,' +0000')" />
</xsl:template>
<xsl:template name="mod-timezone">
<!-- Change timezone in file by + 8 hours 2 etc lines down -->
<xsl:choose>
<xsl:when test="substring(.,9,2) + 8 >= 24">
<!-- Need to increment day -->
<xsl:choose>
<xsl:when test="substring(.,7,2)=31 and substring(.,5,2)=12">
<!-- Need to increment year -->
<xsl:value-of select="substring(.,1,4) + 1" />0101<xsl:if test="substring(.,9,2) + 8 mod 24 < 10">0</xsl:if><xsl:value-of select="substring(.,9,2) + 8 mod 24" /><xsl:value-of select="substring(.,11)" /><xsl:text> +0800</xsl:text>
</xsl:when>
<xsl:when test="(substring(.,7,2)=31 and (substring(.,5,2)=1 or substring(.,5,2)=3 or substring(.,5,2)=5 or substring(.,5,2)=7 or substring(.,5,2)=8 or substring(.,5,2)=10)) or (substring(.,7,2)=30 and (substring(.,5,2)=4 or substring(.,5,2)=6 or substring(.,5,2)=9 or substring(.,5,2)=11))">
<!-- Need to increment month -->
<xsl:value-of select="substring(.,1,4)" /><xsl:if test="substring(.,5,2) + 1 < 10">0</xsl:if><xsl:value-of select="substring(.,5,2) + 1" />01<xsl:if test="substring(.,9,2) + 8 mod 24 < 10">0</xsl:if><xsl:value-of select="substring(.,9,2) + 8 mod 24" /><xsl:value-of select="substring(.,11)" /><xsl:text> +0800</xsl:text>
</xsl:when>
<xsl:when test="(substring(.,7,2)=29 and (substring(.,5,2)=2)) or (substring(.,7,2)=28 and substring(.,5,2)=2 and (substring(.,1,4) mod 4!=0 or substring(.,1,4) mod 400=0))">
<!-- Need to increment month -->
<xsl:value-of select="substring(.,1,4)" />0301<xsl:if test="substring(.,9,2) + 8 mod 24 < 10">0</xsl:if><xsl:value-of select="substring(.,9,2) + 8 mod 24" /><xsl:value-of select="substring(.,11)" /><xsl:text> +0800</xsl:text>
</xsl:when>
<xsl:otherwise>
<!-- Nothing to do but day and add -->
<xsl:value-of select="substring(.,1,6)" /><xsl:if test="substring(.,7,2) + 1 < 10">0</xsl:if><xsl:value-of select="substring(.,7,2) + 1"/><xsl:if test="substring(.,9,2) + 8 mod 24 < 10">0</xsl:if><xsl:value-of select="substring(.,9,2) + 8 mod 24" /><xsl:value-of select="substring(.,11)" /><xsl:text> +0800</xsl:text>
</xsl:otherwise>
<xsl:value-of select="substring(.,1,6)" /><xsl:if test="substring(.,7,2) + 1 < 10">0</xsl:if><xsl:value-of select="substring(.,7,2) + 1"/><xsl:if test="substring(.,9,2) + 8 mod 24 < 10">0</xsl:if><xsl:value-of select="substring(.,9,2) + 8 mod 24" /><xsl:value-of select="substring(.,11)" /><xsl:text> +0800</xsl:text>
</xsl:otherwise>
</xsl:choose>
</xsl:when>
<xsl:otherwise>
<!-- Nothing to do but add -->
<xsl:value-of select="substring(.,1,8)" /><xsl:if test="substring(.,9,2) + 8 < 10">0</xsl:if><xsl:value-of select="substring(.,9,2) + 8" /><xsl:value-of select="substring(.,11)" /><xsl:text> +0800</xsl:text>
</xsl:otherwise>
</xsl:choose>
</xsl:template>
<xsl:template name="copy">
<xsl:element name="{name()}">
<xsl:for-each select="@*">
<xsl:attribute name="{name()}">
<xsl:choose>
<xsl:when test="name(..)='programme' and name()='channel'"><xsl:call-template name="mod-channel" /></xsl:when>
<xsl:when test="name(..)='channel' and name()='id'"><xsl:call-template name="mod-channel" /></xsl:when>
<!-- Uncomment here to try out the time filters
<xsl:when test="name(..)='programme' and name()='start'"><xsl:call-template name="mod-timezone" /></xsl:when>
<xsl:when test="name(..)='programme' and name()='stop'"><xsl:call-template name="mod-timezone" /></xsl:when>
-->
<xsl:otherwise><xsl:value-of select="." /></xsl:otherwise>
</xsl:choose>
</xsl:attribute>
</xsl:for-each>
<xsl:for-each select="*|text()">
<xsl:choose>
<xsl:when test="name()">
<xsl:call-template name="copy" />
</xsl:when>
<xsl:otherwise>
<xsl:value-of select="string()" />
</xsl:otherwise>
</xsl:choose>
</xsl:for-each>
</xsl:element>
</xsl:template>
<xsl:template match="/">
<xsl:for-each select="*">
<xsl:call-template name="copy" />
</xsl:for-each>
</xsl:template>
</xsl:stylesheet>
To load the guide data into the MythTV database:
mythfilldatabase --file 1 7 /usr/local/bin/mythtv/xmltvSydney.xml
Running automatically
An easy way to run this automatically is to run this from a daily cron job. That can be set up either in the crontab for a particular user, or /etc/crontab or /etc/cron.daily/. Whichever is best may depend on your linux distribution. One thing to note when running it from cron is that you should put a randomiser into the call so that it doesn't do it at exactly the same time each day. Eg:
0 1 * * * sleep $((RANDOM % 18000)) && <whatever the download command is>
This helps to reduce the peak loads put on the tvguide.org.au servers by running the script sometime between 1am and 6am every day.
