XMLTV

From MythTV Official Wiki
Jump to: navigation, search


Introduction

XMLTV is the name of a system which obtains and processes program guide information. If you are in North America you can use the Schedules Direct grabber which now supports XMLTV format as well as the old Data Direct format (until v31 and above.) Using the XMLTV format with Schedules Direct gets you more days of data than the old Data Direct option. All users in the USA and other countries need to install and configure XMLTV in order to obtain program guide listings.

Installation

Install the xmltv package on your master backend server, using your distribution's package manager.

The ubuntu packages and others e.g. Debian, Raspbian are likely to have out of date version of xmltv. You can get the latest version as follows in a terminal session:

sudo apt update
sudo apt install git make xmltv perl-doc
# Note: xmltv gets the required perl modules

mkdir -p build
cd build
git clone https://github.com/XMLTV/xmltv.git
cd xmltv
perl Makefile.PL
make
make test
sudo make install

The latest version is installed in /usr/local/bin, which will be used in preference to the original versions in /usr/bin/

use tv_find_grabbers to list all grabbers, including location

Info on XMLTV is here: https://github.com/XMLTV/xmltv.

Configuration

Configuration of XMLTV must be done on the system where mythfilldatabase will be run. Normally that will be your primary backend machine, but you can run it in other places if you prefer.

MythTV Setup

If user mythtv is allowed to login and start the window manager, XMLTV can be configured through the Video Sources Setup page in mythtv-setup. Otherwise, follow the steps below and do the configuration from the command line. Depending on the grabber, it will prompt for any information it requires. If it comes down to selecting channels one at a time manually that can be a challenge if there are many channels.


Information.png Tip: Setup of XMLTV via command line is less challenging and more consistent with XMLTV instructions/usage.


Warning.png Warning: It seems users are missing the step that says run the setup as the user that will run mythfilldatabase (which is frequently user mythtv.)

Some grabbers may not be able to be fully configured through the MythTV setup interface.

The general steps are:

  1. Logon or su to the user that will run mythfilldatabase.
    • In the case of mythbuntu installation that is the user id mythtv.
  2. Run mythtv-setup using that user id.
    • If you do not want to logon to that id or you cannot, you can run the mythtv-setup from another id on the same backend.
  3. Setup Myth to use XMLTV following the instructions at Setup Video Sources.

The next steps are optional, and not recommended. By running all programs as user mythtv, as described above, duplicating the involved directories and files is not required.

  4. After completing setup, copy or link the contents of the $HOME/.xmltv directory to the mythtv home directory
  • The configuration file in $HOME/.xmltv MUST be renamed to the name supplied to mythtv-setup Video Source name. Example: Assuming you named the video source as xmltv, the configuration file name must be named $HOME/.mythtv/xmltv.xmltv.
  5. If copying or linking the files please make sure that the files and directories are read-write permitted by the mythtv user id.

Command Line Setup

Additional configuration can be done from the command line. As long as you have selected your xmltv grabber in MythTV Setup you can perform some or all configuration via the command line.

Logon to the user that will run mythfilldatabase (normally mythtv). As with mythtv setup if you cannot or will not, you can use any user id and then copy or link directories and files as described above.

List Grabbers

tv_find_grabbers

This gives a listing of grabbers, corresponding to the list you see in MythTV Setup. here is a small sample:

/usr/bin/tv_grab_zz_sdjson|Schedules Direct JSON API
/usr/bin/tv_grab_is|Iceland
/usr/bin/tv_grab_es_laguiatv|Spain (laguiatv.com)
/usr/bin/tv_grab_se_swedb|Sweden (swedb/tvsajten)
/usr/bin/tv_grab_combiner|Combine data from several other grabbers
/opt/mythtv/bin/tv_grab_zz_sdjson_sqlite|Multinational (Schedules Direct JSON web services with SQLite DB)

Find in the list the grabber you selected in MythTV Setup, and get the name of the executable from the beginning of the line. Find out information about it with the --info option.

tv_grab_zz_sdjson --info

That will give information on the usage of that grabber. Most options will be common to all grabbers. To use the --info option you need the perl-doc package installed.

Run the configuration

This is the same process that is run from MythTV Setup when you use the configure button in Video Source Setup.

tv_grab_xx_xxxxxx --configure --config-file $HOME/.mythtv/YYYY.xmltv

Use the name of the selected grabber. The value YYYY in the configuration file name corresponds to the Video Source Name. Use your video source name here. If your video source name includes spaces put quotes around it.

The grabber will prompt you required information. If your lineup includes a large number of channels and you do not subscribe to all of them, some grabbers will allow you to select the desired channels.

You should not include channels in your xmltv config which are not required. This places an unnecessarily high load on both the source websites and your own system.

For more information on editing channels see the Channel Editor [1]

Schedules Direct Setup

Schedules Direct now uses XMLTV only. In the USA the XMLTV format provides 18 days of data.

For a list of countries/regions supported see https://www.schedulesdirect.org/regions Up to 4 different lineups are supported at any one time.

Here are detailed steps for setting up to use the XMLTV format. In the following,
$HOME/.mythtv/SD.xmltv
is used as an example. That works for a user logged in (super used to) mythtv. Another example:
~mythtv/.mythtv/SD.xmltv

eliminated the confusion of the $HOME value. Users on systems that have a mythtv user that has no home directory/shell should simply use the full path to the .xmltv file. the sdjson grabbers don't understand $MYTHCONFDIR.

Be sure to read: #MythTV_Setup.

Warning.png
Beginning in v31, the Data Direct format is no longer supported. Also, the mythfilldatabase --dd-grab-all switch is no longer valid. Be sure to remove it with mythtv-setup (General -> Program Schedule Downloading Options -> Guide Data Arguements).

Types of Schedules Direct Grabbers

As XMLTV-from-Schedules Direct is supported using JSON, there are two grabbers that can be used for downloading from Schedules Direct:

  • tv_grab_zz_sdjson_sqllite - a JSON grabber that utilizes a SQL Lite database-based configuration file
  • tv_grab_zz_sdjson - a JSON grabber that utilizes a text-based configuration file

If you run tv_find_grabbers you should see these included in the list:

 tv_find_grabbers | grep Schedules
 /usr/bin/tv_grab_zz_sdjson|Schedules Direct JSON API
 /usr/bin/tv_grab_zz_sdjson_sqllite|Multinational (Schedules Direct JSON web services with SQLite DB)


Information.png Tip: You should obtain the latest version, since there have been recent changes. When you download the a more recent version, you must ensure your copies of the programs are found before the version provided for by your Linux distro - do this by placing ensuring you either replace the versions directly in /usr/bin, or adjusting PATH so that your downloaded version is found before the existing version in /usr/bin.

Use one or the other: The tv_grab_zz_sdjson_sqlite has more options and needs some command line setup that cannot be done from the mythtv-setup program. Both of them cache data allowing them to download a minimal amount of data when running. Only changes since the previous download are downloaded.

Setup with tv_grab_zz_sdjson_sqlite

If you want the very latest version of this you can download it from github.

Run MythTV Setup and select Multinational (Schedules Direct JSON web services with SQLite DB) for your listings source.

From the command line

tv_grab_zz_sdjson_sqlite --manage-lineups --config-file $HOME/.mythtv/SD.xmltv

Replace SD with the name of the source you had set up in MythTV Setup.

  • Enter your Schedules Direct user and password when prompted.
  • Select the Initialize/update the local database option.
  • Accept the default database name.
  • Accept the password hash question.
  • Add or delete lineups using the prompts.

Do not go into channel selection yet. Run this:

tv_grab_zz_sdjson_sqlite --configure --config-file $HOME/.mythtv/SD.xmltv
  • Supply the database name you selected above.
  • Accept defaults for the prompted questions.

Run these:

tv_grab_zz_sdjson_sqlite --days 0 --config-file $HOME/.mythtv/SD.xmltv
tv_grab_zz_sdjson_sqlite --manage-lineups --config-file $HOME/.mythtv/SD.xmltv
  • If you get all channels in the lineup, then select the options that will make current and future channels selected.
  • If you only get a few of the channels you will need to select the appropriate channels. Select a default for existing and future channels, then use option 5 to select which channels you need. This goes though all channels one at a time. You have to complete this for every channel in the list. You cannot stop partway through.
  • If you need to do the channel selection again because of errors or channel changes, your previous values are remembered and used as defaults.

Channel selection shortcut

The channel selection process with tv_grab_zz_sdjson_sqlite is tedious and error prone. Here is a quick and dirty python script to make it easier:

#!/usr/bin/python

# -*- coding: utf-8 -*-
#---------------------------
# Name: sdjson-sqlite-select
#---------------------------
__title__   = "sdjson-sqlite-select"
__version__ = "v0.1.0"

'''
List and update selected status of available callsigns
'''


import time
import argparse
from argparse import RawDescriptionHelpFormatter
import os
import sys
import logging
import traceback

import pymysql
import sqlite3
import json


def process_command_line():
    '''All command line processing is done here.'''

    appname = os.path.basename(sys.argv[0])

    examples = (('\nExamples:\n'
                 '  Extract all callsigns from /home/mythtv/SchedulesDirect.DB '
                 'and place them in /tmp/callsigns:\n'
                 '    {0} --list --database /home/mythtv/SchedulesDirect.DB '
                 '--file /tmp/callsigns\n'
                 '\n'
                 '  Set selected status in /home/mythtv/SchedulesDirect.DB '
                 'from callsigns in /tmp/callsigns:\n'
                 '    {0} --set --database /home/mythtv/SchedulesDirect.DB '
                 '--file /tmp/callsigns\n'
                 '\n'
                 'The callsigns file will list all the callsigns. Any callsign '
                 'with a leading explamation point (aka bang: "!") is/will be '
                 'deselected.\n\n'
                 'When callsigns are listed to the file, The callsigns will be '
                 'sorted, with selected callsigns at the bottom of the file'
                 '\n\n'
                 'Use --list to generate file, then edit the file to add "!" '
                 'in front of any callsigns you do not want downloaded. Use '
                 '--set to apply those changed to the database.')
                .format(appname)
                )

    parser = argparse.ArgumentParser(description='List and update selected '
                                     'status of available callsigns.',
                                     formatter_class=RawDescriptionHelpFormatter,
                                     epilog='{}'.format(examples))

    parser.add_argument('--debug', action='store_true',
                        help='turn on debug messages (%(default)s)')

    parser.add_argument('--version', action='version',
                        version='%(prog)s 0.1.0')

    parser.add_argument('--file', type=str, required=True,
                            metavar='<filename>',
                            help='File for callsigns')

    parser.add_argument('--database', type=str, required=True,
                            metavar='<filename>',
                            help='File for callsigns')

    parser.add_argument('--set', action='store_true',
                        help='Set selected status in SchedulesDirect.DB '
                        'from <file>')

    parser.add_argument('--list', action='store_true',
                        help='List available callsigns into <file>')

    return vars(parser.parse_args())



def list_callsigns(args):
    db = sqlite3.connect(args['database'])
    cursor = db.cursor()

    channels = {}

    for row in cursor.execute('SELECT station,selected from channels'):
        channels[row[0]] = bool(row[1])

    result = []
    for row in cursor.execute('SELECT details from stations'):
        data = json.loads(row[0])
        v = '' if channels[data['stationID']] else '!'
        result.append('{}{}'.format(v, data['callsign']))

    result.sort()
    with open(args['file'], 'w') as f:
        for callsign in result:
            f.write('{}\n'.format(callsign))
        
    return True

def set_selected(args):
    db = sqlite3.connect(args['database'])
    cursor = db.cursor()

    changed = 0

    channels = {}
    with open(args['file'], 'r') as file:
        for callsign in file:
            if callsign[0] == '!':
                selected = False
                callsign = callsign[1:]
            else:
                selected = True
            channels[callsign.strip()] = selected

    stations = {}
    for row in cursor.execute('SELECT details from stations'):
        data = json.loads(row[0])
        stations[data['callsign']] = data['stationID']
        if args['debug']:
             print('callsign {} = station {}'
                  .format(data['callsign'], data['stationID']))

    for chan in channels.keys():
        if args['debug']:
            print('Checking {}'.format(chan))
        st = (stations[chan], )
        for row in cursor.execute('SELECT selected from channels '
                                  'where station=?', st):
            if bool(row[0]) != channels[chan]:
                print('{}: selected {} -> {}'.format(chan, bool(row[0]),
                                                     channels[chan]))
                st = (channels[chan], stations[chan], )
                cursor.execute('UPDATE channels SET selected=? WHERE '
                               'station=?', st)
                db.commit()
                changed += 1

    print('Modified {} channels'.format(changed))
    return True

    
def main():
    args = process_command_line()

    opts = dict()

    if args['set']:
        return set_selected(args)
    elif args['list']:
        return list_callsigns(args)

    return False


if __name__ == '__main__':
    if main():
        sys.exit(0)
    else:
        sys.exit(1)

Use --help to get examples on how to run it.

An alternative method is as follows. First configure the grabber to select no channels. Install sqlite3. List the tables in the sqllite database. If you did not use the default database name, enter the one that you used. You may have to specify your database name with /home/mythtv instead of $HOME. In that case first make sure the file is writeable by your login user id.

sqlite3 $HOME/.xmltv/SchedulesDirect.DB 
SELECT name FROM sqlite_master WHERE type='table';

List the channels.

sqlite3 $HOME/.xmltv/SchedulesDirect.DB 
select * from channels;

You can copy the results from the screen, or pipe the results to a file. The value for selected is 0 if unselected or 1 if selected. Edit the file and produce an xml file like this to set the ones you want to select:

update channels set selected = 1 where channum = 2;
update channels set selected = 1 where channum = 3;
...
update channels set selected = 1 where channum = 983;
update channels set selected = 1 where channum = 986;

Update them with a command like this:

sqlite3 $HOME/.xmltv/SchedulesDirect.DB < selected.sql

Converting from DataDirect to XMLTV

There are several reasons to convert your Datadirect setup to use XMLTV.

  • You get 18 days of data instead of 14.
  • You get icons for all channels.
  • Season and episode information are included in listings.

If you decide to do this, first complete the setup of tv_grab_zz_sdjson_sqlite described above.

When you switch, the xmltvid's for the existing channels will change. Using this grabber the xmltvid has the format I11460.json.schedulesdirect.org. Using DD the xmltvid's are a numeric value.

There are two options for the conversion:

  1. Delete all your channels and let mythfilldatabase add them back. This option is not available in cases where you have to scan for channels using DVB or ATSC cards.
  2. Manually update each channel xmltvid.

I recommend that you backup your database and make sure there are no recordings scheduled for the near future before starting.

Deleting and re-adding channels

This is only possible for card types that do not need scanning for channels.

Before attempting this bear in mind the following. If any channel's chanid changes, your existing programs will show the wrong channel that the program was recorded on.

The mechanisms in MythTV for adding channels use a format of sourceid and channel number to create a chanid wherever possible. So channel 2 on source id 2 will be chanid 2002, for example. If it cannot do that then it will allocate a chanid number which is unrelated to the channel number or source id.

In order to see the chanids you will have to look at the channel table in the database. See if your existing chanid numbers match the scheme described in the previous paragraph.

If the chanids match the scheme above you should be able to safely delete them and let mythfilldatabase add them.

You can delete all of the existing channels for each video source in mythtv-setup, channel editor. Select the Video source and click Delete.

To add all of the channels you have selected as well as listings, run mythfilldatabase. Default options will add channels and listings.

Manually Updating xmltvids

If you prefer not to delete your channels or you have a DVB type tuner you will have to update the xmltvids in all of them. Using this grabber the xmltvid has the format Innnnn.json.schedulesdirect.org. You will have to change all of your existing ids to this format. The easiest place to make this change is with mythweb channel editor, although you can use the mythtv-setup channel editor as well. You can get the list of channels with their new format xmltvids by running tv_grab_zz_sdjson_sqlite --days 0 --config-file $HOME/.mythtv/SD.xmltv, which is one step of the setup process (see above). If you did not capture the output you can run this again at any time, no harm will be done.

Once you have updated all of the xmltvid's, run mythfilldatabase. Your channel names and icons will be updated to match the values supplied by schedules direct.

In a tuner type that does not use scanning, such as a cable card tuner, any channels not found with matching xmltvids are added. If you have missed updating some xmltvids that could result in duplication of the channel.

Setup with tv_grab_zz_sdjson

Run MythTV Setup and select Schedules Direct JSON API for your listings source.

You can complete the setup using the Configure button in MythTV Setup. If you prefer to use the command line, run

tv_grab_zz_sdjson --configure --config-file $HOME/.mythtv/SD.xmltv

Replace SD with the name of the source you had set up in MythTV Setup.

  • Enter your Schedules Direct user and password when prompted.
  • For channel id format you can select any of the supplied options. Option 0 is the same format as used by tv_grab_zz_sdjson_sqlite. Option 2 is the format used by Schedules Direct Data Direct.
  • Follow the prompts to complete setup.
  • When selecting channels the selection process allows you to skip remaining channels in the list and mark them all as yes or no.
  • If you need to do the channel selection again because of errors or channel changes, your previous values are lost. You have to do the entire selection again.

Channel selection shortcut

The channel selection process with tv_grab_zz_sdjson is tedious and error prone. An alternative method is as follows. First configure the grabber to select all channels. Edit the file $HOME/.mythtv/SD.xmltv (replace SD with the name of the source you had set up in MythTV Setup). You will see a bunch of lines like this.

channel=74348
channel=11460

When a channel is selected it has =, when unselected it has !. You need to set the channels you want to use = and the others to !.

Get a list of channels by running

tv_grab_zz_sdjson --days 0 --config-file $HOME/.mythtv/SD.xmltv > list.xml

Although this requests 0 days it actually produces 1 day of programs. At the start of the file before the programs is the channel listing. Delete the program schedules and keep the channel listing at the top of the file. The channel id and display names identify each channel by its XMLTV id and the number and name. Note that the channel id in this file may have extra characters appended depending on the format you chose during configure. For purpose of the configuration file you only need the 5 numeric digit of the channel id. Using your favorite editor and your provider's channel list update the channel lines in the configuration file to have = for ones you want and ! for ones you do not.

List of Grabbers

Important.png Note: Not all grabbers on this page are XMLTV compliant. Only fully compliant grabbers are supported by MythTV and we advise against the use of the rest.

If you know a grabber listed here does not conform entirely to both the xmltv grabber specifications (XMLTV grabber spec) and XMLTV xml format (XMLTV Format) then please mark it as unsupported. If in doubt XMLTV provide a script to validate a grabber (XMLTV Validator)

There is also a list on the XMLTV wiki.

EPG Collector

EPG Collector is a .NET/Mono program for collecting the transmitted guide data from DVB or ATSC streams in various formats and creating an XMLTV file.

Multinational

Schedules Direct supports countries in North America, South America, Caribbean and Europe for $25 (US) per year. For a list of supported countries see http://schedulesdirect.org/regions. See #Schedules_Direct_Setup above

Australia

Refer to Australian TV Listings for further details.

  1. OzTivo Guide (Free, requires registration)
  2. http://www.icetv.com.au (AU$99 per year subscription)
  3. Shepherd (Free: hybrid multi source)

See also this (largely historical) list of Australian grabbers.

Belgium

The following is a list of grabbers for Belgium. It is presented only for historical purposes, because at the time of writing this (January 2011) none of them work.

  1. https://launchpad.net/belguide
  2. http://users.skynet.be/jxmltv
  3. http://pytvgrab.sourceforge.net/

Brazil

There are a working grabber, tv_grab_br_net a perl scripts (or any other Linux native solutions) for Brazilian TV programming. This grabber get data from Net [2] cable TV. No open to air TV on this source, just paid tv programme. It is detected by mythtvsetup and you are prompted (on background windows) to make your choice about your city, line-up and channels. It provide both, simple and complete description for programs.

Other functional alternative is to use TVxb, which isn't open source or even Linux, but works well with wine (at least as far as version 1.0085).

TVxb is configurable for different countries via ini-files. Check the following page for Brazilian templates. As of today, there are some for NET (PR), and others for TVA as well.

http://www.tvxb.com/country/br-ini.html

For information on how to set it up with MythTV, check out TVxb.

Croatia

The grabber in the official XMLTV distribution works correctly. More information is available on http://www.gonix.net/.

Czech Republic

The XMLTV grabber for "Hungary/Romania" was extended to the Czech Republic.

Another way to get TV listings in Czech Republic is TVxb.

Not public XMLTV file for czech and slovak channels (UPC channels included) can be send after request to rfordinal.

Denmark

In Denmark there is a group maintaining several grabbers. This group has created the following:

Currently three grabbers are maintained:

  • DR 2012 grabber written in perl (dr.dk)
  • An additional written in python (yousee.dk)
  • An universal grabber combining data from mutiple sources (tvtid.tv2.dk, yousee.dk, tv-guiden.dk, jubii.dk, ontv.dk, ahot.dk, dr.dk and swedb)

Finland

The grabbers in the official XMLTV distribution work correctly. Alternate source: http://xmltv.se

France

The grabber in the upcoming XMLTV release works correctly. More information is available on [3].

Germany/Austria

  1. XMLTV.se offers many of the main channels for personal use, the grabber from the main xmltv distribution works.
  2. Some channels are available via Egon Zappt in the upcoming release of xmltv.
  3. TV Movie can not give out their data without additional license, see Ben Bucksch's log.

Greece

Try this link for Nova Greece: http://sgcpm.com/epg/devilcosta.gz, for example using mythfilldatabase. (Be aware that its just a dump of DVB-EIT using the service_id as channel_id which is not proper xmltv. Please consider writing a proper grabber for the file so it can be configured via the GUI or simply provide a proper guide feed with tools like NonameTV.)

India

The grabber for What's-On-India has been added to the official XMLTV distribution.

An XMLTV grabber for all channels in India can be found at http://ravihtpc.wordpress.com/2010/01/09/india-epg-data-for-mythtv-xmltv-listings-grabber/.

Italy

The grabber in the official XMLTV distribution works correctly.

Hong Kong

First thanks to http://www.akbkhome.com/blog.php/View/119/Mythtv_xmltv_importer_for_HK.html.

I modify his source code to get information from TVB new web site.

   download Tv.php http://mythtv-xmltvhk.googlecode.com/files/Tv.php
   download hongkong.ini http://mythtv-xmltvhk.googlecode.com/files/hongkong.ini
   build xml from tvb, atv website using following command, just entry 
          php ./Tv.php hongkong.ini > hongkong.xml
   using 'myfilldatabase' to update mythtv schedule program, just entry
          mythfilldatabase --file 1 hongkong.xml
   (* it is my first development on php, so it may be simplified *)

[Last Update by hwkit (17/04/2010)]

Since TVB web site is changed, the program need to be updated. (My php skill is poor, so there are error when running, however it also can get schedule from TVB web site), I also change the program name called 'xmltvhk.php'. Please download it from google.

   download xmltvhk.php and copy it to /etc/xmltvhk (http://mythtv-xmltvhk.googlecode.com/files/xmltvhk.php)
   download updated hongkong.ini and copy it to /etc/xmltvhk (http://mythtv-xmltvhk.googlecode.com/files/hongkong.ini)
   download filltv  and copy it to /etc/cron.weekly (make it executable (chmod +x /etc/cron.weekly/filltv) (http://mythtv-xmltvhk.googlecode.com/files/filltv)
   Source code 
   svn checkout http://mythtv-xmltvhk.googlecode.com/svn/trunk/ mythtv-xmltvhk-read-only

[Last Update by hwkit (07/07/2010)]

Netherlands

Netherlands (Holland) specific info can be found here: Netherlands

New Zealand

Sky TV threatened legal action to get the xmltv grabber for NZ taken down. You can still receive the transmitted guide in MHEG5 with tools like EPG Collector.

Norway

The grabber in the official XMLTV distribution works correctly. Another source of info: http://xmltv.se

Slovakia

The XMLTV grabber for "Hungary/Romania" was extended to Slovakia.

South Africa

See the South Africa MUG WIKI page for details.

Spain

Both grabbers in the official XMLTV distribution work correctly.

Sweden

The grabber in the official XMLTV distribution works correctly. More information is available on http://xmltv.se.

Switzerland

The grabber in the official XMLTV distribution works correctly.

United Kingdom

The grabbers in the official XMLTV distribution work correctly. A brief guide to configuring tv_grab_uk_rt is available, and one for Metabroadcast Atlas.

Ukraine, Russia, Belarus

  1. tv_grab_ru Converts JTV-formatted archive with about 200 channels to XMLTV. But this application does not show additional info about programmes.
  2. tv_grab_ua Downloads XMLTV-formatted data from sites like teleguide, izbushka or star.poltava.ua. In addition it parses existing data and adds category/rating/credits into output XMLTV file.
  3. teleguide.info XMLTV-formatted zip archive with programmes inside.

Latvia