NexTView Import

From MythTV Official Wiki
Jump to: navigation, search

This How-To describes a procedure to harvest EPG data for MythTV using the nexTView protocol. This method is only interesting for those who have access to the European continent. This data is not sent in the rest of the world.


Important.png Note: The scripts and methods used to import Nextview data into MythTV shown on this page are not officially supported or recommended. The preferred solution would be to create a fully XMLTV compliant grabber. This would be much easier to setup and changes within mythtv would not require scripts to be changed or re-written.


Required Software

nxtvepg - This software provides an interface to the nexTView protocol, both in console and as a GUI (X) application. 2.7.5 should be the lowest version that you should install.

Install 'nxtvepg'
sudo apt-get install nxtvepg

Install 'scantv'
sudo apt-get install scantv

Setup and Testing nxtvepg

Get the latest software as described above. Now start 'nxtvepg' to do some tests, try to get your EPG data.

nxtvepg

If this works you should now have a new window, there go to
Configure -> Provider scan -> start-scan
Configure -> Select Provider

you can exit nxtvepg. Now try to harvest data to be exported to XMLTV format. This is done with the following command:

nxtvepg -daemon -acqonce full -provider d92 -nodetach

This runs nxtvepg without GUI and acquires the provider d92 (Kabel1 in Germany) and leaves the console locked until the harvesting is done. Be patient - this may take a long time (in the range of 20 minutes to 50min). Now we will try to export the harvested data to a XML file.

nxtvepg -provider d92 -dump xml > /tmp/nxtvepgtest.xml

You should now have a file in /tmp of about 1 Megabyte size. Open it up in your favorite editor to check for correctness.

Scripts

At first copy the cnilist and the 2 Scripts(autochannel.sh & harvest) to a folder e.g. /home/yourname/epg give them the name they have. After that, come back here and follow the rest of the How to.
Ensure that you have an empty line at the end of every script

Channel list

Following list of channels should be extended to include those you might need:

Script.png cnilist

# CNI-Code | Name (must fit to scantvdump and is Case Sensitive) | callsign  
# This list is based on SF 1, you may have to change to your needs.
CNI04C1=SF 1=SF 1
CNI04C7=SF 2=SF 2
CNI04CC=SFi=SFi
CNI0DC1=ARD=ARD
CNI0DC2=ZDF=ZDF
CNI0497=3PLUS=3+
CNI0DC7=3sat=3SAT
CNI0D85=Arte=ARTE
CNI0DC9=ARD/ZDF Kinderkanal=[[Ki Ka]]
CNI0DCB=BR-3=BR 3
CNI0DFE=MDR-3 Landesweit=MDR3
CNI0DE0=SW 3 Baden-Wuerttemberg=SW 3
CNI1DE6=WDR-3 Landesweit=WDR3
CNI0D8D=DSF=DSF
CNI0D91=Eurosport=EUSP
CNI0D92=Kabel 1=KAB1
CNI0D8C=n-tv=NTV
CNI0487=PRO 7=PRO7
CNI0DAB=RTL=RTL
CNI0D8F=RTL 2=RTL2
CNI0DB9=SAT 1=SAT1
CNI0D8A=Super RTL=SRTL
CNI0D8E=VOX=VOX
CNI0AC1=ORF 1=ORF1
CNI0AC2=ORF 2=ORF2
CNI04C2=TSR 1=TSR1
CNI04C8=TSR 2=TSR2
CNI33F1=TF1=TF 1
CNI33F2=France 2=FR 2
CNI33F3=France 3=FR 3
CNI33F6=M6=M6
CNI33F8=RTL 9=RTL9
CNIF500=TV5=TV 5
CNI04C3=TSI 1=TSI1
CNI04C9=TSI 2=TSI2
CNI3901=RAI 1=RAI1
CNI3902=RAI 2=RAI2
CNIFA05=Canale 5=CNL5
CNIFA06=Italia 1=IT 1
CNIFA04=RETE 4=RET4
CNI9001=TRT-1=TRT1
CNI0DD4=Nord-3 (NDR/SFB/RB)=N3
CNI0DBA=NEUN LIVE=9LIV 
CNI0DCF=Hessischer Rundfunk=HR 3
CNI0D82=Radio Berlin-Brandenburg=RBB
CNI0D88=VIVA 1=VIVA
CNIFFEF=MTV=MTV
CNI0D7A=N 24=N24
CNI0D78=Tele 5=Tele5
???????=BBC Prime=BBC1
???????=RTP1=RTP1
CNI0x0486=Star TV=STAR
CNI0x01F2=CNN International=CNNI
???????=Canal+=CNL+
CNI0x0481=TeleZüri=TZ


Automatic import channel data

The following script scans for channels then combines the channels with the cnilist and adds these to MythTV.
To test it run the command: (for Hauppauge read further down)
scantv -o /home/yours/scantv_output


Hauppauge PVR owner

I testet the script with PVR-350 and PVR-500(2x150inside). I had to do this change in the script autochannel.sh:
(You still get an error-message, but it works anyway)

# Scan input for channel information and dump it into a file
#scantv -a -f europe-west -n PAL-BGH -o $autochannel
# comment out the line above for PVR and use this instead
scantv -a -f europe-west -n PAL-BGH -c /dev/video0 -C /dev/vbi0 -o $autochannel

The script autochannel.sh will run a full scan and takes 30 - 60min, you will see the lines in the bash passing by, while scanning. In the path you start it, it will create a file called scantvdump_(day-number).txt .

It is recommended using this only if you have not added channels manually, this is because it will mix up your already entered channels. If you can live with it, go ahead.
First you need to make it executeable sudo chmod 777 autochannel.sh. Then enter your password in this script (it is in the 4 line before bottom) for the MySQL-Database you can find it in /etc/mythtv/mysql.txt.


Script.png autochannel.sh

#!/bin/bash
# Scan tv source for channel information and insert it into MythTV's database
# Written by zagibu@gmx.ch
# Distributed under the terms of the GNU General Public License v2
# Date: 21.05.2005
# Script update: 27.01.2008 added line for HPG PVR's, MySQl options changed to REPLACE, hint about password, removed CH for channel entry

version=v0.1
command_list="scantv mysql head tail cut wc"
autochannel=scantvdump_$(date +\%d).txt

# Verify that each command we use exists
for command in $command_list; do
        path=$(which $command | grep "no $command in")

        if [ ! -x $(which $command) -a "$path" ]; then
                echo -e "\n\nERROR: $command not found! Check your \
                $command_list and/or your \$PATH"
                exit -1
        fi
done

echo -e "\nScanner $version"
echo "============"

# Scan input for channel information and dump it into a file
scantv -a -f europe-west -n PAL-BGH -o $autochannel
# comment out the line above for PVR and use this instead
#scantv -a -f europe-west -n PAL-BGH -c /dev/video0 -C /dev/vbi0 -o $autochannel

# Count lines of scantv dump
lines=$(cat $autochannel | wc -l | cut -f1 -d' ')
# Subtract amount of header lines (7)
lines=$(expr $lines - 7)

# Read each channel entry of scantv dump
for (( i=0; i<$lines; i=$i+3 ))
do
        chanid=$(expr $(expr $i + 3) / 3)
        channum=$chanid
        freqid=$(tail -n$(expr $lines - $(expr $i + 1)) $autochannel | head -n1 | cut -f3 -d' ')
        name=$(tail -n$(expr $lines - $i) $autochannel | head -n1)
        name=$(echo $name | cut -f2 -d'[' | cut -f1 -d']')
        callsign=$(cat cnilist | grep "=$name=" | cut -f3 -d'=')
        xmltvid=$(cat cnilist | grep "=$name=" | cut -f1 -d'=')

   # Insert gathered channel info into mythtv's db
   # If you use the short option form (-p), you CANNOT have a space between the option and the password
   mysql -u mythtv -pyour_passw -D mythconverg -e"INSERT INTO channel 
(chanid, channum, freqid, sourceid, callsign, name, xmltvid, tvformat) VALUES 
($chanid, '$channum', '$freqid', 1, '$callsign', '$name', '$xmltvid', 'PAL');"
done 


Check the import of channels and xmltvID

Instead of doing it one by one with mythtv-setup, we can read out the mySQL-Database. Please be very careful, it is YOUR database!
To do that, you need to know the password for your database. You already read it out before to run the script autochannel.sh or you look in /etc/mythtv/ there you find the file mysql.txt inside is your password (on a mythbuntu).

Login to your mySQL type:
mysql -u mythtv -pyour_passw -D mythconverg

select channum, xmltvid, name, callsign, chanid, tvformat, freqid from channel where sourceid = "1";

Now you should get a pretty output like:

Script.png mySQL output of channels
+---------+-----------+-------------------------+---------------+--------+----------+--------+

channum : xmltvid  : name  : callsign  : chanid : tvformat : freqid :

+---------+-----------+-------------------------+---------------+--------+----------+--------+

1  : CNI04C1  : SF 1  : SF 1  : 1 : PAL  : 175600 :
2  : CNI04C7  : SF 2  : SF 2  : 2 : PAL  : 210910 :
3  : CNI0DC7  : 3sat  : 3SAT  : 3 : PAL  : S33  :
4  : CNI0AC1  : ORF 1  : ORF1  : 4 : PAL  : SE5  :
5  : CNI0497  : 3+  : 3Plus  : 5 : PAL  : 260310 :
6  : CNI0D8E  : VOX  : VOX  : 6 : PAL  : 30  :
7  : CNI0487  : PRO 7  : PRO7  : 7 : PAL  : 27  :

...

36  :  : Home Shopping Europe  : Home SE  : 36 : PAL  : 495230 :

+---------+-----------+-------------------------+---------------+--------+----------+--------+

Do a clean logout of mySQL with: quit

Acquiring EPG-data and Sync Clock

The following script allows to automate the process we tested above of harvesting EPG-data. Make sure to change the provider code in the script, to your preferred provider - see nexTView Provider Table
First you need to make the script executeable sudo chmod 777 harvest

It must be tested (started/run) as myth-user. Go to the to the directory you saved it and start it with:
./harvest

While it is acquiring data, you can stop it any time with Ctrl+c to see if it works instead of waiting 20-50min.
A Logfile helps us to keep track of the EPG-Data updates. Create an empty text file in /var/log
sudo touch /var/log/epg_data.log

Then change the owner of the file
sudo chown yourusername:mythtv /var/log/epg_data.log


Script.png harvest

SHELL=/bin/bash
#!/bin/bash
# Get time and epg data from teletext, then insert it into MythTV's database
# Written by zagibu@gmx.ch
# Distributed under the terms of the GNU General Public License v2
# Date: 21.05.2005
# Script update: 27.01.2008 added start-/finshtime to print
# Script update: 07.04.2008 removed the '-1' at the last command
# Script update: 13.08.2008 added: SHELL=/bin/bash at top, code for start/stop output new into logfile.

version=v0.1
command_list="nxtvepg hwclock mythfilldatabase"
epgdump=nxtvepgdump_$(date +\%d).txt

# verify that each command we use exists
for command in $command_list; do
        path=$(which $command | grep "no $command in")

        if [ ! -x $(which $command) -a "$path" ]; then
                echo -e "\n\nERROR: $command not found! Check your \
                $command_list and/or your $PATH"
                exit -1
        fi
done

echo "Adjusting time (using teletext as reference)..."
# Set system time to current teletext time
nxtvepg -clock set
# Save system time to hardware clock
hwclock --systohc

# Remember to change this to your NexTView provide of choice
echo "* * Acquiring NexTView EPG from provider Kabel 1 & SF 1... started `date '+%R %a %d %b'` * *" >> /var/log/epg_data.log
echo "* * Acquiring NexTView EPG from provider Kabel 1 & SF 1... started `date '+%R %a %d %b'` * *"
nxtvepg -acqonce full -daemon -nodetach -provider 0D92
# Dump current EPG data into a file (xmltv5 format must be set in .nxtvepgrc)
echo "* Acquiring NexTView EPG from provider Kabel 1 & SF 1...finished   `date '+%R %a %d %b'` *" >> /var/log/epg_data.log
echo "* finished acquiring  `date +%R` *"
nxtvepg -dump xml -provider 0D92 > $epgdump

echo "Loading [[Next View]] EPG data into MythTV..."
# Insert EPG data into MythTV's database, do not add more channels
mythfilldatabase --update --file 1 $epgdump

echo "All processes completed. Enjoy your [[Myth Box]]!" 


Cron Job to automate the process

How to automate the harvest.
It depends on your use, if the machine is running all day long you may want it in cron to be done at 3 o'clock in the morning. If you shutdown your machine daily you may want it to run right in the boot-process.
cron and/or anacron

Anacron is meant for machines that don't run 24x7.

  • cron - daemon to execute scheduled commands
  • anacron - runs commands periodically

What is anacron:

Anacron (like `anac(h)ronistic') is a periodic command scheduler.
It executes commands at intervals specified in days. Unlike cron, it does not assume that the system is running continuously.

It can therefore be used to control the execution of daily, weekly and monthly jobs (or anything with a period of n days), on systems that don't run 24 hours a day. When installed and configured properly, Anacron will make sure that the commands are run at the specified intervals as closely as machine-uptime permits.

more details you can find in your command-console 'bash'

  • man cron
  • man 5 crontab
  • man anacron
  • man anacrontab

Required Software? None, cron is a standard and anacron too for missed jobs.

cron while booting and daily

Every user has a separate crontab stored in /var/spool/cron/crontabs/ or /var/cron/tabs
To edit the crontab simply type:

crontab -e

Then enter the information for reboot and every 24h at 3 o'clock in the morning.
The file is at first a /tmp but, when you save it, it will tell you:

no crontab for yourusername - using an empty one

crontab: installing new crontab


Script.png crontab script:

#-----------------------------------------------------------------
# This script shall start a script at boot/reboot to get the 
# EPG-Data for the MediaCenter

# Shell variable for cron
SHELL=/bin/bash
# PATH variable for cron
PATH=/usr/local/bin:/usr/local/sbin:/sbin:/usr/sbin:/bin:/usr/bin:/usr/bin/X11
#M   S    T M W   Befehl
#-----------------------------------------------------------------
@reboot /home/username/epg/cronjob_harvest
# run at 03:01am every night
1  3  *  *  * /home/username/epg/cronjob_harvest
#-----------------------------------------------------------------


script called-up by the cronjob

At first copy this sript to a folder e.g. /home/yourname/epg give it the name it has. Change 3times the yourname in the script to your directory.
Then you need to make it executeable sudo chmod 777 cronjob_harvest
After that, you can test it by starting it.

/home/yourname/epg/cronjob_harvest


Script.png cronjob_harvest script:

SHELL=/bin/bash
#!/bin/bash
# remove old files and start harvesting EGP-Data, then insert it into MythTV's database
# Written by reto007@hotm....com
# Distributed under the terms of the GNU General Public License v2
# Date: 21.08.2008
# changes:



version=v0.1
command_list="crontab nxtvepg mythfilldatabase"

# verify that each command we use exists
for command in $command_list; do
        path=$(which $command | grep "no $command in")

        if [ ! -x $(which $command) -a "$path" ]; then
                echo -e "\n\nERROR: $command not found! Check your \
                $command_list and/or your $PATH"
                exit -1
        fi
done

# remove old files 
rm -v /home/yourname/epg/nxtve* >> /var/log/epg_data.log

# go to directory
cd /home/yourname/epg/

# start harvesting EPG-Data
/home/yourname/epg/harvest

experiences - problems

27. Jan. 2008, by Tido, I tested it with Mythbuntu 7.10 PVR-500 and PVR-350 it runs fine. Aferwards I worked a lot on this page to make it better for the next. So please leave comment here.

18. Sep. 2010, by Tido, it still works on my old machine like a charme. No Website updating or anything. Just runs :-)

About

This Howto is based on serveral forum threads: