Difference between revisions of "Mythfilldatabase"

From MythTV Official Wiki
Jump to: navigation, search
(Run mythfilldatabase with the --file flag: don't use with Schedules Direct)
(Add information on configuration with xmltv)
 
(14 intermediate revisions by 8 users not shown)
Line 1: Line 1:
{{Cleanup}}
 
 
{{Wrongtitle|mythfilldatabase}}
 
{{Wrongtitle|mythfilldatabase}}
'''mythfilldatabase''' fills the Myth database [[program table]] with upcoming shows.
+
{{UpToDate 0.28}}
  
= Automatic use of mythfilldatabase =
+
==Introduction==
Normally mythfilldatabase is run automatically by [[mythbackend]] every 24 hours. For most people allowing mythfilldatabase to run automatically will be sufficient, however in areas with no XMLTV grabber or an unusual channel setup it may be run manually.
+
'''mythfilldatabase''' fills the Myth database [[program table]] with upcoming shows. It downloads data from schedule providers and loads channel names and program details into the MythTV database.
  
= Manual use of mythfilldatabase =
+
There are two data formats supported.
To run an extra time out of sequence (for example, because your subscription had run out, and you just renewed it), use no command line options:
+
# ''Data Direct'' format is an older format that is supported but not recommended. Only one provider of schedules information, ''Schedules Direct'' supports this format and only for USA TV programs. Since Schedules Direct now also supports XMLTV format, and provides better data that way, that is to be preferred.
      mythfilldatabase
+
# [[XMLTV]] format is supported by many providers for many countries.
 +
 
 +
==Setup==
 +
When running mythtv-setup, the [[Setup Video Sources]] section will configure information necessary for running mythfilldatabase. The [[Setup General#Program Schedule Downloading Options|Program Schedule Downloading Options]] section of Setup General tells the mythtv backend whether to run mythfilldatabase, when to run it and what parameters to use.
 +
 
 +
==Running==
 +
If you set up the mythfilldatabase options in MythTV setup, the backend will run it once a day, based on your settings. When you are using [[XMLTV]], mythfilldatabase internally invokes the XMLTV grabber you have selected, downloads that data into a temporary file, then reads that data and processes it into the database.
 +
 
 +
If you prefer, you can uncheck ''Automatically update program listings'' in setup, and use any method you prefer to run mythfilldatabase once a day (e.g. cron, anacron). You can also do a manual run, even if you already ran it that day, at any time. For example if your daily run failed, or you just added some new channels, you can run it manually to get the latest schedules updated.
 +
 
 +
==Memory Usage==
 +
When using XMLTV, mythfilldatabase can use a lot of main memory. If you have 200 or more channels and you are processing 21 days of schedules it can use 5 GB of resident memory. If your backend only has 4 GB of memory this will start swapping, causing mythfilldatabase to take a very long time and possibly causing recordings to fail. You should do a manual run of mythfilldatabase and watch the memory usage using top to determine if you have a problem. This is particularly true if you previously used the Data Direct format, which does not have this problem.
 +
 
 +
To reduce the memory usage, you can run a few days of data at a time. I have found that running 3 days of data at a time uses around 500 MB of main memory. To achieve this you will need to set up a script that calls the xmltv grabber and mythfilldatabase repeatedly. Instead of having mythfilldatabase call the grabber internally you will do that in the script. You can name this script in the backend setup so that the backend calls it automatically every day. See the following example. It is very important to get the correct SOURCEID. See [[#Find out the source id of your card]].
 +
 
 +
<pre>
 +
#!/bin/bash
 +
SOURCEID=2
 +
grabber="tv_grab_zz_sdjson_sqlite"
 +
rm -f /tmp/tv_grab_off*.xml
 +
for (( offset = 0; offset < 20; offset += 3 )) ; do
 +
    "$grabber" --days 3 --offset $offset > /tmp/tv_grab_off$offset.xml
 +
    mythfilldatabase --file --sourceid $SOURCEID --xmlfile /tmp/tv_grab_off$offset.xml
 +
done
 +
</pre>
 +
 
 +
==Channels==
 +
By default, mythfilldatabase will add or update channels that are included in the input file if the tuner type supports it. That is any tuner that does not scan for channels. If you have previously added channels from a different grabber or manually, this may create duplicate channels. In many cases having it add the channels for you is convenient and avoids a lot of manual work. If you find it has duplicated channels you can use ''Delete All'' from [[Setup Channel Editor]] to delete all the channels and allow mythfilldatabase to add them. You can override this behavior in mythfilldatabase with the ''--only-update-guide'' option, in which case you will have to set up your channels manually.
 +
 
 +
If you have a tuner that scans for channels or you use the ''--only-update-guide'' option, you will have to set up your channels manually using [[Setup Channel Editor]] and/or the channel editor in mythweb. The channel editor in mythweb is easier to use but does not allow adding channels. You need to enter the ''xmltv id'' for each channel in a format that corresponds with the xmltv id in the download file.
  
 
== Command line options ==
 
== Command line options ==
 
<pre>
 
<pre>
$ mythfilldatabase --help
+
~$ mythfilldatabase -h
mythfilldatabase version: fixes/0.25 [v0.25-100-gb5b4c48] www.mythtv.org
+
mythfilldatabase version: mythtv-fixes/0.28 [v0.28-42-g63477ef] www.mythtv.org
 
Misc. Options:
 
Misc. Options:
 
--dd-file                      Bypass grabber, and read SD data from file
 
--dd-file                      Bypass grabber, and read SD data from file
Line 25: Line 53:
 
--no-mark-repeats              do not mark repeats
 
--no-mark-repeats              do not mark repeats
 
--preset                        Use channel preset values instead of numbers
 
--preset                        Use channel preset values instead of numbers
-h OR --help OR --usage        Display this help printout, or give detailed
+
-h OR --help OR --usage        Display this help printout, or give detailed information of selected option.
                                information of selected option.
+
 
--version                      Display version information.
 
--version                      Display version information.
 
--sourceid                      Operate on single source
 
--sourceid                      Operate on single source
--update                        Run non-destructive updates
 
--xawchannels                  Read channels from xawtvrc file
 
  --xawtvrcfile                  xawtvrc file to import channels from
 
  
 
Channel List Handling Options:
 
Channel List Handling Options:
Line 42: Line 66:
 
--dont-refresh-tba              don't refresh "To be announced" programs
 
--dont-refresh-tba              don't refresh "To be announced" programs
 
--max-days                      force number of days to update
 
--max-days                      force number of days to update
--refresh                      Provide a day or range of days to refresh. Can
+
--refresh                      Provide a day or range of days to refresh. Can be used repeatedly.
                                be used repeatedly.
+
 
+
Channel Icon Handling Options:
+
--export-icon-map              export icon map to file
+
--import-icon-map              import icon map to file
+
--reset-icon-map                resets icon maps
+
--update-icon-map              updates icon map icons
+
  
 
Logging Options:
 
Logging Options:
--loglevel                      Set the logging level.  All log messages at
+
--enable-dblog                  Enable logging to database.
                                lower levels will be discarded.
+
--loglevel                      Set the logging level.  All log messages at lower levels will be discarded.
                                 In descending order: emerg, alert, crit, err,
+
                                 In descending order: emerg, alert, crit, err, warning, notice, info, debug
                                warning, notice, info, debug
+
 
                                 defaults to info
 
                                 defaults to info
--logpath                      Writes logging messages to a file in the
+
--logpath                      Writes logging messages to a file in the directory logpath with filenames in the format:
                                directory logpath with filenames in the format:
+
 
                                 applicationName.date.pid.log.
 
                                 applicationName.date.pid.log.
                                 This is typically used in combination with
+
                                 This is typically used in combination with --daemon, and if used in combination with --pidfile, this
                                --daemon, and if used in combination with
+
                                can be used with log rotators, using the HUP call to inform MythTV to reload the file
                                --pidfile, this can be used with log rotators,
+
-q OR --quiet                  Don't log to the console (-q).  Don't log anywhere (-q -q)
                                using the HUP call to inform MythTV to reload
+
                                the file
+
--nodblog                      Disable database logging.
+
-q OR --quiet                  Don't log to the console (-q).  Don't log
+
                                anywhere (-q -q)
+
 
--syslog                        Set the syslog logging facility.
 
--syslog                        Set the syslog logging facility.
 
                                 Set to "none" to disable, defaults to none.
 
                                 Set to "none" to disable, defaults to none.
-v OR --verbose                Specify log filtering. Use '-v help' for level
+
-v OR --verbose                Specify log filtering. Use '-v help' for level info.
                                info.
+
 
 +
Guide Data Handling Options:
 +
--only-update-guide            Only update guide data
  
 
</pre>
 
</pre>
  
 
== External XMLTV ==
 
== External XMLTV ==
If you are in a country where mythfilldatabase will not work the normal way you will need to use the --file flag (e.g. XMLTV does not have a grabber included, or the grabber options are not fully supported by MythTV).
+
{{note_box|The following steps are '''only required''' if you are using a grabber which does not comply to the baseline specification-  i.e. It is '''not xmltv compliant'''. If possible inform the grabber author that their script does not comply with the standard so that they may fix it.}}
 +
If you are using a non-compliant XMLTV grabber where mythfilldatabase will not automatically work in the normal way you will need to use the --file flag.
  
 
=== Obtain xml file of tv guide ===
 
=== Obtain xml file of tv guide ===
Line 93: Line 106:
 
<pre><nowiki>
 
<pre><nowiki>
 
$ mysql -u root mythconverg
 
$ mysql -u root mythconverg
mysql> select * from cardinput\G
+
mysql> select * from videosource;
cardinputid
+
+----------+------+---------------+--------+-----------+-----------+----------+--------+------------+------------+
cardid
+
| sourceid | name | xmltvgrabber  | userid | freqtable | lineupid  | password | useeit | configpath | dvb_nit_id |
sourceid=4
+
+----------+------+---------------+--------+-----------+-----------+----------+--------+------------+------------+
etc.
+
|        2 | SD  | tv_grab_na_sd |        | default  | MA20453:X | NULL    |      0 | NULL      |        -1 |
 +
+----------+------+---------------+--------+-----------+-----------+----------+--------+------------+------------+
 +
1 row in set (0.00 sec)
 
</nowiki></pre>
 
</nowiki></pre>
 +
This shows all of your sources and the id's.
 +
Remember the source id.
  
Remember the source id
+
You may need to do multiple runs, e.g. one run with --sourceid 1, and then second run with --sourceid 2.
 +
{{Note box|Be very careful to use the correct sourceid. If you use the wrong one the system will go ahead and add channels and programs to an invalid source id that has not been set up. This will result in inconsistent results and failures.}}
  
=== Run mythfilldatabase with the --file flag ===
+
If you have run with an invalid source id, you will need to go into mythtv-setup and delete all of the incorrect channels that were created.
  
The --file argument allows to directly import XMLTV files. This is useful if you are using programs like [http://nxtvepg.sourceforge.net/ nxtvepg] to grab your EPG data.
+
Run mythfilldatabase as follows:
  
Note that Schedules Direct members should not use the <code>--file</code> argument with mythfilldatabase.  They should instead use --dd-file with data obtained by running <code>tv_grab_na_dd</code> using the <code>--dd-data</code> argument.
+
<pre>
 +
mythfilldatabase --file --sourceid <sourceid> --xmlfile belgium.xml
 +
</pre>
  
'''From 0.21 on:'''
+
Before you run it the first time you will need to setup the grabber. This is explained on the [[XMLTV]] page.
  
From version 0.21 on the arguments of this command changed. The offset argument has been removed and the command changed to:
+
== Troubleshooting ==
<pre><nowiki>
+
$ mythfilldatabase --file <sourceid> belgium.xml
+
</nowiki></pre>
+
  
'''From 0.25 on:'''
+
=== mythfilldatabase causes the system to hang===
 +
This is probably an I/O saturation issue. mythfilldatabase and [[MySQL]] can cause hard disk or ATA bus saturation, causing [[Mythbackend]] to starve. Older versions of MySQL would cause disk thrashing because of the use of write barriers. That could be solved by adding nobarrier to the /etc/fstab entry for the file system that holds your database (normally the root filesystem):
  
From version 0.25, the arguments of this command changed:
+
<code>UUID=900c3f88-ce0c-4a92-8e28-9ac04fdd1b09 / ext4 '''nobarrier''',errors=remount-ro 0 1</code>
<pre><nowiki>
+
$ mythfilldatabase --file --sourceid <sourceid> --xmlfile belgium.xml
+
</nowiki></pre>
+
  
 +
Note that this could cause file system corruption if you have power failures or system crashes. If that is a concern you should move the databse to a separate file system. With recent versions of Linux and MySQL barriers no longer cause this problem.
  
The first time you will need to use the --manual flag, but this is better explained on the [[XMLTV]] page.
+
This also be caused by excessive memory usage. See [[#Memory Usage|Memory Usage]] above.
 
+
= Troubleshooting =
+
 
+
=== mythfilldatabase causes the system to hang===
+
This is probably an I/O saturation issue.  mythfilldatabase and [[MySQL]] can cause hard disk or ATA bus saturation, causing [[Mythbackend]] to starve. Read more about [[Troubleshooting:Mythfilldatabase_IO_bottleneck]].
+
  
[[Category:MythTV_Software]]
+
[[Category:MythTV Software]]

Latest revision as of 16:40, 21 August 2016

Important.png Note: The correct title of this article is mythfilldatabase. It appears incorrectly here due to technical restrictions.

Software-update-available.png This page is up-to-date as of MythTV version 0.28, the current release is 0.28


Introduction

mythfilldatabase fills the Myth database program table with upcoming shows. It downloads data from schedule providers and loads channel names and program details into the MythTV database.

There are two data formats supported.

  1. Data Direct format is an older format that is supported but not recommended. Only one provider of schedules information, Schedules Direct supports this format and only for USA TV programs. Since Schedules Direct now also supports XMLTV format, and provides better data that way, that is to be preferred.
  2. XMLTV format is supported by many providers for many countries.

Setup

When running mythtv-setup, the Setup Video Sources section will configure information necessary for running mythfilldatabase. The Program Schedule Downloading Options section of Setup General tells the mythtv backend whether to run mythfilldatabase, when to run it and what parameters to use.

Running

If you set up the mythfilldatabase options in MythTV setup, the backend will run it once a day, based on your settings. When you are using XMLTV, mythfilldatabase internally invokes the XMLTV grabber you have selected, downloads that data into a temporary file, then reads that data and processes it into the database.

If you prefer, you can uncheck Automatically update program listings in setup, and use any method you prefer to run mythfilldatabase once a day (e.g. cron, anacron). You can also do a manual run, even if you already ran it that day, at any time. For example if your daily run failed, or you just added some new channels, you can run it manually to get the latest schedules updated.

Memory Usage

When using XMLTV, mythfilldatabase can use a lot of main memory. If you have 200 or more channels and you are processing 21 days of schedules it can use 5 GB of resident memory. If your backend only has 4 GB of memory this will start swapping, causing mythfilldatabase to take a very long time and possibly causing recordings to fail. You should do a manual run of mythfilldatabase and watch the memory usage using top to determine if you have a problem. This is particularly true if you previously used the Data Direct format, which does not have this problem.

To reduce the memory usage, you can run a few days of data at a time. I have found that running 3 days of data at a time uses around 500 MB of main memory. To achieve this you will need to set up a script that calls the xmltv grabber and mythfilldatabase repeatedly. Instead of having mythfilldatabase call the grabber internally you will do that in the script. You can name this script in the backend setup so that the backend calls it automatically every day. See the following example. It is very important to get the correct SOURCEID. See #Find out the source id of your card.

#!/bin/bash
SOURCEID=2
grabber="tv_grab_zz_sdjson_sqlite"
rm -f /tmp/tv_grab_off*.xml
for (( offset = 0; offset < 20; offset += 3 )) ; do
    "$grabber" --days 3 --offset $offset > /tmp/tv_grab_off$offset.xml
    mythfilldatabase --file --sourceid $SOURCEID --xmlfile /tmp/tv_grab_off$offset.xml
done

Channels

By default, mythfilldatabase will add or update channels that are included in the input file if the tuner type supports it. That is any tuner that does not scan for channels. If you have previously added channels from a different grabber or manually, this may create duplicate channels. In many cases having it add the channels for you is convenient and avoids a lot of manual work. If you find it has duplicated channels you can use Delete All from Setup Channel Editor to delete all the channels and allow mythfilldatabase to add them. You can override this behavior in mythfilldatabase with the --only-update-guide option, in which case you will have to set up your channels manually.

If you have a tuner that scans for channels or you use the --only-update-guide option, you will have to set up your channels manually using Setup Channel Editor and/or the channel editor in mythweb. The channel editor in mythweb is easier to use but does not allow adding channels. You need to enter the xmltv id for each channel in a format that corresponds with the xmltv id in the download file.

Command line options

~$ mythfilldatabase -h
mythfilldatabase version: mythtv-fixes/0.28 [v0.28-42-g63477ef] www.mythtv.org
Misc. Options:
--dd-file                       Bypass grabber, and read SD data from file
  --lineupid                      DataDirect lineup of input xml file
  --offset                        Day offset of input xml file
  --xmlfile                       XML file to import manually
--dd-grab-all                   refresh full data using DataDirect
--file                          Bypass grabbers and define sourceid and file
  --xmlfile                       XML file to import manually
--manual                        Run interactive configuration
--no-mark-repeats               do not mark repeats
--preset                        Use channel preset values instead of numbers
-h OR --help OR --usage         Display this help printout, or give detailed information of selected option.
--version                       Display version information.
--sourceid                      Operate on single source

Channel List Handling Options:
--do-channel-updates            update channels using datadirect
--do-not-filter-new-channels    don't filter ATSC channels for addition
--only-update-channels          only update channel lineup
--remove-new-channels           disable new channels on datadirect web interface

Filtering Options:
--dont-refresh-tba              don't refresh "To be announced" programs
--max-days                      force number of days to update
--refresh                       Provide a day or range of days to refresh. Can be used repeatedly.

Logging Options:
--enable-dblog                  Enable logging to database.
--loglevel                      Set the logging level.  All log messages at lower levels will be discarded.
                                In descending order: emerg, alert, crit, err, warning, notice, info, debug
                                defaults to info
--logpath                       Writes logging messages to a file in the directory logpath with filenames in the format:
                                applicationName.date.pid.log.
                                This is typically used in combination with --daemon, and if used in combination with --pidfile, this
                                can be used with log rotators, using the HUP call to inform MythTV to reload the file
-q OR --quiet                   Don't log to the console (-q).  Don't log anywhere (-q -q)
--syslog                        Set the syslog logging facility.
                                Set to "none" to disable, defaults to none.
-v OR --verbose                 Specify log filtering. Use '-v help' for level info.

Guide Data Handling Options:
--only-update-guide             Only update guide data

External XMLTV

Important.png Note: The following steps are only required if you are using a grabber which does not comply to the baseline specification- i.e. It is not xmltv compliant. If possible inform the grabber author that their script does not comply with the standard so that they may fix it.

If you are using a non-compliant XMLTV grabber where mythfilldatabase will not automatically work in the normal way you will need to use the --file flag.

Obtain xml file of tv guide

Here in Belgium, I use pytvgrab to get my tv guide and output to an xml file:

$ tv_grab_be_tvb --configure  (you only need this the first time)
$ tv_grab_be_tvb -o belgium.xml

Find out the source id of your card

This is normally '1' if you have only one input card, but it is best to check it out because I have found if I have run mythtv setup more than once it can end up being something other than 1. To check, you need to look at the database:

$ mysql -u root mythconverg
mysql> select * from videosource;
+----------+------+---------------+--------+-----------+-----------+----------+--------+------------+------------+
| sourceid | name | xmltvgrabber  | userid | freqtable | lineupid  | password | useeit | configpath | dvb_nit_id |
+----------+------+---------------+--------+-----------+-----------+----------+--------+------------+------------+
|        2 | SD   | tv_grab_na_sd |        | default   | MA20453:X | NULL     |      0 | NULL       |         -1 |
+----------+------+---------------+--------+-----------+-----------+----------+--------+------------+------------+
1 row in set (0.00 sec)

This shows all of your sources and the id's. Remember the source id.

You may need to do multiple runs, e.g. one run with --sourceid 1, and then second run with --sourceid 2.

Important.png Note: Be very careful to use the correct sourceid. If you use the wrong one the system will go ahead and add channels and programs to an invalid source id that has not been set up. This will result in inconsistent results and failures.

If you have run with an invalid source id, you will need to go into mythtv-setup and delete all of the incorrect channels that were created.

Run mythfilldatabase as follows:

mythfilldatabase --file --sourceid <sourceid> --xmlfile belgium.xml

Before you run it the first time you will need to setup the grabber. This is explained on the XMLTV page.

Troubleshooting

mythfilldatabase causes the system to hang

This is probably an I/O saturation issue. mythfilldatabase and MySQL can cause hard disk or ATA bus saturation, causing Mythbackend to starve. Older versions of MySQL would cause disk thrashing because of the use of write barriers. That could be solved by adding nobarrier to the /etc/fstab entry for the file system that holds your database (normally the root filesystem):

UUID=900c3f88-ce0c-4a92-8e28-9ac04fdd1b09 / ext4 nobarrier,errors=remount-ro 0 1

Note that this could cause file system corruption if you have power failures or system crashes. If that is a concern you should move the databse to a separate file system. With recent versions of Linux and MySQL barriers no longer cause this problem.

This also be caused by excessive memory usage. See Memory Usage above.