Difference between revisions of "Mythfilldatabase"

From MythTV Official Wiki
Jump to: navigation, search
(Update info on how to find the source id)
(51 intermediate revisions by 21 users not shown)
Line 1: Line 1:
{{Cleanup}}
 
 
{{Wrongtitle|mythfilldatabase}}
 
{{Wrongtitle|mythfilldatabase}}
= About =
 
'''mythfilldatabase''' fills the Myth programs table with upcoming shows.
 
  
= Manual use of mythfilldatabase =
+
'''mythfilldatabase''' fills the Myth database [[program table]] with upcoming shows.
  
== Command Line Options ==
+
= Automatic use of mythfilldatabase =
--manual
+
Normally mythfilldatabase is run automatically by [[mythbackend]] every 24 hours and needs no additional configuration. For most people allowing mythfilldatabase to run automatically will be sufficient, however in areas with no XMLTV grabber or an unusual channel setup it may be run manually.
  Run in manual channel configuration mode
 
  This will ask you questions about every channel
 
  
--update
+
= Manual use of mythfilldatabase =
  For running non-destructive updates on the database for
+
To run an extra time out of sequence (for example, because your subscription had run out, and you just renewed it), use no command line options:
  users in xmltv zones that do not provide channel data
+
      mythfilldatabase
  For example, to update two datasources with a single xmltv file, use :
 
      mythfilldatabase --update --file 1 -1 tvguide.xml
 
      mythfilldatabase --update --file 2 -1 tvguide.xml
 
  See below for help about  --file.
 
 
 
--preset
 
  Use it in case that you want to assign a preset number for
 
  each channel, useful for non US countries where people
 
  are used to assigning a sequenced number for each channel, i.e.:
 
  1->TVE1(S41), 2->La 2(SE18), 3->TV3(21), 4->Canal 33(60)...
 
 
 
--no-delete
 
  Do not delete old programs from the database until 7 days old.
 
 
 
--file <sourceid> <offset> <xmlfile>
 
  Bypass the grabbers and read data directly from a file
 
  <sourceid> = number for the video source to use with this file
 
  <offset>  = days from today that xmlfile defines
 
                (-1 means to replace all data, up to 10 days)
 
  <xmlfile>  = file to read
 
 
 
--dd-file <sourceid> <offset> <lineupid> <xmlfile>
 
  <sourceid> = see --file
 
  <offset>  = see --file
 
  <lineupid> = the lineup id
 
  <xmlfile>  = see --file
 
 
 
--xawchannels <sourceid> <xawtvrcfile>
 
  (--manual flag works in combination with this)
 
  Read channels as defined in xawtvrc file given
 
  <sourceid>    = cardinput
 
  <xawtvrcfile> = file to read
 
 
 
--do_channel_updates
 
  When using DataDirect, ask mythfilldatabase to
 
  overwrite channel names, frequencies, etc. with the
 
  values available from the data source. This will
 
  override custom channel names, which is why it is
 
  off by default.
 
  This may manage to fix a corrupted channel lineup.
 
 
 
--remove-new-channels
 
  When using DataDirect, ask mythfilldatabase to
 
  remove new channels (those not in the database)
 
  from the DataDirect lineup.  These channels are
 
  removed from the lineup as if you had done so
 
  via the DataDirect website's Lineup Wizard, but
 
  may be re-added manually and incorporated into
 
  MythTV by running mythfilldatabase without this
 
  option.  New channels are automatically removed
 
  for DVB and HDTV sources that use DataDirect.
 
 
 
--graboptions <"options">
 
  Pass options to grabber
 
 
 
--sourceid <number>
 
  Only refresh data for sourceid given
 
 
 
--max-days <number>
 
  Force the maximum number of days, counting today,
 
  for the grabber to check for future listings
 
--only-update-channels
 
  Get as little listings data as possible to update channels
 
--refresh-today
 
 
 
--refresh-second
 
 
 
--refresh-all
 
  (Only valid for grabbers: DataDirect, se_swedb, no, ee, de_tvtoday)
 
  Force a refresh today or two days (or every day) from now,
 
  to catch the latest changes.
 
  These may be used to attempt to fix corrupted program guide data.
 
--dont-refresh-tomorrow
 
  Tomorrow will always be refreshed unless this argument is used
 
--dont-refresh-tba
 
  "To be announced" programs will always be refreshed
 
  unless this argument is used
 
 
 
--export-icon-map [<filename>]
 
  Exports your current icon map to <filename> (default: iconmap.xml)
 
--import-icon-map [<filename>]
 
  Imports an icon map from <filename> (default: iconmap.xml)
 
--update-icon-map
 
  Updates icon map icons only
 
--reset-icon-map [all]
 
  Resets your icon map (pass all to reset channel icons as well)
 
  
--mark-repeats
+
== Command line options ==
  Marks any programs with a OriginalAirDate earlier
+
<pre>
  than their start date as a repeat
+
~$ mythfilldatabase -h
 +
mythfilldatabase version: fixes/0.27 [v0.27-159-g9bf1070] www.mythtv.org
 +
Misc. Options:
 +
--dd-file                      Bypass grabber, and read SD data from file
 +
  --lineupid                      DataDirect lineup of input xml file
 +
  --offset                        Day offset of input xml file
 +
  --xmlfile                      XML file to import manually
 +
--dd-grab-all                  refresh full data using DataDirect
 +
--file                          Bypass grabbers and define sourceid and file
 +
  --xmlfile                      XML file to import manually
 +
--manual                        Run interactive configuration
 +
--no-mark-repeats               do not mark repeats
 +
--preset                        Use channel preset values instead of numbers
 +
-h OR --help OR --usage        Display this help printout, or give detailed
 +
                                information of selected option.
 +
--version                      Display version information.
 +
--sourceid                      Operate on single source
  
-v or --verbose debug-level
+
Channel List Handling Options:
   Use '-v help' for level info
+
--do-channel-updates            update channels using datadirect
 +
--do-not-filter-new-channels   don't filter ATSC channels for addition
 +
--only-update-channels          only update channel lineup
 +
--remove-new-channels          disable new channels on datadirect web interface
  
--help
+
Filtering Options:
  This text
+
--dont-refresh-tba              don't refresh "To be announced" programs
 +
--max-days                      force number of days to update
 +
--refresh                      Provide a day or range of days to refresh. Can
 +
                                be used repeatedly.
  
 +
Logging Options:
 +
--enable-dblog                  Enable logging to database.
 +
--loglevel                      Set the logging level.  All log messages at
 +
                                lower levels will be discarded.
 +
                                In descending order: emerg, alert, crit, err,
 +
                                warning, notice, info, debug
 +
                                defaults to info
 +
--logpath                      Writes logging messages to a file in the
 +
                                directory logpath with filenames in the format:
 +
                                applicationName.date.pid.log.
 +
                                This is typically used in combination with
 +
                                --daemon, and if used in combination with
 +
                                --pidfile, this can be used with log rotators,
 +
                                using the HUP call to inform MythTV to reload
 +
                                the file
 +
--nologserver                  Disable all logging but console.
 +
-q OR --quiet                  Don't log to the console (-q).  Don't log
 +
                                anywhere (-q -q)
 +
--syslog                        Set the syslog logging facility.
 +
                                Set to "none" to disable, defaults to none.
 +
-v OR --verbose                Specify log filtering. Use '-v help' for level
 +
                                info.
  
  --manual and --update can not be used together.
+
Guide Data Handling Options:
 +
--only-update-guide            Only update guide data
 +
</pre>
  
 
== External XMLTV ==
 
== External XMLTV ==
If you are in a country where mythfilldatabase will not work the normal way you will need to use the --file flag (e.g. XMLTV does not have a grabber included, or the grabber options are not fully supported by MythTV).
+
{{note_box|The following steps are '''only required''' if you are using a grabber which does not comply to the baseline specification-  i.e. It is '''not xmltv compliant'''. If possible inform the grabber author that their script does not comply with the standard so that they may fix it.}}
 +
If you are using a non-compliant XMLTV grabber where mythfilldatabase will not automatically work in the normal way you will need to use the --file flag.
  
=== 1. Obtain xml file of tv guide ===
+
=== Obtain xml file of tv guide ===
  
 
Here in Belgium, I use [http://pytvgrab.sourceforge.net pytvgrab] to get my tv guide and output to an xml file:
 
Here in Belgium, I use [http://pytvgrab.sourceforge.net pytvgrab] to get my tv guide and output to an xml file:
Line 126: Line 82:
 
</nowiki></pre>
 
</nowiki></pre>
  
=== 2. Find out the source id of your card ===
+
=== Find out the source id of your card ===
  
 
This is normally '1' if you have only one input card, but it is best to check it out because I have found if I have run mythtv setup more than once it can end up being something other than 1. To check, you need to look at the database:
 
This is normally '1' if you have only one input card, but it is best to check it out because I have found if I have run mythtv setup more than once it can end up being something other than 1. To check, you need to look at the database:
Line 132: Line 88:
 
<pre><nowiki>
 
<pre><nowiki>
 
$ mysql -u root mythconverg
 
$ mysql -u root mythconverg
mysql> select * from cardinput\G
+
mysql> select * from videosource;
cardinputid
+
+----------+------+---------------+--------+-----------+-----------+----------+--------+------------+------------+
cardid
+
| sourceid | name | xmltvgrabber  | userid | freqtable | lineupid  | password | useeit | configpath | dvb_nit_id |
sourceid=4
+
+----------+------+---------------+--------+-----------+-----------+----------+--------+------------+------------+
etc.
+
|        2 | SD  | tv_grab_na_sd |        | default  | MA20453:X | NULL    |      0 | NULL      |        -1 |
 +
+----------+------+---------------+--------+-----------+-----------+----------+--------+------------+------------+
 +
1 row in set (0.00 sec)
 
</nowiki></pre>
 
</nowiki></pre>
 +
This shows all of your sources and the id's.
 +
Remember the source id.
  
Remember the source id
+
You may need to do multiple runs, e.g. one run with --sourceid 1, and then second run with --sourceid 2.
 +
{{Note box|Be very careful to use the correct sourceid. If you use the wrong one the system will go ahead and add channels and programs to an invalid source id that has not been set up. This will result in inconsistent results and failures.}}
  
=== 3. Run mythfilldatabase with the --file flag ===
+
If you have run with an invalid source id, you will need to go into mythtv-setup and delete all of the incorrect channels that were created.
  
The -1 means to replace all data, belgium.xml is the file name for the output of the grabber as above:
+
=== Run mythfilldatabase with the --file flag ===
<pre><nowiki>
 
$ mythfilldatabase --file <sourceid> -1 belgium.xml
 
</nowiki></pre>
 
The first time you will need to use the --manual flag, but this is better explained on the [[XmlTv]] page.
 
  
== Troubleshooting ==
+
The --file argument allows to directly import XMLTV files. This is useful if you are using programs like [http://nxtvepg.sourceforge.net/ nxtvepg] to grab your EPG data.
=== 1. LinkSys or Fedora Core ===
 
I spent the past 3 weeks trying to troubleshoot a problem I was having with mythfilldatabase. I would run it and it would download between 5K and 3K and then just timeout. If I tried downloading the NVidia drivers, the nVidia page would not even come up.
 
  
I found this solution [http://www.gossamer-threads.com/lists/engine?do=post_view_flat;post=219553;page=1;sb=post_latest_reply;so=ASC;mh=25;list=mythtv http://www.gossamer-threads.com]
+
Note that Schedules Direct members should not use the <code>--file</code> argument with mythfilldatabase. They should instead use --dd-file with data obtained by running <code>tv_grab_na_dd</code> using the <code>--dd-data</code> argument.
  
which says to do this:
+
'''From 0.21 on:'''
  
add this line in /etc/sysctl.conf
+
From version 0.21 on the arguments of this command changed. The offset argument has been removed and the command changed to:
 +
<pre><nowiki>
 +
$ mythfilldatabase --file <sourceid> belgium.xml
 +
</nowiki></pre>
  
net.ipv4.tcp_window_scaling = 0
+
'''From 0.25 on:'''
  
execute sysctl -p /etc/sysctl.conf
+
From version 0.25, the arguments of this command changed:
or restart your computer
+
<pre><nowiki>
 +
$ mythfilldatabase --file --sourceid <sourceid> --xmlfile belgium.xml
 +
</nowiki></pre>
  
Weeks of frustration fixed.  Hope this helps someone.
+
Before you run it the first time you will need to use the --manual flag, but this is better explained on the [[XMLTV]] page.
  
Note from a network techie.... Please be aware that disabling tcp window scaling has a system wide impact and will probably result in much slower bulk transfer speeds. Just thought you should be aware.
+
= Troubleshooting =
  
=== 2. INSERT IGNORE INTO SQL Errors ===
+
=== mythfilldatabase causes the system to hang===
If you find in mythfilldatabase's logs something like:
+
This is probably an I/O saturation issue.  mythfilldatabase and [[MySQL]] can cause hard disk or ATA bus saturation, causing [[Mythbackend]] to starve.  Read more about [[Troubleshooting:Mythfilldatabase_IO_bottleneck]].
<pre>2006-11-21 22:33:22.474 DB Error (Inserting into program table):
 
Query was:
 
INSERT IGNORE INTO program (chanid, starttime, endtime, title, subtitle, description,
 
showtype, category, category_type, airdate, stars, previouslyshown, stereo, subtitled,
 
hdtv, closecaptioned, partnumber, parttotal, seriesid, originalairdate, colorcode,
 
syndicatedepisodenumber, programid) SELECT dd_v_program.chanid, DATE_ADD(starttime,
 
INTERVAL channel.tmoffset MINUTE), DATE_ADD(endtime, INTERVAL channel.tmoffset MINUTE),
 
title, subtitle, description, showtype, dd_genre.class, category_type, airdate, stars,
 
previouslyshown, stereo, subtitled, hdtv, closecaptioned, partnumber, parttotal, seriesid,
 
originalairdate, colorcode, syndicatedepisodenumber, dd_v_program.programid FROM (dd_v_program,
 
channel) LEFT JOIN dd_genre ON (dd_v_program.programid = dd_genre.programid AND
 
dd_genre.relevance = '0') WHERE dd_v_program.chanid = channel.chanid;
 
Driver error was [2/1105]:
 
QMYSQL3: Unable to execute query
 
Database error was:
 
Unknown error</pre>
 
  
It most likely means your database has become corrupted.  To fix the corrupted database run the following command
+
[[Category:MythTV_Software]]
<pre>mysqlcheck -r -u <username> -p <password> mythconverg</pre>
 
 
 
===3. mythfilldatabase causes the system to hang===
 
This is probably an I/O saturation issue.  mythfilldatabase and [[MySQL]] can cause hard disk or ATA bus saturation, causing [[Mythbackend]] to starve.  Read more about [[Troubleshooting:Mythfilldatabase_IO_bottleneck]].
 
[[Category:Utilities]]
 

Revision as of 18:51, 18 July 2016

Important.png Note: The correct title of this article is mythfilldatabase. It appears incorrectly here due to technical restrictions.


mythfilldatabase fills the Myth database program table with upcoming shows.

Automatic use of mythfilldatabase

Normally mythfilldatabase is run automatically by mythbackend every 24 hours and needs no additional configuration. For most people allowing mythfilldatabase to run automatically will be sufficient, however in areas with no XMLTV grabber or an unusual channel setup it may be run manually.

Manual use of mythfilldatabase

To run an extra time out of sequence (for example, because your subscription had run out, and you just renewed it), use no command line options:

     mythfilldatabase

Command line options

~$ mythfilldatabase -h
mythfilldatabase version: fixes/0.27 [v0.27-159-g9bf1070] www.mythtv.org
Misc. Options:
--dd-file                       Bypass grabber, and read SD data from file
  --lineupid                      DataDirect lineup of input xml file
  --offset                        Day offset of input xml file
  --xmlfile                       XML file to import manually
--dd-grab-all                   refresh full data using DataDirect
--file                          Bypass grabbers and define sourceid and file
  --xmlfile                       XML file to import manually
--manual                        Run interactive configuration
--no-mark-repeats               do not mark repeats
--preset                        Use channel preset values instead of numbers
-h OR --help OR --usage         Display this help printout, or give detailed
                                information of selected option.
--version                       Display version information.
--sourceid                      Operate on single source

Channel List Handling Options:
--do-channel-updates            update channels using datadirect
--do-not-filter-new-channels    don't filter ATSC channels for addition
--only-update-channels          only update channel lineup
--remove-new-channels           disable new channels on datadirect web interface

Filtering Options:
--dont-refresh-tba              don't refresh "To be announced" programs
--max-days                      force number of days to update
--refresh                       Provide a day or range of days to refresh. Can
                                be used repeatedly.

Logging Options:
--enable-dblog                  Enable logging to database.
--loglevel                      Set the logging level.  All log messages at
                                lower levels will be discarded.
                                In descending order: emerg, alert, crit, err,
                                warning, notice, info, debug
                                defaults to info
--logpath                       Writes logging messages to a file in the
                                directory logpath with filenames in the format:
                                applicationName.date.pid.log.
                                This is typically used in combination with
                                --daemon, and if used in combination with
                                --pidfile, this can be used with log rotators,
                                using the HUP call to inform MythTV to reload
                                the file
--nologserver                   Disable all logging but console.
-q OR --quiet                   Don't log to the console (-q).  Don't log
                                anywhere (-q -q)
--syslog                        Set the syslog logging facility.
                                Set to "none" to disable, defaults to none.
-v OR --verbose                 Specify log filtering. Use '-v help' for level
                                info.

Guide Data Handling Options:
--only-update-guide             Only update guide data

External XMLTV

Important.png Note: The following steps are only required if you are using a grabber which does not comply to the baseline specification- i.e. It is not xmltv compliant. If possible inform the grabber author that their script does not comply with the standard so that they may fix it.

If you are using a non-compliant XMLTV grabber where mythfilldatabase will not automatically work in the normal way you will need to use the --file flag.

Obtain xml file of tv guide

Here in Belgium, I use pytvgrab to get my tv guide and output to an xml file:

$ tv_grab_be_tvb --configure  (you only need this the first time)
$ tv_grab_be_tvb -o belgium.xml

Find out the source id of your card

This is normally '1' if you have only one input card, but it is best to check it out because I have found if I have run mythtv setup more than once it can end up being something other than 1. To check, you need to look at the database:

$ mysql -u root mythconverg
mysql> select * from videosource;
+----------+------+---------------+--------+-----------+-----------+----------+--------+------------+------------+
| sourceid | name | xmltvgrabber  | userid | freqtable | lineupid  | password | useeit | configpath | dvb_nit_id |
+----------+------+---------------+--------+-----------+-----------+----------+--------+------------+------------+
|        2 | SD   | tv_grab_na_sd |        | default   | MA20453:X | NULL     |      0 | NULL       |         -1 |
+----------+------+---------------+--------+-----------+-----------+----------+--------+------------+------------+
1 row in set (0.00 sec)

This shows all of your sources and the id's. Remember the source id.

You may need to do multiple runs, e.g. one run with --sourceid 1, and then second run with --sourceid 2.

Important.png Note: Be very careful to use the correct sourceid. If you use the wrong one the system will go ahead and add channels and programs to an invalid source id that has not been set up. This will result in inconsistent results and failures.

If you have run with an invalid source id, you will need to go into mythtv-setup and delete all of the incorrect channels that were created.

Run mythfilldatabase with the --file flag

The --file argument allows to directly import XMLTV files. This is useful if you are using programs like nxtvepg to grab your EPG data.

Note that Schedules Direct members should not use the --file argument with mythfilldatabase. They should instead use --dd-file with data obtained by running tv_grab_na_dd using the --dd-data argument.

From 0.21 on:

From version 0.21 on the arguments of this command changed. The offset argument has been removed and the command changed to:

$ mythfilldatabase --file <sourceid> belgium.xml

From 0.25 on:

From version 0.25, the arguments of this command changed:

$ mythfilldatabase --file --sourceid <sourceid> --xmlfile belgium.xml

Before you run it the first time you will need to use the --manual flag, but this is better explained on the XMLTV page.

Troubleshooting

mythfilldatabase causes the system to hang

This is probably an I/O saturation issue. mythfilldatabase and MySQL can cause hard disk or ATA bus saturation, causing Mythbackend to starve. Read more about Troubleshooting:Mythfilldatabase_IO_bottleneck.