Difference between revisions of "Mythfilldatabase"

From MythTV Official Wiki
Jump to: navigation, search
m (1. LinkSys or Fedora Core: After reading the thread cited, I feel the urge to smack some people for not recognizing the obvious.)
m (Memory Usage: Typo.)
(48 intermediate revisions by 14 users not shown)
Line 1: Line 1:
{{Cleanup}}
 
 
{{Wrongtitle|mythfilldatabase}}
 
{{Wrongtitle|mythfilldatabase}}
'''mythfilldatabase''' fills the Myth database [[program table]] with upcoming shows.
+
{{UpToDate 0.28}}
  
= Automatic use of mythfilldatabase =
+
==Introduction==
Normally mythfilldatabase is run automatically by [[mythbackend]] every 24 hours. It uses an [[XMLTV]] or [[zap2it]] grabber as defined in [[mythtv-setup]]. For most people allowing mythfilldatabase to run automatically will be sufficient, however in areas with no XMLTV grabber or an unusual channel setup it may be run manually.
+
'''mythfilldatabase''' fills the MythTV database [[program table]] with upcoming shows. It downloads data from schedule providers and loads channel names and program details into the MythTV database.
  
= Manual use of mythfilldatabase =
+
There are two data formats supported.
 +
# ''Data Direct'' format is an older format that is supported but not recommended. Only one provider of schedules information, ''Schedules Direct'' supports this format and only for USA TV programs. Since Schedules Direct now also supports XMLTV format, and provides better data that way, that is to be preferred.
 +
# [[XMLTV]] format is supported by many providers for many countries.
  
== Command line options ==
+
==Setup==
--manual
+
When running mythtv-setup, the [[Setup Video Sources]] section will configure information necessary for running mythfilldatabase. The [[Setup General#Program Schedule Downloading Options|Program Schedule Downloading Options]] section of Setup General tells the mythtv backend whether to run mythfilldatabase, when to run it and what parameters to use.
:Run in manual channel configuration mode. This will ask you questions about every channel.
 
  
--update
+
==Running==
:For running non-destructive updates on the database for users in xmltv zones that do not provide channel data. For example, to update two datasources with a single xmltv file, use:
+
If you set up the mythfilldatabase options in MythTV setup, the backend will run it once a day, based on your settings. When you are using [[XMLTV]], mythfilldatabase internally invokes the XMLTV grabber you have selected, downloads that data into a temporary file, then reads that data and processes it into the database.
      mythfilldatabase --update --file 1 -1 tvguide.xml
 
      mythfilldatabase --update --file 2 -1 tvguide.xml
 
See below for help about --file.
 
  
--preset
+
You can also do a manual run, even if you already ran it that day, at any time. For example if your daily run failed, or you just added some new channels, you can run it manually to get the latest schedules updated.
:Use it in case that you want to assign a preset number for each channel, useful for non US countries where people  are used to assigning a sequenced number for each channel, i.e.: 1->TVE1(S41), 2->La 2(SE18), 3->TV3(21), 4->Canal 33(60)...
 
  
--no-delete
+
==Memory Usage==
:Do not delete old programs from the database until 7 days old. Since 0.21 MythTV will always keep the last 7 days of programs so this option is deprecated.
+
When using XMLTV, mythfilldatabase can use a lot of main memory. If you have 200 or more channels and you are processing 21 days of schedules it can use 5 GB of resident memory. If your backend only has 4 GB of memory this will start swapping, causing mythfilldatabase to take a very long time and possibly causing recordings to fail. You should do a manual run of mythfilldatabase and watch the memory usage using top to determine if you have a problem. This is particularly true if you previously used the Data Direct format, which does not have this problem.
  
--file <sourceid> <offset> <xmlfile>
+
To reduce the memory usage, you can run a few days of data at a time. I have found that running 3 days of data at a time uses around 500 MB of main memory. To achieve this you will need to set up a script that calls the xmltv grabber and mythfilldatabase repeatedly. Instead of having mythfilldatabase call the grabber internally you will do that in the script. You can name this script in the backend setup so that the backend calls it automatically every day. See the following example. It is very important to get the correct SOURCEID. See [[#Find out the source id of your card]].
:Bypass the grabbers and read data directly from a file. <sourceid> = number for the video source to use with this file. <xmlfile>  = file to read.
 
  
--dd-file <sourceid> <offset> <lineupid> <xmlfile>
+
<pre>
:<sourceid> = see --file
+
#!/bin/bash
:<offset>   = see --file
+
SOURCEID=2
:<lineupid> = the lineup id
+
grabber="tv_grab_zz_sdjson_sqlite"
:<xmlfile> = see --file
+
rm -f /tmp/tv_grab_off*.xml
 +
for (( offset = 0; offset < 20; offset += 3 )) ; do
 +
    "$grabber" --days 3 --offset $offset > /tmp/tv_grab_off$offset.xml
 +
    mythfilldatabase --file --sourceid $SOURCEID --xmlfile /tmp/tv_grab_off$offset.xml
 +
done
 +
</pre>
  
--xawchannels <sourceid> <xawtvrcfile>
+
Another option is to use the ''--no-allatonce'' option which is new in version 29, and which forces it to process one day at a time, using around 250MB of memory. This way you do not need a script, nor do you need to find the source id.
:(--manual flag works in combination with this). Read channels as defined in xawtvrc file given <sourceid> = cardinput, <xawtvrcfile> = file to read
 
  
--do-channel-updates
+
==Channels==
:When using DataDirect, ask mythfilldatabase to overwrite channel names, frequencies, etc. with the values available from the data source. This will override custom channel names, which is why it is off by default. This may manage to fix a corrupted channel lineup. You will also likely need to use this switch if your cable company does a channel realignment, once your program listings source catches up.
+
By default, mythfilldatabase will add or update channels that are included in the input file if the tuner type supports it. That is any tuner that does not scan for channels. If you have previously added channels from a different grabber or manually, this may create duplicate channels. In many cases having it add the channels for you is convenient and avoids a lot of manual work. If you find it has duplicated channels you can use ''Delete All'' from [[Setup Channel Editor]] to delete all the channels and allow mythfilldatabase to add them. You can override this behavior in mythfilldatabase with the ''--only-update-guide'' option, in which case you will have to set up your channels manually.
  
--remove-new-channels
+
If you have a tuner that scans for channels or you use the ''--only-update-guide'' option, you will have to set up your channels manually using [[Setup Channel Editor]] and/or the channel editor in mythweb. The channel editor in mythweb is easier to use but does not allow adding channels. You need to enter the ''xmltv id'' for each channel in a format that corresponds with the xmltv id in the download file.
:When using DataDirect, ask mythfilldatabase to remove new channels (those not in the database) from the DataDirect lineup. These channels are removed from the lineup as if you had done so via the DataDirect website's Lineup Wizard, but may be re-added manually and incorporated into MythTV by running mythfilldatabase without this option. New channels are automatically removed for DVB and HDTV sources that use DataDirect.
 
  
--graboptions <"options">
+
== Command line options ==
:Pass options to grabber.
 
 
 
--sourceid <number>
 
:Only refresh data for sourceid given.
 
 
 
--max-days <number>
 
:Force the maximum number of days, counting today, for the grabber to check for future listings.
 
 
 
--only-update-channels
 
:Get as little listings data as possible to update channels.
 
 
 
--refresh-today
 
 
 
--refresh-second
 
 
 
--refresh-all
 
:(Only valid for grabbers: DataDirect, se_swedb, no, ee, de_tvtoday). Force a refresh today or two days (or every day) from now, to catch the latest changes. These may be used to attempt to fix corrupted program guide data.
 
 
 
--dont-refresh-tomorrow
 
:Tomorrow will always be refreshed unless this argument is used.
 
 
 
--dont-refresh-tba
 
:"To be announced" programs will always be refreshed unless this argument is used.
 
 
 
--export-icon-map [<filename>]
 
:Exports your current icon map to <filename> (default: iconmap.xml).
 
 
 
--import-icon-map [<filename>]
 
:Imports an icon map from <filename> (default: iconmap.xml).
 
 
 
--update-icon-map
 
:Updates icon map icons only.
 
 
 
--reset-icon-map [all]
 
:Resets your icon map (pass all to reset channel icons as well).
 
 
 
--mark-repeats
 
:Marks any programs with a OriginalAirDate earlier than their start date as a repeat.
 
  
-v or --verbose debug-level
+
This information is available from <code>mythfilldatabase --help</code>. The details for each option can be displayed with <code>mythfilldatabase --help option</code>.
:Use '-v help' for level info.
 
  
--help
+
{| class="wikitable" style="width: 100%;"
:This text.
+
|-
 
+
|style="width: 20%; background-color: #E6E6FF;" align="center"|'''Option'''
--manual and --update can not be used together.
+
|style="width: 30%; background-color: #E6E6FF;" align="center"|'''Description'''
 +
|style="width: 50%; background-color: #E6E6FF;" align="center"|'''Details'''
 +
|-
 +
| -h OR --help OR --usage
 +
| Display this help printout, or give detailed information of selected option.
 +
|
 +
Displays a list of all commands available for use with
 +
this application. If another option is provided as an
 +
argument, it will provide detailed information on that
 +
option.
 +
|-
 +
| --version
 +
| Display version information.
 +
|
 +
Display informtion about build, including:<br>
 +
version, branch, protocol, library API, Qt
 +
and compiled options.
 +
|-
 +
| -v OR --verbose
 +
| Specify log filtering. Use '-v help' for level info.
 +
|
 +
|-
 +
| --logpath
 +
|
 +
|
 +
Writes logging messages to a file in the directory logpath with
 +
filenames in the format: applicationName.date.pid.log.<br>
 +
This is typically used in combination with --daemon, and if used
 +
in combination with --pidfile, this can be used with log
 +
rotators, using the HUP call to inform MythTV to reload the
 +
file
 +
|-
 +
| -q OR --quiet
 +
| Don't log to the console (-q).  Don't log anywhere (-q -q)
 +
|
 +
|-
 +
| --loglevel
 +
|
 +
Set the logging level.  All log messages at lower levels will be
 +
discarded.<br>
 +
In descending order: emerg, alert, crit, err, warning, notice,
 +
info, debug
 +
|
 +
|-
 +
| --syslog
 +
|
 +
Set the syslog logging facility. <br>Set to none to disable,
 +
defaults to none.
 +
|
 +
|-
 +
| --nodblog
 +
| Disable database logging.
 +
| this is now the default, see --enable-dblog
 +
|-
 +
| --enable-dblog
 +
| Enable logging to database.
 +
|
 +
|-
 +
| --manual
 +
| Run interactive configuration
 +
|
 +
Manual mode will interactively ask you questions about
 +
each channel as it is processed, to configure for
 +
future use.
 +
|-
 +
| --preset
 +
| Use channel preset values instead of numbers
 +
|
 +
For use with assigning preset numbers for each
 +
channel. Useful for non-US countries where people
 +
are used to assigning a sequenced number for each
 +
channel:<br>1->TVE1(S41), 2->La 2(SE18), 3->TV(21)...
 +
|-
 +
| --file
 +
| Bypass grabbers and define sourceid and file
 +
|
 +
Directly define the sourceid and XMLTV file to
 +
import.
 +
|-
 +
| --dd-file
 +
| Bypass grabber, and read SD data from file
 +
|
 +
Directly define the data needed to import a local
 +
DataDirect download.
 +
|-
 +
| --sourceid
 +
| Operate on single source
 +
|
 +
Limit mythfilldatabase to only operate on the
 +
specified channel source.
 +
|-
 +
| --offset
 +
| Day offset of input xml file
 +
|
 +
Specify how many days offset from today is the
 +
information in the given XML file.
 +
|-
 +
| --lineupid
 +
| DataDirect lineup of input xml file
 +
|
 +
Specify the DataDirect lineup that corresponds to
 +
the information in the given XML file.
 +
|-
 +
| --xmlfile
 +
| XML file to import manually
 +
|
 +
Specify an XML guide data file to import directly
 +
rather than pull data through the specified grabber.
 +
|-
 +
| --update
 +
| Run non-destructive updates
 +
|
 +
Run non-destructive updates on the database for
 +
users in xmltv zones that do not provide channel
 +
data. Stops the addition of new channels and the
 +
changing of channel icons.
 +
|-
 +
| --only-update-guide
 +
| Only update guide data
 +
| Only update the guide data, do not alter channels or icons.
 +
''Note this only applies to xmltv grabbers. For DataDirect see the remove-new-channels option''
 +
|-
 +
| --do-channel-updates
 +
| update channels using datadirect
 +
|
 +
When using DataDirect, ask mythfilldatabase to
 +
overwrite channel names, frequencies, etc. with
 +
values available from the data source. This will
 +
override custom channel names, which is why it
 +
is disabled by default.
 +
|-
 +
| --remove-new-channels
 +
| disable new channels on datadirect web interface
 +
|
 +
When using DataDirect, ask mythfilldatabase to
 +
mark any new channels as disabled on the remote
 +
lineup. Channels can be manually enabled on the
 +
website at a later time, and incorporated into
 +
MythTV by running mythfilldatabase without this
 +
option. New digital channels cannot be directly
 +
imported and thus are disabled automatically.
 +
|-
 +
| --do-not-filter-new-channels
 +
| don't filter ATSC channels for addition
 +
|
 +
Normally, MythTV tries to avoid adding ATSC
 +
channels to NTSC channel lineups. This option
 +
restores the behavior of adding every channel in
 +
the downloaded channel lineup to MythTV's lineup,
 +
in case MythTV's smarts fail you.
 +
|-
 +
| --cardtype
 +
| No information.
 +
|
 +
|-
 +
| --refresh
 +
| Provide a day or range of days to refresh. Can be used repeatedly.
 +
|
 +
Provide days to refresh during the grabber run.  Multiple <br>
 +
days or ranges can be supplied by multiple instances of the <br>
 +
option. Supported days are:<br>
 +
[not]today<br>
 +
[not]tomorrow<br>
 +
[not]second<br>
 +
<nowiki>#[-#]</nowiki><br>
 +
all<br>
 +
Note that if all is specified any others will be ingored.<br><br>
 +
example:<br>
 +
--refresh today --refresh 4-8 --refresh nottomorrow
 +
|-
 +
| --max-days
 +
| force number of days to update
 +
|
 +
Force the maximum number of days, counting today,
 +
for the guide data grabber to check for future
 +
listings.
 +
|-
 +
| --no-allatonce
 +
''New in version 29''
 +
| Do not use allatonce even if the grabber prefers it.
 +
|
 +
This option prevents mythfilldatabase from utlizing
 +
the advertised grabber preference of 'allatonce'.
 +
This may be necessary for grabbers that return a large
 +
amount of data
 +
|-
 +
| --dd-grab-all
 +
| refresh full data using DataDirect
 +
|
 +
This option is only valid for selected grabbers (DataDirect).<br>
 +
This option is the preferred way of updating guide data from
 +
DataDirect, and pulls all fourteen days of guide data at once.
 +
|-
 +
| --only-update-channels
 +
| only update channel lineup
 +
|
 +
Download as little listings data as possible to update the
 +
channel lineup.
 +
|-
 +
| --no-mark-repeats
 +
| do not mark repeats
 +
|
 +
|-
 +
| --
 +
| Grabber Options
 +
| Any text or options after this is passed directly to the grabber. Consequently this must be the last option.
 +
|-
 +
| --dont-refresh-tba
 +
|
 +
|
 +
This option is only valid for selected grabbers.<br>
 +
Prevent mythfilldatabase from automatically refreshing any
 +
programs marked as "To be announced".<br>
 +
If being used with datadirect, this option should not be
 +
used, rather use --dd-grab-all to pull all listings each time.
 +
|-
 +
| colspan=4 align=center | '''Deprecated options'''
 +
|-
 +
| -l OR --logfile
 +
|
 +
|
 +
This option has been removed as part of
 +
rewrite of the logging interface. Please update your init
 +
scripts to use --syslog to interface with your system's
 +
existing system logging daemon, or --logpath to specify a
 +
dirctory for MythTV to write its logs to.
 +
|-
 +
| --refresh-today
 +
|
 +
|
 +
This option is only valid for selected grabbers.<br>
 +
Force a refresh for today's guide data.<br>This can be used
 +
in combination with other --refresh-<n> options.<br>
 +
If being used with datadirect, this option should not be
 +
used, rather use --dd-grab-all to pull all listings each time.
 +
|-
 +
| --dont-refresh-tomorrow
 +
|
 +
|
 +
This option is only valid for selected grabbers.<br>
 +
Prevent mythfilldatabase from pulling information for
 +
tomorrow's listings. Data for tomorrow is always pulled
 +
unless specifically specified otherwise.<br>
 +
If being used with datadirect, this option should not be
 +
used, rather use --dd-grab-all to pull all listings each time.
 +
|-
 +
| --refresh-second
 +
|
 +
|
 +
This option is only valid for selected grabbers.<br>
 +
Force a refresh for guide data two days from now. This can  
 +
be used in combination with other --refresh-<n> options.<br>
 +
If being used with datadirect, this option should not be  
 +
used, rather use --dd-grab-all to pull all listings each time.
 +
|-
 +
| --refresh-day
 +
|
 +
|
 +
This option is only valid for selected grabbers.<br>
 +
Force a refresh for guide data on a specific day. This can
 +
be used in combination with other --refresh-<n> options.<br>
 +
If being used with datadirect, this option should not be
 +
used, rather use --dd-grab-all to pull all listings each time.
 +
|-
 +
| --refresh-all
 +
|
 +
|
 +
This option is only valid for selected grabbers.<br>
 +
This option forces a refresh of all guide data, but does so
 +
with fourteen downloads of one day each.<br>
 +
If being used with datadirect, this option should not be
 +
used, rather use --dd-grab-all to pull all listings each time.
 +
|-
 +
| --graboptions
 +
|
 +
|
 +
mythfilldatabase now passes any text after an<br>
 +
independent '--' directly to the external grabber.<br>
 +
e.g. mythfilldatabase -- --daily
 +
|-
 +
| --mark-repeats
 +
|
 +
|
 +
This is now the default behavior. Use<br>
 +
--no-mark-repeats to disable.
 +
|-
 +
|}
  
 
== External XMLTV ==
 
== External XMLTV ==
If you are in a country where mythfilldatabase will not work the normal way you will need to use the --file flag (e.g. XMLTV does not have a grabber included, or the grabber options are not fully supported by MythTV).
+
{{note_box|The following steps are '''only required''' if you are using a grabber which does not comply to the baseline specification-  i.e. It is '''not xmltv compliant'''. If possible inform the grabber author that their script does not comply with the standard so that they may fix it.}}
 +
If you are using a non-compliant XMLTV grabber where mythfilldatabase will not automatically work in the normal way you will need to use the --file flag.
  
=== 1. Obtain xml file of tv guide ===
+
=== Obtain xml file of tv guide ===
  
 
Here in Belgium, I use [http://pytvgrab.sourceforge.net pytvgrab] to get my tv guide and output to an xml file:
 
Here in Belgium, I use [http://pytvgrab.sourceforge.net pytvgrab] to get my tv guide and output to an xml file:
Line 102: Line 348:
 
</nowiki></pre>
 
</nowiki></pre>
  
=== 2. Find out the source id of your card ===
+
=== Find out the source id of your card ===
  
 
This is normally '1' if you have only one input card, but it is best to check it out because I have found if I have run mythtv setup more than once it can end up being something other than 1. To check, you need to look at the database:
 
This is normally '1' if you have only one input card, but it is best to check it out because I have found if I have run mythtv setup more than once it can end up being something other than 1. To check, you need to look at the database:
Line 108: Line 354:
 
<pre><nowiki>
 
<pre><nowiki>
 
$ mysql -u root mythconverg
 
$ mysql -u root mythconverg
mysql> select * from cardinput\G
+
mysql> select * from videosource;
cardinputid
+
+----------+------+---------------+--------+-----------+-----------+----------+--------+------------+------------+
cardid
+
| sourceid | name | xmltvgrabber  | userid | freqtable | lineupid  | password | useeit | configpath | dvb_nit_id |
sourceid=4
+
+----------+------+---------------+--------+-----------+-----------+----------+--------+------------+------------+
etc.
+
|        2 | SD  | tv_grab_na_sd |        | default  | MA20453:X | NULL    |      0 | NULL      |        -1 |
 +
+----------+------+---------------+--------+-----------+-----------+----------+--------+------------+------------+
 +
1 row in set (0.00 sec)
 
</nowiki></pre>
 
</nowiki></pre>
 +
This shows all of your sources and the id's.
 +
Remember the source id.
  
Remember the source id
+
You may need to do multiple runs, e.g. one run with --sourceid 1, and then second run with --sourceid 2.
 +
{{Note box|Be very careful to use the correct sourceid. If you use the wrong one the system will go ahead and add channels and programs to an invalid source id that has not been set up. This will result in inconsistent results and failures.}}
  
=== 3. Run mythfilldatabase with the --file flag ===
+
If you have run with an invalid source id, you will need to go into mythtv-setup and delete all of the incorrect channels that were created.
  
The --file argument allows to directly import xmltv files. This is useful if you are using programs like [http://nxtvepg.sourceforge.net/ nxtvepg] to grab your EPG data.
+
Run mythfilldatabase as follows:
  
'''Prior 0.21:'''
+
<pre>
 +
mythfilldatabase --file --sourceid <sourceid> --xmlfile belgium.xml
 +
</pre>
  
The offset argument the defines the number of days from today on to be updated. "-1" will refresh all. In this example "belgium.xml" is the file name for the output of the grabber:
+
Before you run it the first time you will need to setup the grabber. This is explained on the [[XMLTV]] page.
<pre><nowiki>
 
$ mythfilldatabase --file <sourceid> offset belgium.xml
 
</nowiki></pre>
 
 
 
'''From 0.21 on:'''
 
 
 
From version 0.21 on the arguments of this command changed. The offset argument has been removed and the command changed to:
 
<pre><nowiki>
 
$ mythfilldatabase --file <sourceid> belgium.xml
 
</nowiki></pre>
 
 
 
The first time you will need to use the --manual flag, but this is better explained on the [[XmlTv]] page.
 
  
 
== Troubleshooting ==
 
== Troubleshooting ==
=== 1. LinkSys or Fedora Core ===
 
I spent the past 3 weeks trying to troubleshoot a problem I was having with mythfilldatabase.  I would run it and it would download between 5K and 3K and then just timeout.  If I tried downloading the NVidia drivers, the nVidia page would not even come up.
 
 
I found this solution [http://www.gossamer-threads.com/lists/engine?do=post_view_flat;post=219553;page=1;sb=post_latest_reply;so=ASC;mh=25;list=mythtv http://www.gossamer-threads.com]
 
 
which says to do this:
 
 
add this line in /etc/sysctl.conf
 
 
net.ipv4.tcp_window_scaling = 0
 
 
execute sysctl -p /etc/sysctl.conf
 
or restart your computer
 
 
Weeks of frustration fixed.  Hope this helps someone.
 
 
Note from a network techie.... Please be aware that disabling tcp window scaling has a system wide impact and will probably result in much slower bulk transfer speeds. Just thought you should be aware.
 
 
Note from another network techie: if you're scripting this to auto-download then run mythfilldatabase, you could instead say `/sbin/sysctl -w net.ipv4.tcp_window_scaling = 0` before your download, and then `/sbin/sysctl -w net.ipv4.tcp_window_scaling = 1` again afterwards, to make the setting temporary while downloading, rather than putting the setting in /etc/sysctl.conf and making it permanent.
 
  
Note from another techie: '''Don't do this.'''  That a normal download over HTTP from another site didn't work is a sure sign that something completely independent from mythfilldatabase is wrong''Fix or replace your horribly broken/misconfigured router instead.'' (...which very likely means taking out that firewall rule you deployed in an attempt to evade your ISP's tampering with BitTorrent traffic.)
+
=== mythfilldatabase causes the system to hang===
 +
This is probably an I/O saturation issuemythfilldatabase and [[MySQL]] can cause hard disk or ATA bus saturation, causing [[Mythbackend]] to starve. Older versions of MySQL would cause disk thrashing because of the use of write barriers. That could be solved by adding nobarrier to the /etc/fstab entry for the file system that holds your database (normally the root filesystem):
  
=== 2. INSERT IGNORE INTO SQL Errors ===
+
<code>UUID=900c3f88-ce0c-4a92-8e28-9ac04fdd1b09 / ext4 '''nobarrier''',errors=remount-ro 0 1</code>
If you find in mythfilldatabase's logs something like:
 
<pre>2006-11-21 22:33:22.474 DB Error (Inserting into program table):
 
Query was:
 
INSERT IGNORE INTO program (chanid, starttime, endtime, title, subtitle, description,
 
showtype, category, category_type, airdate, stars, previouslyshown, stereo, subtitled,
 
hdtv, closecaptioned, partnumber, parttotal, seriesid, originalairdate, colorcode,
 
syndicatedepisodenumber, programid) SELECT dd_v_program.chanid, DATE_ADD(starttime,
 
INTERVAL channel.tmoffset MINUTE), DATE_ADD(endtime, INTERVAL channel.tmoffset MINUTE),
 
title, subtitle, description, showtype, dd_genre.class, category_type, airdate, stars,
 
previouslyshown, stereo, subtitled, hdtv, closecaptioned, partnumber, parttotal, seriesid,
 
originalairdate, colorcode, syndicatedepisodenumber, dd_v_program.programid FROM (dd_v_program,
 
channel) LEFT JOIN dd_genre ON (dd_v_program.programid = dd_genre.programid AND
 
dd_genre.relevance = '0') WHERE dd_v_program.chanid = channel.chanid;
 
Driver error was [2/1105]:
 
QMYSQL3: Unable to execute query
 
Database error was:
 
Unknown error</pre>
 
  
It most likely means your database has become corrupted. To fix the corrupted database run the following command
+
Note that this could cause file system corruption if you have power failures or system crashes. If that is a concern you should move the databse to a separate file system. With recent versions of Linux and MySQL barriers no longer cause this problem.
<pre>mysqlcheck -r -u <username> -p <password> mythconverg</pre>
 
  
=== 3. mythfilldatabase causes the system to hang===
+
This also be caused by excessive memory usage. See [[#Memory Usage|Memory Usage]] above.
This is probably an I/O saturation issue.  mythfilldatabase and [[MySQL]] can cause hard disk or ATA bus saturation, causing [[Mythbackend]] to starve. Read more about [[Troubleshooting:Mythfilldatabase_IO_bottleneck]].
 
[[Category:Utilities]]
 
  
=== 4. mythfilldatabase is scheduled to run, but doesn't ===
+
[[Category:MythTV Software]]
One cause is invalid value in the "mythfilldatabase Log Path".  This must be set to a writable filename, not a directory name.  If it's set to a directory name (as "Log Path" might imply), or an invalid/non-writable filename, it simply won't run, and doesn't report the error anywhere.
 

Revision as of 16:03, 26 April 2017

Important.png Note: The correct title of this article is mythfilldatabase. It appears incorrectly here due to technical restrictions.

Software-update-available.png This page is up-to-date as of MythTV version 0.28, the current release is 34.0


Introduction

mythfilldatabase fills the MythTV database program table with upcoming shows. It downloads data from schedule providers and loads channel names and program details into the MythTV database.

There are two data formats supported.

  1. Data Direct format is an older format that is supported but not recommended. Only one provider of schedules information, Schedules Direct supports this format and only for USA TV programs. Since Schedules Direct now also supports XMLTV format, and provides better data that way, that is to be preferred.
  2. XMLTV format is supported by many providers for many countries.

Setup

When running mythtv-setup, the Setup Video Sources section will configure information necessary for running mythfilldatabase. The Program Schedule Downloading Options section of Setup General tells the mythtv backend whether to run mythfilldatabase, when to run it and what parameters to use.

Running

If you set up the mythfilldatabase options in MythTV setup, the backend will run it once a day, based on your settings. When you are using XMLTV, mythfilldatabase internally invokes the XMLTV grabber you have selected, downloads that data into a temporary file, then reads that data and processes it into the database.

You can also do a manual run, even if you already ran it that day, at any time. For example if your daily run failed, or you just added some new channels, you can run it manually to get the latest schedules updated.

Memory Usage

When using XMLTV, mythfilldatabase can use a lot of main memory. If you have 200 or more channels and you are processing 21 days of schedules it can use 5 GB of resident memory. If your backend only has 4 GB of memory this will start swapping, causing mythfilldatabase to take a very long time and possibly causing recordings to fail. You should do a manual run of mythfilldatabase and watch the memory usage using top to determine if you have a problem. This is particularly true if you previously used the Data Direct format, which does not have this problem.

To reduce the memory usage, you can run a few days of data at a time. I have found that running 3 days of data at a time uses around 500 MB of main memory. To achieve this you will need to set up a script that calls the xmltv grabber and mythfilldatabase repeatedly. Instead of having mythfilldatabase call the grabber internally you will do that in the script. You can name this script in the backend setup so that the backend calls it automatically every day. See the following example. It is very important to get the correct SOURCEID. See #Find out the source id of your card.

#!/bin/bash
SOURCEID=2
grabber="tv_grab_zz_sdjson_sqlite"
rm -f /tmp/tv_grab_off*.xml
for (( offset = 0; offset < 20; offset += 3 )) ; do
    "$grabber" --days 3 --offset $offset > /tmp/tv_grab_off$offset.xml
    mythfilldatabase --file --sourceid $SOURCEID --xmlfile /tmp/tv_grab_off$offset.xml
done

Another option is to use the --no-allatonce option which is new in version 29, and which forces it to process one day at a time, using around 250MB of memory. This way you do not need a script, nor do you need to find the source id.

Channels

By default, mythfilldatabase will add or update channels that are included in the input file if the tuner type supports it. That is any tuner that does not scan for channels. If you have previously added channels from a different grabber or manually, this may create duplicate channels. In many cases having it add the channels for you is convenient and avoids a lot of manual work. If you find it has duplicated channels you can use Delete All from Setup Channel Editor to delete all the channels and allow mythfilldatabase to add them. You can override this behavior in mythfilldatabase with the --only-update-guide option, in which case you will have to set up your channels manually.

If you have a tuner that scans for channels or you use the --only-update-guide option, you will have to set up your channels manually using Setup Channel Editor and/or the channel editor in mythweb. The channel editor in mythweb is easier to use but does not allow adding channels. You need to enter the xmltv id for each channel in a format that corresponds with the xmltv id in the download file.

Command line options

This information is available from mythfilldatabase --help. The details for each option can be displayed with mythfilldatabase --help option.

Option Description Details
-h OR --help OR --usage Display this help printout, or give detailed information of selected option.

Displays a list of all commands available for use with this application. If another option is provided as an argument, it will provide detailed information on that option.

--version Display version information.

Display informtion about build, including:
version, branch, protocol, library API, Qt and compiled options.

-v OR --verbose Specify log filtering. Use '-v help' for level info.
--logpath

Writes logging messages to a file in the directory logpath with filenames in the format: applicationName.date.pid.log.
This is typically used in combination with --daemon, and if used in combination with --pidfile, this can be used with log rotators, using the HUP call to inform MythTV to reload the file

-q OR --quiet Don't log to the console (-q). Don't log anywhere (-q -q)
--loglevel

Set the logging level. All log messages at lower levels will be discarded.
In descending order: emerg, alert, crit, err, warning, notice, info, debug

--syslog

Set the syslog logging facility.
Set to none to disable, defaults to none.

--nodblog Disable database logging. this is now the default, see --enable-dblog
--enable-dblog Enable logging to database.
--manual Run interactive configuration

Manual mode will interactively ask you questions about each channel as it is processed, to configure for future use.

--preset Use channel preset values instead of numbers

For use with assigning preset numbers for each channel. Useful for non-US countries where people are used to assigning a sequenced number for each channel:
1->TVE1(S41), 2->La 2(SE18), 3->TV(21)...

--file Bypass grabbers and define sourceid and file

Directly define the sourceid and XMLTV file to import.

--dd-file Bypass grabber, and read SD data from file

Directly define the data needed to import a local DataDirect download.

--sourceid Operate on single source

Limit mythfilldatabase to only operate on the specified channel source.

--offset Day offset of input xml file

Specify how many days offset from today is the information in the given XML file.

--lineupid DataDirect lineup of input xml file

Specify the DataDirect lineup that corresponds to the information in the given XML file.

--xmlfile XML file to import manually

Specify an XML guide data file to import directly rather than pull data through the specified grabber.

--update Run non-destructive updates

Run non-destructive updates on the database for users in xmltv zones that do not provide channel data. Stops the addition of new channels and the changing of channel icons.

--only-update-guide Only update guide data Only update the guide data, do not alter channels or icons.

Note this only applies to xmltv grabbers. For DataDirect see the remove-new-channels option

--do-channel-updates update channels using datadirect

When using DataDirect, ask mythfilldatabase to overwrite channel names, frequencies, etc. with values available from the data source. This will override custom channel names, which is why it is disabled by default.

--remove-new-channels disable new channels on datadirect web interface

When using DataDirect, ask mythfilldatabase to mark any new channels as disabled on the remote lineup. Channels can be manually enabled on the website at a later time, and incorporated into MythTV by running mythfilldatabase without this option. New digital channels cannot be directly imported and thus are disabled automatically.

--do-not-filter-new-channels don't filter ATSC channels for addition

Normally, MythTV tries to avoid adding ATSC channels to NTSC channel lineups. This option restores the behavior of adding every channel in the downloaded channel lineup to MythTV's lineup, in case MythTV's smarts fail you.

--cardtype No information.
--refresh Provide a day or range of days to refresh. Can be used repeatedly.

Provide days to refresh during the grabber run. Multiple
days or ranges can be supplied by multiple instances of the
option. Supported days are:
[not]today
[not]tomorrow
[not]second
#[-#]
all
Note that if all is specified any others will be ingored.

example:
--refresh today --refresh 4-8 --refresh nottomorrow

--max-days force number of days to update

Force the maximum number of days, counting today, for the guide data grabber to check for future listings.

--no-allatonce

New in version 29

Do not use allatonce even if the grabber prefers it.

This option prevents mythfilldatabase from utlizing the advertised grabber preference of 'allatonce'. This may be necessary for grabbers that return a large amount of data

--dd-grab-all refresh full data using DataDirect

This option is only valid for selected grabbers (DataDirect).
This option is the preferred way of updating guide data from DataDirect, and pulls all fourteen days of guide data at once.

--only-update-channels only update channel lineup

Download as little listings data as possible to update the channel lineup.

--no-mark-repeats do not mark repeats
-- Grabber Options Any text or options after this is passed directly to the grabber. Consequently this must be the last option.
--dont-refresh-tba

This option is only valid for selected grabbers.
Prevent mythfilldatabase from automatically refreshing any programs marked as "To be announced".
If being used with datadirect, this option should not be used, rather use --dd-grab-all to pull all listings each time.

Deprecated options
-l OR --logfile

This option has been removed as part of rewrite of the logging interface. Please update your init scripts to use --syslog to interface with your system's existing system logging daemon, or --logpath to specify a dirctory for MythTV to write its logs to.

--refresh-today

This option is only valid for selected grabbers.
Force a refresh for today's guide data.
This can be used in combination with other --refresh-<n> options.
If being used with datadirect, this option should not be used, rather use --dd-grab-all to pull all listings each time.

--dont-refresh-tomorrow

This option is only valid for selected grabbers.
Prevent mythfilldatabase from pulling information for tomorrow's listings. Data for tomorrow is always pulled unless specifically specified otherwise.
If being used with datadirect, this option should not be used, rather use --dd-grab-all to pull all listings each time.

--refresh-second

This option is only valid for selected grabbers.
Force a refresh for guide data two days from now. This can be used in combination with other --refresh-<n> options.
If being used with datadirect, this option should not be used, rather use --dd-grab-all to pull all listings each time.

--refresh-day

This option is only valid for selected grabbers.
Force a refresh for guide data on a specific day. This can be used in combination with other --refresh-<n> options.
If being used with datadirect, this option should not be used, rather use --dd-grab-all to pull all listings each time.

--refresh-all

This option is only valid for selected grabbers.
This option forces a refresh of all guide data, but does so with fourteen downloads of one day each.
If being used with datadirect, this option should not be used, rather use --dd-grab-all to pull all listings each time.

--graboptions

mythfilldatabase now passes any text after an
independent '--' directly to the external grabber.
e.g. mythfilldatabase -- --daily

--mark-repeats

This is now the default behavior. Use
--no-mark-repeats to disable.

External XMLTV

Important.png Note: The following steps are only required if you are using a grabber which does not comply to the baseline specification- i.e. It is not xmltv compliant. If possible inform the grabber author that their script does not comply with the standard so that they may fix it.

If you are using a non-compliant XMLTV grabber where mythfilldatabase will not automatically work in the normal way you will need to use the --file flag.

Obtain xml file of tv guide

Here in Belgium, I use pytvgrab to get my tv guide and output to an xml file:

$ tv_grab_be_tvb --configure  (you only need this the first time)
$ tv_grab_be_tvb -o belgium.xml

Find out the source id of your card

This is normally '1' if you have only one input card, but it is best to check it out because I have found if I have run mythtv setup more than once it can end up being something other than 1. To check, you need to look at the database:

$ mysql -u root mythconverg
mysql> select * from videosource;
+----------+------+---------------+--------+-----------+-----------+----------+--------+------------+------------+
| sourceid | name | xmltvgrabber  | userid | freqtable | lineupid  | password | useeit | configpath | dvb_nit_id |
+----------+------+---------------+--------+-----------+-----------+----------+--------+------------+------------+
|        2 | SD   | tv_grab_na_sd |        | default   | MA20453:X | NULL     |      0 | NULL       |         -1 |
+----------+------+---------------+--------+-----------+-----------+----------+--------+------------+------------+
1 row in set (0.00 sec)

This shows all of your sources and the id's. Remember the source id.

You may need to do multiple runs, e.g. one run with --sourceid 1, and then second run with --sourceid 2.

Important.png Note: Be very careful to use the correct sourceid. If you use the wrong one the system will go ahead and add channels and programs to an invalid source id that has not been set up. This will result in inconsistent results and failures.

If you have run with an invalid source id, you will need to go into mythtv-setup and delete all of the incorrect channels that were created.

Run mythfilldatabase as follows:

mythfilldatabase --file --sourceid <sourceid> --xmlfile belgium.xml

Before you run it the first time you will need to setup the grabber. This is explained on the XMLTV page.

Troubleshooting

mythfilldatabase causes the system to hang

This is probably an I/O saturation issue. mythfilldatabase and MySQL can cause hard disk or ATA bus saturation, causing Mythbackend to starve. Older versions of MySQL would cause disk thrashing because of the use of write barriers. That could be solved by adding nobarrier to the /etc/fstab entry for the file system that holds your database (normally the root filesystem):

UUID=900c3f88-ce0c-4a92-8e28-9ac04fdd1b09 / ext4 nobarrier,errors=remount-ro 0 1

Note that this could cause file system corruption if you have power failures or system crashes. If that is a concern you should move the databse to a separate file system. With recent versions of Linux and MySQL barriers no longer cause this problem.

This also be caused by excessive memory usage. See Memory Usage above.