[mythtv-users] (SOLVED) Mythfilldatabase returns incomplete schedule data (Patched)

Raymond Boettcher raymondboettcher at yahoo.com
Sat Jan 19 12:29:32 UTC 2013


    I actually was running the grabber at the suggested time.  I figured out that it was erroring on occasion because of my ISP's rather slow and non-responsive DNS Server.  So I reset my DNS settings manually to google's 8.8.8.8 server.  This seems to have cleared a few things up.

    There is an inherent problem with telling the grabber to download the full workload of data at once every night.  This will eventually take up a lot more bandwidth on Schedules Direct side when you really just need a day or two worth of updates.  Everyone will eventually switch over to --dd-grab-all and then MythTV will eventually remove the option because of the excessive workload on Schedules Direct.  

    The original code in MythTV 0.23.1 used wget as it's download agent.  This had its benefits because wget wasn't so easy to give up trying to download the data.  Now that this functionality was moved into mythfilldatabase directly. So it only makes 1 attempt to download the data and if that fails, it aborts that block of data.  However, based on source code, it looks as if they converted the code for Schedules Direct to use an internal grabber but it seems the xmltv code still uses the external shell routine to grab data.

    No doubt the MythTV developers are busy working on more important matters.  So, I wrote an update for mythfilldatabase that will allow 5 attempts at downloading a block of data before giving up.  It's not the cleanest code in the world and no doubt the code should have been located directly inside the routine that handles the download itself instead of the function that calls for the download.  However, I don't know if the same grabber code is used by Stream Music, etc that may call upon the routine to download the M3U Playlist file or by the grabber for the Network Tuner.  So I placed the code where it is because I know that at that location it will not affect anything except mythfilldatabase.

    For the moment the retries is fixed at 5 but can be changed inside the source file and recompiled.  I did at least assign a variable for that value instead of hard coding it.  I was going to assign a parm to mythfilldatabase to make the value changeable, I just haven't got to it yet.  I added the code for both the normal download by block routine and the --dd-grab-all routine as well.

    I've tested the code and it works perfectly!  The diff must be applied against /mythtv-fixes-0-26/mythtv/programs/mythfilldatabase/filldata.cpp.  It should diff against the latest release as well.  I just downloaded the 0.26+fixes branch from github the other day.  I linked the fully patched filldata.cpp file along with a diff file that reflects the changes against the original source.  I compressed it into a single file since the mailing list decided to silently rejected my mail with both source files attached...

http://www.raylinenetworks.com/downloads/mythfilldatabase-patch.tar.bz2
 
    Hopefully, this helps out some people out there that are having problems with Schedules Direct and we can prevent a few hundred thousand people from using --dd-grab-all causing excessive bandwidth usage on Schedules Direct's part causing more people to have problems grabbing their schedule.

Raymond Boettcher

> --- On Wed, 1/16/13, Michael T. Dean <mtdean at thirdcontact.com>
> wrote:
> 
> > From: Michael T. Dean <mtdean at thirdcontact.com>
> > Subject: Re: [mythtv-users] Mythfilldatabase returns
> incomplete schedule data
> > To: "Discussion about MythTV" <mythtv-users at mythtv.org>
> > Date: Wednesday, January 16, 2013, 11:38 AM
> > On 01/16/2013 03:24 AM, Raymond
> > Wagner wrote:
> > > On 1/16/2013 03:19, Raymond Boettcher wrote:
> > >> Blocks of data after the missing time seem to
> > populate correctly. So I
> > >> think it just fails to download a specific
> block of
> > guide data once in a
> > >> while. You would think mythfilldatabase would
> retry
> > that block during
> > >> the next scheduled run but never does. I think
> it
> > automatically flags
> > >> that time range as already downloaded and
> fails to
> > try again.
> > > 
> > > You should be having mythfilldatabase run with
> the
> > parameter '--dd-grab-all', so that there are no
> "specific
> > blocks" of data, but rather you download the entire
> thing
> > all at once, and refresh all fourteen days of data on
> each
> > run.
> > 
> > Exactly.  The problem is that you're running
> > mythfilldatabase at a "less than ideal" time, and--due
> to
> > the number of channels you have and the number of
> programs
> > on them--mythfilldatabase is saying, "Well, I got
> sufficient
> > listings data for that last data that I shouldn't
> waste
> > resources by re-grabbing it until I need to (until that
> day
> > is "tomorrow")."  So, you end up with holes in your
> > data.
> > 
> > To fix this:  a) make sure you let Schedules Direct
> > tell you when to run mythfilldatabase by enabling the
> > mythtv-setup setting, "Run mythfilldatabase at time
> > suggested by the grabber." and b) run mythfilldatabase
> with
> > --dd-grab-all (assuming your backend and MySQL server
> > host(s) are sufficiently powerful so that you always
> get all
> > of the most-current listings data available and use
> fewer
> > server resources than you do with a default
> mythfilldatabase
> > run.
> > 
> > See
> > http://www.gossamer-threads.com/lists/mythtv/users/449426#449426
> > and
> > http://www.gossamer-threads.com/lists/mythtv/commits/449416#449416
> > for details about using --dd-grab-all.
> > 
> > Mike
> > _______________________________________________
> > mythtv-users mailing list
> > mythtv-users at mythtv.org
> > http://www.mythtv.org/mailman/listinfo/mythtv-users
> >
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mythtv.org/pipermail/mythtv-users/attachments/20130119/5286d420/attachment.html>


More information about the mythtv-users mailing list