[mythtv-users] WARNING: SchedulesDirect users: server currently returning bogus results
f-myth-users at media.mit.edu
f-myth-users at media.mit.edu
Sun Dec 20 08:21:58 UTC 2009
> Date: Sun, 20 Dec 2009 01:42:29 -0600
> From: Robert Eden <rmeden at yahoo.com>
> I looked at my code, and appear to have run into this problem with
> _na_dd in the (distant) past. In addition to parsing the XML, I
> search the raw XML for <faultstring>. If it exists, I report the error
> and die. From Tribune's point of view it makes sense; they were
> processing/transmitting data when they hit an error. It would be
> difficult to close all the open tags so they just add the error message
> and die. When I dealt with this before, the XML wasn't valid and I
> couldn't parse. it.
Yes, makes sense. My automation now also checks for the fault string;
that was the first thing I added.
> Since you use "--download-only" my check can't help you since it
> just.... downloads the data. If something downloaded (even with an
> embedded error) to me it's a success under --download-only. Whatever
> parses the output should check for errors just like I do when I process
> the downloaded data.
Also makes sense. (And implies that if I hadn't thrown away the exit
value from tv_grab_na_dd by running it by hand, it still would have
appeared to have succeeded.)
Before today, I merely looked for the existence of </SOAP-ENV:Envelope>
because a month of extraordinarily poor ISP network issues led to the
file occasionally being truncated during the download, so that told me
that the file was complete. However, today's behavior emitted the
fault code -and- then properly closed the file, so my automation said
hunky-dory and went and called mfdb on a bad input.
Now that I see the behavior, I check for at least one each of all the
likely elements a valid file would contain, both singular and plural,
both open and closed, e.g., <stations>, <station>, </station>, </stations>,
and etc for schedule, program, etc. It's ultra-cheesy compared to actually
parsing the XML, but it's one line of code and a dumb list of names, so it's
good enough to catch the very rare case of a totally implausible file; worth
the few minutes it took to write.
> In the case of MythTV (when it reads a raw data direct formatted file),
> it appears to blow away the database before it fully parses the file,
> ensuring it's valid. I suspect many XML errors could cause the "zero'd
> database" symptom.
Whoops. I wonder if there's a way to defer the deletion that wouldn't
unacceptably complicate the code for a rare case.
> The outage should not have impacted many people. Most download at least
> 7 days of data and a single days' outage *should* be tolerable. IMHO
Oh, absolutely, assuming that Myth correctly did -not- remove any
pre-existing scheduling data after downloading the truncated XML.
In fact, it only just now occurred to me that most people wouldn't
have seen a totally blank schedule even if they -were- impacted,
because by default I believe that Myth downloads "today" (only in
some versions of Myth?), "tomorrow", and "today+13" (plus any day
which doesn't seem to have enough hours filled in, I believe), so
even if 2 or 3 days got nuked, the -rest- would still show up in
the schedule, masking the problem. But since I ran my test using
"--days 14", it blew away everything.
> you have a valid gripe with the Myth loader, not the service. (as
I may not even have a valid gripe with the Myth loader; I haven't
looked closely at the code to see if it has enough error-checking
that today's issue would have just caused it to give up -before-
deleting any scheduling data. My automation certainly didn't handle
it well, but I've fixed it. :)
[If anyone decides to add more error-checking and needs my defective
XML, I can make it available, but you can make yours just as easily---
look at my original post and truncate the XML as shown, adding the
trailing error lines.]
> history shows, we do have a pretty good track record minimizing
> outages). Hopefully the stock Myth download behaved better and only a
I think it must have or we'd have heard more about it by now.
> few people were impacted. Good idea to warn the list.
Thanks, and thank you again for your time (and for the service!).
More information about the mythtv-users