[mythtv-users] UK Radio Times XMLTV [was: Reliability of USB capture cards]

[Nick Morrott]

On 26 July 2010 01:27, Jason Chambers <lists at purplish-monkey.com> wrote:
> On 25/07/2010 21:07, Keith Edmunds wrote:
>>
>> On Sun, 25 Jul 2010 20:55:31 +0100, lists at purplish-monkey.com said:
>>>
>>> I'm in the UK using the Radio Times* xmltv feed so I don't have to
>>> use EIT.
>>
>> I'm in the UK also, and I'd be interested in your experiences with
>> the Radio Times data (but perhaps that's a subject for another
>> thread).
>
> Generally its very good.  There are a few minor problems which I'll get to
> later but none of them are annoying enough for me to start sending
> threatening letters to the BBC.

Please don't! They provide the service gratis but do not create the
listings content themeselves - they pass them on from their listings
provider after adding their own programme reviews where necessary.

> I originally started using XMLTV for two reasons.  Firstly there's the
> problem that started your original thread where the cards are not reliable
> when continuously gathering EIT data.

Disabling active EIT collection on one or both of the tuners has
historically been the accepted "fix" for the instability. If you
record frequently from DVB-T, passive EIT collection during recordings
should keep your listings fresh.

> Secondly when I tried, the EIT data was missing the category, making it
> harder to search for the programmes I was likely to be interested in as
> everything was lumped together in one long list.  Because of the problem
> with the cards I never bothered to find out whether this was a due to my
> setup, a bug in MythTV or the EIT data itself.

EIT (and the MythTV implementation for the last few releases) does
(and has) provide programme categorisation where available.

> With the Radio Times data you get 14 days of programme data which includes
> episode titles (in most cases), cast & director details along with good
> descriptions.  The descriptions - for films in particular - are a not just a
> synopsis but actually a 1 or 2 paragraph review by the Radio Times (very
> helpful if you have never heard of it before).
>
> I only have Freeview and then generally only watch films, documentaries and
> drama/comedy series.  Its very rare that I see shows with generic
> descriptions, and I don't recall it ever providing me with the wrong program
> details or getting the start & end times wrong except where late running
> sporting events have affected the schedule on that day.

The quality of this data is likely due to the work of major listings
provider the Radio Times use for their data. The RT data is generated
daily in the early hours, so may not catch last-minute programme
changes.

> I'm not sure how accurate it is for soaps & talk-shows etc that air new
> content every day though as I never watch any of that
> rubbish^H^H^H^H^H^H^Hhigh quality entertainment.
>
> The XMLTV takes a few extra steps to setup as you have to specify the
> xmltvid for each channel manually but obviously that's a one-off task so its
> not worth counting against it.

This is being worked on (slowly, but it is very high on my list).

> The few problems I have found are:
>
> 1.  There are no programids in the data so you have to rely on descriptions
> and episode titles to detect duplicates.  Which leads me onto 2 ...

The program/seriesids are broadcast over EIT and are probably
unavailable to the listings provider for 14 days in advance.

However, I was talking about this very issue at a LUG meeting
recently, and we got to talking about running tv_grab_dvb and
tv_grab_uk_rt outside of MythTV and then blending the data together to
get the best of both worlds:

i) program/seriesid information from the EIT data to enable
serieslink-type things to work (and to improve dupe detection), and
ii) the rich descriptions, reviews and cast information from the RT data.

I've put a lot of work into tv_grab_uk_rt in order to allow
post-processing of the data provided by the Radio Times XMLTV service
to improve its consistency, and this approach would additionally bring
the benefits of the Freeview programme ids into the mix.

> 2. The descriptions and episode titles and not always consistent when shows
> are repeated.

This can be - and is currently - improved if you provide details of
those shows where you see inconsistencies.

> Descriptions vary between in-depth reviews and short synopsis.  And this can
> mean that repeat showings (whether in other time-slots the same week, or
> months later when the entire series is re-run) can have rewritten
> descriptions for the same episode. A human can easily spot that they are
> they a referring to the same episode but Myth has no chance.  "Being Human"
> early in the year sticks in my mind as a example of this, but I've seen it
> on others too.

Recent and ongoing work on the XMLTV DTD will allow for reviews to be
held separately from descriptions. When the RT reviews a programme,
the review comes first, and subsequent repeats have a regular
description for the episode - sometimes weeks later. It may be
possible to look forward within the two week listings window to check
whether a reviewed programme is being repeated and then crib the
regular description for use in lieu of the RT review. The RT review
could then be stored in the separate review element. This is not
currently implemented in tv_grab_uk_rt but is something I will look
at.

> Because of this I would recommend that you use "Subtitle /then/ description"
> as your default duplicate detection method.

Agreed. I've used this since it became available for almost all rules
when using tv_grab_uk_rt data.

> Also when programmes do have episode titles there can be inconsistent
> punctuation/abbreviation which leads to Myth re-recording episodes you've
> already seen.  For instance "House" had episodes "Euphoria, Pt1" and
> "Euphoria, Pt2" back in January but then aired "Euphoria - Part 1" &
> "Euphoria - Part 2" in April.  Similar things happen with and/&, or
> Mr/Mr./Mister etc.
>
> This doesn't happen that often so its not a big problem - just don't be
> surprised when you get the occasional re-recording of an old show due to a
> slight difference in the episode title.

The current release of tv_grab_uk_rt should handle - and correct -
most, if not all, variants of the Part 1/Pt. 1/Part One issue seen in
programme episode information. It can't change historical information
in your previously recorded table, but going forwards should improve
things a lot. Additionally, specific updates can now be made to
listings on a per-programme basis - but it requires notification. I
run the grabber with full debug info enabled, which normally points
out listings which could do with being fixed.

> 3. Inconsistent assignment of program categories.
>
> Films are films, and drama is drama.  But a sitcom could be a sitcom or a
> comedy or entertainment.  An animation is an animation, unless its a comedy
> or a sitcom.
>
> This is what annoys me the most as its not even consistent for the /same/
> show - for instance "Simspsons" & "King of the Hill" both alternate between
> animation and comedy.  And "Men Behaving Badly" has been sitcom, comedy,
> entertainment and (very bizarrely) a game-show.
>
> So if you group or filter your recordings screen by category you may have to
> look in more than one place for certain shows.

The current release of tv_grab_uk_rt allows for blanket category
updating on a per-title basis. At the time of writing, 493 different
programme titles are updated. The debug output of the grabber
currently reports whenever the same programme title is assigned to
different categories within the 14-day listings window. I am intending
to do a historical trawl through old listings data to catch instances
where this has happening in the past. Although I try to pick the best
category for programmes when adding the fixups, it is quite possible
for me to categorise something as comedy when it's a sitcom. Going
forwards, updates will be consistent, and if I do miscategorise
something, it can always be corrected.

With regards to your specific examples, the current release of the
grabber already updates these titles to the following categories:

- King of the Hill~Animation
- The Simpsons~Animation

Men Behaving Badly is not currently handled explicitly (although "Men
Brewing Badly" is!) but I'll add it to the sitcom category (which is
how all current G.O.L.D./Dave showings for the next 14 days are
categorised at time of writing).

> 4. Documentaries do not have cast lists
>
> So you can't do a recording rule to look for shows from people like David
> Attenborough, Rich Hall, or Billy Connolly.  Usually the narrator/presenter
> will be mentioned in the description if they are famous enough but this is
> not always the case.

This is at the mercy of the listings provider. I find power seaches
checking title and description to work very well for cases like this,
and find myself creating power searches almost as frequently as
regular recording rules as broad catch-alls. If your power search
highlights one programme in a series featuring a person of interest,
you can always create a regular recording rule for the title to catch
all showings (and repeats of earlier missed showings) when that
person's name is not explicitly mentioned in the description.

> 5. It does not provide listings for radio stations
>
> Even though their website gives radio station listings they are not included
> in the raw data at xmltv.radiotimes.com that the grabber is allowed to use.

I've asked the RT about this a few times. I was told it would take a
lot of additional work on their part to integrate the radio listings
with the TV data, and they don't have the time/manpower. I've used a
mixed XMLTV/EIT system for years, and find it works very well.

If you do notice inconsistencies in the data you see (and you're
running the current stable release of XMLTV), please post details to
the ongoing uk_rt updates thread on the xmltv-users list. I update the
fixups frequently...

Cheers,
Nick
tv_grab_uk_rt developer

-- 
Nick Morrott

MythTV Official wiki: http://mythtv.org/wiki/
MythTV users list archive: http://www.gossamer-threads.com/lists/mythtv/users

"An investment in knowledge always pays the best interest." - Benjamin Franklin