[mythtv-users] Mooting architecture for a DataDirect replacement

[Rod Smith]

On Saturday 23 June 2007 09:23, Peter Schachte wrote:
> Rod Smith wrote:
> >>> I used net news back in the day.  My memories are of out-of-order
> >>> messages, replies to messages that never-ever came through, and other
> >>> general annoyances.
> >
> > For the application under discussion, out-of-order postings don't matter;
>
> If there are multiple updates to the schedule in separate postings, I don't
> think this is true.  If there's an update that says that station X is
> airing program Y at time Z, and another update sent an hour later that says
> no, it'll be at time Q instead, then you don't want to handle them in the
> wrong order.

This would only be a problem if the code were poorly written (you snipped my 
original comment to that effect) or if the original data never arrived. Each 
message could have a list of the original files and patches upon which it 
relies, and if messages arrive out of order, the client-side software will 
know to delay processing of the later data until the earlier data is 
available and/or to not apply earlier changes that would overwrite more 
recent ones. Alternatively, the software could cache all the original update 
files, detect the out-of-order arrivals, and re-run the whole sequence from 
the start. Obviously a lot of details would depend on the exact file formats 
used, but I doubt if this problem would be all that difficult to solve.

> I believe the OP suggested we consider requirements for this system.  As
> much as I like usenet for discussion forums, I think the requirements for
> our purposes here are rather different.  One key requirement for us is that
> distribution be reliable; ie, at all times, every subscriber should have
> the correct schedule for their lineup as at the time of their last update.

I agree that this isn't the strongest point of Usenet, but the more reliable 
Usenet providers are very reliable. In a worst-case scenario, a user with a 
poor Usenet provider could switch to a more reliable one.

> Another is that if something goes wrong and somehow myth doesn't have the
> correct schedule, it should be able to tell, and be able to correct it. 
> It's hard for me to see how to satisfy that with NNTP.

NNTP isn't the issue here; it's error detection and correction (or lack 
thereof) in the encapsulated files.

> If postings get  
> lost or come in out of order, the schedule winds up wrong.  The problem
> could be detected if each posting contain a checksum for the whole lineup
> after the change in the posting is applied (not just for that posting). 
> But if the checksum fails, what can you do?  If you drop the whole schedule
> and reread all the postings from the same host, you'll just get back the
> same wrong schedule.  You need an authority to go to to get the correct
> data.

I mentioned par2 files as a possible means of error correction in an earlier 
post. Figuring out precisely how to apply this technology would require some 
thought, but it would probably be do-able. Another possibility would be to 
provide a central repository or "master" NNTP server to which Myth boxes 
could connect as a last resort in the event of missing files. The code would 
wait some period of time for missing files to appear on the user's primary 
NNTP server and if they don't, connect to the master server to retrieve them.

> Someone else suggested bittorrent for this, and it actually has some
> properties that make it quite appropriate.

One problem with BitTorrent is that it's strongly associated with piracy. That 
could make it a hard sell to potential content providers, such as TV stations 
("you're going to be distributing the data to video pirates?! Get outta 
here!"). I haven't looked into the issue in any detail, but I believe some 
ISPs are blocking BitTorrent traffic, which if true is another big downside 
for BitTorrent. Finally, I've heard stories of people getting DMCA takedown 
notices for posting ANYTHING on BitTorrent, even family photos. Given the 
frequency with which we'd need to post new data, this could become a very 
serious problem if the RIAA (or whoever) is targeting BitTorrent users in a 
wholesale way. Granted they'd have no legal leg to stand on, but defending 
each and every takedown notice could become an unmanageable legal task.

-- 
Rod Smith
http://www.rodsbooks.com