[mythtv-users] Re: Bleb.org down??

[Simon Kenyon]

On Tuesday 20 April 2004 11:58, Rob Willett wrote:
> Simon,
>
> Most of the software needed is readily available.
>
> 1. My understanding is that bleb.org doesn't simply use one of the existing
> trawl programs. I *thought* I had read somewhere that Andrew stated this
> wasn;t the case. I could be wrong though...

it would be good if he made his software available

>
> 2. If the XML file is generated automatically each day then all we are
> really doing is spreading the download across many servers. The
> syncronisation of these servers is easy using something like wget. If the
> XML file is generated on each request for the XML file then we have to sync
> the databases. A little difficult but not that bad, simply sqldump the data
> into a file, wget it and then reinsert into the mirror. (assuming we're
> running mySQL)

that would work

>
> 3. My first thought is that something like bittorrent
> http://bitconjurer.org/BitTorrent/ might be the quickest and easiest way to
> make this available. Admittedly bittorrent is somewhat overkill for a small
> file but it's the thought that counts. <grin>. This addresses most of your
> last few points.

i've being doing some small research and so far come up wih nothing 100% 
suitable. bittorrent has a couple of characteristics which would lead me to 
believe that it is not suitable:

- it uses its own port number. this makes it firewall unfriendly
- it has a single point of failure; the tracker

>
> If we use bittorrent then the problem becomes a little easier. I've never
> set bittorrent up but will have a look at it later this week. (Just after
> I've solved world hunger...)
>
> Rob.

it seems like there are a number of discrete problems to solve:

- define a database schema which can store xmltv data
- define a mechanism for the exchange of (improper subsets of) the data
- define a mechanism for the administration (addition and deletion) of 
participating nodes with no central registry
- define a mechanism for the discovery of servers
- define a set of grabbers

the network (serious overkill) that i have in mind is contained in the 
attached diagram. comments welcome (including - you're mad!)
--
simon

>
> Quoting Simon Kenyon <simon at koala.ie>:
> > On Tuesday 20 April 2004 08:01, Rob Willett wrote:
> > > It may be that bleb.org actually constructs the XML files out of a DB
> > > to
> >
> > be
> >
> > > returned each time, I thought that this would be inefficient since we
> >
> > could
> >
> > > generate the entire XML ech day, but I don't know how it's setup
> > > either.
> > >
> > > Who needs to approach Andrew about this?
> >
> > a while ago i asked andrew about the software he runs
> > i was concerned about the single point of failure issue
> > i also wanted to spread the load and increase the range of channels
> > supported
> > [i have ntl cable and sky digital]
> >
> > he ignored the question (for whatever reason)
> >
> > i would like to build a network of servers to provide this facility
> > that way it would be fault tolerant
> >
> > required:
> >
> > a set of grabbers from the various tv stations (tvlive, bbc, etc)
> > a database to store them
> > a web system for providing the listsings in xmltv format
> > a mechanism for providing the list of servers
> > a mechanism for keeping the server list current
> > a mechanism for sharing the load
> >
> > i am willing and capable of helping with the design, code and hosting of
> > such
> >
> > a system. i need partners to help in the design.
> >
> > i would suggest the following:
> >
> > a unix like os (i use linux but any should be acceptable - bsd, solaris,
> > aix,
> >
> > etc)
> > mysql at least, but possibly postgres for the database
> > perl or python for the grabbers
> > php or mod_perl for the servers
> >
> > ? for the server sync - there must be software in this space already!
> >
> > --
> > simon
> > _______________________________________________
> > mythtv-users mailing list
> > mythtv-users at mythtv.org
> > http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
>
> -------------------------------------------------
> This mail sent through IMP: http://horde.org/imp/
> _______________________________________________
> mythtv-users mailing list
> mythtv-users at mythtv.org
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dxmltv.png
Type: image/png
Size: 19083 bytes
Desc: not available
Url : http://mythtv.org/pipermail/mythtv-users/attachments/20040420/e79bad40/dxmltv.png