[mythtv-users] Re: Bleb.org down??

Rob Willett rob.mythtv at robertwillett.com
Tue Apr 20 10:34:18 EDT 2004


Jeff,

Sounds like what we're trying to do. Always happy to come out to the Caribbean
to help if you need it! Beats wet and windy London.

I've looked more into the distribution method and just find Mutella. A command
line based version of Gnutella. I'll download it later this evening and have a
play with it. This may provide the automated load balancing distribution I think
we need. 

Rob.

Quoting Jeff C <myth at express.tc>:

> Just as a note, I wanted to chime in on this thread as I'm reading it with
> some interest.
> 
> I'm currently in the Caribbean and am working on something similar to
> generate listings for my local cableco in the Turks and Caicos.
> 
> All of our signals come from North America, but I need to hit 5-6 cities to
> get them all.  What I have done is write some scripts that call the
> tv_grab_na grabber for each of the cities, and then I have only the channels
> I want in each city being collected.  From there I strip off the header and
> footer of each file and combine them together with a master head and footer.
> 
> Then I can do a mythfilldatabase --file 1 0 file.xml
> 
> That works great for the full pass, but when I try and do my dailies it
> dies.  I'm investigating why and will report back, probably a weird XML
> parsing issue somewhere, a step I missed.
> 
> The reason I mention it in this thread is once I get it sorted I intend to
> place the daily files and a master 14 day file on a server in country so
> others can just grab it locally, the same idea as you are thinking, so the
> distribution method is of interest as well as the tribulations of getting
> the file formatted.:)
> 
> Thanks
> 
> -Jeff
> 
> -----Original Message-----
> From: mythtv-users-bounces at mythtv.org
> [mailto:mythtv-users-bounces at mythtv.org] On Behalf Of Rob Willett
> Sent: Tuesday, April 20, 2004 9:41 AM
> To: Discussion about mythtv
> Subject: Re: [mythtv-users] Re: Bleb.org down??
> 
> Simon,
> 
> I'm not a Bittorrent expert, I do have a faint recollection of opening up my
> firewall tough. I wouldn't say this was a major problem though. The single
> point
> of failure of the tracker is! If we have a SPF we're back to what we had
> before
> <sigh>.
> 
> I think we have two very distinct problems to solve:
> 
> 1. Getting the EPG data.
> 
> 2. Distributing the information such that it's:
>    a) easy (no difficult to install software)
>    b) low cost. Free (!), low usage of bandwidth, http connections
>    c) quick
>    d) reliable
> 
> So let's look at 1. As I understand it we already have a program to grab The
> Radio Times listings. I don't know the output of the grabbers are as I'm at
> work. Are they XML? If they are then I think we have a solution to 1. if
> they
> aren't then we have to parse the data and write XML suitable for MythTV to
> handle. Whether it's a single file or multiple files in multiple directories
> or
> even a database is neither here nor there for the moment. 
> 
> 2. The second stage is how we distribute the information. None of us want to
> be
> the master source for all users even if there are only 20 of us. Single
> points
> of failure etc come to mind. We need the system to be auto-load balance, be
> resiliant and easy to use and administer. I think we need some sort of P2P
> program. Never having used Kazaa or Napster I have no idea how they work but
> I'm
> fairly sure that we could use their ideas. Perhaps Gnutella (or some other
> P2P
> program) may be of some use to us. What we really want is peer to peer
> sharing
> with a number of nodes acting as the master source to 'seed' the rest of the
> network. 
> 
> Comments welcomed.
> 
> Rob.
> 
> Quoting Simon Kenyon <simon at koala.ie>:
> 
> > On Tuesday 20 April 2004 11:58, Rob Willett wrote:
> > > Simon,
> > >
> > > Most of the software needed is readily available.
> > >
> > > 1. My understanding is that bleb.org doesn't simply use one of the
> > existing
> > > trawl programs. I *thought* I had read somewhere that Andrew stated this
> > > wasn;t the case. I could be wrong though...
> > 
> > it would be good if he made his software available
> > 
> > >
> > > 2. If the XML file is generated automatically each day then all we are
> > > really doing is spreading the download across many servers. The
> > > syncronisation of these servers is easy using something like wget. If
> the
> > > XML file is generated on each request for the XML file then we have to
> > sync
> > > the databases. A little difficult but not that bad, simply sqldump the
> > data
> > > into a file, wget it and then reinsert into the mirror. (assuming we're
> > > running mySQL)
> > 
> > that would work
> > 
> > >
> > > 3. My first thought is that something like bittorrent
> > > http://bitconjurer.org/BitTorrent/ might be the quickest and easiest way
> > to
> > > make this available. Admittedly bittorrent is somewhat overkill for a
> > small
> > > file but it's the thought that counts. <grin>. This addresses most of
> your
> > > last few points.
> > 
> > i've being doing some small research and so far come up wih nothing 100% 
> > suitable. bittorrent has a couple of characteristics which would lead me
> to 
> > believe that it is not suitable:
> > 
> > - it uses its own port number. this makes it firewall unfriendly
> > - it has a single point of failure; the tracker
> > 
> > >
> > > If we use bittorrent then the problem becomes a little easier. I've
> never
> > > set bittorrent up but will have a look at it later this week. (Just
> after
> > > I've solved world hunger...)
> > >
> > > Rob.
> > 
> > it seems like there are a number of discrete problems to solve:
> > 
> > - define a database schema which can store xmltv data
> > - define a mechanism for the exchange of (improper subsets of) the data
> > - define a mechanism for the administration (addition and deletion) of 
> > participating nodes with no central registry
> > - define a mechanism for the discovery of servers
> > - define a set of grabbers
> > 
> > the network (serious overkill) that i have in mind is contained in the 
> > attached diagram. comments welcome (including - you're mad!)
> > --
> > simon
> > 
> > >
> > > Quoting Simon Kenyon <simon at koala.ie>:
> > > > On Tuesday 20 April 2004 08:01, Rob Willett wrote:
> > > > > It may be that bleb.org actually constructs the XML files out of a
> DB
> > > > > to
> > > >
> > > > be
> > > >
> > > > > returned each time, I thought that this would be inefficient since
> we
> > > >
> > > > could
> > > >
> > > > > generate the entire XML ech day, but I don't know how it's setup
> > > > > either.
> > > > >
> > > > > Who needs to approach Andrew about this?
> > > >
> > > > a while ago i asked andrew about the software he runs
> > > > i was concerned about the single point of failure issue
> > > > i also wanted to spread the load and increase the range of channels
> > > > supported
> > > > [i have ntl cable and sky digital]
> > > >
> > > > he ignored the question (for whatever reason)
> > > >
> > > > i would like to build a network of servers to provide this facility
> > > > that way it would be fault tolerant
> > > >
> > > > required:
> > > >
> > > > a set of grabbers from the various tv stations (tvlive, bbc, etc)
> > > > a database to store them
> > > > a web system for providing the listsings in xmltv format
> > > > a mechanism for providing the list of servers
> > > > a mechanism for keeping the server list current
> > > > a mechanism for sharing the load
> > > >
> > > > i am willing and capable of helping with the design, code and hosting
> of
> > > > such
> > > >
> > > > a system. i need partners to help in the design.
> > > >
> > > > i would suggest the following:
> > > >
> > > > a unix like os (i use linux but any should be acceptable - bsd,
> solaris,
> > > > aix,
> > > >
> > > > etc)
> > > > mysql at least, but possibly postgres for the database
> > > > perl or python for the grabbers
> > > > php or mod_perl for the servers
> > > >
> > > > ? for the server sync - there must be software in this space already!
> > > >
> > > > --
> > > > simon
> > > > _______________________________________________
> > > > mythtv-users mailing list
> > > > mythtv-users at mythtv.org
> > > > http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
> > >
> > > -------------------------------------------------
> > > This mail sent through IMP: http://horde.org/imp/
> > > _______________________________________________
> > > mythtv-users mailing list
> > > mythtv-users at mythtv.org
> > > http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
> > 
> 
> 
> 
> 
> -------------------------------------------------
> This mail sent through IMP: http://horde.org/imp/
> _______________________________________________
> mythtv-users mailing list
> mythtv-users at mythtv.org
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
> 
> 
> 
> 




-------------------------------------------------
This mail sent through IMP: http://horde.org/imp/


More information about the mythtv-users mailing list