[mythtv-users] Mooting architecture for a DataDirect replacement
Jay R. Ashworth
jra at baylink.com
Fri Jun 22 01:46:36 UTC 2007
On Thu, Jun 21, 2007 at 08:19:27PM -0400, Christopher X. Candreva wrote:
> On Thu, 21 Jun 2007, Jay R. Ashworth wrote:
> > I was merely trying to propose an architecture that would make
> > practical the distribution of the load of 200,000 Mythboxen looking for
> > guide data every day. NNTP would.
>
> I used to run an NNTP system. I'm going to assume INN or it's replacement
> has gotten better, but it wasn't easy, and there is going to be
> significantly less experience in it today, especially most ISPs are
> outsourcing it.
Centralized traffic could be *run* via the commercial providers.
Local machines wouldn't need to carry but a couple newsgroups, and not
a lot of data traffic.
> There is another distributed-database system in place, complete with local
> caching and variable cache time: DNS .
Not built for it. Our data objects are too big, DNS is synchronous,
and it's also non-flooding, in any real sense. It doesn't keep the
back data the way NNTP does either.
> The anti-spam community has used DNS for years for real-time blacklists of
> IP address. In it's simplest form: Lets say you want to know if IP address
> 1.2.3.4 is something you should accept mail from. Do a dns lookup on
> 4.3.2.1.blacklistexample.com . If it returns a value, it's blacklisted. DNS
> not found, it's OK . Usually the IP returns is in 127.0.0.x , and by the
> last digit information as to the type of listing is returned.
Sure.
> Another example: The Clam AntiVirus project distributed the virus databases
> via simple http. They were doing http HEAD requests to check if a database
> was available, but the bandwidth was killing them. I suggesting using DNS to
> publish the current serial number of the database. If that number indicates
> a new version is available, the updater downloads a new copy of the virus
> DB. In this way, you can check for new virus signatures every 5 minutes, and
> it only costs a 58 byte UDP packet.
>
> I can see a similar method, using DNS to indicate an update is available,
> triggering the transfer of an updated data set/file.
My intuition is that that's not practical, because the entire idea is
to *deaggregate*: I don't *want* One Big Database, that has to process
specific queries for 200K users. eBay does it pretty well, but they
make a lot more money.
NNTP servers will do all the work for you, leveraging properties
already built into the protocol.
I dunno; maybe I'm way off base; I don't seem to be getting any
traction. But it seems pretty obvious to me...
Cheers,
-- jra
--
Jay R. Ashworth Baylink jra at baylink.com
Designer The Things I Think RFC 2100
Ashworth & Associates http://baylink.pitas.com '87 e24
St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
More information about the mythtv-users
mailing list