[mythtv-users] Little OT: RealTime Parallel Multi-PC Transcoding
nick.rout at gmail.com
Wed Jun 16 03:47:59 UTC 2010
On Wed, Jun 16, 2010 at 12:14 PM, Raymond Wagner <raymond at wagnerrp.com> wrote:
> On 6/15/2010 17:05, belcampo wrote:
>>> Do you actually need to have realtime h264, or do you just want to speed
>>> up transcodes? If the latter, is there any reason why you cannot just run
>>> two transcodes in parallel?
>> If I want a speedup as you say I should run 2 transcodes in parallel, how
>> would I do that, independantly, on the same source.
>> As mentioned in the 1st post multithreading is very inefficient. Also see
>> http://saintdevelopment.com/codecs/bfs-vs-cfs.txt which I found here
>> After 2 Cores the efficiency drops dramatically. Where my approach would
>> /should be more efficient, at least I think and hope so.
> I would have to see more information surrounding that test to draw any
> conclusions from it. Certainly on a quad core processor, running more than
> four threads is going to provide negligable, if any, gains.
> The Core2Quad running DDR2 memory is starved for bandwidth on a number of
> applications. A couple years back, we did testing with a Q6600, using a
> turbomachinery solver that we knew was good for efficient parallelization
> across hundreds of machines. Two processes ran with 75% speedup over one,
> while four processes only ran with 20% speedup over two. DDR3, and
> especially triple channel i7s should significantly improve these numbers.
> Video encoding using 'slices' is what is called 'ridiculously parallel'.
> Aside from some preprocessing of the raw video, the encoding process is
> batch. The encoding threads are completely independent, working on their
> own data set, and there is no need for communication between them. Ignoring
> bottlenecks on shared resources like memory or communication buses, the
> encoder should scale linearly with the number of cores.
>>> You would want the video cuts to be tens of seconds long to reduce data
>>> and startup overhead.
>> Could you be more specific on this ?
> Starting up processes is expensive. The startup and teardown of something
> complex like x264 is probably going to be a couple seconds. If you're only
> going to be encoding chunks of a couple seconds long, that is a huge amount
> of overhead.
In terms of looking at an existing implementation, you might want to
take a look at the code in dvd:rip http://www.exit1.org/dvdrip/
dvd:rip is a perl dvdripper that uses transcode for transcoding. The
point is that it does have a cluster mode where you can distribute
your encoding round a lan. You may get some ideas from their code, or
be able to hack it to do your bidding.
More information about the mythtv-users