[mythtv-users] Little OT: RealTime Parallel Multi-PC Transcoding

[belcampo]

Raymond Wagner wrote:
> On 6/15/2010 17:05, belcampo wrote:
>>> Do you actually need to have realtime h264, or do you just want to 
>>> speed up transcodes?  If the latter, is there any reason why you 
>>> cannot just run two transcodes in parallel?
>> If I want a speedup as you say I should run 2 transcodes in parallel, 
>> how would I do that, independantly, on the same source.
>> As mentioned in the 1st post multithreading is very inefficient. Also 
>> see http://saintdevelopment.com/codecs/bfs-vs-cfs.txt which I found 
>> here http://x264dev.multimedia.cx/
>> After 2 Cores the efficiency drops dramatically. Where my approach 
>> would /should be more efficient, at least I think and hope so.
> 
> I would have to see more information surrounding that test to draw any 
> conclusions from it.  Certainly on a quad core processor, running more 
> than four threads is going to provide negligable, if any, gains.
> 
> The Core2Quad running DDR2 memory is starved for bandwidth on a number 
> of applications.  A couple years back, we did testing with a Q6600, 
> using a turbomachinery solver that we knew was good for efficient 
> parallelization across hundreds of machines.  Two processes ran with 75% 
> speedup over one, while four processes only ran with 20% speedup over 
> two.  DDR3, and especially triple channel i7s should significantly 
> improve these numbers.
I think "the problems" are of another nature, but that's what I think. 
I'll do some testing to see if my gutfeeling tells me I'm wrong or right.
> 
> Video encoding using 'slices' is what is called 'ridiculously 
> parallel'.  Aside from some preprocessing of the raw video, the encoding 
> process is batch.  The encoding threads are completely independent, 
> working on their own data set, and there is no need for communication 
> between them.  Ignoring bottlenecks on shared resources like memory or 
> communication buses, the encoder should scale linearly with the number 
> of cores.
Have a look at 
http://alvarez.site.ac.upc.edu/research.html#Technical_Publications
from the University of Barcelona/Spain.
The parallelization 'problems' are more or less still there.
> 
>>> You would want the video cuts to be tens of seconds long to reduce 
>>> data and startup overhead.
>> Could you be more specific on this ?
> 
> Starting up processes is expensive.  The startup and teardown of 
> something complex like x264 is probably going to be a couple seconds.  
> If you're only going to be encoding chunks of a couple seconds long, 
> that is a huge amount of overhead.
I'll investigate to see if that's a 'real' problem.
Thank you for your input so far, really appreciate it.

Henk
> _______________________________________________
> mythtv-users mailing list
> mythtv-users at mythtv.org
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users