Talk:MythGrid

From MythTV Official Wiki
Revision as of 04:22, 17 May 2012 by Wagnerrp (Talk | contribs)

Jump to: navigation, search

Can you even run your own BOINC server? wagnerrp 15:46, 14 May 2012 (UTC)

Yes although I haven't finished the configuration yet, see http://boinc.berkeley.edu/trac/wiki/ServerIntro
Link is bad.
It looks like BOINC's site is down, so try it later. It was working when I posted it.
Why not using one of the existing batch schedulers designed for local use, like OpenPBS, or its offspring TORQUE, or even just Myth's internal Job Queue? wagnerrp 03:29, 16 May 2012 (UTC)
I don't know anything about OpenPBS or TORQUE, I did consider openstack, but part of this is about promoting the use of BOINC, to claim some unused cycles from all of those unused frontends for other projects. You also get the use of machines that are not mythTV boxes you have when you need the extra processing. Can mythTVs job queue facilitate doing distributed processing?
OpenPBS and TORQUE are common job schedulers in the high performance computing arena. They're designed to be used on private networks, although they're generally used for clustered jobs, stuff communicating through MPI/LAM/PVM, not the "ridiculously parallel" task you are describing. However all of this is superfluous, as the task you are describing can be performed by the internal jobqueue. Schedule one job that performs the initial file splitting, and fires off N more jobs to individually transcode each. Add one final task to stitch them together, but place it in stopped status. Have each individual transcode job update it to queued status, conditionally on whether all the transcode tasks are marked complete. The current jobqueue is distributed, but a "free for all" rather than scheduled. Each jobqueue instance runs twice a minute, and checks the database to see if there are any jobs available it is allowed to run, at which point it grabs the first in the list and runs it. Each backend runs one jobqueue instance, and additional machines can be added in with the mythjobqueue application. wagnerrp 06:33, 16 May 2012 (UTC)
It maybe "ridiculously parallel" and superfluous, based upon an arbitrary definition of those terms, but you are missing the point. If you want to create a high performance computing platform and purchase a large amount of new hardware for processing jobs, no one is stopping you. That is not the point of this. The point of this is to use cycles from various computers that people have around which may or may not be mythtv backends. It is also about incentivizing people who don't currently have BOINC clients installed on those machines to do so.
I think you misunderstand and took offense. "Ridiculously Parallel" is a technical term for the type of task you are describing, and the type of task BOINC is suited to performing. It is also the type of task the internal jobqueue is designed to perform. It describes a computational task in which the individual compute nodes require zero communication, and complete their task independently. I was differentiating from the types of tasks PBS and TORQUE are designed for, where multiple nodes must be allocated simultaneously, and there is heavy communication between them. wagnerrp 04:22, 17 May 2012 (UTC)
What I was getting at is that for MythTV's own internal use, there is the jobqueue. It is not limited to backends, but can be run on any machine you want with the standalone `mythjobqueue` application. For some kind of incentive to install and run BOINC, you may be interested in revamping MythBOINC. It is a long since abandoned 3rd party frontend plugin, to allow one to view their current BOINC profile status. wagnerrp 04:22, 17 May 2012 (UTC)


Taking a completely different look on this, if you're batch transcoding, you run as many slices/cores as your system supports, you run multiple independent files simultaneously on multiple systems, and you just let it churn away in the background for however long it takes. You're doing this in batch for later use, so you don't care to get one done fast. On the other hand, if you do want one transcode done fast, it's because you want to use it now, and so you would want those chunks to be transcoded sequentially so you can start viewing them as they are finished, rather than waiting for the whole. Take a look into the new HLS code in 0.25, as that is exactly how the HLS protocol is designed to operate, with a playlist to stitch the segments together in the player, rather than the container. Right now, it is only designed to run on a single system, but that would be a very interesting behavior if it were modified to run multiple instances of mythtranscode across several machines, and stagger those segments between them. wagnerrp 06:54, 16 May 2012 (UTC)

The stuff I write for this project will be public domain, and of course anything else I wind up modifying will carry its licensing GPL, BSD, etc... You are more than welcome to use them however you wish.