[mythtv-users] nuvexport command line interface (CLI)

[Chris Pinkham]

> > Once the job queue stuff I'm working on is in place, queueing up a nuvexport job
> > from mythweb would be as easy as inserting a record into the jobqueue table if
> > you had setup a custom nuvexport job previously.  The JobQueue would take care
> > of running the job when its turn came up in the queue.  You'll also be able to
> > fire off commercial flagging from MythWeb by inserting a record into this
> > table.  I want to convert the current transcoding queue over to using this
> > as well.
> 
> Could you mail me the spec from this table?   I have my own nuvexport 
> table ready to be implemented, and it contains things like a progress 
> indicator so the status can be watched from mythweb (or another source). 
>   If you want me to use this table, we'll need to colaborate so that it 
> supports the features I want for nuvexport.

Here's the current info from dbcheck.cpp:

CREATE TABLE jobqueue (
    id INTEGER NOT NULL AUTO_INCREMENT,
    chanid INTEGER(10) NOT NULL,
    starttime DATETIME NOT NULL,
    inserttime DATETIME NOT NULL,
    type INTEGER NOT NULL,
    status INTEGER NOT NULL default 0,
    statustime TIMESTAMP NOT NULL,
    hostname VARCHAR(255) NOT NULL default '',
    args VARCHAR(255) NOT NULL default '',
    comment VARCHAR(128) NOT NULL default '',
    PRIMARY KEY(id),
    UNIQUE(chanid, starttime, type, inserttime)
);

CREATE TABLE jobtype (
    type INTEGER NOT NULL,
    description VARCHAR(255) NOT NULL default '',
    command VARCHAR(255) NOT NULL default '',
    PRIMARY KEY(type)
);

INSERT jobtype (type, description) VALUES (1, 'Transcode');
INSERT jobtype (type, description) VALUES (2, 'Flag Commercials');
INSERT jobtype (type, description) VALUES (3, 'Archive');

- chanid/starttime are the normal chanid/starttime as in the recorded
	table.
- inserttime is the time the job was scheduled
- type is the Job Type (enums below for standard jobs)
- status is the current job status)
- statustime is the last time this entry was updated
- hostname is the hostname where this job should run or "master" for
	the master server.  I think also that an empty hostname or some
	other keyword could indicate to just run this job on the first
	available server.
- args could be anything, not using it now, may delete this field.
- comment is currently set by the job as it runs, for instance I have
	the commercial flagger putting a percentage in here as it runs
	and error messages if there are any errors.

Some enums from jobqueue.h:

enum JobStatus {
    JOB_STATUS    = 0x00ff,
    JOB_QUEUED    = 0x0001,
    JOB_PENDING   = 0x0002,
    JOB_STARTING  = 0x0003,
    JOB_RUNNING   = 0x0004,
    JOB_STOPPING  = 0x0005,

    JOB_DONE      = 0x0010,
    JOB_FINISHED  = 0x0011,
    JOB_ABORTED   = 0x0012,
    JOB_ERRORED   = 0x0013,

    JOB_FLAG      = 0xff00,
    JOB_STOP      = 0x0100,
    JOB_RESTART   = 0x0200,

    JOB_UNKNOWN   = 0x0000
};

enum JobTypes {
    JOB_TRANSCODE = 0x001,
    JOB_COMMFLAG  = 0x002,
    JOB_ARCHIVE   = 0x004,
    JOB_USERJOB   = 0x100
};

The JobQueue is meant to be a generic queue that can call other jobs.  I
think that if the job being run needs it's own info, then it might be good
to keep that separate, otherwise we could end up with a "generic" table that
is 50 columns wide.

A job is inserted with a JOB_QUEUED status.  Jobs are processed in ID order.
Only one job for a given chanid/starttime can be running at a time, but
multiple jobs may be running for different chanid/starttime combinations.
There is a "MaxJobs" that specifies how many jobs to run concurrently on a
server.

Each server looks at the queue table and checks to see if there are any
jobs that are supposed to run on that server and that can be started.  If
another job is running for that chanid/starttime, then the server will not
startup a new job for the same chanid/starttime even if the current job is
running on a different server.  This insures that the for instance the
commercial flagging job will not be run until after the transoding job
finishes even if commercial flagging always runs on a different machine
than the commercial flagging does.  I think eventually there could be a
way to specify what jobs a host can run via a mask of the job types.

When the job queue thread determines that a job can be run, it changes the
job status to JOB_PENDING and fires off a thread to handle the job.
The job itself is responsible for setting the status to (optionally)
JOB_STARTING and JOB_RUNNING (required).  If the job finishes, aborts, or
errors, it can set one of the appropriate statuses.  A job can also be
aborted by setting the status to JOB_STOP or by sending a STOP event to
the JobQueue.  The Job Queue picks up jobs with a status of JOB_STOP and
sets a flag to tell the job to abort if it can.  Currently this is
just a pointer to a boolean that is passed in to the job.  The commercial
flagger uses this currently.  It checks the boolean while it is flagging
and as soon as the Job Queue sets the boolean to false, the flagger
sees that and aborts flagging and sets the status to JOB_ABORTED.

> > Once you have MythWeb capable of queueing jobs, then batching them is done
> > as well because you just queue up multiple jobs and each one is handled
> > in-turn.
> 
> That's the idea.  The problem is that nuvexport still needs to be ABLE 
> to be run in standalone mode, but data still needs to go into the table 
> so job progress can be tracked..   Happy to work with you on this, 
> otherwise, I'll stick to what I have on my own.

As long as you have a command-line mode that takes a chanid and startime
then it could be fired off from the JobQueue.

The jobtype table has 3 columns:

type - the job type used in the jobqueue table
description - a human-readable description of the job
command - the command to run to perform the job.  For transcoding, archive,
	and commercial flagging, this would be empty but potentially could be
	filled in if the user wanted to run some external program to do
	one of those two actions.  For user-jobs, this would be the command
	to run something like this:

	"nuvexport --mode svcd --chanid %C --starttime %S --outfile
		/path/to/some/dir/%C_%S.mpg --use_cutlist 1 --noise_reduction 0
		--quantisation 4 --v_bitrate 2400"

	In this case, %C would get substituted with the chanid, and %S with
	the starttime in YYYYMMDDHHMMSS format.


Currently I only have the commercial flagging using the JobQueue.  I
haven't delved into converting over the transcoding stuff yet because I
only started working on this about a little over a week ago and haven't
conversed with Geoffrey yet about hooking the transcoder in.  I plan
on adding a simple "Archive" job feature as well but am still tossing
around ideas in my head about that.  Archive in this sense isn't
doing something like transcoding to avi or something, it's more like
"move this file to this directory which is off on my bigger slower
storage rather than keeping it on fast local storage.

Let me know if you have any ideas, either here or if you see me on irc,
and Geoffrey, if you see this, please give your comments as well.
-- 

Chris