Difference between revisions of "Job queue idle.py"
(Actually add the exit status (equal to job count). Update to use Job.getAllEntries() instead of an "empty" search.) |
|||
| Line 89: | Line 89: | ||
</pre>}} | </pre>}} | ||
| + | Since the above is enough to query the job queue, but not actually do anything about it, I wrote the following wrapper which uses the above and then starts and stops mythjobqueue (mythjobqueue prevents the backend from going idle, so it has to actually stopped when the queue is empty if you want your backend to be able to idle and sleep) when there is something to be processed: | ||
| + | |||
| + | <pre> | ||
| + | #!/bin/bash -e | ||
| + | |||
| + | export PATH=$PATH:~/bin | ||
| + | |||
| + | jobqueue_pid="" | ||
| + | |||
| + | while true; do | ||
| + | num_jobs=0 | ||
| + | if ! job_queue_idle.py; then | ||
| + | num_jobs=${PIPESTATUS[0]} | ||
| + | fi | ||
| + | |||
| + | if [ $num_jobs -gt 0 ]; then | ||
| + | if [ -z "$jobqueue_pid" ]; then | ||
| + | mythjobqueue & | ||
| + | jobqueue_pid=$! | ||
| + | fi | ||
| + | else | ||
| + | if [ -n "$jobqueue_pid" ]; then | ||
| + | kill $jobqueue_pid | ||
| + | jobqueue_pid="" | ||
| + | fi | ||
| + | fi | ||
| + | sleep 60 | ||
| + | done | ||
| + | </pre> | ||
| + | |||
| + | I also patched the above job_queue_idle.py with the following to deal with the backend sleeping and the database not being available. Basically, when that happens job_queue_idle.py returns 0 (queue is empty) which seems appropriate since the backend is not running to have the queue processed anyway. I think I recall that the backend won't even idle if the queue is not empty anyway, so if it is sleeping, that implies the queue is empty and therefore returning 0 if the backend is sleeping is appropriate. | ||
| + | |||
| + | <pre> | ||
| + | --- a/job_queue_idle.py 2011-05-29 13:52:40.000000000 -0400 | ||
| + | +++ b/job_queue_idle.py 2011-05-26 12:33:08.209777516 -0400 | ||
| + | @@ -36,7 +36,12 @@ | ||
| + | global verbose_level; | ||
| + | verbose_level = opts.verbose_level | ||
| + | |||
| + | - DB = MythDB() | ||
| + | + try: | ||
| + | + DB = MythDB() | ||
| + | + except: | ||
| + | + verbose(VERBOSE_LEVEL.DEBUG, 'Failed to connect to the database server') | ||
| + | + sys.exit(0) | ||
| + | + | ||
| + | currenthost = DB.gethostname() | ||
| + | verbose(VERBOSE_LEVEL.DEBUG, 'Checking job queue status for host ', | ||
| + | currenthost) | ||
| + | </pre> | ||
[[Category:Python Scripts]] | [[Category:Python Scripts]] | ||
Revision as of 17:57, 29 May 2011
Note: The correct title of this article is job_queue_idle.py. It appears incorrectly here due to technical restrictions.
| Author | Mike Dean |
| Description | A python script that checks the job queue status. Its exit status indicates the number of jobs available for processing on this host. A zero exit status (success) means the job queue is idle and may be terminated. |
| Supports | 
|
This python script checks to see if a job queue server is idle. Its exit status indicates the number of jobs available for processing on this host. A zero exit status (success) means the job queue is idle and may be terminated.
It can be used in a cron job or in a loop in a script that starts and backgrounds mythjobqueue to determine if there are any jobs currently being processed by this host's mythjobqueue or available for this host's mythjobqueue to process. If no jobs are in progress or available for processing, mythjobqueue may be shut down so that the master backend may automatically shut itself down when idle.
job_queue_idle.py
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
#---------------------------
# Name: job_queue_idle.py
# Purpose
# This python script checks to see if a job queue server is idle.
# It can be used in a cron job or in a loop in a script that starts
# and backgrounds mythjobqueue to determine if there are any jobs
# currently being processed by this host's mythjobqueue or available
# for this host's mythjobqueue to process. If no jobs are in
# progress or available for processing, mythjobqueue may be shut down
# so that the master backend may automatically shut itself down when
# idle.
#---------------------------
from MythTV import MythDB, Job
from optparse import OptionParser
import sys
class VERBOSE_LEVEL:
DEFAULT = 0
VERBOSE = 1
DEBUG = 2
def verbose(level, *messages):
if (verbose_level >= level):
print ''.join(map(str,messages))
def main():
parser = OptionParser(usage="usage: %prog [options]")
parser.set_defaults(verbose_level=VERBOSE_LEVEL.DEFAULT)
parser.add_option('-v', '--verbose', action='store', type='int',
dest='verbose_level', help='Verbosity level')
opts, args = parser.parse_args();
global verbose_level;
verbose_level = opts.verbose_level
DB = MythDB()
currenthost = DB.gethostname()
verbose(VERBOSE_LEVEL.DEBUG, 'Checking job queue status for host ',
currenthost)
jobs = list(Job.getAllEntries(db=DB))
jobCount=0
if len(jobs) == 0:
verbose(VERBOSE_LEVEL.VERBOSE, 'No jobs found')
else:
for job in jobs:
if (len(job.hostname) == 0):
verbose(VERBOSE_LEVEL.VERBOSE, 'Job ',job.id,
'available for processing on any host (',
job.comment,')')
jobCount+=1
elif ((job.hostname != currenthost)):
verbose(VERBOSE_LEVEL.DEBUG, 'Job ',job.id,' claimed by ',
job.hostname,' (',job.comment,')')
elif (not(job.status & job.DONE)):
verbose(VERBOSE_LEVEL.VERBOSE, 'Currently processing job ',
job.id,' (',job.comment,')')
jobCount+=1
else:
verbose(VERBOSE_LEVEL.DEBUG, 'Finished processing job ',
job.id,' (',job.comment,')')
verbose(VERBOSE_LEVEL.VERBOSE, 'Found ',jobCount,' jobs.')
if (jobCount > 0):
verbose(VERBOSE_LEVEL.DEFAULT, "mythjobqueue is not idle.")
else:
verbose(VERBOSE_LEVEL.DEFAULT, "mythjobqueue is idle.")
sys.exit(jobCount)
if __name__ == '__main__':
main()
Since the above is enough to query the job queue, but not actually do anything about it, I wrote the following wrapper which uses the above and then starts and stops mythjobqueue (mythjobqueue prevents the backend from going idle, so it has to actually stopped when the queue is empty if you want your backend to be able to idle and sleep) when there is something to be processed:
#!/bin/bash -e
export PATH=$PATH:~/bin
jobqueue_pid=""
while true; do
num_jobs=0
if ! job_queue_idle.py; then
num_jobs=${PIPESTATUS[0]}
fi
if [ $num_jobs -gt 0 ]; then
if [ -z "$jobqueue_pid" ]; then
mythjobqueue &
jobqueue_pid=$!
fi
else
if [ -n "$jobqueue_pid" ]; then
kill $jobqueue_pid
jobqueue_pid=""
fi
fi
sleep 60
done
I also patched the above job_queue_idle.py with the following to deal with the backend sleeping and the database not being available. Basically, when that happens job_queue_idle.py returns 0 (queue is empty) which seems appropriate since the backend is not running to have the queue processed anyway. I think I recall that the backend won't even idle if the queue is not empty anyway, so if it is sleeping, that implies the queue is empty and therefore returning 0 if the backend is sleeping is appropriate.
--- a/job_queue_idle.py 2011-05-29 13:52:40.000000000 -0400
+++ b/job_queue_idle.py 2011-05-26 12:33:08.209777516 -0400
@@ -36,7 +36,12 @@
global verbose_level;
verbose_level = opts.verbose_level
- DB = MythDB()
+ try:
+ DB = MythDB()
+ except:
+ verbose(VERBOSE_LEVEL.DEBUG, 'Failed to connect to the database server')
+ sys.exit(0)
+
currenthost = DB.gethostname()
verbose(VERBOSE_LEVEL.DEBUG, 'Checking job queue status for host ',
currenthost)
