Using pcsk to Supervise mythbackend

From MythTV Official Wiki
Jump to: navigation, search

If mythbackend has ever crashed on you, odds are that you didn't know and you missed out on recording a favourite programme. I find this can happen if there are too many errors in the TV signal, or signal strength is too low, or maybe even if the phase of the moon is wrong... :)

Anyway, here is a nifty rc script I wrote using the default /etc/init.d/mythtv-backend script in Ubuntu 7.04 (Feisty) as a base, but modified to have the following improvements:

1) The Process Creation's Swiss-Army Knife (PCSK) is used to supervise the mythbackend process and respawn it if it dies. Alternatives to pcsk include daemontools and psmon, but I like pcsk as it is small and simple and does exactly what I need.
2) The script is more clever about whether a process is actually running, rather than just checking for the existence of a .pid file (which will never work if the process crashed instead of being deliberately stopped).
3) You can simply run /etc/init.d/mythtv-backend start and know that no matter what the status was previously, mythbackend will be running afterwards (useful if another process or script is used to start mythbackend).
4) Non-root users stand a good chance of being able to determine the status of mythbackend without requiring sudo.
5) Additional status information is provided using Andrew Ruthven's mythtv-status perl script. This isn't essential, but is used if you have it installed.
6) Behaviour of force-reload is clearer.

A disadvantage for you may be that this method loses the ability to 'nice' the mythbackend process (eg using /etc/default/mythtv-backend). I wasn't using nice, so it was no problem for me.

pcsk

First, download, compile and install pcsk from http://www.nix.hu/projects/pcsk/. Something along the lines of:

cd ~
mkdir pcsk
cd pcsk
wget http://downloads.nix.hu/downloads/pcsk/pcsk-0.0.5.tar.bz2
tar xjf pcsk-0.0.5.tar.bz2
cd pcsk-0.0.5
make install


mythtv-status

If you want to get more useful info about the recording status of mythbackend, I highly recommend mythtv-status. You can follow the instructions to install the complete package, but I just needed the script, which I placed in /usr/local/bin:

cd ~
mkdir mythtv-status
cd mythtv-status
wget http://www.etc.gen.nz/projects/mythtv/tarballs/mythtv-status-latest.tar.gz
tar xzf mythtv-status-latest.tar.gz
cd mythtv-status-0.5.1    # or whatever the directory containing the latest version is called
sudo cp bin/mythtv-status /usr/local/bin


mythtv-backend.pcsk

Then, using your favourite text editor, create the following script in the appropriate place for your distro. For my Ubuntu installation the default directory is /etc/init.d/ and I called the script mythtv-backend.pcsk.

Update: This script has been updated to force pcsk to respawn mythbackend even if it exits with return code 0, which it seems to do even when catching a signal and core dumping.

Script.png /etc/init.d/mythtv-backend.pcsk

#! /bin/sh
### BEGIN INIT INFO
# Provides:          mythtv-backend
# Required-Start:    $local_fs
# Required-Stop:     $local_fs
# Default-Start:     24
# Default-Stop:      S
# Short-Description: Start/Stop the MythTV server.
### END INIT INFO
#
# modified mythtv-backend rc script
# v0.21 by DJK 3rd Dec 2007:
#       Removed -0 option for pcsk
#               mythbackend seems to catch all signals (even when core dumping) and still exits 0!
#               so we actually need to restart it no matter what its exit code was...
# v0.2  by DJK 28th Nov 2007:
#       Added pcsk supervision to restart if it dies
#       Had to lose NICE for now, sorry.  Might be able to use renice sometime in the future?
#       Call mythtv-status script for extra details
#

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/bin/mythbackend
NAME="mythbackend"
DESC="MythTV server"

# Check that the mythbackend binary exists
if ! [ -x "$DAEMON" ] ; then
        echo "Could not find $DAEMON binary"
        exit 1
fi

# Check that the pcsk binary exists somewhere in the path (I choose /usr/local/bin)
if ! PCSKBINARY="`which pcsk`" > /dev/null 2>&1; then
        echo "Could not find pcsk binary to supervise $NAME.  Try www.nix.hu/projects/pcsk..."
        exit 1
fi

# Arguments to give to the mythtv-status script
MYTHTVSTATUSARGS="-c"

# Check that the  mythtv-status perl script exists somewhere in the path (I choose /usr/local/bin)
if ! MYTHTVSTATUSBINARY="`which mythtv-status`" > /dev/null 2>&1; then
        # It's not disasterous if it's not there - just let the user know where to find it if they try status
        MYTHTVSTATUSBINARY="echo mythtv-status perl script not found.  Try www.etc.gen.nz/projects/mythtv/mythtv-status.html"
        MYTHTVSTATUSARGS=
fi

set -e

. /lib/lsb/init-functions

USER=mythtv
RUNDIR=/var/run/mythtv
# mythbackend arguments - I've removed --daemon since pcsk will daemonise it for me
ARGS=" --logfile /var/log/mythtv/mythbackend.log --pidfile $RUNDIR/$NAME.pid"
EXTRA_ARGS=""
NICE=0

# The pid and log files for pcsk to use
PCSKPIDFILE="$RUNDIR/$NAME-pcsk.pid"
PCSKLOGFILE="/var/log/mythtv/mythbackend-pcsk.log"

# Command line options for pcsk:
# -r: restart the program if it dies
# -e: log stderr, if there is any
# -o: log stdout, if there is any
## -0: don't restart if the exit code was 0 - not using this at the moment
# -w3: wait 3 sec before restart (default in original script)
# -i10: increment each wait between restarts by 10s
# -c10: restart up to 10 times if the restart is unsuccessful, before giving up
# -m60: restart wasn't successful if the program exits within 60 sec
# -p $PIDFILE: use that file for the pid
# -l $PCSKLOGFILE: log file to keep track of things - good to set up logrotate
PCSKOPTS="-reo -w3 -i10 -c10 -m60 -p $PCSKPIDFILE -l $PCSKLOGFILE"
# Check for alternate options for mythtv-backend
if [ -f /etc/default/mythtv-backend ]; then
  . /etc/default/mythtv-backend
fi

ARGS="$ARGS $EXTRA_ARGS"

PCSKARGS="$PCSKOPTS $DAEMON -- $ARGS"

# If the run directory doesn't exist, then create it if we can and chown it to $USER
# I'm assuming if it's already there its permissions and ownership are OK (cheat...)
# This makes it easier for non-root users to still run this script to get status info
if ! [ -d "$RUNDIR" ] ; then
        if [ "`id -un`" = "root" ] ; then
                mkdir -p "$RUNDIR"
                chown -R "$USER" "$RUNDIR"
        else
                echo "Could not create $RUNDIR.  You need to be root to do that.  Try sudo."
                exit 1
        fi
fi

unset DISPLAY
unset SESSION_MANAGER

case "$1" in
  start)
        [ "`id -un`" = "root" ] || {
                echo "Failed.  You need to be root to $1 $DESC.  Try sudo."
                exit 1
        }
        # Check if pcsk is already supervising this process
        # by checking for the existance of a pidfile and that it's a currently running process
        PCSKPID=
        DAEMONPID=
        if [ -e "$PCSKPIDFILE" ] ; then
                read PCSKPID < "$PCSKPIDFILE"
                if [ -n "$PCSKPID" ] && [ -d "/proc/$PCSKPID" ] ; then
                        # pcsk is running, what about our daemon?
                        if [ -e "$RUNDIR/$NAME.pid" ] ; then
                                read DAEMONPID < "$RUNDIR/$NAME.pid"
                                if [ -n "$DAEMONPID" ] && [ -d "/proc/$DAEMONPID" ] ; then
                                        # daemon is running too
                                        echo "pcsk($NAME) (pid $PCSKPID) is already running, supervising $NAME (pid $DAEMONPID) "
                                        echo "Use restart if you need."
                                        exit 1
                                fi
                                # daemon isn't running at the moment (pcsk might be waiting to respawn it)
                                # anyway, take the opportunity to restart everything now
                                echo "pcsk($NAME) (pid $PCSKPID) is running but $NAME is currently stopped.  Restarting now..."
                                $0 restart
                                exit $?
                        fi
                fi
        fi
        echo -n "Starting $DESC: pcsk($NAME) "
        # NICE gets ignored because pcsk doesn't support it and start-stop-daemon would just nice pcsk, not the daemon.
        # Maybe we could somehow use renice?  Later...
        if ! start-stop-daemon --start --pidfile "$PCSKPIDFILE" --chuid $USER --exec "$PCSKBINARY" -- $PCSKARGS ; then
                echo "failed (start-stop-daemon exited with code $?)"
                exit 1
        fi
        echo "."
        ;;
  stop)        [ "`id -un`" = "root" ] || {
                echo "Failed.  You need to be root to $1 $DESC.  Try sudo."
                exit 1
        }
        echo -n "Stopping $DESC: pcsk($NAME) "
        start-stop-daemon --stop --oknodo --pidfile "$PCSKPIDFILE" --chuid $USER --retry 5 --exec "$PCSKBINARY" -- $PCSKARGS
        [ -e "$PCSKPIDFILE" ] && rm -f "$PCSKPIDFILE"
        [ -e "$RUNDIR/$NAME.pid" ] && rm -f "$RUNDIR/$NAME.pid"
        echo "."
        ;;
  restart)
        $0 stop
        sleep 3
        $0 start
        ;;
  force-reload)
        echo "force-reload not supported, restarting instead..."
        $0 restart
        ;;
  status)
        # We have an advantage in this mess that we know what the pidfiles (and therefore pids) are
        # Poor-man's status can be had by checking for the existance of a pidfile
        # and then that it's a currently running process by looking for that directory in /proc
        PCSKPID=
        DAEMONPID=
        EXITBAD=FALSE
        if [ -e "$PCSKPIDFILE" ] ; then
                read PCSKPID < "$PCSKPIDFILE"
                if [ -n "$PCSKPID" ] && [ -d "/proc/$PCSKPID" ] ; then
                        echo "pcsk($NAME) (pid $PCSKPID) is running..."
                else
                        echo "pcsk($NAME) is dead but pid file exists"
                        EXITBAD=TRUE
                fi
        else
                echo "pcsk($NAME) is stopped"
                EXITBAD=TRUE
        fi
        if [ -e "$RUNDIR/$NAME.pid" ] ; then
                read DAEMONPID < "$RUNDIR/$NAME.pid"
                if [ -n "$DAEMONPID" ] && [ -d "/proc/$DAEMONPID" ] ; then
                        echo "$NAME (pid $DAEMONPID) is running..."
                else
                        echo "$NAME is dead but pid file exists"
                        EXITBAD=TRUE
                fi
        else
                echo "$NAME is stopped"
                EXITBAD=TRUE
        fi
        [ "$EXITBAD" = "TRUE" ] && exit 1
        $MYTHTVSTATUSBINARY $MYTHTVSTATUSARGS
        ;;
  *)
        echo "Usage: $0 {start|stop|restart|status|force-reload(unsupported)}" >&2
        exit 1
        ;;
esac

exit 0


Stop mythbackend before making any changes:

sudo /etc/init.d/mythtv-backend stop



Keep a copy of the original rc script in case you need it, but make the new one the default:

cd /etc/init.d
sudo mv mythtv-backend mythtv-backend.original
sudo ln -s mythtv-backend.pcsk mythtv-backend


Do NOT forget to make executable the script, otherwise your backend will not start!

sudo chmod +x mythtv-backend.pcsk


It's probably best to set up logrotate on the log file that pcsk creates to prevent it getting too big. Assuming the default log location of /var/log/mythtv/mythtv-backend-pcsk.log, modify your /etc/logrotate.d/mythtv-backend file by appending the following lines to the end of it:

/var/log/mythtv/mythbackend-pcsk.log {
        daily
        rotate 7
        notifempty
        copytruncate
}


Start mythbackend again.

sudo /etc/init.d/mythtv-backend start


Explore the options available using:

sudo /etc/init.d/mythtv-backend restart
sudo /etc/init.d/mythtv-backend stop
sudo /etc/init.d/mythtv-backend force-reload  # not supported by mythbackend, so it just calls restart
/etc/init.d/mythtv-backend status             # sudo not required assuming the process is actually running

Note the pretty colours and useful info about current and future mythtv recordings in the status output - useful if you were planning on updating the bios and just want to check what you'd miss out on...