Difference between revisions of "Status Monitoring How To"

From MythTV Official Wiki
Jump to: navigation, search
(Cleaned up headers based on MythTV:Manual of Style)
Line 3: Line 3:
 
This page has a few approaches to monitoring Myth's processes
 
This page has a few approaches to monitoring Myth's processes
  
= Monit =
+
== Monit ==
  
 
Monit (http://www.tildeslash.com/monit/)
 
Monit (http://www.tildeslash.com/monit/)
Line 13: Line 13:
 
* Can also monitor anything else (eg listings downloads, disk space etc)
 
* Can also monitor anything else (eg listings downloads, disk space etc)
  
== Installation ==
+
=== Installation ===
  
 
Debian/knoppmyth install : <code><nowiki>apt-get install monit</nowiki></code>
 
Debian/knoppmyth install : <code><nowiki>apt-get install monit</nowiki></code>
  
== Configuration ==
+
=== Configuration ===
  
 
On Debian create <code><nowiki>/etc/monit/monitrc</nowiki></code>
 
On Debian create <code><nowiki>/etc/monit/monitrc</nowiki></code>
Line 78: Line 78:
 
Once all this works, you can consider moving on to the next bit...
 
Once all this works, you can consider moving on to the next bit...
  
== Paranoia ==
+
=== Paranoia ===
 
So what if monit crashes??? asks Craig Partin...
 
So what if monit crashes??? asks Craig Partin...
  
Line 94: Line 94:
 
You could - however, it's not the best way to do it - the backend may start before mysql - or your frontend may start before the backend does. For this reason the rc3.d and rc5.d scripts are numbered to start in a certain order to ensure that mysql starts before the backend which starts before the frontend etc.
 
You could - however, it's not the best way to do it - the backend may start before mysql - or your frontend may start before the backend does. For this reason the rc3.d and rc5.d scripts are numbered to start in a certain order to ensure that mysql starts before the backend which starts before the frontend etc.
  
= Daemon Tools  =
+
== Daemon Tools  ==
  
 
Daemontools (http://cr.yp.to/daemontools.html)
 
Daemontools (http://cr.yp.to/daemontools.html)
Line 136: Line 136:
 
As soon as you make this file executable supervise will try and start up mythbackend, so make sure you've stopped the existing process (eg: <code><nowiki>/etc/init.d/mythbackend stop</nowiki></code>) and removed the existing startup script (I moved mythbackend from <code><nowiki>/etc/init.d/</nowiki></code> to <code><nowiki>/root</nowiki></code>).
 
As soon as you make this file executable supervise will try and start up mythbackend, so make sure you've stopped the existing process (eg: <code><nowiki>/etc/init.d/mythbackend stop</nowiki></code>) and removed the existing startup script (I moved mythbackend from <code><nowiki>/etc/init.d/</nowiki></code> to <code><nowiki>/root</nowiki></code>).
  
= Quick and Dirty =
+
== Quick and Dirty ==
 
So if all that gets to complicated then try running a script like this (from Will Dorman) periodically from cron:
 
So if all that gets to complicated then try running a script like this (from Will Dorman) periodically from cron:
 
<pre><nowiki>
 
<pre><nowiki>
Line 154: Line 154:
 
</nowiki></pre>
 
</nowiki></pre>
  
= Even Dirtier =
+
== Even Dirtier ==
 
A simpler way to see if a process is running is using ps -C <command> and look at the exit code of ps; ps will error if it can't find the process.
 
A simpler way to see if a process is running is using ps -C <command> and look at the exit code of ps; ps will error if it can't find the process.
 
Much like the Quick and Dirty, this can be run from root's crontab:
 
Much like the Quick and Dirty, this can be run from root's crontab:

Revision as of 18:54, 14 May 2006

So, for whatever reason, your Myth processes have occasional crashes that you haven't diagnosed yet (you are trying to diagnose them aren't you?) and you'd like it to restart when it does...

This page has a few approaches to monitoring Myth's processes

Monit

Monit (http://www.tildeslash.com/monit/)

  • It monitors processes and restarts them if they fail.
  • Can be configured to alert you (I like to know if it crashes in case it gets serious!)
  • It is 'secure'able
  • Has a web interface to check status and log
  • Can also monitor anything else (eg listings downloads, disk space etc)

Installation

Debian/knoppmyth install : apt-get install monit

Configuration

On Debian create /etc/monit/monitrc

# Monit control file
#
# Comments begin with a '#' and extend through the end of the line.
# Blank lines between program entries are ignored. Keywords are case
# insensitive. All path's MUST BE FULLY QUALIFIED, starting with '/'
#
set daemon  30
set logfile /var/log/monit.log
set mailserver smtp.dgreaves.com
set mail-format
 { from: monit@bao.dgreaves.com }
set alert root@dgreaves.com  # Send alert to system admin on any event
set httpd port 2812 and
    allow ash.dgreaves.com
    allow 10.0.0.90
    allow 127.0.0.1
    allow haze.dgreaves.com
    allow admin:monit     # user 'admin' with password 'monit'


check process mythbackend with pidfile /var/run/mythtv/mythbackend.pid
 group mythtv
 start program = "/etc/init.d/mythtv-backend start"
 stop program  = "/etc/init.d/mythtv-backend stop"
 if failed port 6544 then restart
 mode manual
 depends on mysql

check process mysql with pidfile /var/run/mysqld/mysqld.pid
 group mythtv
 start program = "/etc/init.d/mysql start"
 stop program = "/etc/init.d/mysql stop"
 if failed port 3306 then restart
 mode manual

Obivously change the names/ip addresses as needed!

Note that this tells monit to start in 'manual' mode. To actually start monitoring I have this script in /etc/monit/monit_delay.

sleep 60
monit monitor mythbackend

and call it from the monit init.d script after monit is started. This ensures that when monit starts up it allows enough time for all the other services to get going before it kills them or anything...

Now, check it's all working by connecting to http://yourbox:2812/

You should see a web page.

Now try doing an /etc/init.d/mythtv-backend stop. You should get a mail and the backend should restart.

Now try doing an /etc/init.d/mysql stop. You should get a mail and the backend should stop, mysql should start and then the backend should restart.

Once all this works, you can consider moving on to the next bit...

Paranoia

So what if monit crashes??? asks Craig Partin...

Well, if you like you can run monit from init.

In this case, just put the following line in your /etc/inittab

moni:235:respawn:/usr/sbin/monit

Then init watches and makes sure that monit is running (and if init dies you've got a whole heapload of problems!!) But make sure you don't start monit from your init.d script (but don't forget the monit_delay...)

So, why not run mythbackend like this?
You could - however, it's not the best way to do it - the backend may start before mysql - or your frontend may start before the backend does. For this reason the rc3.d and rc5.d scripts are numbered to start in a certain order to ensure that mysql starts before the backend which starts before the frontend etc.

Daemon Tools

Daemontools (http://cr.yp.to/daemontools.html)

Being OSS, Monit isn't the only game in town - daemontools is another very similar mechanism that you may prefer. Daemontools has the advantage of starting and monitoring the process, so there's no danger of mythbackend starting up before daemontools.


Install

apt-get install daemontools-installer

This will set up your inittab, start the supervise process, and create the appropriate directories (assuming you accepted the defaults during the install). Nothing will start, however, until you create the appropriate files under the /service directory. There's no support in daemontools for service startup order (ie. dependencies), so you have to check in the run script.

Configure

Daemontools looks for a file called /service/servicename/run to start the program. So we'll call our service mythbackend, hence

/service/mythbackend/run

#!/bin/sh

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/bin/mythbackend

MYSQLHOST=localhost
MYSQLADMIN=/usr/bin/mysqladmin

if [ -x $MYSQLADMIN ]
then    
        $MYSQLADMIN -h$MYSQLHOST ping &> /dev/null
fi

if [ $? -eq 0 ] # ping worked
then
        exec $DAEMON --logfile /var/log/mythtv/mythbackend.log
else    
        logger "Mysqld not running yet. Waiting..."
fi

As soon as you make this file executable supervise will try and start up mythbackend, so make sure you've stopped the existing process (eg: /etc/init.d/mythbackend stop) and removed the existing startup script (I moved mythbackend from /etc/init.d/ to /root).

Quick and Dirty

So if all that gets to complicated then try running a script like this (from Will Dorman) periodically from cron:

#!/bin/bash

pidno=$( ps ax | grep mythbackend | grep -v grep | grep -v mythmon)
# echo $pidno
# Checks for pid in "ps" listing, field #1.
# Then makes sure it is the actual process, not the process invoked by this script.
# The last "grep $1" filters out this possibility.
if [ -z "$pidno" ]  # If, after all the filtering, the result is a zero-length string,
then                # no running process corresponds to the pid given.
  echo "No such process running."
  mail <mynumber>@mobile.att.net -s Uh-oh < /tmp/crap
  exit $E_NOSUCHPROCESS
fi

Even Dirtier

A simpler way to see if a process is running is using ps -C <command> and look at the exit code of ps; ps will error if it can't find the process. Much like the Quick and Dirty, this can be run from root's crontab:

0-59/5 * * * * /root/bin/mythmon || /sbin/service mythbackend restart > /dev/null;

A nice feature of bash is that you can OR two commands together. The second command is tried iff (if and only if) the first command fails. Similarly, ANDing two commands, the second is executed iff the first is successful. /sbin/service is a feature of Fedora/RedHat.

#!/bin/bash
#
# /root/bin/mythmon
# This is run from cron every 5 minutes.
#

ps -C mythbackend > /dev/null && exit 0;

# append message to log file...
# send email notification...

echo "`date +%Y-%m-%d\ %T.%-3N` /root/bin/mythmon: mythbackend not running!!!" \
         >> /var/log/mythtv/mythbackend.log;
exit 1;