Difference between revisions of "Status Monitoring How To"
(Cleaned up wikicode) |
|||
Line 15: | Line 15: | ||
=== Installation === | === Installation === | ||
− | Debian/knoppmyth install : <code | + | Debian/knoppmyth install : <code>apt-get install monit</code> |
=== Configuration === | === Configuration === | ||
− | * On Debian create <code | + | * On Debian create <code>/etc/monit/monitrc</code> |
− | * On openSUSE create <code | + | * On openSUSE create <code>/etc/monitrc</code> |
− | <pre | + | <pre> |
# Monit control file | # Monit control file | ||
# | # | ||
Line 55: | Line 55: | ||
if failed port 3306 then restart | if failed port 3306 then restart | ||
mode manual | mode manual | ||
− | + | </pre> | |
Obviously change the names/ip addresses as needed! | Obviously change the names/ip addresses as needed! | ||
NOTE: It has been reported that mythbackend could crash from repeated pinging of http port (I didn't see anything in my mythbackend log but my backend crashed almost daily until I made the following change). If you find that monit frequently reports failed protocol test, it may be monit causing the problem and you may want to consider commenting out the line: | NOTE: It has been reported that mythbackend could crash from repeated pinging of http port (I didn't see anything in my mythbackend log but my backend crashed almost daily until I made the following change). If you find that monit frequently reports failed protocol test, it may be monit causing the problem and you may want to consider commenting out the line: | ||
− | + | if failed port 6544 proto http then restart | |
If your backend log file starts showing error messages like this: | If your backend log file starts showing error messages like this: | ||
− | + | QSocketDevice::writeBlock: Invalid socket | |
, then removing the "proto http" in your monit configuration may help also. | , then removing the "proto http" in your monit configuration may help also. | ||
Line 70: | Line 70: | ||
Note that this tells monit to start in 'manual' mode. To actually start monitoring I have this script in <code><nowiki>/etc/monit/monit_delay</nowiki></code>. | Note that this tells monit to start in 'manual' mode. To actually start monitoring I have this script in <code><nowiki>/etc/monit/monit_delay</nowiki></code>. | ||
− | + | sleep 60 | |
− | sleep 60 | + | monit monitor mythbackend |
− | monit monitor mythbackend | + | |
− | |||
and call it from the monit init.d script after monit is started. | and call it from the monit init.d script after monit is started. | ||
This ensures that when monit starts up it allows enough time for all the other services to get going before it kills them or anything... | This ensures that when monit starts up it allows enough time for all the other services to get going before it kills them or anything... | ||
Line 81: | Line 80: | ||
You should see a web page. | You should see a web page. | ||
− | Now try doing an <code | + | Now try doing an <code>/etc/init.d/mythtv-backend stop</code>. You should get a mail and the backend should restart. |
− | Now try doing an <code | + | Now try doing an <code>/etc/init.d/mysql stop</code>. You should get a mail and the backend should stop, mysql should start and then the backend should restart. |
Once all this works, you can consider moving on to the next bit... | Once all this works, you can consider moving on to the next bit... | ||
Line 90: | Line 89: | ||
So what if monit crashes??? asks Craig Partin... | So what if monit crashes??? asks Craig Partin... | ||
− | Well, if you like you can run monit from <code | + | Well, if you like you can run monit from <code>init</code>. |
− | In this case, just put the following line in your <code | + | In this case, just put the following line in your <code>/etc/inittab</code> |
− | + | moni:235:respawn:/usr/sbin/monit | |
− | moni:235:respawn:/usr/sbin/monit | ||
− | |||
− | Then <code | + | Then <code>init</code> watches and makes sure that monit is running (and if init dies you've got a whole heapload of problems!!) |
But make sure you don't start monit from your init.d script (but don't forget the monit_delay...) | But make sure you don't start monit from your init.d script (but don't forget the monit_delay...) | ||
Line 112: | Line 109: | ||
=== Install === | === Install === | ||
− | + | apt-get install daemontools-installer | |
− | + | This will set up your inittab, start the supervise process, and create the appropriate directories (assuming you accepted the defaults during the install). Nothing will start, however, until you create the appropriate files under the <code>/service</code> directory. There's no support in daemontools for service startup order (ie. dependencies), so you have to check in the run script. | |
− | This will set up your inittab, start the supervise process, and create the appropriate directories (assuming you accepted the defaults during the install). Nothing will start, however, until you create the appropriate files under the <code | ||
=== Configure === | === Configure === | ||
− | Daemontools looks for a file called <code | + | Daemontools looks for a file called <code>/service/servicename/run</code> to start the program. So we'll call our service mythbackend, hence |
− | + | /service/mythbackend/run | |
− | <pre | + | <pre> |
#!/bin/sh | #!/bin/sh | ||
Line 141: | Line 137: | ||
logger "Mysqld not running yet. Waiting..." | logger "Mysqld not running yet. Waiting..." | ||
fi | fi | ||
− | + | </pre> | |
− | As soon as you make this file executable supervise will try and start up mythbackend, so make sure you've stopped the existing process (eg: <code | + | As soon as you make this file executable supervise will try and start up mythbackend, so make sure you've stopped the existing process (eg: <code>/etc/init.d/mythbackend stop</code>) and removed the existing startup script (I moved mythbackend from <code>/etc/init.d/</code> to <code></root</code>). |
== Quick and Dirty == | == Quick and Dirty == | ||
So if all that gets to complicated then try running a script like this (from Will Dorman) periodically from [[cron]]: | So if all that gets to complicated then try running a script like this (from Will Dorman) periodically from [[cron]]: | ||
− | <pre | + | <pre> |
#!/bin/bash | #!/bin/bash | ||
Line 161: | Line 157: | ||
exit $E_NOSUCHPROCESS | exit $E_NOSUCHPROCESS | ||
fi | fi | ||
− | + | </pre> | |
== Even Dirtier == | == Even Dirtier == | ||
A simpler way to see if a process is running is using ps -C <command> and look at the exit code of ps; ps will error if it can't find the process. | A simpler way to see if a process is running is using ps -C <command> and look at the exit code of ps; ps will error if it can't find the process. | ||
Much like the Quick and Dirty, this can be run from root's crontab: | Much like the Quick and Dirty, this can be run from root's crontab: | ||
− | + | */5 * * * * /root/bin/mythmon || /sbin/service mythbackend restart > /dev/null; | |
− | */5 * * * * /root/bin/mythmon || /sbin/service mythbackend restart > /dev/null; | ||
− | |||
A nice feature of bash is that you can OR two commands together. The second command is tried iff (if and only if) the first command fails. Similarly, ANDing two commands, the second is executed iff the first is successful. | A nice feature of bash is that you can OR two commands together. The second command is tried iff (if and only if) the first command fails. Similarly, ANDing two commands, the second is executed iff the first is successful. | ||
/sbin/service is a feature of Fedora/RedHat. | /sbin/service is a feature of Fedora/RedHat. | ||
− | <pre | + | <pre> |
#!/bin/bash | #!/bin/bash | ||
# | # | ||
Line 187: | Line 181: | ||
exit 1; | exit 1; | ||
− | + | </pre> | |
== Or Quick == | == Or Quick == |
Revision as of 23:43, 28 November 2008
So, for whatever reason, your Myth processes have occasional crashes that you haven't diagnosed yet (you have submitted a bug report, haven't you?) and you'd like it to restart when it does...
This page has a few approaches to monitoring Myth's processes
Contents
Monit
Monit (http://www.tildeslash.com/monit/)
- It monitors processes and restarts them if they fail.
- Can be configured to alert you (I like to know if it crashes in case it gets serious!)
- It is 'secure'able
- Has a web interface to check status and log
- Can also monitor anything else (eg listings downloads, disk space etc)
Installation
Debian/knoppmyth install : apt-get install monit
Configuration
- On Debian create
/etc/monit/monitrc
- On openSUSE create
/etc/monitrc
# Monit control file # # Comments begin with a '#' and extend through the end of the line. # Blank lines between program entries are ignored. Keywords are case # insensitive. All path's MUST BE FULLY QUALIFIED, starting with '/' # set daemon 30 set logfile /var/log/monit.log set mailserver smtp.mydomain.com set mail-format { from: monit@bao.mydomain.com } set alert root@mydomain.com # Send alert to system admin on any event set httpd port 2812 and allow ash.mydomain.com allow 10.0.0.90 allow 127.0.0.1 allow haze.mydomain.com allow admin:monit # user 'admin' with password 'monit' check process mythbackend with pidfile /var/run/mythtv/mythbackend.pid group mythtv start program = "/etc/init.d/mythtv-backend restart" stop program = "/etc/init.d/mythtv-backend stop" if failed port 6544 proto http then restart mode manual depends on mysql check process mysql with pidfile /var/run/mysqld/mysqld.pid group mythtv start program = "/etc/init.d/mysql start" stop program = "/etc/init.d/mysql stop" if failed port 3306 then restart mode manual
Obviously change the names/ip addresses as needed!
NOTE: It has been reported that mythbackend could crash from repeated pinging of http port (I didn't see anything in my mythbackend log but my backend crashed almost daily until I made the following change). If you find that monit frequently reports failed protocol test, it may be monit causing the problem and you may want to consider commenting out the line:
if failed port 6544 proto http then restart
If your backend log file starts showing error messages like this:
QSocketDevice::writeBlock: Invalid socket
, then removing the "proto http" in your monit configuration may help also.
Now, sudo monit to start monit.
Note that this tells monit to start in 'manual' mode. To actually start monitoring I have this script in /etc/monit/monit_delay
.
sleep 60 monit monitor mythbackend
and call it from the monit init.d script after monit is started. This ensures that when monit starts up it allows enough time for all the other services to get going before it kills them or anything...
Now, check it's all working by connecting to http://yourbox:2812/
You should see a web page.
Now try doing an /etc/init.d/mythtv-backend stop
. You should get a mail and the backend should restart.
Now try doing an /etc/init.d/mysql stop
. You should get a mail and the backend should stop, mysql should start and then the backend should restart.
Once all this works, you can consider moving on to the next bit...
Paranoia
So what if monit crashes??? asks Craig Partin...
Well, if you like you can run monit from init
.
In this case, just put the following line in your /etc/inittab
moni:235:respawn:/usr/sbin/monit
Then init
watches and makes sure that monit is running (and if init dies you've got a whole heapload of problems!!)
But make sure you don't start monit from your init.d script (but don't forget the monit_delay...)
So, why not run mythbackend like this?
You could - however, it's not the best way to do it - the backend may start before mysql - or your frontend may start before the backend does. For this reason the rc3.d and rc5.d scripts are numbered to start in a certain order to ensure that mysql starts before the backend which starts before the frontend etc.
Daemon Tools
Daemontools (http://cr.yp.to/daemontools.html)
Though not OSS like Monit, daemontools is another very similar mechanism that you may prefer. Daemontools has the advantage of starting and monitoring the process, so there's no danger of mythbackend starting up before daemontools.
Install
apt-get install daemontools-installer
This will set up your inittab, start the supervise process, and create the appropriate directories (assuming you accepted the defaults during the install). Nothing will start, however, until you create the appropriate files under the /service
directory. There's no support in daemontools for service startup order (ie. dependencies), so you have to check in the run script.
Configure
Daemontools looks for a file called /service/servicename/run
to start the program. So we'll call our service mythbackend, hence
/service/mythbackend/run
#!/bin/sh PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin DAEMON=/usr/bin/mythbackend MYSQLHOST=localhost MYSQLADMIN=/usr/bin/mysqladmin if [ -x $MYSQLADMIN ] then $MYSQLADMIN -h$MYSQLHOST ping &> /dev/null fi if [ $? -eq 0 ] # ping worked then exec $DAEMON --logfile /var/log/mythtv/mythbackend.log else logger "Mysqld not running yet. Waiting..." fi
As soon as you make this file executable supervise will try and start up mythbackend, so make sure you've stopped the existing process (eg: /etc/init.d/mythbackend stop
) and removed the existing startup script (I moved mythbackend from /etc/init.d/
to </root
).
Quick and Dirty
So if all that gets to complicated then try running a script like this (from Will Dorman) periodically from cron:
#!/bin/bash pidno=$( ps ax | grep mythbackend | grep -v grep | grep -v mythmon) # echo $pidno # Checks for pid in "ps" listing, field #1. # Then makes sure it is the actual process, not the process invoked by this script. # The last "grep $1" filters out this possibility. if [ -z "$pidno" ] # If, after all the filtering, the result is a zero-length string, then # no running process corresponds to the pid given. echo "No such process running." mail <mynumber>@mobile.att.net -s Uh-oh < /tmp/crap exit $E_NOSUCHPROCESS fi
Even Dirtier
A simpler way to see if a process is running is using ps -C <command> and look at the exit code of ps; ps will error if it can't find the process. Much like the Quick and Dirty, this can be run from root's crontab:
*/5 * * * * /root/bin/mythmon || /sbin/service mythbackend restart > /dev/null;
A nice feature of bash is that you can OR two commands together. The second command is tried iff (if and only if) the first command fails. Similarly, ANDing two commands, the second is executed iff the first is successful. /sbin/service is a feature of Fedora/RedHat.
#!/bin/bash # # /root/bin/mythmon # This is run from cron every 5 minutes. # ps -C mythbackend > /dev/null && exit 0; # append message to log file... # send email notification... echo "`date +%Y-%m-%d\ %T.%-3N` /root/bin/mythmon: mythbackend not running!!!" \ >> /var/log/mythtv/mythbackend.log; exit 1;
Or Quick
This script restarts the mythbackend process right away without polling, sends you an email and creates handy numerated log files.
Notes:
- This script assumes you have email working on your box (like postfix). If not, just delete the mail line.
- Run the script at startup, i.e. add "/home/mythtv/gomythbe.sh &" to /etc/rc.local (Fedora)
- The script will restart mythbackend 10 times before stopping and is a fail safe in case something is really wrong...
#!/bin/bash # # Press Ctrl+C to stop the restarting echo "###############################" echo "# MythBackEnd Server Starting #" echo "###############################" echo To stop the restarting press Ctrl+C when the server is being restarted echo trap 'echo; echo $SRV Server Restarter has been STOPPED!; exit 1' 2 C1=0 # Repeat the loop 10 times before exiting while [ $C1 -le 10 ] do C1=$((C1+=1)) # Modify the line below to match your server execution command line mythbackend > /home/mythtv/mythbackend_$C1.log echo "MythBackend server restarted $C1 time(s)!" # Send an email - put in your email address!! mail -s "MythBackEnd - Process restarted $C1 Times" my@emailaddress.com < /home/mythtv/mythbackend_$C1.log sleep 10 done