[mythtv] getting the log server started

[Andy Leiserson]

Since upgrading to 0.26 I have been having problems with the backend not
starting on boot. When this happens there is no indication of what went
wrong -- no log, no console output.

I have identified a few issues, and there still seem to be more
problems. Some of the races are just theories.

 * http://code.mythtv.org/trac/ticket/10924 (not my ticket, but I think
   it's the same as what I observed)

 * a crash in zmq that looks similar to
   https://zeromq.jira.com/browse/LIBZMQ-356 (unfortunately the solution
   offered is to upgrade to zmq v3.x)

 * possible race between initialization of gCoreContext in the main
   thread and use of gCoreContext by MythSystem::ProcessFlags in the
   logging thread when starting the logging server

 * possible race between InitializeMythDirs() in the main thread and use
   of GetInstallPrefix() to construct the path to mythlogserver in the
   logging thread

I have managed to get some stdio messages from the backend when it fails
to start, but I can't seem to get a core dump. (I'm not doing setuid in
the daemon and I've confirmed I can get a core dump if I manually send
it SIGABRT.)

In addition to the issues above, there's also a race between starting
the frontend and the backend on my system. If the frontend starts
first, then the frontend's mythlogserver "wins". Since the frontend
doesn't pass the syslog option to the log server, if the frontend's log
server wins, then the backend's logging is broken.

I would understand a requirement that the backend be started first (i.e.
my setup is broken), but if that's the case, then there should be an
obvious error thrown, rather than silently causing the backend to stop
logging.

I've looked at the logging discussion at http://www.pastebin.ca/2330737.
That seems reasonable, but doesn't resolve the issue of configuring the
log server consistently regardless of who starts it.

I have a couple ideas:

 * make the log server a separate daemon, so init scripts can reliably
   ensure that it starts before any other myth processes

 * move the bifurcation of console/server log output upstream, so that
   console logs are not blocked waiting for the log server to be ready.
   Once this is done, it's not so bad to configure the log server in
   config.xml or the database, since log messages related to broken log
   server configuration can still be output to the console. And it
   should also be okay to delay starting the log server until the core
   context is fully initialized.

I can probably work on patches for either of these if there's agreement
that it makes sense.