Debugging

From MythTV Official Wiki
Revision as of 18:52, 13 June 2011 by Sphery (talk | contribs) (Installing Debug-enabled MythTV from Source: Encourage use of packages)

Jump to: navigation, search

If any of your MythTV programs crash (segfault), you must generate a proper backtrace before submitting a new bug report. Please also see the Ticket HOWTO for more information before submitting a bug report.

To generate a backtrace, you will need to install debug-enabled MythTV from source or packages and then create the backtrace using one of the methods described in "Getting a Backtrace".


Installing Debug-enabled MythTV from Source

Important.png Note: If you have been using packaged versions of MythTV, there is no need to compile from source to obtain a backtrace. Instead, install debug-enabled MythTV from packages. The approach in this section should only be used on systems where MythTV was already compiled and installed from source.

Debugging a compiled-from-source version of MythTV requires compiling and installing MythTV with debug symbols. This means that MythTV must be reconfigured and recompiled.

First, you need to clean the source directory to ensure all of MythTV is rebuilt with debug symbols:

 $ make distclean

Then, rerun the configure script and specify --compile-type=profile or --compile-type=debug in addition to any other configuration options you may have used. Check the config.log file if you have forgotten those options. Note, though, that MythTV should generally only be compiled with the options --prefix, --compile-type, and --enable-proc-opt--and using any other configure arguments may actually be causing the issue you're seeing.

In many cases, a profile build will be acceptable for creating backtraces. The main difference between the debug and profile builds is that the debug build disables optimizations. Therefore, a debug build may perform significantly worse than a profile build. If you need to run MythTV under gdb for a long time, it may be worthwhile to run a profile build. If, however, a developer asks you to re-create the backtrace using a debug build, please do so without argument as some portions of the code can only be reliably debugged with a debug build. However, if your system is sufficiently powerful to run a debug build, you may want to start with that so you can be certain the backtrace will be useful.

 $ ./configure --compile-type=profile

Then, continue with make and make install as usual.

Installing Debug-enabled MythTV from Packages

Note that rather than replacing a package-provided installation of MythTV with a compiled-from-source installation of MythTV to perform the debugging, you may be able to use "debug" packages provided by your packagers. Debug packages simply install the debugging symbols for the already-installed packages you're using, meaning you do not have to uninstall or reinstall MythTV. Often using the debug packages requires enabling a special repo that contains the debug packages.

Once the debug packages have been obtained, you can generally create a backtrace as described, above, for source-based installs, skipping the section on compiling and installing and, instead, only performing the actions described for using gdb. Some distributions provide other tools that may help create and/or submit backtraces and associated logging information. See below for distribution-specific details.


ATrpms packages

yum --enablerepo=atrpms-debuginfo install mythtv-debuginfo

or for all other environments eg testing/bleeding etc..

yum --enablerepo=atrpms\*-debuginfo install mythtv-debuginfo


RPM Fusion packages

yum --enablerepo=\*-debuginfo install mythtv-debuginfo

Ubuntu packages

Although it's possible to debug with the packages in the Ubuntu archive, it is first recommended that you enable the Mythbuntu autobuilds packages and make sure that the issue is occurring with them. These are built from the latest upstream snapshots of either -fixes or -trunk, whichever you choose. You can enable autobuilds at http://mythbuntu.org/auto-builds .

If you can still reproduce the problem with the latest autobuilds packages, please follow these steps to grab everything necessary for a bug report:

Basic backtrace

For most people, this is all that needs to be followed.

1) Install the 'mythtv-dbg' package to add debugging symbols to your system.

# sudo apt-get install mythtv-dbg

Note: if you are using the auto-build packages, make sure your system is up to date. Run the following before installing mythtv-dbg. It's also ok to run this command after the above installation command if you get an error about mismatched versions.

# sudo apt-get upgrade

If you have problems with this and want to back out you can do the following.

# sudo apt-get remove mythtv-dbg
# sudo apt-get --reinstall install mythtv-frontend mythtv-backend

2) Modify /etc/default/apport to set "enabled" to 1

3) Start the apport service. "sudo start apport"

4) When your crash happens, it will ask you to submit a bug report. Submit the full report. You'll need a launchpad account for this.

Advanced retrace

If you are asked to rerun the trace with symbols for the dependencies, you can follow these directions. This is *not* necessary for most people. Take note of the bug number you just above filed, you'll need it here.

1) Install the apport-retrace tool so you can locally run a backtrace on a core dump.

# sudo apt-get install apport-retrace

2) Enable the debug debs repository.

# echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
# deb http://ddebs.ubuntu.com $(lsb_release -cs)-security main restricted universe multiverse
# deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" |
# sudo tee -a /etc/apt/sources.list.d/ddebs.list

3) Grab the key for the debug debs repository

# gpg --keyserver keyserver.ubuntu.com --recv-key 428D7C01 5E0577F2
# gpg --check-sigs 428D7C01 # signed by key of Martin Pitt
# gpg -o - --export 428D7C01 | sudo apt-key add -

4) Update your package lists

# sudo apt-get update

5) Run the apport-retrace tool.

# sudo apport-retrace --auth -v BUGNUMBER

6) Your web browser should open up. You'll need to give the apport-collect "Full Access".

7) Return to apport-retrace invokation and press Enter.

Locally a trace will be performed and uploaded to the bug.

Gentoo ebuilds

Rebuild with debugging symbols intact

for package in `equery l -i myth | awk "{print $3}" | tail -n+2`
do
 USE="debug" && FEATURES="splitdebug" && emerge --oneshot =$package
done

Getting a Backtrace

Once you have installed the debug symbols for MythTV using a source install or packages, you're ready to get create the backtrace. You may choose from several methods for obtaining the backtrace--the method you choose is likely to be determined by which application is failing, how it was called, and how easily reproducible the error is.

Running gdb directly

Important.png Note: You still need to have installed a debug build or debug packages to create a useful backtrace with a core file.

Running gdb directly allows you to control the process of running the application and capturing the backtrace. This approach is most sensibly used in situations where the error is easily reproducible and occurs within an application that is called directly by the user (versus an application called automatically by, e.g., mythbackend). Running a MythTV application on a production system long term under gdb may affect performance, so for not-easily-reproduced errors, you may want to consider using using core files.

Create a gdb script file

To make sure that you don't forget to type a command required for debugging, it's best to setup a gdb script file. This will be read by gdb when it's started. Copy and paste the following into a terminal to create a file called gdbcommands in your home directory:

cat << "EOF" | tee ${HOME}/gdbcommands 
handle SIGPIPE nostop noprint
handle SIG33 nostop noprint
set logging on
set pagination off
set breakpoint pending on
break qFatal
set args -l myth.log -v important,general
run
thread apply all bt full
set logging off
EOF

This gdb script file can be reused for multiple different MythTV applications. Simply edit the argument list in the set args line of the script to specify the required command-line arguments. (Note that gdb has a number of other commands; see the man page for more information.)

Executing the application under gdb

To execute the application, start gdb with the application name and use the -x argument to specify the gdb script file name. Assuming that the problem you're having is in mythbackend, you would execute:

 $ gdb mythbackend -x gdbcommands

which would start mythbackend running under gdb and show the following output (with a lot more information after it).

GNU gdb 6.3-debian
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-linux".Using host libthread_db library "/lib/tls/libthread_db.so.1".
[Thread debugging using libthread_db enabled]

gdb will automatically read the commands that you've placed in the gdbcommands file and begin running the program you specified on the command line. (Note that gdb has a number of other command-line arguments available; see the man page for more information.)

When the application crashes or is terminated, gdb will automatically output a backtrace to the file gdb.txt in the current working directory. If the MythTV program appears to be locked up and does not crash, press CTRL-C to create the backtrace file.

All of the output from gdb.txt should be posted to the mythtv-users or mythtv-dev mailing list along with the steps you followed to get the program to crash. Or, if the backtrace was requested in IRC, please post the entire gdb.txt to a pastebin site and provide the link to the developer who requested the backtrace in IRC.


Important.png Note: Generally, you should not trim any information from the gdb.txt file, unless you had an old gdb.txt file and gdb appended the current run's output to the end. If you do trim the gdb.txt file, make sure you include at least 10 lines prior to the point where the backtrace actually begins. This ensures that there is some context to the backtrace, and so that it's possible to see what exactly caused the segfault.

Troubleshooting from a remote system

If you're trying to troubleshoot and you can't get back to the gdb window for some reason, it may be easier to use two systems or to start mythfrontend from the text console.

If you're going to troubleshoot from a remote system, connect to the machine that you're going to test using ssh or telnet. Next, type:

 $ export DISPLAY=localhost:0.0

This will allow the graphics to be displayed on the X console (usually ALT-F6 or ALT-F7) and still give you output and control of mythfrontend, either from the ssh session, or by switching back to the text console by pressing CTRL-ALT-F1. You can now continue troubleshooting using gdb as detailed above.

Using core files

Important.png Note: You still need to have installed a debug build or debug packages to create a useful backtrace with a core file.

Core files can be created by telling the system to dump the working memory of the running program to a file when the application crashes. The core file may be used with gdb to create a backtrace whenever is convenient for you. This approach is most sensibly used in situations where the error may not be easily reproduced, occurs in an application called automatically by, e.g., mythbackend, requires running the application for a long time (and you do not want to incur performance penalties by running that application under gdb), or when having difficulties running the application under gdb.

Getting a core file

To ensure core files are created when an application crashes, disable the limit on core file size by running the following line in the environment where you run the application--e.g. add the command to the mythbackend startup script or execute it in the terminal you're using to start mythbackend:

 ulimit -c unlimited

You may also specify information that controls the core file name, such as:

 echo 1 > /proc/sys/kernel/core_uses_pid
 echo "/tmp/core.%t.%u.%p.%e" > /proc/sys/kernel/core_pattern

The first line allows the %p format specifier, used in the third line, to place the process ID (PID) in the core file name. The second line creates individual core files for each program crash in the /tmp directory with the program name, time, and other useful information in the file name.

Creating a backtrace from a core file

To produce the backtrace run gdb with the application name and use the -c argument to specify the core file name and location:

$ gdb mythbackend -c ./core 2>&1 | tee mythtv_backtrace.txt

If you changed the format or location of the core file name, change the core file name, above, as appropriate.

Once gdb has started, specify the appropriate gdb commands to output a backtrace to the file gdb.txt in the current directory:

set pagination off
thread apply all bt
quit