Difference between revisions of "Debugging"

From MythTV Official Wiki
Jump to: navigation, search
m (Create a gdb script file: update -l/--logpath arg)
(Executing the application under gdb)
Line 176: Line 176:
which would start mythbackend running under gdb and show the following output (with a lot more information after it).
which would start mythbackend running under gdb and show the following output (with a lot more information after it).
{{Note box|On Ubuntu/Mythbuntu note that mythfrontend is a script and gdb needs to be started with $ gdb mythfrontend.real -x ${HOME}/gdbcommands}}

Revision as of 16:05, 23 August 2012

If any of your MythTV programs crash (segfault), you must generate a proper backtrace before submitting a new bug report. Please also see the Ticket HOWTO for more information before submitting a bug report.

To generate a backtrace, you will need to install debug-enabled MythTV from source or packages and then create the backtrace using one of the methods described in "Getting a Backtrace".

Installing Debug-enabled MythTV from Source

Important.png Note: If you have been using packaged versions of MythTV, there is no need to compile from source to obtain a backtrace. Instead, install debug-enabled MythTV from packages. The approach in this section should only be used on systems where MythTV was already compiled and installed from source.

Debugging a compiled-from-source version of MythTV requires compiling and installing MythTV with debug symbols. This means that MythTV must be reconfigured and recompiled.

First, you need to clean the source directory to ensure all of MythTV is rebuilt with debug symbols:

 $ make distclean

Then, rerun the configure script and specify --compile-type=profile or --compile-type=debug in addition to any other configuration options you may have used. Check the config.log file if you have forgotten those options. Note, though, that MythTV should generally only be compiled with the options --prefix, --compile-type, and --enable-proc-opt--and using any other configure arguments may actually be causing the issue you're seeing.

In many cases, a profile build will be acceptable for creating backtraces. The main difference between the debug and profile builds is that the debug build disables optimizations. Therefore, a debug build may perform significantly worse than a profile build. If you need to run MythTV under gdb for a long time, it may be worthwhile to run a profile build. If, however, a developer asks you to re-create the backtrace using a debug build, please do so without argument as some portions of the code can only be reliably debugged with a debug build. However, if your system is sufficiently powerful to run a debug build, you may want to start with that so you can be certain the backtrace will be useful.

 $ ./configure --compile-type=profile

Then, continue with make and make install as usual.

Installing Debug-enabled MythTV from Packages

Note that rather than replacing a package-provided installation of MythTV with a compiled-from-source installation of MythTV to perform the debugging, you may be able to use "debug" packages provided by your packagers. Debug packages simply install the debugging symbols for the already-installed packages you're using, meaning you do not have to uninstall or reinstall MythTV. Often using the debug packages requires enabling a special repo that contains the debug packages.

Once the debug packages have been obtained, you can generally create a backtrace as described, above, for source-based installs, skipping the section on compiling and installing and, instead, only performing the actions described for using gdb. Some distributions provide other tools that may help create and/or submit backtraces and associated logging information. See below for distribution-specific details.

ATrpms packages

yum --enablerepo=atrpms-debuginfo install mythtv-debuginfo

or for all other environments eg testing/bleeding etc..

yum --enablerepo=atrpms\*-debuginfo install mythtv-debuginfo

RPM Fusion packages

yum --enablerepo=\*-debuginfo install mythtv-debuginfo

Ubuntu packages

Although it's possible to debug with the packages in the Ubuntu archive, it is first recommended that you enable the Mythbuntu autobuilds packages and make sure that the issue is occurring with them. These are built from the latest upstream snapshots of either -fixes or -trunk, whichever you choose. You can enable autobuilds at http://mythbuntu.org/auto-builds .

If you can still reproduce the problem with the latest autobuilds packages, please follow these steps to grab everything necessary for a bug report:

Basic backtrace

For most people, this is all that needs to be followed.

1) Install the 'mythtv-dbg' package to add debugging symbols to your system.

# sudo apt-get install mythtv-dbg

Note: if you are using the auto-build packages, make sure your system is up to date. Run the following before installing mythtv-dbg. It's also ok to run this command after the above installation command if you get an error about mismatched versions.

# sudo apt-get upgrade

If you have problems with this and want to back out you can do the following.

# sudo apt-get remove mythtv-dbg
# sudo apt-get --reinstall install mythtv-frontend mythtv-backend

2) Modify /etc/default/apport to set "enabled" to 1

3) Start the apport service. "sudo start apport"

4) When your crash happens a crash report file will be created in /var/crash

5) Submit the crash report:

# sudo apport-cli <crash file>


# sudo apport-bug <crash file>

you will see

What would you like to do? Your options are:
  S: Send complete report (recommended; 151.9 MiB)
  R: Send reduced report (slow Internet connection; 54.9 MiB)
  V: View report
  K: Keep report file for sending later or copying to somewhere else
  C: Cancel
Please choose (S/R/V/K/C):

you will want to submit a complete report

6) You will be asked to fill out the bug information, you can either open a browser window now or copy the given URL into any browser to do so.

7) Once the bug submission is complete you will need to make it public. To do so, on the upper right of the bug page under "Report a Bug" you will see "This Report is Private", click on the icon next to it to edit the state to make it public.

8) Copy/paste the URL into the MythTV bug.

Advanced retrace

If you are asked to rerun the trace with symbols for the dependencies, you can follow these directions. This is *not* necessary for most people. Take note of the bug number you just above filed, you'll need it here.

1) Install the apport-retrace tool so you can locally run a backtrace on a core dump.

# sudo apt-get install apport-retrace

2) Enable the debug debs repository.

# echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
# deb http://ddebs.ubuntu.com $(lsb_release -cs)-security main restricted universe multiverse
# deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" |
# sudo tee -a /etc/apt/sources.list.d/ddebs.list

3) Grab the key for the debug debs repository

# gpg --keyserver keyserver.ubuntu.com --recv-key 428D7C01 5E0577F2
# gpg --check-sigs 428D7C01 # signed by key of Martin Pitt
# gpg -o - --export 428D7C01 | sudo apt-key add -

4) Update your package lists

# sudo apt-get update

5) Run the apport-retrace tool.

# sudo apport-retrace --auth -v BUGNUMBER

6) Your web browser should open up. You'll need to give the apport-collect "Full Access".

7) Return to apport-retrace invokation and press Enter.

Locally a trace will be performed and uploaded to the bug.

Gentoo ebuilds

Rebuild with debugging symbols intact

for package in `equery l -i myth | awk "{print $3}" | tail -n+2`
 USE="debug" && FEATURES="splitdebug" && emerge --oneshot =$package

Getting a Backtrace

Once you have installed the debug symbols for MythTV using a source install or packages, you're ready to get create the backtrace. You may choose from several methods for obtaining the backtrace--the method you choose is likely to be determined by which application is failing, how it was called, and how easily reproducible the error is.

Running gdb directly

Important.png Note: You still need to have installed a debug build or debug packages to create a useful backtrace with a core file.

Running gdb directly allows you to control the process of running the application and capturing the backtrace. This approach is most sensibly used in situations where the error is easily reproducible and occurs within an application that is called directly by the user (versus an application called automatically by, e.g., mythbackend). Running a MythTV application on a production system long term under gdb may affect performance, so for not-easily-reproduced errors, you may want to consider using using core files.

Create a gdb script file

To make sure that you don't forget to type a command required for debugging, it's best to setup a gdb script file. This will be read by gdb when it's started. Copy and paste the following into a terminal to create a file called gdbcommands in your home directory:

cat << "EOF" | tee ${HOME}/gdbcommands 
handle SIGPIPE nostop noprint
handle SIG33 nostop noprint
set logging on
set pagination off
set breakpoint pending on
break qFatal
set args --logpath /tmp -v important,general
thread apply all bt full
set logging off

This gdb script file can be reused for multiple different MythTV applications. Simply edit the argument list in the set args line of the script to specify the required command-line arguments. (Note that gdb has a number of other commands; see the man page for more information.)

Executing the application under gdb

To execute the application, start gdb with the application name and use the -x argument to specify the gdb script file name. Assuming that the problem you're having is in mythbackend, you would execute:

 $ gdb mythbackend -x ${HOME}/gdbcommands

which would start mythbackend running under gdb and show the following output (with a lot more information after it).

Important.png Note: On Ubuntu/Mythbuntu note that mythfrontend is a script and gdb needs to be started with $ gdb mythfrontend.real -x ${HOME}/gdbcommands

GNU gdb 6.3-debian
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-linux".Using host libthread_db library "/lib/tls/libthread_db.so.1".
[Thread debugging using libthread_db enabled]

gdb will automatically read the commands that you've placed in the gdbcommands file and begin running the program you specified on the command line. (Note that gdb has a number of other command-line arguments available; see the man page for more information.)

When the application crashes or is terminated, gdb will automatically output a backtrace to the file gdb.txt in the current working directory. If the MythTV program appears to be locked up and does not crash, press CTRL-C to create the backtrace file.

All of the output from gdb.txt should be posted to the mythtv-users or mythtv-dev mailing list along with the steps you followed to get the program to crash. Or, if the backtrace was requested in IRC, please post the entire gdb.txt to a pastebin site and provide the link to the developer who requested the backtrace in IRC.

Important.png Note: Generally, you should not trim any information from the gdb.txt file, unless you had an old gdb.txt file and gdb appended the current run's output to the end. If you do trim the gdb.txt file, make sure you include at least 10 lines prior to the point where the backtrace actually begins. This ensures that there is some context to the backtrace, and so that it's possible to see what exactly caused the segfault.

Troubleshooting from a remote system

If you're trying to troubleshoot and you can't get back to the gdb window for some reason, it may be easier to use two systems or to start mythfrontend from the text console.

If you're going to troubleshoot from a remote system, connect to the machine that you're going to test using ssh or telnet. Next, type:

 $ export DISPLAY=localhost:0.0

This will allow the graphics to be displayed on the X console (usually ALT-F6 or ALT-F7) and still give you output and control of mythfrontend, either from the ssh session, or by switching back to the text console by pressing CTRL-ALT-F1. You can now continue troubleshooting using gdb as detailed above.

Attaching gdb to a running process

If you need to get a backtrace for an application which has hung or become deadlocked, but didn't first execute the application via gdb as outlined in the previous section, you can attempt to attach to the already running application using gdb's attach command.

First, find the PID of the application that you want to attach to:

 $ ps aux | grep <application name>

... for example, if you are trying to find the pid for a deadlocked mythfrontend, you would run:

 $ ps aux | grep mythfrontend

... the output of which would look something like this:

$ ps aux | grep mythfrontend
mythtv    8181  3.8 11.4 788912 235212 ?       Sl   20:01   0:41 mythfrontend
mythtv    8332  0.0  0.0   7748   872 pts/2    S+   20:19   0:00 grep mythfrontend

The PID for the running mythfrontend would be 8181. You would ignore 8332 because that would be the PID for the search itself that we were just running.

Once you have the PID, make sure you are in a working directory in which you have appropriate rights to create and write to a file. The user's current home directory works nicely. Then, launch gdb:

$ gdb
GNU gdb (GDB) 7.2-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:

You will see a (gdb) prompt. At the prompt, enter the following two commands - "set logging on" and "set pagination off". The first will enable the output to be logged to gdb.txt in the current working directory, and the second will turn pagination of the output off:

(gdb) set logging on
Copying output to gdb.txt.
(gdb) set pagination off

Now you are ready to attach to the running process. Enter "attach <pid>" where <pid> is the PID we looked up earlier, in this case it would look like:

(gdb) attach 8181

Once succesfully attached, you will now need to start collecting the backtrace. At the gdb prompt, enter the following to collect the backtrace:

(gdb) thread apply all bt full

A backtrace will be collected and output to the console, as well as to a gdb.txt in the current working directory. Once the backtrace is complete, you can enter the following to stop the logging, and exit gdb:

(gdb) set logging off
Done logging to gdb.txt.
(gdb) detach
Detaching from program: /usr/local/bin/mythfrontend, process 8181
(gdb) quit

Using core files

Important.png Note: You still need to have installed a debug build or debug packages to create a useful backtrace with a core file.

Core files can be created by telling the system to dump the working memory of the running program to a file when the application crashes. The core file may be used with gdb to create a backtrace whenever is convenient for you. This approach is most sensibly used in situations where the error may not be easily reproduced, occurs in an application called automatically by, e.g., mythbackend, requires running the application for a long time (and you do not want to incur performance penalties by running that application under gdb), or when having difficulties running the application under gdb.

Getting a core file

To ensure core files are created when an application crashes, disable the limit on core file size by running the following line in the environment where you run the application--e.g. add the command to the mythbackend startup script or execute it in the terminal you're using to start mythbackend:

 ulimit -c unlimited

If you or your system start scripts are running a daemon application, such as mythbackend, with the --user argument to drop privileges, you must also tell the kernel to allow creating a core file. You may do so with:

sudo sysctl -w fs.suid_dumpable=2

For mythbackend, this is no longer required after 0.25pre [4a515ba].

See /usr/src/linux-`uname -r`/Documentation/sysctl/fs.txt for more information. Note, however, that allowing a setuid process to create core files may have security ramifications. Therefore, you may not want to allow this. Alternatively, just start the application as the desired user without using the system start script/without using the --user argument.

You may also specify information that controls the core file name, such as:

 sudo sysctl -w kernel.core_uses_pid=1
 sudo sysctl -w kernel.core_pattern='/tmp/core.%t.%u.%p.%e'

Or, to make the settings survive across reboots, add the following to your /etc/sysctl.conf file or the appropriate file under /etc/sysctl.d (such as /etc/sysctl.d/10-corefiles.conf ).

kernel.core_uses_pid = 1
kernel.core_pattern = /tmp/core.%t.%u.%p.%e

The first line allows the %p format specifier, used in the second line, to place the process ID (PID) in the core file name. The second line creates individual core files for each program crash in the /tmp directory with the time, user id, PID and program name in the file name.

Creating a backtrace from a core file

To produce the backtrace run gdb with the application name and use the -c argument to specify the core file name and location:

$ gdb mythbackend -c ./core 2>&1 | tee mythtv_backtrace.txt

If you changed the format or location of the core file name, change the core file name, above, as appropriate.

Once gdb has started, specify the appropriate gdb commands to output a backtrace to the file gdb.txt in the current directory:

set pagination off
thread apply all bt