Difference between revisions of "Debugging"

From MythTV Official Wiki
Jump to: navigation, search
(Getting a Backtrace: "Import" the initial copy of GDB-running instructions from the HOWTO.)
(Removed some very old instructions about getting cores)
(28 intermediate revisions by 6 users not shown)
Line 1: Line 1:
 
If any of your MythTV programs crash (segfault), you must generate a proper backtrace before submitting a new bug report.  Please also see the [http://svn.mythtv.org/trac/wiki/TicketHowTo Ticket HOWTO] for more information before submitting a bug report.
 
If any of your MythTV programs crash (segfault), you must generate a proper backtrace before submitting a new bug report.  Please also see the [http://svn.mythtv.org/trac/wiki/TicketHowTo Ticket HOWTO] for more information before submitting a bug report.
 +
 +
To generate a backtrace, you will need to install debug-enabled MythTV from [[Debugging#Installing_Debug-enabled_MythTV_from_Source|source]] or [[Debugging#Installing_Debug-enabled_MythTV_from_Packages|packages]] and then create the backtrace using one of the methods described in "[[Debugging#Getting_a_Backtrace|Getting a Backtrace]]".
  
  
 
= Installing Debug-enabled MythTV from Source =
 
= Installing Debug-enabled MythTV from Source =
 +
{{Note box|If you have been using packaged versions of MythTV, there is no need to compile from source to obtain a backtrace. Instead, [[Debugging#Installing_Debug-enabled_MythTV_from_Packages|install debug-enabled MythTV from packages]]. The approach in this section should only be used on systems where MythTV was already compiled and installed from source.}}
 +
 +
Debugging a compiled-from-source version of MythTV requires compiling and installing MythTV with debug symbols. This means that MythTV must be reconfigured and recompiled.
 +
 +
First, you need to clean the source directory to ensure all of MythTV is rebuilt with debug symbols:
 +
 +
  $ make distclean
 +
 +
Then, rerun the configure script and specify <code>--compile-type=profile</code> or <code>--compile-type=debug</code> in addition to any other configuration options you may have used. Check the config.log file if you have forgotten those options. Note, though, that MythTV should generally only be compiled with the options <code>--prefix</code>, <code>--compile-type</code>, and <code>--enable-proc-opt</code>--and using any other configure arguments may actually be causing the issue you're seeing.
  
[http://www.mythtv.org/docs/mythtv-HOWTO-22.html#ss22.2 Section 22.2] of the [http://www.mythtv.org/docs/ MythTV Documentation], linked from the [http://www.mythtv.org MythTV website] provides details of debugging source-based installs.
+
In many cases, a profile build will be acceptable for creating backtraces. The main difference between the <code>debug</code> and <code>profile</code> builds is that the <code>debug</code> build disables optimizations. Therefore, a <code>debug</code> build may perform significantly worse than a <code>profile</code> build. If you need to run MythTV under gdb for a long time, it may be worthwhile to run a <code>profile</code> build. If, however, a developer asks you to re-create the backtrace using a <code>debug</code> build, please do so without argument as some portions of the code can only be reliably debugged with a <code>debug</code> build. However, if your system is sufficiently powerful to run a <code>debug</code> build, you may want to start with that so you can be certain the backtrace will be useful.
  
Note, also, that in addition to <code>--compile-type=debug</code>, the MythTV configure script now supports an option, <code>--compile-type=profile</code>.  In many cases, a profile build will be acceptable for creating backtraces.  The main difference between the <code>debug</code> and <code>profile</code> builds is that the <code>debug</code> build disables optimizations.  Therefore, a <code>debug</code> build may perform significantly worse than a <code>profile</code> build.  If you need to run MythTV under gdb for a long time, it may be worthwhile to run a <code>profile</code> build.  If, however, a developer asks you to re-create the backtrace using a <code>debug</code> build, please do so without argument as some portions of the code can only be reliably debugged with a <code>debug</code> build.  However, if your system is sufficiently powerful to run a <code>debug</code> build, you may want to start with that so you can be certain the backtrace will be useful.
+
  $ ./configure --compile-type=profile
  
 +
Then, continue with <code>make</code> and <code>make install</code> as usual.
  
 
= Installing Debug-enabled MythTV from Packages =
 
= Installing Debug-enabled MythTV from Packages =
Line 32: Line 44:
 
== Ubuntu packages ==
 
== Ubuntu packages ==
  
Although it's possible to debug with the packages in the Ubuntu archive, it is first recommended that you enable the Mythbuntu autobuilds packages and make sure that the issue is occurring with them.  These are built from the latest upstream snapshots of either -fixes or -trunk, whichever you choose.  You can enable autobuilds at http://mythbuntu.org/auto-builds .
+
Although it's possible to debug with the packages in the Ubuntu archive, it is first recommended that you enable the Mythbuntu autobuilds packages and make sure that the issue is occurring with them.  These are built from the latest upstream snapshots of either -fixes or -trunk, whichever you choose.  You can enable autobuilds at http://www.mythbuntu.org/repos .
  
 
If you can still reproduce the problem with the latest autobuilds packages, please follow these steps to grab everything necessary for a bug report:
 
If you can still reproduce the problem with the latest autobuilds packages, please follow these steps to grab everything necessary for a bug report:
  
=== Basic Backtrace ===
+
=== Basic backtrace ===
 
For most people, this is all that needs to be followed.
 
For most people, this is all that needs to be followed.
  
Line 56: Line 68:
 
3) Start the apport service. "sudo start apport"
 
3) Start the apport service. "sudo start apport"
  
4) When your crash happens, it will ask you to submit a bug report. Submit the full report.  You'll need a launchpad account for this.
+
4) When your crash happens a crash report file will be created in /var/crash
 +
 
 +
5) Submit the crash report:
 +
 
 +
# sudo apport-cli <crash file>
 +
or
 +
# sudo apport-bug <crash file>
 +
 +
you will see
 +
What would you like to do? Your options are:
 +
  S: Send complete report (recommended; 151.9 MiB)
 +
  R: Send reduced report (slow Internet connection; 54.9 MiB)
 +
  V: View report
 +
  K: Keep report file for sending later or copying to somewhere else
 +
  C: Cancel
 +
  Please choose (S/R/V/K/C):
  
=== Advanced Retrace ===
+
you will want to submit a complete report
 +
 
 +
6) You will be asked to fill out the bug information, you can either open a browser window now or copy the given URL into any browser to do so.
 +
 
 +
7) Once the bug submission is complete you will need to make it public. To do so, on the upper right of the bug page under "Report a Bug" you will see "This Report is Private", click on the icon next to it to edit the state to make it public.
 +
 
 +
8) Copy/paste the URL into the MythTV bug.
 +
 
 +
=== Advanced retrace ===
 
If you are asked to rerun the trace with symbols for the dependencies, you can follow these directions.  This is *not* necessary for most people.
 
If you are asked to rerun the trace with symbols for the dependencies, you can follow these directions.  This is *not* necessary for most people.
 
Take note of the bug number you just above filed, you'll need it here.
 
Take note of the bug number you just above filed, you'll need it here.
Line 109: Line 144:
  
 
== Running gdb directly ==
 
== Running gdb directly ==
 +
{{Note box|You still need to have installed a [http://www.mythtv.org/docs/mythtv-HOWTO-22.html#ss22.2 debug build] or [[Debugging#Debugging_package-based_installs|debug packages]] to create a useful backtrace with a core file.}}
  
Running gdb directly allows you to control the process of running the application and capturing the backtrace. This approach is most sensibly used in situations where the error is easily reproducible and results within an application that is called directly by the user (versus an application called automatically by, e.g., mythbackend). Running a MythTV application on a production system long term under gdb may affect performance, so for not-easily-reproduced errors, you may want to consider using [[Debugging#Using_core_files|using core files]].
+
Running gdb directly allows you to control the process of running the application and capturing the backtrace. This approach is most sensibly used in situations where the error is easily reproducible and occurs within an application that is called directly by the user (versus an application called automatically by, e.g., mythbackend). Running a MythTV application on a production system long term under gdb may affect performance, so for not-easily-reproduced errors, you may want to consider using [[Debugging#Using_core_files|using core files]].
  
 
=== Create a gdb script file ===
 
=== Create a gdb script file ===
Line 124: Line 160:
 
set breakpoint pending on
 
set breakpoint pending on
 
break qFatal
 
break qFatal
set args -l myth.log -v important,general
+
set args --logpath /tmp -v general
 
run
 
run
 
thread apply all bt full
 
thread apply all bt full
Line 131: Line 167:
 
</pre>
 
</pre>
  
This gdb script file can be reused for multiple different MythTV applications. Simply edit the argument list in the <code>set args</code> line of the script to specify the required command-line arguments. gdb has a number of commands, read the man page for more information.
+
This gdb script file can be reused for multiple different MythTV applications. Simply edit the argument list in the <code>set args</code> line of the script to specify the required command-line arguments. (Note that gdb has a number of other commands; see the man page and from within gdb you can type help, for more information.)
  
 
=== Executing the application under gdb ===
 
=== Executing the application under gdb ===
Line 137: Line 173:
 
To execute the application, start gdb with the application name and use the <code>-x</code> argument to specify the gdb script file name. Assuming that the problem you're having is in <code>mythbackend</code>, you would execute:
 
To execute the application, start gdb with the application name and use the <code>-x</code> argument to specify the gdb script file name. Assuming that the problem you're having is in <code>mythbackend</code>, you would execute:
  
   $ gdb mythbackend -x gdbcommands
+
   $ gdb mythbackend -x ${HOME}/gdbcommands
  
 
which would start mythbackend running under gdb and show the following output (with a lot more information after it).
 
which would start mythbackend running under gdb and show the following output (with a lot more information after it).
  
</pre>
+
{{Note box|On Ubuntu/Mythbuntu note that mythfrontend is a script and gdb needs to be started with $ gdb mythfrontend.real -x ${HOME}/gdbcommands}}
 +
 
 +
<pre>
 
GNU gdb 6.3-debian
 
GNU gdb 6.3-debian
 
Copyright 2004 Free Software Foundation, Inc.
 
Copyright 2004 Free Software Foundation, Inc.
Line 152: Line 190:
 
</pre>
 
</pre>
  
gdb will automatically read the commands that you've placed in the <code>gdbcommands</code> file and begin running the program you specified on the command line. gdb has a number of options, read the man page for more information.
+
gdb will automatically read the commands that you've placed in the <code>gdbcommands</code> file and begin running the program you specified on the command line. (Note that gdb has a number of other command-line arguments available; see the man page for more information.)
  
 
When the application crashes or is terminated, gdb will automatically output a backtrace to the file <code>gdb.txt</code> in the current working directory. If the MythTV program appears to be locked up and does not crash, press CTRL-C to create the backtrace file.
 
When the application crashes or is terminated, gdb will automatically output a backtrace to the file <code>gdb.txt</code> in the current working directory. If the MythTV program appears to be locked up and does not crash, press CTRL-C to create the backtrace file.
  
 
All of the output from <code>gdb.txt</code> should be posted to the [http://www.mythtv.org/mailman/listinfo/mythtv-users mythtv-users] or [http://www.mythtv.org/mailman/listinfo/mythtv-dev mythtv-dev] mailing list along with the steps you followed to get the program to crash. Or, if the backtrace was requested in IRC, please post the entire <code>gdb.txt</code> to a pastebin site and provide the link to the developer who requested the backtrace in IRC.
 
All of the output from <code>gdb.txt</code> should be posted to the [http://www.mythtv.org/mailman/listinfo/mythtv-users mythtv-users] or [http://www.mythtv.org/mailman/listinfo/mythtv-dev mythtv-dev] mailing list along with the steps you followed to get the program to crash. Or, if the backtrace was requested in IRC, please post the entire <code>gdb.txt</code> to a pastebin site and provide the link to the developer who requested the backtrace in IRC.
{{Note box|Generally you should not trim any information from the <code>gdb.txt</code> file, unless you had an old <code>gdb.txt</code> file and gdb appended the current run's output to the end. If you do trim the <code>gdb.txt</code> file, make sure you include at least 10 lines prior to the point where the backtrace actually begins. This ensures that there is some context to the backtrace, and so that it's possible to see what exactly caused the segfault.}}
+
 
 +
{{Note box|Generally, you should not trim any information from the <code>gdb.txt</code> file, unless you had an old <code>gdb.txt</code> file and gdb appended the current run's output to the end. If you do trim the <code>gdb.txt</code> file, make sure you include at least 10 lines prior to the point where the backtrace actually begins. This ensures that there is some context to the backtrace, and so that it's possible to see what exactly caused the segfault.}}
  
 
=== Troubleshooting from a remote system ===
 
=== Troubleshooting from a remote system ===
Line 168: Line 207:
  
 
This will allow the graphics to be displayed on the X console (usually ALT-F6 or ALT-F7) and still give you output and control of mythfrontend, either from the ssh session, or by switching back to the text console by pressing CTRL-ALT-F1. You can now continue troubleshooting using gdb as detailed above.
 
This will allow the graphics to be displayed on the X console (usually ALT-F6 or ALT-F7) and still give you output and control of mythfrontend, either from the ssh session, or by switching back to the text console by pressing CTRL-ALT-F1. You can now continue troubleshooting using gdb as detailed above.
 +
 +
== Attaching gdb to a running process ==
 +
 +
If you need to get a backtrace for an application which has hung or become deadlocked, but didn't first execute the application via gdb as outlined in the previous section, you can attempt to attach to the already running application using gdb's '''attach''' command.
 +
 +
First, find the PID of the application that you want to attach to:
 +
 +
  $ ps aux | grep <application name>
 +
 +
... for example, if you are trying to find the pid for a deadlocked mythfrontend, you would run:
 +
 +
  $ ps aux | grep mythfrontend
 +
 +
... the output of which would look something like this:
 +
 +
<pre>$ ps aux | grep mythfrontend
 +
mythtv    8181  3.8 11.4 788912 235212 ?      Sl  20:01  0:41 mythfrontend
 +
mythtv    8332  0.0  0.0  7748  872 pts/2    S+  20:19  0:00 grep mythfrontend
 +
</pre>
 +
 +
The PID for the running mythfrontend would be 8181.  You would ignore 8332 because that would be the PID for the search itself that we were just running.
 +
 +
Once you have the PID, make sure you are in a working directory in which you have appropriate rights to create and write to a file.  The user's current home directory works nicely.  Then, launch gdb:
 +
 +
  <pre>$ gdb
 +
GNU gdb (GDB) 7.2-ubuntu
 +
Copyright (C) 2010 Free Software Foundation, Inc.
 +
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
 +
This is free software: you are free to change and redistribute it.
 +
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
 +
and "show warranty" for details.
 +
This GDB was configured as "x86_64-linux-gnu".
 +
For bug reporting instructions, please see:
 +
<http://www.gnu.org/software/gdb/bugs/>.
 +
(gdb)</pre>
 +
 +
You will see a (gdb) prompt.  At the prompt, enter the following two commands - "'''set logging on'''" and "'''set pagination off'''".  The first will enable the output to be logged to gdb.txt in the current working directory, and the second will turn pagination of the output off:
 +
 +
<pre>(gdb) set logging on
 +
Copying output to gdb.txt.
 +
(gdb) set pagination off
 +
</pre>
 +
 +
Now you are ready to attach to the running process.  Enter "attach <pid>" where <pid> is the PID we looked up earlier, in this case it would look like:
 +
 +
  <pre>(gdb) attach 8181</pre>
 +
 +
Once succesfully attached, you may either collect the backtrace or continue running the application to get to the point of failure.  To continue running the application, type the following at the gdb prompt:
 +
 +
  <pre>(gdb) continue</pre>
 +
 +
Once you've gotten to the point where you want to create a backtrace, enter the following at the gdb prompt:
 +
 +
  <pre>(gdb) thread apply all bt full</pre>
 +
 +
A backtrace will be collected and output to the console, as well as to a gdb.txt in the current working directory.  Once the backtrace is complete, you can enter the following to stop the logging, and exit gdb:
 +
 +
  <pre>(gdb) set logging off
 +
Done logging to gdb.txt.
 +
(gdb) detach
 +
Detaching from program: /usr/local/bin/mythfrontend, process 8181
 +
(gdb) quit
 +
</pre>
  
 
== Using core files ==
 
== Using core files ==
Line 173: Line 275:
 
{{Note box|You still need to have installed a [http://www.mythtv.org/docs/mythtv-HOWTO-22.html#ss22.2 debug build] or [[Debugging#Debugging_package-based_installs|debug packages]] to create a useful backtrace with a core file.}}
 
{{Note box|You still need to have installed a [http://www.mythtv.org/docs/mythtv-HOWTO-22.html#ss22.2 debug build] or [[Debugging#Debugging_package-based_installs|debug packages]] to create a useful backtrace with a core file.}}
  
One way to get backtraces for the backend is to dump core on crashes and then use <code>gdb mythbackend name_of_core_file</code>, this allows you to restart the backend on crashes and still get a backtrace at your leisure. To ensure core files are created when mythbackend crashes, add these three lines to your mythbackend startup script:
+
Core files can be created by telling the system to dump the working memory of the running program to a file when the application crashes.  The core file may be used with gdb to create a backtrace whenever is convenient for you. This approach is most sensibly used in situations where the error may not be easily reproduced, occurs in an application called automatically by, e.g., mythbackend, requires running the application for a long time (and you do not want to incur performance penalties by running that application under gdb), or when having difficulties running the application under gdb.
  
<code>
+
=== Getting a core file ===
ulimit -c unlimited
+
 
echo 1 > /proc/sys/kernel/core_uses_pid
+
To ensure core files are created when an application crashes, disable the limit on core file size by running the following line in the environment where you run the application--e.g. add the command to the mythbackend startup script or execute it in the terminal you're using to start mythbackend:
  echo "/tmp/core.%t.%u.%p.%e" > /proc/sys/kernel/core_pattern
+
 
</code>
+
  ulimit -c unlimited
 +
 
 +
See <code>/usr/src/linux-`uname -r`/Documentation/sysctl/fs.txt</code> for more information.  Note, however, that allowing a setuid process to create core files may have security ramifications.  Therefore, you may not want to allow this.  Alternatively, just start the application as the desired user without using the system start script/without using the <code>--user</code> argument.
 +
 
 +
You may also specify information that controls the core file name, such as:
 +
 
 +
<pre>
 +
sudo sysctl -w kernel.core_uses_pid=1
 +
  sudo sysctl -w kernel.core_pattern='/tmp/core.%t.%u.%p.%e'
 +
</pre>
 +
 
 +
Or, to make the settings survive across reboots, add the following to your <code>/etc/sysctl.conf</code> file or the appropriate file under <code>/etc/sysctl.d</code> (such as <code>/etc/sysctl.d/10-corefiles.conf</code> ).
 +
 
 +
<pre>
 +
kernel.core_uses_pid = 1
 +
kernel.core_pattern = /tmp/core.%t.%u.%p.%e
 +
</pre>
 +
 
 +
The first line allows the %p format specifier, used in the second line, to place the process ID (PID) in the core file name.  The second line creates individual core files for each program crash in the /tmp directory with the time, user id, PID and program name in the file name.
 +
 
 +
=== Creating a backtrace from a core file ===
 +
 
 +
To produce the backtrace run gdb with the application name and use the -c argument to specify the core file name and location:
 +
 
 +
$ gdb mythbackend -c ./core 2>&1 | tee mythtv_backtrace.txt
  
The first allows core files to be used, the second allows the %p in the third line, and the third line creates individual core dumps for each program crash in tmp so they are easy to locate, and have the program that crashed and the time it crashed along with other useful information right in the name.
+
If you changed the format or location of the core file name, change the core file name, above, as appropriate.
  
To finally produce the backtrace run:
+
Once gdb has started, specify the appropriate gdb commands to output a backtrace:
  
 
<code>
 
<code>
$ sudo gdb mythbackend -c /tmp/core.[rest_of_the_file_name] 2>&1 | tee mythtv_backtrace.txt
 
 
  set pagination off
 
  set pagination off
 
  thread apply all bt
 
  thread apply all bt
 
  quit
 
  quit
 
</code>
 
</code>
 
Unless there was memory overwriting going on or there was a serious hardware failure, the output will tell us exactly what line of code the backend crashed on, and what other threads were doing at the time.
 
  
 
[[Category:HOWTO]]
 
[[Category:HOWTO]]

Revision as of 16:12, 28 October 2013

If any of your MythTV programs crash (segfault), you must generate a proper backtrace before submitting a new bug report. Please also see the Ticket HOWTO for more information before submitting a bug report.

To generate a backtrace, you will need to install debug-enabled MythTV from source or packages and then create the backtrace using one of the methods described in "Getting a Backtrace".


Installing Debug-enabled MythTV from Source

Important.png Note: If you have been using packaged versions of MythTV, there is no need to compile from source to obtain a backtrace. Instead, install debug-enabled MythTV from packages. The approach in this section should only be used on systems where MythTV was already compiled and installed from source.

Debugging a compiled-from-source version of MythTV requires compiling and installing MythTV with debug symbols. This means that MythTV must be reconfigured and recompiled.

First, you need to clean the source directory to ensure all of MythTV is rebuilt with debug symbols:

 $ make distclean

Then, rerun the configure script and specify --compile-type=profile or --compile-type=debug in addition to any other configuration options you may have used. Check the config.log file if you have forgotten those options. Note, though, that MythTV should generally only be compiled with the options --prefix, --compile-type, and --enable-proc-opt--and using any other configure arguments may actually be causing the issue you're seeing.

In many cases, a profile build will be acceptable for creating backtraces. The main difference between the debug and profile builds is that the debug build disables optimizations. Therefore, a debug build may perform significantly worse than a profile build. If you need to run MythTV under gdb for a long time, it may be worthwhile to run a profile build. If, however, a developer asks you to re-create the backtrace using a debug build, please do so without argument as some portions of the code can only be reliably debugged with a debug build. However, if your system is sufficiently powerful to run a debug build, you may want to start with that so you can be certain the backtrace will be useful.

 $ ./configure --compile-type=profile

Then, continue with make and make install as usual.

Installing Debug-enabled MythTV from Packages

Note that rather than replacing a package-provided installation of MythTV with a compiled-from-source installation of MythTV to perform the debugging, you may be able to use "debug" packages provided by your packagers. Debug packages simply install the debugging symbols for the already-installed packages you're using, meaning you do not have to uninstall or reinstall MythTV. Often using the debug packages requires enabling a special repo that contains the debug packages.

Once the debug packages have been obtained, you can generally create a backtrace as described, above, for source-based installs, skipping the section on compiling and installing and, instead, only performing the actions described for using gdb. Some distributions provide other tools that may help create and/or submit backtraces and associated logging information. See below for distribution-specific details.


ATrpms packages

yum --enablerepo=atrpms-debuginfo install mythtv-debuginfo

or for all other environments eg testing/bleeding etc..

yum --enablerepo=atrpms\*-debuginfo install mythtv-debuginfo


RPM Fusion packages

yum --enablerepo=\*-debuginfo install mythtv-debuginfo

Ubuntu packages

Although it's possible to debug with the packages in the Ubuntu archive, it is first recommended that you enable the Mythbuntu autobuilds packages and make sure that the issue is occurring with them. These are built from the latest upstream snapshots of either -fixes or -trunk, whichever you choose. You can enable autobuilds at http://www.mythbuntu.org/repos .

If you can still reproduce the problem with the latest autobuilds packages, please follow these steps to grab everything necessary for a bug report:

Basic backtrace

For most people, this is all that needs to be followed.

1) Install the 'mythtv-dbg' package to add debugging symbols to your system.

# sudo apt-get install mythtv-dbg

Note: if you are using the auto-build packages, make sure your system is up to date. Run the following before installing mythtv-dbg. It's also ok to run this command after the above installation command if you get an error about mismatched versions.

# sudo apt-get upgrade

If you have problems with this and want to back out you can do the following.

# sudo apt-get remove mythtv-dbg
# sudo apt-get --reinstall install mythtv-frontend mythtv-backend

2) Modify /etc/default/apport to set "enabled" to 1

3) Start the apport service. "sudo start apport"

4) When your crash happens a crash report file will be created in /var/crash

5) Submit the crash report:

# sudo apport-cli <crash file>

or

# sudo apport-bug <crash file>

you will see

What would you like to do? Your options are:
  S: Send complete report (recommended; 151.9 MiB)
  R: Send reduced report (slow Internet connection; 54.9 MiB)
  V: View report
  K: Keep report file for sending later or copying to somewhere else
  C: Cancel
Please choose (S/R/V/K/C):

you will want to submit a complete report

6) You will be asked to fill out the bug information, you can either open a browser window now or copy the given URL into any browser to do so.

7) Once the bug submission is complete you will need to make it public. To do so, on the upper right of the bug page under "Report a Bug" you will see "This Report is Private", click on the icon next to it to edit the state to make it public.

8) Copy/paste the URL into the MythTV bug.

Advanced retrace

If you are asked to rerun the trace with symbols for the dependencies, you can follow these directions. This is *not* necessary for most people. Take note of the bug number you just above filed, you'll need it here.

1) Install the apport-retrace tool so you can locally run a backtrace on a core dump.

# sudo apt-get install apport-retrace

2) Enable the debug debs repository.

# echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
# deb http://ddebs.ubuntu.com $(lsb_release -cs)-security main restricted universe multiverse
# deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" |
# sudo tee -a /etc/apt/sources.list.d/ddebs.list

3) Grab the key for the debug debs repository

# gpg --keyserver keyserver.ubuntu.com --recv-key 428D7C01 5E0577F2
# gpg --check-sigs 428D7C01 # signed by key of Martin Pitt
# gpg -o - --export 428D7C01 | sudo apt-key add -

4) Update your package lists

# sudo apt-get update

5) Run the apport-retrace tool.

# sudo apport-retrace --auth -v BUGNUMBER

6) Your web browser should open up. You'll need to give the apport-collect "Full Access".

7) Return to apport-retrace invokation and press Enter.

Locally a trace will be performed and uploaded to the bug.

Gentoo ebuilds

Rebuild with debugging symbols intact

for package in `equery l -i myth | awk "{print $3}" | tail -n+2`
do
 USE="debug" && FEATURES="splitdebug" && emerge --oneshot =$package
done

Getting a Backtrace

Once you have installed the debug symbols for MythTV using a source install or packages, you're ready to get create the backtrace. You may choose from several methods for obtaining the backtrace--the method you choose is likely to be determined by which application is failing, how it was called, and how easily reproducible the error is.

Running gdb directly

Important.png Note: You still need to have installed a debug build or debug packages to create a useful backtrace with a core file.

Running gdb directly allows you to control the process of running the application and capturing the backtrace. This approach is most sensibly used in situations where the error is easily reproducible and occurs within an application that is called directly by the user (versus an application called automatically by, e.g., mythbackend). Running a MythTV application on a production system long term under gdb may affect performance, so for not-easily-reproduced errors, you may want to consider using using core files.

Create a gdb script file

To make sure that you don't forget to type a command required for debugging, it's best to setup a gdb script file. This will be read by gdb when it's started. Copy and paste the following into a terminal to create a file called gdbcommands in your home directory:

cat << "EOF" | tee ${HOME}/gdbcommands 
handle SIGPIPE nostop noprint
handle SIG33 nostop noprint
set logging on
set pagination off
set breakpoint pending on
break qFatal
set args --logpath /tmp -v general
run
thread apply all bt full
set logging off
EOF

This gdb script file can be reused for multiple different MythTV applications. Simply edit the argument list in the set args line of the script to specify the required command-line arguments. (Note that gdb has a number of other commands; see the man page and from within gdb you can type help, for more information.)

Executing the application under gdb

To execute the application, start gdb with the application name and use the -x argument to specify the gdb script file name. Assuming that the problem you're having is in mythbackend, you would execute:

 $ gdb mythbackend -x ${HOME}/gdbcommands

which would start mythbackend running under gdb and show the following output (with a lot more information after it).


Important.png Note: On Ubuntu/Mythbuntu note that mythfrontend is a script and gdb needs to be started with $ gdb mythfrontend.real -x ${HOME}/gdbcommands

GNU gdb 6.3-debian
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-linux".Using host libthread_db library "/lib/tls/libthread_db.so.1".
[Thread debugging using libthread_db enabled]

gdb will automatically read the commands that you've placed in the gdbcommands file and begin running the program you specified on the command line. (Note that gdb has a number of other command-line arguments available; see the man page for more information.)

When the application crashes or is terminated, gdb will automatically output a backtrace to the file gdb.txt in the current working directory. If the MythTV program appears to be locked up and does not crash, press CTRL-C to create the backtrace file.

All of the output from gdb.txt should be posted to the mythtv-users or mythtv-dev mailing list along with the steps you followed to get the program to crash. Or, if the backtrace was requested in IRC, please post the entire gdb.txt to a pastebin site and provide the link to the developer who requested the backtrace in IRC.


Important.png Note: Generally, you should not trim any information from the gdb.txt file, unless you had an old gdb.txt file and gdb appended the current run's output to the end. If you do trim the gdb.txt file, make sure you include at least 10 lines prior to the point where the backtrace actually begins. This ensures that there is some context to the backtrace, and so that it's possible to see what exactly caused the segfault.

Troubleshooting from a remote system

If you're trying to troubleshoot and you can't get back to the gdb window for some reason, it may be easier to use two systems or to start mythfrontend from the text console.

If you're going to troubleshoot from a remote system, connect to the machine that you're going to test using ssh or telnet. Next, type:

 $ export DISPLAY=localhost:0.0

This will allow the graphics to be displayed on the X console (usually ALT-F6 or ALT-F7) and still give you output and control of mythfrontend, either from the ssh session, or by switching back to the text console by pressing CTRL-ALT-F1. You can now continue troubleshooting using gdb as detailed above.

Attaching gdb to a running process

If you need to get a backtrace for an application which has hung or become deadlocked, but didn't first execute the application via gdb as outlined in the previous section, you can attempt to attach to the already running application using gdb's attach command.

First, find the PID of the application that you want to attach to:

 $ ps aux | grep <application name>

... for example, if you are trying to find the pid for a deadlocked mythfrontend, you would run:

 $ ps aux | grep mythfrontend

... the output of which would look something like this:

$ ps aux | grep mythfrontend
mythtv    8181  3.8 11.4 788912 235212 ?       Sl   20:01   0:41 mythfrontend
mythtv    8332  0.0  0.0   7748   872 pts/2    S+   20:19   0:00 grep mythfrontend

The PID for the running mythfrontend would be 8181. You would ignore 8332 because that would be the PID for the search itself that we were just running.

Once you have the PID, make sure you are in a working directory in which you have appropriate rights to create and write to a file. The user's current home directory works nicely. Then, launch gdb:

$ gdb
GNU gdb (GDB) 7.2-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
(gdb)

You will see a (gdb) prompt. At the prompt, enter the following two commands - "set logging on" and "set pagination off". The first will enable the output to be logged to gdb.txt in the current working directory, and the second will turn pagination of the output off:

(gdb) set logging on
Copying output to gdb.txt.
(gdb) set pagination off

Now you are ready to attach to the running process. Enter "attach <pid>" where <pid> is the PID we looked up earlier, in this case it would look like:

(gdb) attach 8181

Once succesfully attached, you may either collect the backtrace or continue running the application to get to the point of failure. To continue running the application, type the following at the gdb prompt:

(gdb) continue

Once you've gotten to the point where you want to create a backtrace, enter the following at the gdb prompt:

(gdb) thread apply all bt full

A backtrace will be collected and output to the console, as well as to a gdb.txt in the current working directory. Once the backtrace is complete, you can enter the following to stop the logging, and exit gdb:

(gdb) set logging off
Done logging to gdb.txt.
(gdb) detach
Detaching from program: /usr/local/bin/mythfrontend, process 8181
(gdb) quit

Using core files

Important.png Note: You still need to have installed a debug build or debug packages to create a useful backtrace with a core file.

Core files can be created by telling the system to dump the working memory of the running program to a file when the application crashes. The core file may be used with gdb to create a backtrace whenever is convenient for you. This approach is most sensibly used in situations where the error may not be easily reproduced, occurs in an application called automatically by, e.g., mythbackend, requires running the application for a long time (and you do not want to incur performance penalties by running that application under gdb), or when having difficulties running the application under gdb.

Getting a core file

To ensure core files are created when an application crashes, disable the limit on core file size by running the following line in the environment where you run the application--e.g. add the command to the mythbackend startup script or execute it in the terminal you're using to start mythbackend:

 ulimit -c unlimited

See /usr/src/linux-`uname -r`/Documentation/sysctl/fs.txt for more information. Note, however, that allowing a setuid process to create core files may have security ramifications. Therefore, you may not want to allow this. Alternatively, just start the application as the desired user without using the system start script/without using the --user argument.

You may also specify information that controls the core file name, such as:

 sudo sysctl -w kernel.core_uses_pid=1
 sudo sysctl -w kernel.core_pattern='/tmp/core.%t.%u.%p.%e'

Or, to make the settings survive across reboots, add the following to your /etc/sysctl.conf file or the appropriate file under /etc/sysctl.d (such as /etc/sysctl.d/10-corefiles.conf ).

kernel.core_uses_pid = 1
kernel.core_pattern = /tmp/core.%t.%u.%p.%e

The first line allows the %p format specifier, used in the second line, to place the process ID (PID) in the core file name. The second line creates individual core files for each program crash in the /tmp directory with the time, user id, PID and program name in the file name.

Creating a backtrace from a core file

To produce the backtrace run gdb with the application name and use the -c argument to specify the core file name and location:

$ gdb mythbackend -c ./core 2>&1 | tee mythtv_backtrace.txt

If you changed the format or location of the core file name, change the core file name, above, as appropriate.

Once gdb has started, specify the appropriate gdb commands to output a backtrace:

set pagination off
thread apply all bt
quit