[mythtv] Denoise3D filter causing segfaults

[Andrew Mahone]

Trying to debug this D Banerjee's reported bug, here's what I have so far:

1) Nothing seems to go wrong unless I use the denoise3d filter.  Denoise3d 
works once, but if I exit and re-enter live tv, it crashes the backend.  With 
the more extreme CFLAGS I have for much of the rest of my system, this only 
takes one or two tries.  If I use just "-O2 -march=athlon-xp" it takes 4 or 5 
tries to get it to crash.

2) Hacking the CPU tests in NewDenoise3DFilter to always use the non-MMX 
filter seems to reliably prevent it from crashing.

3) If I use a debug build of MythTV, it *always* crashes the second time I 
enter live tv mode.  It does this with any filter, with no filters, or with 
NVR and NVP hacked to to never use the filter API.  Gdb won't tell me 
anything.  If I use a normal build of MythTV and a debug build of the filter, 
I'm back to the original behavior, but I still can't get a trace.  The gdb 
session is attached to this mail.

So, it looks like the bug is in the MMX code in denoise3d, but gdb isn't able 
to tell me anything, so it will be very difficult for me to debug.  Anyone 
have any other suggestions for debugging that could be helpful?  I know about 
valgrind, but I can't get it to build on my system, despite the fact that 
it's supposed to work with 2.6 kernels and with glibc with NPTL.  For now, 
I'm going on the assumption that the MMX code somehow runs off the end of one 
of its buffers and overwrites something else to cause the crash later, so I 
guess I'll see if I can find anything wrong in there by hand.

-- 
  Andrew Mahone
  andrewmahone AT eml DOT cc
-------------- next part --------------
GNU gdb 6.0
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...(no debugging symbols found)...Using host libthread_db library "/lib/libthread_db.so.1".

(gdb) r
Starting program: /usr/bin/mythbackend
(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...
(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...[Thread debugging using libthread_db enabled]
[New Thread 1090294176 (LWP 28962)]
Starting up as the master server.
[New Thread 1101347760 (LWP 28972)]
[New Thread 1109740464 (LWP 28973)]
[New Thread 1118133168 (LWP 28974)]
[New Thread 1126525872 (LWP 28975)]
2003-11-16 16:49:01 Enabled verbose msgs : important general
[New Thread 1134922672 (LWP 28976)]
[New Thread 1143315376 (LWP 28977)]
[New Thread 1151708080 (LWP 28978)]
[New Thread 1160100784 (LWP 28979)]
[New Thread 1168493488 (LWP 28980)]
2003-11-16 16:49:03 Found changes in the todo list.
2003-11-16 16:49:06 MainServer::HandleAnnounce Playback
2003-11-16 16:49:06 adding: chshrcat as a player 1
2003-11-16 16:49:06 MainServer::HandleAnnounce Playback
2003-11-16 16:49:06 adding: chshrcat as a player 0
[New Thread 1178987440 (LWP 28985)]
2003-11-16 16:49:06 MainServer::HandleAnnounce Playback
2003-11-16 16:49:06 adding: chshrcat as a player 0
2003-11-16 16:49:06 adding: chshrcat as a remote ringbuffer
2003-11-16 16:49:06 Changing from None to WatchingLiveTV
[New Thread 1229122480 (LWP 28986)]
[New Thread 1241504688 (LWP 28987)]
[New Thread 1249897392 (LWP 28988)]
[New Thread 1258290096 (LWP 28989)]
2003-11-16 16:49:42 Changing from WatchingLiveTV to None
2003-11-16 16:49:47 MainServer::HandleAnnounce Playback
2003-11-16 16:49:47 adding: chshrcat as a player 0
2003-11-16 16:49:47 MainServer::HandleAnnounce Playback
2003-11-16 16:49:47 adding: chshrcat as a player 0
2003-11-16 16:49:47 adding: chshrcat as a remote ringbuffer
2003-11-16 16:49:47 Changing from None to WatchingLiveTV
Couldn't get registers: No such process.
(gdb) bt full
Cannot fetch general-purpose registers for thread 1101347760: generic error
Cannot fetch general-purpose registers for thread 1101347760: generic error
(gdb)