[mythtv] Cause of denoise3d black spots
jrgreen at MIT.EDU
Wed Jan 21 04:22:55 UTC 2009
I think I've figured out what causes the black spots when using
denoise3d with MMX and compiler optimization.
The MMX code makes use of macros in libavcodec/i386/mmx.h like the
#define mmx_r2m(op,reg,mem) \
asm volatile (#op " %%" #reg ", %0" \
: "=m" (mem) \
: /* nothing */ )
#define movq_r2m(reg,var) mmx_r2m (movq, reg, var)
In filter_denoise3d.c these are used in code of this sort:
movq_r2m (mm0, wbuf);
The problem here is that since wbuf is of type int16_t, gcc assumes
that the asm statement will only output to two bytes at the address of
wbuf, whereas the output is actually eight bytes. When using -O3,
gcc stores other entries of wbuf into registers after the first time
this asm macro is called but after the second time, only the new value
for wbuf is retrieved and not wbuf, etc.
A solution is to cast to a data type of the correct size:
movq_r2m (mm0, *((mmx_t *) &wbuf);
I don't know if this should be done in mmx.h (a handful of changes) or
in filter_denoise3d.c (many changes).
Some cursory searches revealed many other places where care is not taken
to cast MMX asm input or output operands to the correct size.
I have tested that this does eliminate the black spots using denoise3d.
More information about the mythtv-dev