[mythtv-commits] Ticket #3464: MythTranscode segfaults on resize when src=dst then src!=dst

[MythTV]

#3464: MythTranscode segfaults on resize when src=dst then src!=dst
-----------------------+----------------------------------------------------
 Reporter:  anonymous  |       Owner:  ijr    
     Type:  patch      |      Status:  new    
 Priority:  minor      |   Milestone:  unknown
Component:  mythtv     |     Version:  unknown
 Severity:  medium     |  
-----------------------+----------------------------------------------------
 There is a bug in mythtranscode which generates a segfault when resizing
 the frame.  The problem only occurs when the source file changes
 resolutions and only after the source resolution is the same as the
 destination and then changes again.

 In my test case initial source resolution was 544x480 with a "noresize"
 profile.  The source resolution changed to 528x480 (source is mpeg2 ts)
 and mythtranscode would segfault every time in libmythavcodec.so
 img_resample.  If you look at programs/mythtranscode/transcode.cpp
 {{{
             if ((video_width == newWidth) && (video_height == newHeight))
             {
                 frame.buf = lastDecode->buf;
             }
             else
             {
                 avpicture_fill(&imageIn, lastDecode->buf, PIX_FMT_YUV420P,
                                video_width, video_height);
                 avpicture_fill(&imageOut, frame.buf, PIX_FMT_YUV420P,
                                newWidth, newHeight);
              ...
 }}}
 You can see that when the frame dimensions are the same, the frame buffer
 is simply passed through to the recorder.  However, when the dimensions
 are different, img_resample writes to frame.buf.  Unfortunately frame.buf
 still points to the previous frame's lastDecode->buf which is invalid.
 The attached patch sets the frame.buf pointer to the proper scratch buffer
 (newFrame).

 The other unrelated line change is in the newFrame allocator.  I don't see
 why the size of the buffer needs to be recalculated considering it was
 calculated previously.

-- 
Ticket URL: <http://svn.mythtv.org/trac/ticket/3464>
MythTV <http://svn.mythtv.org/trac>
MythTV