2019-render

2019 Render branch

This page outlines the development of the 2019-render branch of MythTV - which is not yet merged into the master branch. The intention is to merge the branch into master in time for the v31 release.

Summary - libmythui

The OpenGL portions of libmythui, the library that displays MythTV's user interface, have been substantially re-written to utilise the newer OpenGL functionality provided by Qt5.

This reduces the complexity of the MythTV code base through the removal of code duplicated between MythTV and Qt and greatly improves portability across platforms. The new code should operate without issue on any device that has a compliant OpenGL 2.0 or OpenGL ES 2.0 driver. More modern OpenGL/ES versions are also fully supported but not yet required. OpenGL ES is automatically used on platforms that implement it (e.g. Android, Raspberry Pi and other embedded devices).

The user should notice no difference.

Summary - libmythtv

libmythtv, the library that decodes and displays media, now exclusively uses OpenGL to display video and associated imagery (On Screen Display, Captions etc). This ensures video rendering is consistent across all supported devices and enables the use of highly performant rendering when using hardware based video decoders.

libmythtv now supports fully featured decoding and direct rendering using VDPAU and NVDEC (NVidia), VAAPI (Intel and AMD), VideoToolBox (OSX), MMAL (Raspberry Pi), V4L2 Codecs (Linux - Raspberry Pi and other embedded devices) and MediaCodec (Android). Windows support is not yet implemented. 10, 12 and 16(!) bit decoding and rendering is available where supported by the relevant drivers. All hardware decoders also have an equivalent 'copy back' decoder that will pass decoded video back into main memory for processing and display. These may also be used in the future for hardware accelerated transcoding, commercial flagging etc.

For software based decoding, MythTV now supports an extensive range of frame formats allowing lossless processing and display of 10, 12 and 16bit formats.

Colour space conversion and processing has been extended considerably to ensure accurate rendering in most cases. Colour mapping of High Dynamic Range (HDR) content is a work in progress though Hybrid Log Gamma material should display relatively well on a non-HDR display.

Note Displaying high bit depth material on a 10bit (or higher) display has not been tested (donations welcome) but should, in theory, work without issue.

Deinterlacing support has been re-written. MythTV now uses deinterlacers from FFmpeg/libavfilter for software deinterlacing and the OpenGL shader based deinterlacers have been optimised and improved. Deinterlacer settings now use preferences for methods and quality rather than using explicit deinterlacers. In most cases this will enable the same deinterlacers as the previous approach but allows MythTV to make informed decisions about how and when to deinterlace when the preferred option is not available.

Details - libmythui

OpenGL ES
 - OpenGL ES will generally only be used when required. To try and force the use of OpenGL ES where regular OpenGL would normally be selected set the environment variable 'MYTHTV_OPENGL_ES=true'

OpenGL Core profiles
 - MythTV does not currently require any functionality offered by more modern OpenGL versions. To try and force a more modern profile set the environment variable "MYTHTV_OPENGL_CORE=true". This will attempt to create an OpenGL context that includes compute shader functionality (which will be used for future development).

Debugging OpenGL issues
 - to enable GPU driver logging, use gpu logging verbosity (i.e. mythfrontend -v gpu).
 - GPU logging may produce a lot of log messages for certain drivers. To filter out unwanted, verbose messages use the environment variable "MYTHTV_OPENGL_LOGFILTER" with a combination of other, error, deprecated, undefined, performance, portability, grouppush and grouppop.
 - for advanced debugging of OpenGL errors, set the environment variable MYTHTV_OPENGL_SYNCHRONOUS=true (in combination with GPU logging). Break points can then be set in your debugger of choice that will give a backtrace pointing to exactly which lines of code triggered the error.
 - to debug Qt QPA (Qt Platform Abstraction) issues - set the environment variable 'QT_LOGGING_RULES=qt.qpa.gl=true'.
 - as all drawing now uses OpenGL (user interface and video), grabbing a screenshot (bind the appropriate action to a key in settings) will give an accurate version of what is being displayed in all cases. This may be useful when reporting rendering issues.

Details - libmythtv

Debugging OpenGL performance
 - use gpu and playback logging verbosity (i.e. mythfrontend -v gpu,playback). This will give timing information for the various stages of OpenGL video rendering - texture upload, framebuffer clearing, rendering, flushing and swapping. Note - the timing detail is informative only. Overall performance will be limited by many other factors.

TODO

Colourspace handling
 - Support temporal dithering when displaying content with a higher bit depth than the display.
 - Complete tone mapping for HDR material.
 - Auto detection of display colour primaries and transfer characteristics from EDID.
 - Auto detection of display range (limited v full) from EDID.
 - Validate lossless render path where supported (software decode, VAAPI DRM, NVDEC, VideoToolBox)
 - Possibly fallback to a 'lossy' render path when we know the display cannot handle the full colour depth i.e. rendering 10bit to 8bit framebuffer.

Deinterlacing
 - Add back deinterlacing of HLS material (override deinterlacer).
 - Add an A/V sync adjustment for deinterlacers with multiple reference frames. The displayed frame may not be in sync with the audio.

NVDEC
 - Add support for yadif-cuda post process deinterlace filter

Known limitations

Windows
 - There is currently no Windows support. Any Windows build will fail in multiple places.

VAAPI
 - To get the best VAAPI direct rendering performance and quality (using DRM), the environment variable QT_XCB_GL_INTEGRATION="xcb_egl" must be set to tell Qt to use EGL as the windowing interface. On Wayland desktops, there is no VAAPI direct rendering without setting this variable and on 'regular' X desktops performance will be significantly reduced by using GLX code. Unfortunately, setting this variable breaks OpenGL setup on NVidia systems - with no obvious workaround. It is unclear how this can be resolved as we need to create our OpenGL context before we can check what driver is in use.

Deinterlacing
 - The basic software deinterlacer is very poor quality and needs improvement.

V4L2 Codecs
 - There is no direct rendering support for V4L2 and performance is limited as a result. There are unsupported and possibly unstable patches for FFmpeg to use DRM PRIME buffers for zero copy.
 - On the Raspberry Pi (at least) the driver does not flag whether frames are interlaced. Automatic interlaced detection then fails and deinterlacing is not enabled.

MMAL
 - Performance is not good enough.
 - MMAL deinterlacing is not implemented.

VDPAU
 - No support for other VDPAU video filters (denoise etc). There are no plans to add this functionality back.

OpenGL ES
 - OpenGL ES does not support certain texture formats which are required for uploading video frames with a higher bit depth (e.g. 10bit). Hence certain frame formats are disallowed when using OpenGL ES for rendering and they are converted to 8bit in the decoder. This applies to software decoding only and there is no known workaround.

Possible issues

VideoToolBox
 - HEVC decoding is untested.
 - 10bit HEVC decoding is untested (only available in FFmpeg master - so requires a re-sync).

Known bugs - with resolution

VAAPI
 - intermittent null pointer dereference crash when using VAAPI for decoding and seeking. Requires a simple patch to FFmpeg.

Known bugs - unresolved

OpenGL playback
 - incorrect viewport used with certain windowing/display settings e.g. running mythfrontend in a window and not using the GUI size for playback.

Deinterlacing
 - not enabling single rate deinterlacer when timestretch is enabled and the display cannot support the required rate. Leads to broken A/V sync and no audio.

VAAPI
 - minor scaling issue with certain H.264 (and possibly HEVC) content when using VPP for deinterlacing. The root cause is the old 1080 v 1088 issue. FFmpeg will set the height for some content to 1088 which then causes issues as the frames are passed through the VPP deinterlacing filter.
 - A/V sync issue with VAAPI copy back and VPP deinterlacing with streams who's PTS increments by only 1 (unlikely to be a real world issue).

NVDEC
 - static functionality check at startup sometimes failing and NVDEC decoding is not available when it should be.

MediaCodec (Android)
 - long pauses under certain conditions while the decoder waits for video surfaces to become available (direct rendering only?)
 - deadlock and playback failures when the stream changes.