Difference between revisions of "Lossless Cut"

From MythTV Official Wiki
Jump to: navigation, search
(Supported Recording Devices)
Line 317: Line 317:
 
|
 
|
 
|}<BR>
 
|}<BR>
 +
 +
*"*" - Audio and video tracks are being cut successfully but subtitles are not yet being processed properly or at all
  
 
=Unsupported Recording Devices=
 
=Unsupported Recording Devices=

Revision as of 10:37, 13 October 2012

Important.png Note: Currently Lossless Cut scripts are available as an Alpha release. The intent is to have MythTV users qualify their recorded videos to determine the script's compatibility with a wide variety of recording devices. So far these are the known Supported Recording Devices. Hopefully with the assistance of the MythTV community we can identify other recording devices which work and/or have the script improved to include additional devices and their encoding types.

Alpha test status update (Wed, 10 Oct 2012 16:52 EST):

  • Alpha v0.1.4 has been released to fix the bug where the configuration setting "delete_old" was being ignored when the replace "-r" option is used. This ment even if you changed the "delete_old" setting to "false" the original recording was still deleted.
    Be aware that the default for "delete_old" is "false" which means the original recorded video file is renamed with a ".Old" extension tacked on to the end (e.g. 2296_20120903120000.mpg.Old). You are responsible for dealing with the old recorded file and the disk space it occupies.
    Edits were added to ensure that valid FPS, last and first seek frames were in the DB. If they can not be found messages are displayed and added to the log. Then the script performs a graceful abort returning a failed error return code.
  • Alpha v0.1.3 has been released which fixes a fews bugs no one has reported but also adds a manufacturer and model column to the support and unsupport wiki report "-w" type. No changes were made to subtitle processing.
  • It is likely that subtitle processing will be an issue until the subtitle extraction utilitites mythccextractor and/or CCExtractor work consistently. Right now neither the subtitles from a HDHomerun or a AverMedia AverTV DVB-T 777 can be processed. In the new release (v0.1.3) the code has been modified so that a recorded video will be lossless cut even if the subtitles cannot be included. The CCExtractor developer has been very responsive in the past and hopefully that will continue.


Alpha Testing needs:

  • Recording devices:
  • General recordings:
    • HD recordings any resolutions from a MythTV recording device
    • Specifically a HD 1080p recording
    • NTSC standard definition recordings
    • Recordings from non-North American countries
    • Recordings where EPG metadata is limited, especially for TV Series
  • Slave backend, especially where the recording backend is on a different PC than where the MythVideo "Video" storage group resides
  • NFS mounts:
    • Lossless Cutting of a recording which is mounted on a NFS drive
    • Lossless Cutting while using the Export and Move options to an NFS mount


Subtitles:
See known issues with subtitle processing under "Alpha test status update" above.
Right now the script's subtitle processing has only been tested on UK DVB-T Freeview text format tracks. It is suspected that alternate subtitles track formats will require adjustments to the CCExtractor command line arguments. Those arguments are available in the "~/.mythtv/loss_less.cfg" file so you can experiment.

Subtitle testing needs:

  • Multiple subtitles in one recorded file, especially where tracks contain multiple languages such as English and Spanish.
  • Subtitles tracks containing UTF8 characters sets. e.g. Asian languages, Cyrillic ... etc.


See Bug Reporting for an easy way to provide the developer with a bug report and if required a video sample file


Author R.D. Vaughan (IRC: <IReboot> in #mythtv-users)
Description The Lossless Cut userjobs are used in conjunction with MythTV's commflag and editing UI to create loss less cut mkv video files. Lossless Cut supports multiple video, audio encoding types and subtitles formats.
Supports Version24.png  Version25.png  Version26.png 



Features

  • Supports MythTV versions 0.24+fixes and higher
  • Supports NTSC, PAL, SD, HD 720p and 1080i recordings (this claim needs to be verified by the Alpha testers)
  • A wide variety of Supported Containers, video and audio encoding types and subtitle formats are available
  • Supports automatic subtitle conversion to SRT format, cut and remuxing into the final mkv video
  • Support for recordings with multiple audio, video and subtitle tracks
  • A Debian PPA is available along with source. The PPA has the benefit of always keeping you up to date with the latest enhancements and bug fixes. The source tarball can be easily downloaded and installed on non-Debian based distros. Check Lossless Cut install from PPA or Lossless Cut install from source for details.
  • Includes a userjob that adjusts commercial flag cut frames to their closest keyframes. Which speeds up manual review of commercial flagged cuts points. See: Keyframe Adjustment of Cut and Skip Lists
  • Supports three types of loss less cuts:
  • Full integration with the MythTV metadata grabbers including a fall back to EPG metadata when the program cannot be found by the grabbers, using the grabbers as set in MythTV settings including the language code
  • When exported or moved the mkv file is renamed to a customizable format e.g. "Sanctuary - S02E01 - End of Nights (1).mkv"
  • When exported to MythVideo any missing sub directories will be automatically created according to a customizable path and naming convention
  • The MythVideo Video storage group does not have to be on the same backend as the recording backend. Transfer of the cut mkv file is handled by the MythTV backend.
  • A support script provides text bug Reports, automated video sample creation and wiki compatible text for adding Supported Recording Devices or Unsupported Recording Devices table rows
  • Optionally, metadata will be added to the mkv container (title, subtitles, season and episode numbers, program description)
  • A text configuration file provides a convenient method to Customize lossless_cut.cfg
  • The command line and arguments used during each step of loss less cut processing is added to a Log file, generated with every userjob. The log file is automatically deleted if the job successfully completes. This is essential information for problem analysis.


Prerequisites

  • Python 2.6 or 2.7 (identical to the MythTV python requirements)
  • MythTV v0.24 and higher backend installed and functional
  • MythTV python bindings
  • Python "python-dateutil" package
  • lxml v2.7.2 or higher (identical to the MythTV python requirements)
  • MKVToolnix v5.7.x or higher, see: MKVToolNix
  • Mediainfo v0.7.5 or higher, see: Mediainfo
  • Lossless Cut install from PPA or Lossless Cut install from source on any backend (master and/or slave) which will be executing a Lossless Cut userjob
  • The lossless_cut.py script requires read/write access to the directories containing the recorded videos


Installation Instructions

Installing Dependencies

You must use recent version of the MKVToolnix and Mediainfo or Lossless Cut will not work. Lossless Cut is not compatible with earlier versions of "mkvmerge". Even recent distributions such as Ubuntu 12.04 do not come with the required version of MKVToolNix so use the PPA or install from source.
HP-mini on the user list suggested that if you are using older versions of Ubuntu and need to compile MKVToolNix or MediaInfo from source then you need to update your toolchain. This can be done using ubuntu devs toolchain ppa. Here are the PPA packages.

Mediainfo

To add the Mediainfo PPA as a 3rd Party source and install Mediainfo, type the following in a terminal session. You will need root access:

sudo add-apt-repository ppa:shiki/mediainfo
sudo apt-get update
sudo apt-get install mediainfo

For Debian add the Mediainfo PPA as a 3rd Party source and install the deb, by typing the following in a terminal session. You will need root access:
The example below is specific to Debian 7 "Wheezy" but you can change the two instances of "precise" to the Ubuntu distribution that is closest to what your Debian install reflects.

su root
echo "deb http://ppa.launchpad.net/shiki/mediainfo/ubuntu precise main #MediaInfo" >> /etc/apt/sources.list
echo "deb-src http://ppa.launchpad.net/shiki/mediainfo/ubuntu precise main #MediaInfo Source" >> /etc/apt/sources.list
gpg --keyserver keyserver.ubuntu.com --recv-key 61260473f9d8bc54
gpg -a --export 61260473f9d8bc54 | apt-key add -
apt-get update
apt-get install mediainfo
exit

Mediainfo PPA.Alternately if you cannot or do not want to use a PPA then see: Mediainfo

MKVToolNix

MKVToolNix supports many Linux distributions and multiple releases of those distributions. Installation instructionsand alternately Source downloads
NOTE: You only need to install the "mkvtoolnix" package but NOT the "mkvtoolnix-gui" package.

Lossless Cut install from PPA

Do not install Lossless Cut until you have successfully installed both Mediainfo and MKVToolNix.
The Lossless CutLaunchpad PPA

To add the Lossless Cut PPA as a 3rd Party source and install the deb, type the following in a terminal session. You will need root access:

sudo add-apt-repository ppa:r-d-vaughan/lossless-cut
sudo apt-get update
sudo apt-get install losslesscut

For Debian add the Lossless Cut PPA as a 3rd Party source and install the deb, by typing the following in a terminal session. You will need root access:
The example below is specific to Debian 7 "Wheezy" but as long as you have satisifed the listed dependencies the deb is release/architecture agnostic.

su root
echo "deb http://ppa.launchpad.net/r-d-vaughan/lossless-cut/ubuntu precise main #Lossless Cut scripts" >> /etc/apt/sources.list
echo "deb-src http://ppa.launchpad.net/r-d-vaughan/lossless-cut/ubuntu precise main #Lossless Cut Source" >> /etc/apt/sources.list
gpg --keyserver keyserver.ubuntu.com --recv-key FEB573EB54BB4483
gpg -a --export FEB573EB54BB4483 | apt-key add -
apt-get update
apt-get install losslesscut
exit

Lossless Cut install from source

Install the entier "lossless_cut" directory anywhere as long as the script has read/write access to the recorded videos directory.
To install Lossless Cut from source:

  • Download the latest tarball from Lossless Cut Source
  • Open the downloaded file and extract the entier "lossless_cut" directory where ever you want. Specifically from the tar ball at
    "/~r-d-vaughan/lossless-cut/trunk/lossless_cut/usr/share/lossless_cut"
  • Ensure the following scripts have execute permissions:
lossless_cut/lossless_cut.py
lossless_cut/keyframe_adjust.py
lossless_cut/ll_report.py
lossless_cut/subtitle/ccextractor32bit
lossless_cut/subtitle/ccextractor64bit

Verify that you are ready to use Lossless Cut

Run the following command in a terminal session.

./lossless_cut.py -f "/path to a/recorded.mpg" --test

This command will verify that your MythTV backend has all the required prerequisites. Running this test will also automatically create the Lossless Cut configuration file "~/.mythtv/lossless_cut.cfg".

If you see the "Congratulations!" message at the bottom of the default settings display you are good to go:

Congratulations! All script dependencies have been satisfied.
You are ready to perform loss less cuts on MythTV recorded videos.

Customize lossless_cut.cfg

Assuming you successfully verified that Lossless Cut will run on your installation you should review and alter as required the configuration file "~/.mythtv/lossless_cut.cfg" with a text editor. There is nothing wrong with just using the default configuration. The configuration file contains extensive documentation.

MythTV User Jobs

Add two or three user jobs to MythTV. See: Adding a User Job

Keyframe Adjustment of Cut and Skip Lists

This user job is optional but will help speed up setting accurate cut points after you have commflagged a recording. When you schedule a recording or create a recording rule, enable auto flag commercials and this Keyframe Adjust user job.
If you installed Lossless Cut from the PPA:

User Job description: Keyframe Adjust
User Job command: /usr/bin/keyframe_adjust -f "%DIR%/%FILE%"

If you installed Lossless Cut from source:

User Job description: Keyframe Adjust
User Job command: /path to script/keyframe_adjust.py -f "%DIR%/%FILE%"

When this job runs every skip or cut frame will be adjusted to its nearest keyframe according to the recorded file's seek frames.

  • The start cut frames are adjusted to the nearest keyframe less than or equal to the start skip or cut frame number
  • The end cut frames are adjusted to the nearest keyframe greater than or equal to the end skip or cut frame number

Loss Less Cut user jobs

There are three different options for Lossless Cut user Jobs. The first two are non-destructive, which means that the original recorded video is left untouched.

Export to MythVideo

If you installed Lossless Cut from the PPA:

User Job description: LL Cut to MythVideo
User Job command: /usr/bin/lossless_cut -f "%DIR%/%FILE%" -e

If you installed Lossless Cut from source:

User Job description: LL Cut to MythVideo
User Job command: /path to script/lossless_cut.py -f "%DIR%/%FILE%" -e

Cut and Save

If you installed Lossless Cut from the PPA:

User Job description: LL Cut and Save
User Job command: /usr/bin/lossless_cut -f "%DIR%/%FILE%" -m "/path to save cut mkv video file"

If you installed Lossless Cut from source:

User Job description: LL Cut and Save
User Job command: /path to script/lossless_cut.py -f "%DIR%/%FILE%" -m "/path to save cut mkv video file"

NOTE: Lossless Cut requires read/write access to the "-m" directory path.

Cut and Replace a Recording

Be warned: After the successful creation of the mkv file the old recording will be replaced in the "Watched Recordings" screen and the original mpg file will be deleted. Any cut and/or skip list will be removed. The MKV file will have the same name as the original recording but with a "mkv" file extension.
If you installed Lossless Cut from the PPA:

User Job description: LL Cut & Replace
User Job command: /usr/bin/lossless_cut -f "%DIR%/%FILE%" -r

If you installed Lossless Cut from source:

User Job description: LL Cut & Replace
User Job command: /path to script/lossless_cut.py -f "%DIR%/%FILE%" -r

Lossless Cut Work Flow

  • Assume that you enable the auto commercial detect user job then run/enable the Keyframe Adjust user job
  • For the best lossless cut results use the "Edit the Recording" instructionsand always set your cut points to keyframes
  • After saving the cuts run one of the three Loss Less Cut user jobs
  • Watch the cut video. Assuming you did not replace the original recording you can always go back and fine tune your cuts as described in the previous step. Remember to delete the previous exported or saved cut video file before running the Lossless Cut job again.
  • Assuming you did not replace the original recording and you are satisfied with the cuts delete the original recording

NOTE: If you perform a second cut and export/save but had NOT deleted the previous cut video, a second video file will be created with a date and time tagged on to the end of the file name. In this way you can never accidentally overwrite an existing file with the same name.
E.g. "Boss (2011) - S02E02 - Mania - Thur Sep 20th 12:00:02.mkv

Reports

As it is very difficult to anticipate all the issues that may occur while processing recordings produced by the wide variety of MythTV devices. This is the reason that the bug reporting script ll_report.py has been included. There are four variations of reports that can be produced.

  • Create a text file report that includes detailed information about the OS, the relevant utility versions in your environment, the recording device used and the specifics of the recorded video file (fps, resolutions, etc ...).
    See: Bug Reporting Option: -b
  • Create an archive of the text file mentioned above, the associated log file plus a 25Mg video file containing the first ~30 seconds of the recorded video file. The archive (tar.bz2) should be uploaded to an Internet location or service like mediafire or dropbox so that the developer can download and analyze the issues.
    See: Bug Reporting Option: -B
  • Create an archive of the text file mentioned above, the associated log file plus a 25Mg video file containing ~30 seconds starting at a specific start time into the the recorded video file. The time is expressed in "HH:MM:SS" format. The "HH" and "MM" are optional. The archive (tar.bz2) should be uploaded to an Internet location or service like mediafire or dropbox so that the developer can download and analyze the issues.
    See: Bug Reporting Option: -B -s HH:MM:SS
  • Display text lines which can be cut and pasted into this Wiki's Supported Recording Devices or Unsupported Recording Devices recording device table. That text lines contains information about the device which recorded the video plus track details.
    See: Wiki Table Row Generation Option: -w

Bug Reporting

If you have a problem to report please take the time to report the issue by following these steps.

  • Run the Lossless Cut script as it appears in your User Job but with an additional -k option. This will ensure that the log file is not deleted. For example:
lossless_cut -f "/path to the problem/recordedfile.mpg" -e -k

If the problem does not require a sample video file then:

  • Run the bug reporting script for example:
ll_report -f "/path to the problem/recordedfile.mpg" -b
  • Copy the text bug report to your clipboard
  • Post on the MythTV usermailing list a description of the problem and paste the text bug report into that email

If the problem requires a sample video file then:

  • Run the bug reporting script for example:
ll_report -f "/path to the problem/recordedfile.mpg" -B
  • Upload the resulting archive file to a service like mediafire or dropbox
  • Post on the MythTV usermailing list a brief description of the problem and include the URL of the bug archive

Wiki Table Row Generation

ll_report -f "/path to a/recordedfile.mpg" -w
  • Copy to your clipboard the text which looks like:
|-
|Manufacturer
|Model
|DVB-T
|MPEG Video Version 2
|MPEG Audio Version 1 Layer 2 codec-3
|Teletext, DVB Subtitle codec-6
  • Log on to the MythTV wiki
  • Navigate to this wiki page, click "edit" on the appropriate section containing the table which you want to add a row
  • Paste the copied text into the table making sure the Manufacturer name will maintain row alphabetic order
  • Change the "Manufacturer" and "Model" test to your specific recording device
  • Preview your changes by clicking the "Show Preview" button
  • When you are satisfied with the new row, save the changes by clicking the "Save page" button

Supported Containers, video and audio encoding types and subtitle formats

Despite the formats MKVToolNIX supports, Lossless Cut can only accurately cut a recording which has valid keyframe seek records in the MythTV database. This means it is unlikely that videos which are manually added to MythTV will produce quality results.

How to check what your install supports

You can check what your version of MKVToolNix supports by typing the following in a terminal session:

mkvmerge -l

The containers, video, audio encoding and subtitle formats listed below are specific to "mkvmerge v5.8.0 ('No Sleep / Pillow') built on Sep 2 2012 14:35:40". Use the command above to see what your installed version supports. New capabilities are always being added to MKVToolNix.

Recorded Video Input Containers

Note: All Lossless Cut videos will be output in a MKV container.

  • Matroska audio/video files [mka mks mkv mk3d webm webmv webma]
  • QuickTime audio/video files [mov]
  • Ogg/OGM audio/video files [ogg ogm ogv]
  • RealMedia audio/video files [ra ram rm rmvb rv]
  • AVI (Audio/Video Interleaved) [avi]
  • WebM audio/video files [webm webmv webma]

Video Input encoding

  • AVC/h.264 elementary streams [264 avc h264 x264]
  • Dirac [drc]
  • IVF with VP8 video files [ivf]
  • MP4 audio/video files [mp4 m4v]
  • MPEG program streams [mpg mpeg m2v mpv evo evob vob]
  • MPEG transport streams [ts m2ts mts]
  • MPEG video elementary streams [m1v m2v mpv]
  • VC1 elementary streams [vc1]

Audio Input encoding

  • A/52 (aka AC3) [ac3 eac3]
  • AAC (Advanced Audio Coding) [aac m4a mp4]
  • ALAC (Apple Lossless Audio Codec) [caf m4a mp4]
  • Dolby TrueHD [thd thd+ac3 truehd true-hd]
  • DTS/DTS-HD (Digital Theater System) [dts dtshd dts-hd]
  • FLAC (Free Lossless Audio Codec) [flac ogg]
  • MPEG audio files [mp2 mp3]
  • TTA (The lossless True Audio codec) [tta]
  • WAVE (uncompressed PCM audio) [wav]
  • WAVPACK v4 audio [wv]

Subtitle Input Formats

In addition to the subtitle formats listed below, Lossless Cut will attempt to extract unsupported subtitle tracks and convert them to SRT file then cut and remux the SRT subtitles into the final output MKV video. Any MKVToolNIX supported subtitle tracks will be be cut and copied directly.

  • PGS/SUP subtitles [sup]
  • SRT text subtitles [srt]
  • SSA/ASS text subtitles [ass ssa]
  • USF text subtitles [usf xml]
  • VobSub subtitles [idx]

Current list of MediaInfo supported formats

MediaInfo is used to identify what tracks (Video, Audio and Subtitles) are included in a MythTV recording. This information is used during Lossless Cut processing. See: Supported Formats

Supported Recording Devices

Please add your table row in "Manufacturer" and "Model" alphabetical order. Before adding your device to this table please check that it is not already listed.

Manufacturer Model Recording Device Video Codec(s) Audio Codec(s) Subtitle Formats(s)
?? ?? DVB-T (UK Freeview) MPEG Video Version 2 MPEG Audio Version 1 Layer 2 codec-3 Teletext, DVB Subtitle codec-6
?? ?? DVB-S2 (UK Freesat) AVC Version 2 AC-3 Layer 2 codec-129, MPEG Audio Version 1 Layer 2 codec-4 DVB Subtitle codec-6, Teletext Subtitle
Hauppauge 1212 HDPVR AVC AC-3 codec-129

  • "*" - Audio and video tracks are being cut successfully but subtitles are not yet being processed properly or at all

Unsupported Recording Devices

Please add your table row in "Manufacturer" and "Model" alphabetical order.
Before adding your device to this table please post a bug report to the MythTV mailing list. Follow the Bug Reporting instructions. Also if the script works, most of the time, with the recordings from a particular device then it is likely that the recording itself is the issue. See the first bullet of Known Limitations. Only post a table row for an Unsupported device when Lossless Cut consistently fails.

Manufacturer Model Recording Device Video Codec(s) Audio Codec(s) Subtitle Formats(s)

Known Limitations

  • If the original recording is corrupted then Lossless Cut is unlikely to work. Audio sync and faulty edits have been observed. Likely these type of recordings would suffer the same fate when edited in video editors like Avidemux. MythTV can often play back the original recordings and pass through the area(s) of corruption but you will notice that is where Lossless Cut video runs into problems.
  • Specifically for h.264 encoded videos, any rebuild of the recording's seektable (mythcommflag --rebuild or --video) does not produce valid keyframes. This means that the Lossless Cut script will not likely cut a video accurately even when setting the keyframe cut points manually with the frontend edit UI. The lossless cut results are a hit and miss proposition, which will not change until ticket #11144is fixed. Most of the time this limitation will not effect anyone unless you try to manually add a recording to MythTV and then rebuilding the seek table. Restoring the original seek records from a database backup will work as the keyframes would be accurate. Unless you know what you are doing it is not recommended that you replace/restore specific records in a database.
  • mythccextractor will not be used until ticket #11081is fixed. Currently CCExtractor (included with Lossless Cut) is exclusively used for subtitle extraction and conversion to SRT format.
  • Recorded videos without MythTV seek table records cannot be edited
  • Recorded videos that have been replaced by a loss less cut mkv video cannot be re-edited
  • A user has reported that Lossless Cut works as expected with their HDHomerun device for all but one specific channel (ABC channel 13 WZZM in Grand Rapids). Lossless Cut videos from this specific channel produces only an audio track. This issue is actively being investigated.

Credits

  • The developers at Bunkus.orgfor the MKVToolNix suite of utilities. Without those utilities Lossless Cut would not exist.
  • The MediaInfodevelopers
  • Ian Thiele for his h264cut.sh scriptwhich demonstrated the versatility and maturity of the MKVToolNix "mkvmerge" utility. Also Lossless Cut follows the same general process flow for video cut and merging as in his h264cut.sh script.
  • Raymond Wagner for developing the Python bindings including adding new bindings when requested. Also for putting up with my questions on how to use the python bindings to their best effect.
  • Carlos Fernandez for his versatile subtitle extractor utility CCExtractorand his excellent support
  • Scott Harris for supplying multiple bug sample archives and for doing bug fix testing
  • Andy Boff for providing a sample of UK DVB-T recording which included multiple audio tracks and subtitles tracks