Difference between revisions of "Mythcutprojectx"

From MythTV Official Wiki
Jump to: navigation, search
m
(Added a new version for both fixes and trunk with auto stream selection or user override by command line. Remove trunk-only version.)
Line 6: Line 6:
 
|category=User Job Scripts}}
 
|category=User Job Scripts}}
  
Two versions are provided: I have used mythcutprojectx in 0.23-fixes, many hundreds of times.  I do not run trunk, so trunkcutprojectx is almost untested, but it is intended for use with trunk. The main difference is in the translation from the MythTV cutlist to that used by Project-X.  Using the wrong version will probably give an inverse cut.  The trunk version should work with either mp2 or ac3 audio; fixes without editing is mp2 only.  
+
Two versions are provided: I have used mythcutprojectx in 0.23-fixes, many hundreds of times.  It assumes that the audio format is mp2.
  
 +
I had reports that using it in trunk (July 2010) gave an inverse cut, and I posted a version called trunkcutprojectx, which I had not tested myself, that was intended to cut in the right sense in trunk and to recognize both mp2 and ac3 audio.
 +
 +
I am now posting a new version in which the sense of cut is controlled by editing the variable INVERT.  It is still intended to be run from a terminal but now after listing the video and audio streams found by "ffmpeg -i" it will automatically select the first listed and work with them unless the user intervenes within TIMEOUT seconds.  If the choice is to quit, the two wanted PIDs can be given in the command line.  If the wanted PIDs are known in advance no interaction is needed.  In the UK the two main PIDs are usually determined by the channel number.
 +
 
As stated in the preamble, this script is essentially a replacement for the 'lossless' mythtranscode with cutlist for mpeg-2 recordings known to the mythtv database.  Mythtranscode works well with many recordings, but it has problems with others, maybe because individual data streams start or stop.  One typical failure mode reports that 'one buffer is full and the other is empty', and aborts, but other defects in the input may not be detected until they cause a failure in a later process such as DVD creation or a defect in the DVD that has been created.  It seems that Project-X is more tolerant of defects in its input, and recordings preprocessed by this script usually yield good DVDs.
 
As stated in the preamble, this script is essentially a replacement for the 'lossless' mythtranscode with cutlist for mpeg-2 recordings known to the mythtv database.  Mythtranscode works well with many recordings, but it has problems with others, maybe because individual data streams start or stop.  One typical failure mode reports that 'one buffer is full and the other is empty', and aborts, but other defects in the input may not be detected until they cause a failure in a later process such as DVD creation or a defect in the DVD that has been created.  It seems that Project-X is more tolerant of defects in its input, and recordings preprocessed by this script usually yield good DVDs.
  
 
MythArchive does itself offer the ability to use Project-X to apply the MythTV cutlist.  I have preferred to use this script.  Applying the cutlist via Project-X within MythArchive sometimes gave me the wrong sound channel, and it means that the size of the final file is not known when the recordings are selected.  I also believe that the conversion and application of the cutlist here is more robust than in mythburn.py. It uses byte counts, not frame counts.
 
MythArchive does itself offer the ability to use Project-X to apply the MythTV cutlist.  I have preferred to use this script.  Applying the cutlist via Project-X within MythArchive sometimes gave me the wrong sound channel, and it means that the size of the final file is not known when the recordings are selected.  I also believe that the conversion and application of the cutlist here is more robust than in mythburn.py. It uses byte counts, not frame counts.
 
The script can be run interactively if the two PIDs to be kept (one video, one audio) are not known in advance: it will then show what PIDs are present and which stream is which.  If you already know this, the PIDs to be kept can be entered on the command line and no interaction will be needed.  In the UK the two main PIDs are usually linked to the channel number. 
 
  
 
The cutlist is not applied with single-frame accuracy but I have found it robust and 'good enough'. I have had very few failures when the script has been used on the original recording; it seemed less reliable when used on recordings that had been preprocessed using mencoder.
 
The cutlist is not applied with single-frame accuracy but I have found it robust and 'good enough'. I have had very few failures when the script has been used on the original recording; it seemed less reliable when used on recordings that had been preprocessed using mencoder.
Line 25: Line 27:
  
 
The script (still called mythcutprojectx) uses ideas from scripts found in the mythtv lists: mythcutprojectx and pxcut.sh
 
The script (still called mythcutprojectx) uses ideas from scripts found in the mythtv lists: mythcutprojectx and pxcut.sh
 
I have added a variant, trunkcutprojectx, because the original version performs an inverse cut when used in trunk.  This version should work with ac3 audio as well as mp3, and I have included as comments suggestions by Christopher Meredith for automatically copying ffmpeg-derived PIDs into the script.  These lines are for information only.  I have given no thought to where they might be placed.
 
  
 
This script was initially posted in trac.  The trac page has links explaining the motivation for writing this script and its relationship to MythArchive.
 
This script was initially posted in trac.  The trac page has links explaining the motivation for writing this script and its relationship to MythArchive.
Line 211: Line 211:
 
}}
 
}}
  
{{Code box|trunkcutprojectx.sh|
+
{{Code box|mythcutprojectx_20100716.sh|
 
<pre>
 
<pre>
 +
 
#!/bin/sh -e
 
#!/bin/sh -e
  
Line 218: Line 219:
 
# Largely based on scripts posted by Tino Keitel and Kees Cook in the Mythtv lists.
 
# Largely based on scripts posted by Tino Keitel and Kees Cook in the Mythtv lists.
  
# Usage: ./trunkcutprojectx <recording>
+
# Usage: ./mythcutprojectx <recording>
 
# <recording> is an mpeg2 file recorded by MythTV with a valid DB entry.
 
# <recording> is an mpeg2 file recorded by MythTV with a valid DB entry.
  
# This script is essentially a replacement for the 'lossless' mpeg2 mythtranscode.
+
# This script is essentially a terminal-based replacement for the 'lossless' mpeg2 mythtranscode.
 
# It will pass the recording and the MythTV cutlist to ProjectX.
 
# It will pass the recording and the MythTV cutlist to ProjectX.
 
# If the cutlist is empty the entire recording will be processed.
 
# If the cutlist is empty the entire recording will be processed.
# It uses ffmpeg to report what streams are present and
+
# It uses ffmpeg to report what streams are present, chooses the first video and audio streams listed,
# asks the user to input the PIDs of the video and one audio stream.
+
# and gives the user TIMEOUT seconds to accept that choice or quit and make another.
# It uses ProjectX to demux and mplex (from mjpegtools) to remux.
+
# It uses ProjectX to demux, and mplex (from mjpegtools) to remux.
 
# Output format is DVD compliant without nav packets.
 
# Output format is DVD compliant without nav packets.
 
# It then clears the cutlist, updates the filesize in the database and rebuilds the seek table.
 
# It then clears the cutlist, updates the filesize in the database and rebuilds the seek table.
 
# The result is apparently acceptable as a recording within MythTV and as input to MythArchive.
 
# The result is apparently acceptable as a recording within MythTV and as input to MythArchive.
# The ProjectX log file is kept. Other tempfiles are deleted.
+
# The ProjectX log file and ffmpeg stream analysis are kept. Other tempfiles are deleted.
 +
# The variable INVERT controls the sense in which the cutlist is applied.
  
 
# The script needs to be edited to define some local variables.
 
# The script needs to be edited to define some local variables.
Line 236: Line 238:
 
####################
 
####################
  
# Variables RECDIR1, RECDIR2, TEMPDIR1, TEMPDIR2, PROJECTX, PASSWD need to be customised.
+
# Variables RECDIR1, RECDIR2, TEMPDIR1, TEMPDIR2, PROJECTX, PASSWD, INVERT need to be customised.
 +
# At present (July 2010) MythTV trunk and fixes apparently apply the cutlist in opposite senses.
 +
# TESTRUN is initially set to true so that the polarity of the cutlist that will be passed to Project-X can be checked
 
# RECDIR1 and TEMPDIR1 should if possible be on different drive spindles.  Likewise RECDIR2 and TEMPDIR2.
 
# RECDIR1 and TEMPDIR1 should if possible be on different drive spindles.  Likewise RECDIR2 and TEMPDIR2.
  
Line 251: Line 255:
 
PASSWD=mythtv
 
PASSWD=mythtv
  
 +
INVERT=true  # old MythTV setting, used in "fixes"
 +
# INVERT=false  # setting for use in trunk
 +
 +
TIMEOUT=20  # Longest 'thinking time' in seconds allowed before adopting the automatically selected audio stream.
 +
 +
TESTRUN=true    # cutlists will be shown but the recording will be unchanged 
 +
# TESTRUN=false  # the recording will be processed
 
#################
 
#################
  
if test "$1" = "-h" || test "$1" = "--help" ; then
+
if [ "$1" = "-h" ] || [ "$1" = "--help" ] ; then
echo "Usage: ./trunkcutprojectx <recording>"
+
echo "Usage: "$0" <recording>"
 
echo "<recording> is an mpeg2 file recorded by MythTV with a valid DB entry."
 
echo "<recording> is an mpeg2 file recorded by MythTV with a valid DB entry."
 
echo "e.g. 1234_20100405123400.mpg in one of the defined RECDIRs"
 
echo "e.g. 1234_20100405123400.mpg in one of the defined RECDIRs"
Line 261: Line 272:
 
fi
 
fi
  
 +
# exit if .old file exists
 +
 +
if  [ -f ${RECDIR1}/"$1".old ] ; then
 +
    echo " ${RECDIR1}/"$1".old exists: giving up." ; exit 1
 +
fi
 +
 +
if  [ -f ${RECDIR2}/"$1".old ] ; then
 +
    echo " ${RECDIR2}/"$1".old exists: giving up." ; exit 1
 +
fi
 +
 
# Customize with paths to alternative recording and temp folders
 
# Customize with paths to alternative recording and temp folders
  
Line 270: Line 291:
 
     if  [ ! -f "$1" ] ; then  
 
     if  [ ! -f "$1" ] ; then  
 
       echo " "$1" not found.  Giving up"
 
       echo " "$1" not found.  Giving up"
       cd
+
       cd ~
 
       exit 1
 
       exit 1
 
     fi
 
     fi
 
fi  
 
fi  
 
echo "**** This version is for trunk: if used in fixes the cutlist will be inverted ****"
 
echo "I have not tested it in trunk.  No Guarantees"
 
  
 
if [ $# -lt 3 ]
 
if [ $# -lt 3 ]
Line 282: Line 300:
 
   echo "Error: needs three arguments. Running  ffmpeg -i "$1" 2>&1 | grep -C 4 Video "  
 
   echo "Error: needs three arguments. Running  ffmpeg -i "$1" 2>&1 | grep -C 4 Video "  
 
   echo   
 
   echo   
   ffmpeg -i "$1" 2>&1 | grep -C 4 Video
+
   ffmpeg -i "$1" 2>&1 | grep -C 4 Video | tee temp$$.txt
 
   echo
 
   echo
  echo "Expected Command Line is of the form  ./trunkcutprojectx 1234_20070927190000.mpg  0xvvv 0xaaa "
 
  echo "                                                        filename_in_DB          vPID  aPID  "
 
  cd ~
 
  exit 1
 
fi
 
  
VPID="$2"
+
  # Thanks to Christopher Meredith for the basic parsing magic here.
APID="$3"
+
  VPID=`grep Video  temp$$.txt | head -n1 | cut -f 1,1 -d']' | sed 's+.*\[++g'`
 +
  # It has to be tweaked for multiple audio streams.  This (with head -n1 ) selects the first listed by ffmpeg.
 +
  # You may alternatively wish to select for language, format, etc.  May be channel, programme, user dependent.
 +
  APID=`grep Audio  temp$$.txt | head -n1 | cut -f 1,1 -d']' | sed 's+.*\[++g'`
 +
 
 +
  echo -e "Choosing the first audio track listed by \" ffmpeg -i \".  It may not be the one you want."
 +
  echo -e "\nThe selected values would be "$VPID" and "$APID".  The track info for these is \n"
 +
 
 +
  grep "$VPID" temp$$.txt
 +
  grep "$APID" temp$$.txt
 +
 
 +
  echo -e "\nTo accept these values press \"a\", or wait....\n" 
 +
  echo  "If you want to select other values, or quit to think about it, press another key within $TIMEOUT seconds."
 +
 
 +
  read -t $TIMEOUT -n 1 RESP
 +
  if  [ $? -gt 128 ] ; then   
 +
      RESP="a"
 +
  fi
 +
 
 +
  if [ "$RESP" != "a" ] ; then
 +
      echo -e "Quitting: if you want to select the PIDs from the command line its expected form is  \n"
 +
      echo " "$0" 1234_20070927190000.mpg  0xvvv 0xaaa "
 +
      echo -e "                    filename_in_DB          vPID  aPID \n"
 +
      cd ~
 +
      exit 1
 +
  fi
  
# Auto PID insertion code suggested by Christopher Meredith. May need tweaking for multiple audio streams.
+
  echo -e "Going on: processing with suggested values $VPID $APID \n"
# Lines provided for information only:  I have given no thought to where they might be used.
+
  grep "$VPID" temp$$.txt
# VPID=`ffmpeg -i "$1" 2>&1 | grep Video | cut -f 1,1 -d']' | sed 's+.*\[++g'`
+
  grep "$APID" temp$$.txt
# APID=`ffmpeg -i "$1" 2>&1 | grep Audio | cut -f 1,1 -d']' | sed 's+.*\[++g'`
+
  echo
 +
else
 +
  VPID="$2"
 +
  APID="$3"
 +
fi
  
 +
#Now do the actual processing
 
# chanid and starttime identify the recording in the DB
 
# chanid and starttime identify the recording in the DB
 
chanid=`echo "select chanid from recorded where basename=\"$1\";" |
 
chanid=`echo "select chanid from recorded where basename=\"$1\";" |
Line 306: Line 349:
  
 
# In trunk an initial zero is apparently treated as a normal cut-in point.
 
# In trunk an initial zero is apparently treated as a normal cut-in point.
# cut-in points
+
# list0 shows cut-in points and eof, but in fixes never includes zero
 
list0=`echo "select mark from recordedmarkup
 
list0=`echo "select mark from recordedmarkup
 
where chanid=$chanid and starttime='$starttime' and type=0 order by mark;" |
 
where chanid=$chanid and starttime='$starttime' and type=0 order by mark;" |
 
mysql -N -u mythtv  -p$PASSWD mythconverg `
 
mysql -N -u mythtv  -p$PASSWD mythconverg `
 
+
#list1 shows cut-out points. In fixes an initial here is a cut-in.
# cut-out points   
 
 
list1=`echo "select mark from recordedmarkup
 
list1=`echo "select mark from recordedmarkup
 
where chanid=$chanid and starttime='$starttime' and type=1 order by mark;" |
 
where chanid=$chanid and starttime='$starttime' and type=1 order by mark;" |
Line 317: Line 359:
  
 
echo "CollectionPanel.CutMode=0" > cutlist$$ ;
 
echo "CollectionPanel.CutMode=0" > cutlist$$ ;
 
+
if  ! $INVERT ; then
 
     list=`echo "select mark from recordedmarkup
 
     list=`echo "select mark from recordedmarkup
 
     where chanid=$chanid and starttime='$starttime' and type in (0,1) order by mark;" |
 
     where chanid=$chanid and starttime='$starttime' and type in (0,1) order by mark;" |
 
     mysql -N -u mythtv  -p$PASSWD mythconverg `   
 
     mysql -N -u mythtv  -p$PASSWD mythconverg `   
 +
else
 +
    for i in $list1 ;
 +
      do
 +
        if [ $i = "0" ]
 +
        then 
 +
          list=`echo "select mark from recordedmarkup
 +
              where chanid=$chanid and starttime='$starttime' and type in (0,1) order by mark;" |
 +
              mysql -N -u mythtv  -p$PASSWD mythconverg | tail -n +2 `
 +
              # tail -n +2 drops the initial zero.
 +
        else
 +
          echo "0" >> cutlist$$
 +
          # That isn't quite the same as inserting a leading zero in list.  Does it matter?
 +
          list=`echo "select mark from recordedmarkup
 +
              where chanid=$chanid and starttime='$starttime' and type in (0,1) order by mark;" |
 +
              mysql -N -u mythtv  -p$PASSWD mythconverg ` 
 +
      fi
 +
    #  use only the first element of list1, as a switch.
 +
      break
 +
    done
 +
fi
  
 
# find the key frame (mark type 9) right before each cut mark,
 
# find the key frame (mark type 9) right before each cut mark,
Line 341: Line 403:
 
echo $list
 
echo $list
 
echo
 
echo
 +
 +
echo -e "\"list\" is MythTV's frame-count cutlist that is used to create the byte-count cutlist used here by Project-X."
 +
echo "At the time of writing (July 2010) the internal cutlists used by fixes and trunk appear to work in opposite senses."
 +
echo "The cut can be inverted by adding or removing a leading zero. "
 +
echo -e "That is what the INVERT variable does.  For fixes it should be set to \"true\" \n"
 +
 +
echo "This is the byte-count cutlist for Project-X. The first value is a cut-in point."
 +
echo -e "Cut-out and cut-in then follow in sequence to EOF. \n"
 
cat cutlist$$
 
cat cutlist$$
echo
+
echo -e "\nThis is a test exit point for you to check that INVERT is correctly set."
 +
echo -e "Its value now is  \"${INVERT}\".  When it's OK, edit the script to set TESTRUN=false.\n"
 +
 
 +
if $TESTRUN ; then
 +
  rm -f cutlist$$
 +
  rm -f temp$$.txt 
 +
  cd ~
 +
  exit 0
 +
fi
  
#exit
+
mv  "$1" "$1".old
  
if  [ -f "$1".old ] ; then
 
  echo " "$1".old exists.  Giving up"
 
  exit 1
 
else
 
  mv  "$1" "$1".old
 
 
# use ProjectX to de-multiplex selected streams with the created cutlist
 
# use ProjectX to de-multiplex selected streams with the created cutlist
  ionice -c3 java -jar "$PROJECTX" -name tempcut$$ -id $VPID,$APID -out $TEMP -cut cutlist$$ "$1".old || :
+
ionice -c3 java -jar "$PROJECTX" -name tempcut$$ -id ${VPID},${APID} -out $TEMP -cut cutlist$$ "$1".old || :
 +
 
 
# and pipe for re-multiplexing to mplex. -f 9 is dvd format without navpacks
 
# and pipe for re-multiplexing to mplex. -f 9 is dvd format without navpacks
 
+
DEMUXPREF=$TEMP/tempcut${$}
  DEMUXPREF=$TEMP/tempcut${$}
+
if [ -f $DEMUXPREF.mp2 ] ; then
  if [ -f $DEMUXPREF.mp2 ] ; then
 
 
     DEMUXAUDIO=$DEMUXPREF.mp2
 
     DEMUXAUDIO=$DEMUXPREF.mp2
  else
+
else
 
     DEMUXAUDIO=$DEMUXPREF.ac3
 
     DEMUXAUDIO=$DEMUXPREF.ac3
  fi
+
fi
 
+
ionice -c3  mplex -o "$1" -V -f 9 $DEMUXPREF.m2v $DEMUXAUDIO
  ionice -c3  mplex -o "$1" -V -f 9 $DEMUXPREF.m2v $DEMUXAUDIO
 
 
    
 
    
 
# tell mythDB about new filesize and clear myth cutlist
 
# tell mythDB about new filesize and clear myth cutlist
  FILESIZE=`du -b "$1" | cut -f 1`
+
FILESIZE=`du -b "$1" | cut -f 1`
  if [ "${FILESIZE}" -gt 1000000 ]; then
+
if [ "${FILESIZE}" -gt 1000000 ]; then
       echo "running: update recorded set filesize=${FILESIZE} where basename=\"$1\";"
+
       echo "Running: update recorded set filesize=${FILESIZE} where basename=\"$1\";"
 
       echo "update recorded set filesize=${FILESIZE} where basename=\"$1\";" | mysql -u mythtv -p$PASSWD mythconverg
 
       echo "update recorded set filesize=${FILESIZE} where basename=\"$1\";" | mysql -u mythtv -p$PASSWD mythconverg
       echo "filesize has been reset"
+
       echo "Filesize has been reset"
       echo "running: ionice -c3 mythcommflag -f "$1" --clearcutlist"
+
       echo "Running: ionice -c3 mythcommflag -f "$1" --clearcutlist"
 
       ionice -c3 mythcommflag -f "$1" --clearcutlist
 
       ionice -c3 mythcommflag -f "$1" --clearcutlist
       echo "cutlist has been cleared"
+
       echo "Cutlist has been cleared"
  fi
+
fi
  
 
#rebuild seek table
 
#rebuild seek table
  echo "running: ionice -c3 mythtranscode --mpeg2 --buildindex --showprogress --chanid "$chanid" --starttime "$starttime""
+
echo "Running: ionice -c3 mythtranscode --mpeg2 --buildindex --showprogress --chanid "$chanid" --starttime "$starttime""
  ionice -c3 mythtranscode --mpeg2 --buildindex --showprogress --chanid "$chanid" --starttime "$starttime"
+
ionice -c3 mythtranscode --mpeg2 --buildindex --showprogress --chanid "$chanid" --starttime "$starttime"
  echo "seek table has been rebuilt"
+
echo -e "Seek table has been rebuilt.\n"
  
  echo "Output file is $1  - PID streams $VPID and $APID copied"
+
echo -e "Output file is $1. INVERT is set to \"${INVERT}\". PID streams $VPID and $APID were copied.\n"
  echo
+
if [ -f temp$$.txt ] ; then
  rm -f "$1".png
+
    echo -e "Their original parameters were \n"
#rm -f $TEMP/tempcut${$}*
+
    grep "$VPID" temp$$.txt
  mv "$DEMUXPREF"_log.txt $TEMP/$1_pxlog.txt  
+
    grep "$APID" temp$$.txt
  rm -f $DEMUXPREF.m2v
+
    cat temp$$.txt >> "$DEMUXPREF"_log.txt
  rm -f $DEMUXAUDIO
+
    echo
  rm -f cutlist$$
 
 
fi
 
fi
  
cd
+
rm -f "$1".png
 +
#rm -f $TEMP/tempcut${$}*
 +
mv  ${DEMUXPREF}_log.txt ${TEMP}/"$1"_pxlog.txt
 +
rm -f $DEMUXPREF.m2v
 +
rm -f $DEMUXAUDIO
 +
rm -f cutlist$$
 +
rm -f temp$$.txt
 +
cd ~
 
exit 0
 
exit 0
 +
  
 
</pre>
 
</pre>

Revision as of 12:32, 16 July 2010

Important.png Note: The correct title of this article is mythcutprojectx. It appears incorrectly here due to technical restrictions.


Author John Pilkington
Description Script to preprocess MPEG-2 recordings from DVB-T before running MythArchive
Supports


Two versions are provided: I have used mythcutprojectx in 0.23-fixes, many hundreds of times. It assumes that the audio format is mp2.

I had reports that using it in trunk (July 2010) gave an inverse cut, and I posted a version called trunkcutprojectx, which I had not tested myself, that was intended to cut in the right sense in trunk and to recognize both mp2 and ac3 audio.

I am now posting a new version in which the sense of cut is controlled by editing the variable INVERT. It is still intended to be run from a terminal but now after listing the video and audio streams found by "ffmpeg -i" it will automatically select the first listed and work with them unless the user intervenes within TIMEOUT seconds. If the choice is to quit, the two wanted PIDs can be given in the command line. If the wanted PIDs are known in advance no interaction is needed. In the UK the two main PIDs are usually determined by the channel number.

As stated in the preamble, this script is essentially a replacement for the 'lossless' mythtranscode with cutlist for mpeg-2 recordings known to the mythtv database. Mythtranscode works well with many recordings, but it has problems with others, maybe because individual data streams start or stop. One typical failure mode reports that 'one buffer is full and the other is empty', and aborts, but other defects in the input may not be detected until they cause a failure in a later process such as DVD creation or a defect in the DVD that has been created. It seems that Project-X is more tolerant of defects in its input, and recordings preprocessed by this script usually yield good DVDs.

MythArchive does itself offer the ability to use Project-X to apply the MythTV cutlist. I have preferred to use this script. Applying the cutlist via Project-X within MythArchive sometimes gave me the wrong sound channel, and it means that the size of the final file is not known when the recordings are selected. I also believe that the conversion and application of the cutlist here is more robust than in mythburn.py. It uses byte counts, not frame counts.

The cutlist is not applied with single-frame accuracy but I have found it robust and 'good enough'. I have had very few failures when the script has been used on the original recording; it seemed less reliable when used on recordings that had been preprocessed using mencoder.

The script assumes that the recording will be in one of two folders that must be defined by editing, and uses two other folders for temp storage. The intention is to read from one disc and write to another, but this isn't essential. I hope that modifications for a different number of discs will be obvious.

Commands that cause intensive disc activity are preceded in the script by 'ionice -c3'; this works as intended on my Fedora_12 box, but not under CentOS_5. Just use the command without the 'ionice -c3' preface.

Project-X uses a configuration file, default name X.ini, which may be modified e.g. by use under the GUI. This script, and MythArchive, work as intended if no X.ini is found. Under this script its expected location is in the relevant RECDIR. When Project-X is used within MythArchive the expected location is ~/X.ini.

At the original time of writing 'commflag --clearcutlist' produced a stream of false error messages after completion. This has now (July 2010) been cured.

The script (still called mythcutprojectx) uses ideas from scripts found in the mythtv lists: mythcutprojectx and pxcut.sh

This script was initially posted in trac. The trac page has links explaining the motivation for writing this script and its relationship to MythArchive.

http://svn.mythtv.org/trac/ticket/8347#comment:5


Script.png mythcutprojectx.sh

#!/bin/sh -e

# Copyright (C) 2010 John Pilkington 
# Largely based on scripts posted by Tino Keitel and Kees Cook in the Mythtv lists.

# Usage: ./mythcutprojectx <recording>
# <recording> is an mpeg2 file recorded by MythTV with a valid DB entry.

# This script is essentially a replacement for the 'lossless' mpeg2 mythtranscode.
# It will pass the recording and the MythTV cutlist to ProjectX.
# If the cutlist is empty the entire recording will be processed.
# It uses ffmpeg to report what streams are present and
# asks the user to input the PIDs of the video and one audio stream.
# It uses ProjectX to demux and mplex (from mjpegtools) to remux.
# Output format is DVD compliant without nav packets.
# It then clears the cutlist, updates the filesize in the database and rebuilds the seek table.
# The result is apparently acceptable as a recording within MythTV and as input to MythArchive.
# The ProjectX log file is kept. Other tempfiles are deleted.

# The script needs to be edited to define some local variables.

####################

# Variables RECDIR1, RECDIR2, TEMPDIR1, TEMPDIR2, PROJECTX, PASSWD need to be customised.
# RECDIR1 and TEMPDIR1 should if possible be on different drive spindles.  Likewise RECDIR2 and TEMPDIR2.

RECDIR1=/mnt/f10store/myth/reca
TEMPDIR1=/mnt/sam1/tempb

RECDIR2=/mnt/sam1/recb
TEMPDIR2=/mnt/f10store/myth/tempa

#PROJECTX=/path/to/ProjectX.jar (or to a link to it)
PROJECTX=~/projectx

#PASSWD=`grep "^DBPassword" ~/.mythtv/mysql.txt | cut -d '=' -f 2-`
PASSWD=mythtv

#################

if test "$1" = "-h" || test "$1" = "--help" ; then
echo "Usage: ./mythcutprojectx <recording>"
echo "<recording> is an mpeg2 file recorded by MythTV with a valid DB entry."
echo "e.g. 1234_20100405123400.mpg in one of the defined RECDIRs"
echo "The output file replaces the input file which is renamed to <recording>.old"
exit 0
fi

# Customize with paths to alternative recording and temp folders

cd $RECDIR1
TEMP=$TEMPDIR1
if  [ ! -f "$1" ] ; then
   cd $RECDIR2
   TEMP=$TEMPDIR2
     if  [ ! -f "$1" ] ; then 
       echo " "$1" not found.  Giving up"
       cd
       exit 1
     fi
fi 

if [ $# -lt 3 ]
then
   echo "Error: needs three arguments. Running  ffmpeg -i "$1" 2>&1 | grep -C 4 Video " 
   echo  
   ffmpeg -i "$1" 2>&1 | grep -C 4 Video
   echo
   echo "Expected Command Line is of the form   ./mythcutprojectx 1234_20070927190000.mpg  0xvvv 0xaaa " 
   echo "                                                         filename_in_DB           vPID  aPID  " 
   cd ~
   exit 1
fi

echo "**** This version is for fixes: if used in trunk the cutlist will be inverted ****"

# chanid and starttime identify the recording in the DB
chanid=`echo "select chanid from recorded where basename=\"$1\";" |
mysql -N -u mythtv  -p$PASSWD mythconverg `

starttime=`echo "select starttime from recorded where basename=\"$1\";" |
mysql -N -u mythtv  -p$PASSWD mythconverg `

# list0 gives start (cut-in) points but omits initial zero if start of recording is wanted; includes eof.
list0=`echo "select mark from recordedmarkup
where chanid=$chanid and starttime='$starttime' and type=0 order by mark;" |
mysql -N -u mythtv  -p$PASSWD mythconverg `
  
# list1 gives start of cut (cut-out points). Has initial zero if start of recording is to be cut 
list1=`echo "select mark from recordedmarkup
where chanid=$chanid and starttime='$starttime' and type=1 order by mark;" |
mysql -N -u mythtv  -p$PASSWD mythconverg `  

echo "CollectionPanel.CutMode=0" > cutlist$$ ;
for i in $list1 ;
do
  if [ $i = "0" ] 
    then  
    list=`echo "select mark from recordedmarkup
    where chanid=$chanid and starttime='$starttime' and type in (0,1) order by mark;" |
    mysql -N -u mythtv  -p$PASSWD mythconverg | tail -n +2 ` 
# tail -n +2 drops the initial zero.
  else
    echo "0" >> cutlist$$
    list=`echo "select mark from recordedmarkup
    where chanid=$chanid and starttime='$starttime' and type in (0,1) order by mark;" |
    mysql -N -u mythtv  -p$PASSWD mythconverg `  
  fi
#  use only the first element of list1, as a switch.
  break
done
 
# find the key frame (mark type 9) right before each cut mark,
# extract the byte offset, write it into the ProjectX cutlist
for i in $list ;
do echo "select offset from recordedseek
  where chanid=$chanid and starttime='$starttime' and type=9 and mark >= $i and mark < ($i + 100)
  order by offset;" |
  mysql -N -u mythtv  -p$PASSWD mythconverg | head -n 1
# for each cycle, head -n 1 yields the first line only.
done >> cutlist$$

echo "list0"
echo $list0
echo
echo "list1"
echo $list1
echo
echo "list"
echo $list
echo
cat cutlist$$
echo

#exit

if  [ -f "$1".old ] ; then
  echo " "$1".old exists.  Giving up"
  exit 1
else
  mv  "$1" "$1".old
# use ProjectX to de-multiplex selected streams with the created cutlist
  ionice -c3 java -jar "$PROJECTX" -name tempcut$$ -id $2,$3 -out $TEMP -cut cutlist$$ "$1".old || :
# and pipe for re-multiplexing to mplex. -f 9 is dvd format without navpacks
  ionice -c3  mplex -o "$1" -V -f 9 $TEMP/tempcut${$}.m2v $TEMP/tempcut${$}.mp2 

# tell mythDB about new filesize and clear myth cutlist
  FILESIZE=`du -b "$1" | cut -f 1`
  if [ "${FILESIZE}" -gt 1000000 ]; then
      echo "running: update recorded set filesize=${FILESIZE} where basename=\"$1\";"
      echo "update recorded set filesize=${FILESIZE} where basename=\"$1\";" | mysql -u mythtv -p$PASSWD mythconverg
      echo "filesize has been reset"
      echo "running: ionice -c3 mythcommflag -f "$1" --clearcutlist"
      ionice -c3 mythcommflag -f "$1" --clearcutlist
      echo "cutlist has been cleared"
  fi

#rebuild seek table
  echo "running: ionice -c3 mythtranscode --mpeg2 --buildindex --showprogress --chanid "$chanid" --starttime "$starttime""
  ionice -c3 mythtranscode --mpeg2 --buildindex --showprogress --chanid "$chanid" --starttime "$starttime"
  echo "seek table has been rebuilt"

  echo "Output file is $1  -  PID streams $2 and $3 copied"
  echo
  rm -f "$1".png
#rm -f $TEMP/tempcut${$}*
  mv $TEMP/tempcut${$}_log.txt $TEMP/$1_pxlog.txt 
  rm -f $TEMP/tempcut${$}.m2v
  rm -f $TEMP/tempcut${$}.mp2
  rm -f cutlist$$
fi

cd
exit 0


Script.png mythcutprojectx_20100716.sh


#!/bin/sh -e

# Copyright (C) 2010 John Pilkington 
# Largely based on scripts posted by Tino Keitel and Kees Cook in the Mythtv lists.

# Usage: ./mythcutprojectx <recording>
# <recording> is an mpeg2 file recorded by MythTV with a valid DB entry.

# This script is essentially a terminal-based replacement for the 'lossless' mpeg2 mythtranscode.
# It will pass the recording and the MythTV cutlist to ProjectX.
# If the cutlist is empty the entire recording will be processed.
# It uses ffmpeg to report what streams are present, chooses the first video and audio streams listed, 
# and gives the user TIMEOUT seconds to accept that choice or quit and make another.
# It uses ProjectX to demux, and mplex (from mjpegtools) to remux.
# Output format is DVD compliant without nav packets.
# It then clears the cutlist, updates the filesize in the database and rebuilds the seek table.
# The result is apparently acceptable as a recording within MythTV and as input to MythArchive.
# The ProjectX log file and ffmpeg stream analysis are kept. Other tempfiles are deleted.
# The variable INVERT controls the sense in which the cutlist is applied.

# The script needs to be edited to define some local variables.

####################

# Variables RECDIR1, RECDIR2, TEMPDIR1, TEMPDIR2, PROJECTX, PASSWD, INVERT need to be customised.
# At present (July 2010) MythTV trunk and fixes apparently apply the cutlist in opposite senses.
# TESTRUN is initially set to true so that the polarity of the cutlist that will be passed to Project-X can be checked
# RECDIR1 and TEMPDIR1 should if possible be on different drive spindles.  Likewise RECDIR2 and TEMPDIR2.

RECDIR1=/mnt/f10store/myth/reca
TEMPDIR1=/mnt/sam1/tempb

RECDIR2=/mnt/sam1/recb
TEMPDIR2=/mnt/f10store/myth/tempa

#PROJECTX=/path/to/ProjectX.jar (or to a link to it)
PROJECTX=~/projectx

#PASSWD=`grep "^DBPassword" ~/.mythtv/mysql.txt | cut -d '=' -f 2-`
PASSWD=mythtv

INVERT=true  # old MythTV setting, used in "fixes"
# INVERT=false   # setting for use in trunk

TIMEOUT=20   # Longest 'thinking time' in seconds allowed before adopting the automatically selected audio stream.

TESTRUN=true    # cutlists will be shown but the recording will be unchanged  
# TESTRUN=false  # the recording will be processed
#################

if [ "$1" = "-h" ] || [ "$1" = "--help" ] ; then
echo "Usage: "$0" <recording>"
echo "<recording> is an mpeg2 file recorded by MythTV with a valid DB entry."
echo "e.g. 1234_20100405123400.mpg in one of the defined RECDIRs"
echo "The output file replaces the input file which is renamed to <recording>.old"
exit 0
fi

# exit if .old file exists

if  [ -f ${RECDIR1}/"$1".old ] ; then 
    echo " ${RECDIR1}/"$1".old exists: giving up." ; exit 1
fi

if  [ -f ${RECDIR2}/"$1".old ] ; then 
    echo " ${RECDIR2}/"$1".old exists: giving up." ; exit 1
fi
 
# Customize with paths to alternative recording and temp folders

cd $RECDIR1
TEMP=$TEMPDIR1
if  [ ! -f "$1" ] ; then
   cd $RECDIR2
   TEMP=$TEMPDIR2
     if  [ ! -f "$1" ] ; then 
       echo " "$1" not found.  Giving up"
       cd ~
       exit 1
     fi
fi 

if [ $# -lt 3 ]
then
   echo "Error: needs three arguments. Running  ffmpeg -i "$1" 2>&1 | grep -C 4 Video " 
   echo  
   ffmpeg -i "$1" 2>&1 | grep -C 4 Video | tee temp$$.txt
   echo

   # Thanks to Christopher Meredith for the basic parsing magic here. 
   VPID=`grep Video  temp$$.txt | head -n1 | cut -f 1,1 -d']' | sed 's+.*\[++g'`
   # It has to be tweaked for multiple audio streams.  This (with head -n1 ) selects the first listed by ffmpeg.
   # You may alternatively wish to select for language, format, etc.   May be channel, programme, user dependent.
   APID=`grep Audio  temp$$.txt | head -n1 | cut -f 1,1 -d']' | sed 's+.*\[++g'`

   echo -e "Choosing the first audio track listed by \" ffmpeg -i \".  It may not be the one you want."
   echo -e "\nThe selected values would be "$VPID" and "$APID".  The track info for these is \n"

   grep "$VPID" temp$$.txt
   grep "$APID" temp$$.txt

   echo -e "\nTo accept these values press \"a\", or wait....\n"  
   echo  "If you want to select other values, or quit to think about it, press another key within $TIMEOUT seconds."

   read -t $TIMEOUT -n 1 RESP
   if  [ $? -gt 128 ] ; then    
       RESP="a"
   fi

   if [ "$RESP" != "a" ] ; then
       echo -e "Quitting: if you want to select the PIDs from the command line its expected form is   \n"
       echo " "$0" 1234_20070927190000.mpg  0xvvv 0xaaa " 
       echo -e "                    filename_in_DB           vPID  aPID \n" 
       cd ~
       exit 1
   fi

   echo -e "Going on: processing with suggested values $VPID  $APID \n"
   grep "$VPID" temp$$.txt
   grep "$APID" temp$$.txt
   echo
else
   VPID="$2"
   APID="$3"
fi

#Now do the actual processing
# chanid and starttime identify the recording in the DB
chanid=`echo "select chanid from recorded where basename=\"$1\";" |
mysql -N -u mythtv  -p$PASSWD mythconverg `

starttime=`echo "select starttime from recorded where basename=\"$1\";" |
mysql -N -u mythtv  -p$PASSWD mythconverg `

# In trunk an initial zero is apparently treated as a normal cut-in point.
# list0 shows cut-in points and eof, but in fixes never includes zero
list0=`echo "select mark from recordedmarkup
where chanid=$chanid and starttime='$starttime' and type=0 order by mark;" |
mysql -N -u mythtv  -p$PASSWD mythconverg `
#list1 shows cut-out points.  In fixes an initial here is a cut-in. 
list1=`echo "select mark from recordedmarkup
where chanid=$chanid and starttime='$starttime' and type=1 order by mark;" |
mysql -N -u mythtv  -p$PASSWD mythconverg `  

echo "CollectionPanel.CutMode=0" > cutlist$$ ;
if  ! $INVERT ; then 
    list=`echo "select mark from recordedmarkup
    where chanid=$chanid and starttime='$starttime' and type in (0,1) order by mark;" |
    mysql -N -u mythtv  -p$PASSWD mythconverg `  
else 
    for i in $list1 ;
      do
        if [ $i = "0" ] 
        then  
           list=`echo "select mark from recordedmarkup
              where chanid=$chanid and starttime='$starttime' and type in (0,1) order by mark;" |
              mysql -N -u mythtv  -p$PASSWD mythconverg | tail -n +2 ` 
              # tail -n +2 drops the initial zero.
        else
           echo "0" >> cutlist$$
           # That isn't quite the same as inserting a leading zero in list.  Does it matter?
           list=`echo "select mark from recordedmarkup
              where chanid=$chanid and starttime='$starttime' and type in (0,1) order by mark;" |
              mysql -N -u mythtv  -p$PASSWD mythconverg `  
       fi
     #  use only the first element of list1, as a switch.
       break
     done
fi 

# find the key frame (mark type 9) right before each cut mark,
# extract the byte offset, write it into the ProjectX cutlist
for i in $list ;
do echo "select offset from recordedseek
  where chanid=$chanid and starttime='$starttime' and type=9 and mark >= $i and mark < ($i + 100)
  order by offset;" |
  mysql -N -u mythtv  -p$PASSWD mythconverg | head -n 1
# for each cycle, head -n 1 yields the first line only.
done >> cutlist$$

echo "list0"
echo $list0
echo
echo "list1"
echo $list1
echo
echo "list"
echo $list
echo

echo -e "\"list\" is MythTV's frame-count cutlist that is used to create the byte-count cutlist used here by Project-X."
echo "At the time of writing (July 2010) the internal cutlists used by fixes and trunk appear to work in opposite senses."
echo "The cut can be inverted by adding or removing a leading zero. "
echo -e "That is what the INVERT variable does.  For fixes it should be set to \"true\" \n"

echo "This is the byte-count cutlist for Project-X. The first value is a cut-in point."
echo -e "Cut-out and cut-in then follow in sequence to EOF. \n"
cat cutlist$$
echo -e "\nThis is a test exit point for you to check that INVERT is correctly set."
echo -e "Its value now is  \"${INVERT}\".  When it's OK, edit the script to set TESTRUN=false.\n"

if $TESTRUN ; then
   rm -f cutlist$$
   rm -f temp$$.txt   
   cd ~
   exit 0
fi

mv  "$1" "$1".old

# use ProjectX to de-multiplex selected streams with the created cutlist
ionice -c3 java -jar "$PROJECTX" -name tempcut$$ -id ${VPID},${APID} -out $TEMP -cut cutlist$$ "$1".old || :

# and pipe for re-multiplexing to mplex. -f 9 is dvd format without navpacks
DEMUXPREF=$TEMP/tempcut${$}
if [ -f $DEMUXPREF.mp2 ] ; then
    DEMUXAUDIO=$DEMUXPREF.mp2
else
    DEMUXAUDIO=$DEMUXPREF.ac3
fi
ionice -c3  mplex -o "$1" -V -f 9 $DEMUXPREF.m2v $DEMUXAUDIO
  
# tell mythDB about new filesize and clear myth cutlist
FILESIZE=`du -b "$1" | cut -f 1`
if [ "${FILESIZE}" -gt 1000000 ]; then
      echo "Running: update recorded set filesize=${FILESIZE} where basename=\"$1\";"
      echo "update recorded set filesize=${FILESIZE} where basename=\"$1\";" | mysql -u mythtv -p$PASSWD mythconverg
      echo "Filesize has been reset"
      echo "Running: ionice -c3 mythcommflag -f "$1" --clearcutlist"
      ionice -c3 mythcommflag -f "$1" --clearcutlist
      echo "Cutlist has been cleared"
fi

#rebuild seek table
echo "Running: ionice -c3 mythtranscode --mpeg2 --buildindex --showprogress --chanid "$chanid" --starttime "$starttime""
ionice -c3 mythtranscode --mpeg2 --buildindex --showprogress --chanid "$chanid" --starttime "$starttime"
echo -e "Seek table has been rebuilt.\n"

echo -e "Output file is $1.  INVERT is set to \"${INVERT}\".  PID streams $VPID and $APID were copied.\n"
if [ -f temp$$.txt ] ; then
     echo -e "Their original parameters were \n"
     grep "$VPID" temp$$.txt
     grep "$APID" temp$$.txt
     cat temp$$.txt >> "$DEMUXPREF"_log.txt
     echo
fi

rm -f "$1".png
#rm -f $TEMP/tempcut${$}*
mv  ${DEMUXPREF}_log.txt ${TEMP}/"$1"_pxlog.txt 
rm -f $DEMUXPREF.m2v
rm -f $DEMUXAUDIO
rm -f cutlist$$
rm -f temp$$.txt
cd ~
exit 0