MythDVBcut
From MythTV Official Wiki
This script is based on one I've been using for several years to process my recordings from dvb-t, and another, less often, for recordings from dvb-t2.
This updated script works for me with 0.25-fixes and 0.26-fixes for both dvb-t and dvb-t2. It provides more info about what it has done and, although restricted to cutting at keyframes, usually cuts where I would like it to. It might well be useful for other formats, at least for removing pre-and-post run.
The user interface is largely unchanged; I use it from the keyboard.
The link between the editor-based cutlist and the cutpoints actually used differs slightly from both the earlier versions. I find that edits at the last/first keyframes that show a wanted picture usually give acceptable results. This may well depend on the editing processes at source.
--Johnp 12:08, 8 March 2013 (UTC)
Soon after this was posted, Michael Stucky reported that the call to mplex failed when the script was used on an HD recording from an HDHR device. He gave a workaround and comments that I reproduce here for convenience:
"Further tests indicate that a trial-and-error process is/maybe required for each HD recording (channel??) to get a clean run of mplex. The only way I have been able to get consistent results is to use mythffmpeg to remux into an mkv container.
mythffmpeg -fflags +genpts -i $DEMUXPREF.m2v -i $DEMUXAUDIO -vcodec copy -acodec copy "$OUTFILE"
Where $OUTFILE is the recording name with an .mkv extension. I move the resulting file to my Video storage group, the file is playable on my MythFEs and all my other media players."
Here's the original thread:
http://www.gossamer-threads.com/lists/mythtv/users/540360#540360 --Johnp 20:52, 10 March 2013 (UTC)
Two 'features' have become apparent since the script was first posted:
MythTV 0.26 filenames are based on UTC, while the Mythfrontend display uses local time. I'm still using mental arithmetic to convert, but users not in the UK may find this more inconvenient.
An unannounced change at my local transmitter makes mythffmpeg find multiple 'Video' streams in some DVB-T channels; this causes an error exit. In the 'grep' lines for stream count and video stream selection replace 'mpeg2video' or 'Video' by 'mpeg2video (Main)'.
--Johnp 07:51, 3 July 2013 (UTC)
I'm posting a new version that can create either 'DVD format' or matroska files after cutting by ProjectX. The choice is made by editing the MAKEMKV Boolean variable in the script. Thanks to Michael Stucky for his suggestions.
http://www.gossamer-threads.com/lists/mythtv/users/546679
--Johnp 10:59, 4 July 2013 (UTC)
I'm still using essentially the same process here, but with some changes for 0.28-pre. Current version attached as mythDVBcut_20150728.sh This is a direct paste of the script as I use it with v0.28-pre-2683-g7f68683 under SL7. Editing for local conditions will be required, as before. I have also removed the version from March 2013.
--Johnp (talk) 12:33, 28 July 2015 (UTC)
Builds of MythTV Master after mid-April 2015 (v0.28-pre-2827) may insert a 'mythical' bookmark at the scheduled programme start. Mythfrontend playback of programmes recorded with pre-roll and cut with MythDVBcut would then start at a point some time after the start-of-new-file. v0.28-pre-3055 introduced 'mythutil --clearbookmarks' that will allow restoration of the expected behaviour. It can be called here immediately after, and in the same way as, the existing 'mythutil --clearcutlist' section below. See Ticket #11713.
--Johnp (talk) 18:51, 25 August 2015 (UTC)
I have added an experimental script, mythTScut20180309.sh, originally intended for cutting h264-encoded recordings but potentially useful for other .ts files. Input and output files are in .ts format. It uses tools from mythffmpeg and does not use mkv tools or Project-X. At present it has options to read single video and audio streams either 'together' or 'independently' (1 or 2). I feel that 2 is likely to be better eventually, but at present it fails on raw recordings with 'interactive tv' streams not recognised by mythffmpeg. Both options usually work after a first pass with option 1 and no cutlist.
Processed recordings usually play without trouble in mythfrontend or via UPnP but the positioning of switchpoints is still slightly imprecise and there may be a/v sync transients after them.
I hope that it will be possible to find ffmpeg options to address these issues.
Updated mythTScut.ts to 20180414. Tweaks to cutpoint positioning. Added options for output as Recording, Video, or Test; reading input as 1 or 2 streams, and NC to ignore cutlist.
Added an updated version of mythTScut_20180622.sh
I have not made any changes to the script since early May 2018 and have made few new HD recordings. This is my 'current' version. Playback over cuts is often better via UPnP than in Mythfrontend. Changes between interlaced and progressive formats may not be smoothly handled.
mythDVBcut_20150728.sh
#!/bin/bash
# Copyright (C) 2014 John Pilkington
# Uses ideas from scripts posted by Tino Keitel and Kees Cook in the Mythtv lists.
# The suggestion to convert HD mpeg2 files to .mkv, and code to do it, came from Michael Stucky.
# **** I have not tried running this script on an .mkv file that it has created *****
# Usage: "$0" <recording> ...or...
# ionice -c3 ./"$0" <recording> will reduce io priority and is recommended.
# <recording> is a file recorded by MythTV with a valid DB entry and seektable,
# e.g. 1234_20100405123400.ts or 1234_20100405123400.mpg
# in one of the RECDIRs defined below.
# The output file replaces the input file in the list of recordings, perhaps with a new suffix.
# The input file is renamed to <recording>.old
# This script is essentially a terminal-based replacement for the 'lossless' mpeg2 mythtranscode.
# It was developed from mythcutprojectx but will now cut some recording formats that defeat Project-X.
# All cuts are made at keyframes.
# Project-X is used here to demux and apply a cutlist, and also to discard all but one video and one audio stream.
# If the audio is initially in .wav format it will be converted to .mp2; mp2 or ac3 will be unchanged.
# For non-mpeg2 video all streams are passed through unchanged, but with cuts applied at the video keyframes.
# The script then clears the cutlist, updates some entries in the database, rebuilds the seek table and creates a new preview.
# If the script is edited to have MAKEMKV=false, the result should be acceptable as a recording within MythTV
# and perhaps as an input to MythArchive. After MAKEMKV=true, IIUC, the file must be treated as a Video.
# The logfile includes the positions in the new file at which deletions have been made.
# The script needs to be edited to define some local variables and folders.
# It will not recognise Storage Groups.
# The Project-X java-based demuxer for mpeg2 video may be packaged for your distro, although perhaps not fully updated.
# A tarball of the most recent version of Project-X can be downloaded from the link near the bottom of this webpage:-
# http://project-x.cvs.sourceforge.net/viewvc/project-x/Project-X/
# I'm running it with java-1.7.0-openjdk under SL7; if you need to build it, install java*jdk-devel and run build.sh
# Some of the comments in this script have been left as reminders of things that might be useful.
####################
#. ~/.mythtv/mysql.txt # for DB access info
#PASSWD=`grep "^DBPassword" ~/.mythtv/mysql.txt | cut -d '=' -f 2-`
#PASSWD=mythtv
# mysql.txt is no longer referenced in 0.26+
# The values are now defined within ~/.mythtv/config.xml but it seems easier to
# get them into the script by editing. They are:
# DBUserName, DBPassword, DBLocalHostName, DBName
# Define the variables used for DB access
#
DBUserName="mythtv"
DBPassword="mythtv"
DBLocalHostName="localhost"
DBName="mythconverg"
BN="$1" # "Basename" of file as given in the command line but without special attributes.
MAKEMKV=false # Do not convert to .mkv. After running Project-X use mplex to create a DVD-profile file.
# Remultiplexing with mplex fails at higher bitrates as found in HD mpeg2, but mkv can cope.
# MAKEMKV=true # after ProjectX create an .mkv file
# If TESTRUN is set to true, cutlists will be shown but the recording will be unchanged.
# TESTRUN=true
TESTRUN=false # the recording will be processed
if [ $TESTRUN != "false" ] ; then TESTRUN=true
fi
#CUTMODE="FRAMECOUNT" # Not used here.
CUTMODE="BYTECOUNT"
USEPJX=true # Can be set to false here if you wish; will be set to false if not mpeg2video (Main)
TIMEOUT=20 # Longest 'thinking time' in seconds allowed before adopting the automatically selected audio stream.
# Variables RECDIR1, TEMPDIR1, RECDIR2, TEMPDIR2, LOGDIR, PROJECTX, HOST1, HOST2, HOST3 need to be customised.
# The TEMPDIRs are used to hold the demuxed recording, so should have a few GiB available.
# Use $HOSTNAME (as given by running 'hostname') to find out which machine is being used and assign working directories.
HOST1="PresV5000" # laptop with single disk, FE/BE
HOST2="gateway12" # twin-disk box, FE/BE
#HOST3="HP-kub" #
HOST3="HP_Box" # another single-disk FE/BE
if [ $HOSTNAME = ${HOST1} ] ; then
RECDIR1=/home/john/Mythrecs
# TEMPDIR1=/home/john/MythTmp
TEMPDIR1=/mnt/VidsOnRoot/tmpPX # Not a good location!
RECDIR2=${RECDIR1}
TEMPDIR2=${TEMPDIR1}
LOGDIR=/home/john/Logs
#PROJECTX=/path/to/ProjectX.jar (or to a link to it)
PROJECTX=~/projectx_link
elif [ $HOSTNAME = ${HOST2} ] ; then
# Section below for twin-disk setup using Project-X.
# RECDIR1 and TEMPDIR1 should if possible be on different drive spindles. Likewise RECDIR2 and TEMPDIR2.
# This will reduce the load on individual disks and disk-controllers in the IO-heavy processing.
RECDIR1=/mnt/f10store/myth/reca
TEMPDIR1=/mnt/sam1/tempb
RECDIR2=/mnt/sam1/recb
TEMPDIR2=/mnt/f10store/myth/tempa
LOGDIR=/home/John/Documents/PXcutlogs
#PROJECTX=/path/to/ProjectX.jar (or to a link to it)
PROJECTX=~/projectx_link
elif [ $HOSTNAME = ${HOST3} ] ; then
RECDIR1=/home/john/SGs/RecsSG1
TEMPDIR1=/home/john/MythTmp
# RECDIR2=${RECDIR1}
RECDIR2=/home/john/SGs/LivetvSG1
TEMPDIR2=${TEMPDIR1}
LOGDIR=/home/john/Logs
#PROJECTX=/path/to/ProjectX.jar (or to a link to it)
PROJECTX=~/projectx_link
else
echo "Hostname $HOSTNAME not recognised."
exit 1
fi
if [ "$BN" = "-h" ] || [ "$BN" = "--help" ] ; then
echo "Usage: "$0" <recording>"
echo "<recording> is a file known to MythTV as a 'recording' with a valid DB entry and a seektable."
echo "e.g. 1234_20100405123400.ts or 1234_20100405123400.mpg "
echo "in one of the RECDIRs defined in the script."
echo "The input file will be renamed to <recording>.old and"
echo "MythTV will see the output file instead."
exit 0
fi
# exit if .old file exists
if [ -f ${RECDIR1}/"$BN".old ] ; then
echo " ${RECDIR1}/"$BN".old exists: giving up." ; exit 1
fi
if [ -f ${RECDIR2}/"$BN".old ] ; then
echo " ${RECDIR2}/"$BN".old exists: giving up." ; exit 1
fi
# Customize with paths to alternative recording and temp folders
cd ${RECDIR1}
RECDIR=${RECDIR1}
TEMP=${TEMPDIR1}
if [ ! -f "$BN" ] ; then
cd ${RECDIR2}
RECDIR=${RECDIR2}
TEMP=${TEMPDIR2}
if [ ! -f "$BN" ] ; then
echo " "$BN" not found. Giving up"
cd ~
exit 1
fi
fi
# Is it an mpeg-2 recording?
echo
echo
#mythffmpeg -i "$BN" 2>&1 | grep -C 4 Video | tee "temp$$.txt" # no longer adequate
#mythffmpeg -i "$BN" 2>&1 | grep -B 2 -A 4 Video | tee "temp$$.txt" # no longer adequate
#mythffmpeg -i "$BN" 2>&1 | grep -B 2 -A 4 "mpeg2video (Main)" | tee "temp$$.txt" # a reminder
# mythffmpeg may fail to identify streams if recording started before, or continued after,
# the period of activity of a part-time channel.
# ffmpeg usually works for me, but
# 'dd bs=1M skip=numblock1 count=numblock2 if=infile of=outfile'
# gives a fallback.
# Specify mythffmpeg here because it should be generally available and usually works.
ffmpeg -i "$BN" 2>&1 | grep -B 2 -A 4 "mpeg2video (Main)" | tee "temp$$.txt"
mpeg=$(grep -c "mpeg2video (Main)" temp$$.txt)
echo "mpeg2video (Main) stream count: $mpeg"
if [ $mpeg != 1 ] ; then
echo "Not mpeg2video (Main), or no or multiple video streams"
if [ $mpeg = 0 ] ; then
USEPJX=false # really ought to see if it's a radio channel.
else
exit 1
fi
fi
if [ $# -lt 3 ]
then
echo "Needs one or three arguments."
echo
cat temp$$.txt
echo
# Examples (BBC FOUR SD, March 2015) :
# $ mythffmpeg -version
# ffmpeg version 2.3.1 Copyright (c) 2000-2014 the FFmpeg developers
# built on Feb 19 2015 23:55:22 with gcc 4.8.2 (GCC) 20140120 (Red Hat 4.8.2-16)
# Duration: 01:05:59.66, start: 29170.850711, bitrate: 3786 kb/s
# Program 1
# Stream #0:0[0x191]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p(tv), 704x576 [SAR 16:11 DAR 16:9], max. 15000 kb/s, 25 fps, 25 tbr, 90k tbn, 50 tbc
# Stream #0:1[0x192](eng): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16p, 254 kb/s
# Stream #0:2[0x196](eng): Audio: mp3 ([3][0][0][0] / 0x0003), 0 channels, s16p (visual impaired)
# Stream #0:3[0x195](eng): Subtitle: dvb_subtitle ([6][0][0][0] / 0x0006)
# Stream #0:4[0x1c2]: Unknown: none ([5][0][0][0] / 0x0005)
# Thanks to Christopher Meredith for the basic parsing magic here.
VPID=`grep "mpeg2video (Main)" temp$$.txt | head -n1 | cut -f 1,1 -d']' | sed 's+.*\[++g'`
# It has to be tweaked for multiple audio streams. This (with head -n1 ) selects the first listed by ffmpeg.
# You may alternatively wish to select for language, format, etc. May be channel, programme, user dependent.
APID=`grep Audio temp$$.txt | head -n1 | cut -f 1,1 -d']' | sed 's+.*\[++g'`
echo -e "Choosing the first audio track listed by \" mythffmpeg -i \". It may not be the one you want."
echo -e "\nThe selected values would be "$VPID" and "$APID". The track info for these is \n"
grep "$VPID" temp$$.txt
grep "$APID" temp$$.txt
echo -e "\nTo accept these values press \"a\", or wait....\n"
echo "If you want to select other values, or to quit and think about it, press another key within $TIMEOUT seconds."
echo -e "If the format is not mpeg2video all streams will be passed unchanged.\n"
read -t $TIMEOUT -n 1 RESP
if [ $? -gt 128 ] ; then
RESP="a"
fi
if [ "$RESP" != "a" ] ; then
echo -e "Quitting: if you want to select the PIDs from the command line its expected form is \n"
echo " "$0" 1234_20070927190000.ts (or .mpg) 0xvvv 0xaaa "
echo -e " filename_in_DB vPID aPID \n"
cd ~
exit 1
fi
echo -e "Going on: processing with suggested values $VPID $APID \n"
grep "$VPID" temp$$.txt
grep "$APID" temp$$.txt
echo
else
VPID="$2"
APID="$3"
fi
########################
# Now do the actual processing
# recordedid is now the prime search key in the DB.
recordedid=$(echo "select recordedid from recorded where basename=\"$BN\";" |
mysql -N -u${DBUserName} -p${DBPassword} -h${DBLocalHostName} $DBName )
# but some tables and utils still need chanid and starttime.
chanid=$(echo "select chanid from recorded where recordedid = '$recordedid' ;" |
mysql -N -u${DBUserName} -p${DBPassword} -h${DBLocalHostName} $DBName )
starttime=$(echo "select starttime from recorded where recordedid = '$recordedid' ;" |
mysql -N -u${DBUserName} -p${DBPassword} -h${DBLocalHostName} $DBName )
#exit
echo -e "\nLogfile listing:\n" > log$$
echo -e "Logfile is log$log$$ \n" | tee -a log$$
echo "chanid ${chanid} starttime ${starttime} recordedid ${recordedid} " >> log$$
starttime=$(echo ${starttime} | tr -d ': -')
echo "Reformatted starttime ${starttime} " >> log$$
command="mythutil --getcutlist --chanid $chanid --starttime $starttime -q"
echo "Running: "${command}"" >> log$$
mythutilcutlist=$($(echo ${command}))
echo "${mythutilcutlist}" |tee -a log$$
shortlist=$(echo ${mythutilcutlist} | sed 's/Cutlist://' )
echo -e "\nIf you want to reset this cutlist, after restoring the .old file
and resetting the DB with mythsetsize, you could try: \n
mythutil --setcutlist ${shortlist} --chanid $chanid --starttime $starttime \n
-but seektables from different tools may differ!!" | tee -a log$$
#exit 0
if [ "${mythutilcutlist}" = "Cutlist: " ] ; then
echo "Cutlist was empty; inserting dummy EOF" >> log$$
mythutilcutlist=" 9999999 "
fi
echo -e "\nCutframe list from editor: " >> log$$
echo "${mythutilcutlist}" | tr -d [:alpha:] | tr [:punct:] " " | tee edlist$$ >> log$$
#cat edlist$$ | tee -a log$$
echo
#
#Reverse the sense of the cutlist
#
echo -n > revedlist$$
for i in $(cat edlist$$) ;
do
if [ $i -eq 0 ] ; then
for j in $(cat edlist$$) ;
do
if [ $j != 0 ] ; then echo -n "$j " >> revedlist$$
fi
done
else
echo -n " 0 " >> revedlist$$
for j in $(cat edlist$$) ;
do
echo -n "$j " >> revedlist$$
done
fi
break
done
echo >> revedlist$$
echo -e "\nPassframe (reversed cutframe) list from editor: " >> log$$
cat revedlist$$ | tee -a log$$
# For a byte-count cutlist, (PX CutMode=0)
# mark is the frame count in the seektable and is compared here with the frame count defined by the cutlist editor.
# mark type 9 is MARK_GOP_BYFRAME (see eg trac ticket #1088) with adjacent values typically separated by 12 or more,
# so that is presumably the cutpoint frame granularity in bytecount mode.
# Subjectively the granularity seems smaller, but this may not apply to locally encoded recordings;
# in dvb-t and similar systems the position of keyframes often appears to depend on pre-transmission edits of content.
#
# Find the next keyframe (mark type 9) after a position slightly before the frame identified by the cutlist editor.
# The original version, which I have been using for years, effectively had lag=0.
# The 'recordedseek' table doesn't know about recordedid.
lag=4 # In frames. Best value might depend on recording source, MythTV version and seektable history.
scope=2000 # Sometimes h264 keyframes in the wild are much more widely spaced than expected.
# This might only have been true for 'rebuilt' seektables, but the large value should do no harm.
for frame in $(cat revedlist$$)
do
i=$((${frame} - ${lag}))
j=$((${i}))
k=$((${i} + ${scope}))
echo "select offset, mark from recordedseek
where chanid=$chanid and starttime='$starttime' and type=9
and mark >= ${j} and mark < ${k} order by offset limit 3 ;" |
mysql -N -u${DBUserName} -p${DBPassword} -h${DBLocalHostName} ${DBName}
done > tmp0$$
echo "Full results of DB read:"
cat tmp0$$
cat tmp0$$ | sed -n '1,${p;n;n;}' > tmp1$$ # select lines 1,4,7...
cat tmp0$$ | sed -n '2,${p;n;n;}' > tmp2$$ # 2,5,8...
cat tmp0$$ | sed -n '3,${p;n;n;}' > tmp3$$
rm tmp0$$
echo
# write the byte offset and frame number into one-line cutlists.
echo -e "\nActive keyframe passlist via DB. First is a cut-in:" >> log$$
cut -f2 tmp1$$ | tr "\n" " " | tee -a log$$ > keylist$$
echo >> keylist$$
echo >> log$$
echo -e "\n2nd keyframes, via DB. For info only." >> log$$
cut -f2 tmp2$$ | tr "\n" " " >> log$$
echo >> log$$
echo -e "\n3rd keyframes, via DB. For info only." >> log$$
cut -f2 tmp3$$ | tr "\n" " " >> log$$
echo >> log$$
echo -e "\nByte offsets of switchpoints in original file, via DB. First is a cut-in:" >> log$$
cut -f1 tmp1$$ | tr "\n" " " >> log$$
echo >> log$$
echo -e "\nByte offsets of 2nd keyframes, via DB. Compare with PjX log." >> log$$
cut -f1 tmp2$$ | tr "\n" " " >> log$$
echo >> log$$
mv tmp1$$ tmp$$ # for use
rm tmp2$$ tmp3$$
# Now apply a one-packet offset to the byte positions passed to Project-X.
# With this 188-byte adjustment in place, the byte-positions that PjX reports
# having used are exactly equal to those held in the myth DB for
# the keyframe that next follows the one selected:- i.e. it seems to report the
# end-of-GOP position when that GOP has been processed.
# Editing may be frame-accurate if edit-points are the first or last keyframes
# for which a wanted picture is displayed.
FILESIZE=$( du -bL "$BN" | cut -f 1 )
echo -e "\nCreating the cutlist for Project-X with 188 byte adjustment" >> log$$
cut -f1 tmp$$ > PXraw$$
for i in $( cat PXraw$$ ) ;
do
echo $(( $i + 188 ))
done > PXadj$$
rm tmp$$
echo "PXraw$$"
cat PXraw$$
echo "EOF"
echo "PXadj$$"
cat PXadj$$
echo "EOF"
# Add a limiting filesize value for use if EOF will be reached
J=$( cat PXraw$$ | wc -l )
echo "PXraw$$ has $J lines"
if [ $(( $J % 2 )) -eq 1 ] ; then
echo "Adding EOF endmark"
echo "$FILESIZE" >> PXraw$$
echo "$FILESIZE" >> PXadj$$
fi
echo "########### Start of recording " | tee -a log$$
hexdump -C -n 40 -s 0 ${1} | tee -a log$$
echo "########### PXraw$$ ### " | tee -a log$$
for i in $( cat PXraw$$ ) ;
do
echo ${i} | tee -a log$$
hexdump -C -n 40 -s $i ${1} | tee -a log$$
done
echo "########## PXadj$$ ###" | tee -a log$$
for i in $( cat PXadj$$ ) ;
do
echo ${i} | tee -a log$$
hexdump -C -n 40 -s $i ${1} | tee -a log$$
done
# Set up Project-X for bytecount mode
echo "CollectionPanel.CutMode=0" > projx$$
cat PXadj$$ >> projx$$
echo
echo >> log$$
cat projx$$ | tee -a log$$
# Don't apply the 188-byte offset when using pyscript
cat PXraw$$ | tr "\n" " " > bytelist$$
echo >> bytelist$$
# Prepare pyscript$$, the script that would run pycut.py
echo -e " #!/bin/bash\n mv '$RECDIR/$BN' '$RECDIR/$BN.old' " > pyscript$$
echo -en " ionice -c3 ~/pycut.py '$RECDIR/$BN.old' '$RECDIR/$BN' " >> pyscript$$
cat bytelist$$ >> pyscript$$
if ${USEPJX} ; then
# Do apply the offset
cat PXadj$$ | tr "\n" " " > bytelist$$
echo >> bytelist$$
fi
rm PXraw$$ PXadj$$
# These calculations work only if no streams or frames have been dropped
# so should probably be 'if ! ${USEPJX} ; then'
if ! ${USEPJX} ; then
echo -e "\nSwitchbyte positions in new file:" >> log$$
J=0
S=0 # 0 or 1 for cut or pass lists
for i in $(cat bytelist$$) ;
do
if [ $S -eq 0 ] ; then
J=$((J - i))
S=1
else
J=$((J + i))
S=0
echo -n "$J " >> log$$
fi
done
echo >> log$$
echo -e "\nSwitchframe positions in new file:" >> log$$
J=0
S=0 # 0 or 1 for cut or pass lists
for i in $(cat keylist$$) ;
do
if [ $S -eq 0 ] ; then
J=$((J - i))
S=1
else
J=$((J + i))
S=0
echo -n "$J " >> log$$
fi
done
echo >> log$$
fi
echo
cat log$$
# create script to be run
echo -e "\nThis is pyscript$$, the script that will be run if TESTRUN is false
and you are not trying to use PjX instead:\n"
cat pyscript$$
chmod +x pyscript$$
echo -e "\nTo run this use ./pyscript$$. \n"
if $TESTRUN ; then
echo "Quitting because TESTRUN is ${TESTRUN}"
rm -f cutlist$$
rm -f temp$$
cd ~
exit 0
fi
###########################
# Now do the actual cutting and concatenation
TEMPHEAD=$TEMP/tempcut${$}
OUTFILEHEAD=$( echo $BN | tr -d [:alpha:] | tr -d [.] )
if ${USEPJX} ; then
#################
# For mpeg2 format only....
# use ProjectX to de-multiplex selected streams with the created cutlist
mv "$BN" "$BN".old
CMD="ionice -c3 java -jar "$PROJECTX" -name tempcut$$ -id ${VPID},${APID} \
-out $TEMP -cut projx$$ "$BN".old"
echo "running: "${CMD}""
${CMD}
#
# if demuxed audio is in .wav format (maybe direct camera output) convert it to mp2
if [ -f $TEMPHEAD.wav ] ; then
twolame $TEMPHEAD.wav
fi
if [ -f $TEMPHEAD.mp2 ] ; then
TEMPAUDIO=$TEMPHEAD.mp2
else
TEMPAUDIO=$TEMPHEAD.ac3
fi
if ! ${MAKEMKV} ; then
# Now remux to MPEG2_PS.
# mplex -f 8, mplex -f 9 both use DVD profile.
# mplex -f 8 inserts blank DVD-nav sectors, -f 9 does not.
#
# This seems to be a 'corner case' infrequent warning from mplex.
# I have never detected a problem during playback but it may make MythArchive fail.
# ++ WARN: [mplex] Stream e0: data will arrive too late sent(SCR)=7314 required(DTS)=7200
# ++ WARN: [mplex] Video e0: buf= 101237 frame=000001 sector=00000050
OUTFILE=$OUTFILEHEAD.mpg
echo "Newfile name will be: $OUTFILE "
CMD="ionice -c3 mplex -o "$OUTFILE" -V -f 9 $TEMPHEAD.m2v $TEMPAUDIO"
echo "running: ${CMD}"
${CMD}
else
# Remux to .mkv
# This does play as a recording via uPnP but won't seek or skip in Mythfrontend.
# OUTFILE=$( echo $BN | sed 's/mpg/mkv/')
OUTFILE=$OUTFILEHEAD.mkv
echo "Newfile name will be: $OUTFILE "
CMD="ionice -c3 mythffmpeg -fflags +genpts -i $TEMPHEAD.m2v -i $TEMPAUDIO -vcodec copy -acodec copy $OUTFILE "
echo "running: ${CMD}"
${CMD}
fi
# tell mythDB about new filename and set the 'transcoded' flag
echo "update recorded set basename='${OUTFILE}', transcoded = 1 where recordedid = '$recordedid' ; " |
mysql -N -u${DBUserName} -p${DBPassword} -h${DBLocalHostName} ${DBName}
# Update the container type, needed for uPnP playback. Empty will work too, if the transcoded flag is set.
# This is a HACK pending 0.28 release, when mythutil should do it. See Ticket #12388
echo "update recordedfile set basename = '${OUTFILE}', container = 'MPEG2-PS' where recordedid = '$recordedid' ;" |
mysql -N -u${DBUserName} -p${DBPassword} -h${DBLocalHostName} ${DBName}
CMD="rm $TEMPHEAD.m2v $TEMPAUDIO " # Large; usually best to remove these.
echo "running: "${CMD}""
${CMD}
################
else
################
# Use pyscript, which isn't restricted to mpeg2; but PjX can repair some mpeg2 defects.
echo "Running : "
cat pyscript$$
echo
echo
./pyscript$$
echo
###############
OUTFILE=$BN
fi
# Cutting completed. Now clean up.
CMD="ionice -c3 mythutil --clearcutlist --chanid "$chanid" --starttime "$starttime" -q "
echo "running: "${CMD}""
${CMD}
echo -e "Cutlist has been cleared.\n"
# Rebuild seek table; this now also resets filesize in DB
# TODO? With MKV output, seektable is probably not useful and may cause problems.
CMD="ionice -c3 mythcommflag -q --rebuild --file $OUTFILE "
echo "running: "${CMD}""
${CMD}
#echo -e "Seek table has been rebuilt.\n"
#echo "The cutlist was applied in ** "$CUTMODE" ** mode."
echo -e "Output file is $OUTFILE. \n"
# Get tech details of output file into the log.
echo -e "\nRunning: mythffmpeg -i "$OUTFILE" 2>&1 | grep -C 4 Video" | tee -a log$$
echo
mythffmpeg -i "$OUTFILE" 2>&1 | grep -C 4 Video | tee -a log$$
echo
CMD="grep -A 2 Switch log$$ " # put non-ProjectX outfile switchpoints onto terminal
echo "running: "${CMD}""
${CMD}
# The PjX-listed switchpoints are usually 2 frames further apart than those calculated above.
# The editor often sees them about 1 GOP earlier. Dropped frames and/or seektable problems?
echo "outfile switchpoints listed by ProjectX"
grep 'cut-in' ${TEMPHEAD}_log.txt
grep '.Video (m2v):' ${TEMPHEAD}_log.txt \
| awk '{print "Overall : " $3, $4," " $5}'
cat log$$ >> ${TEMPHEAD}_log.txt
echo -e "\nWhile the .old file still exists you can examine it with\n
hexdump -C -n 40 -s {byteoffset} ${RECDIR}/$BN.old\n
32-bit hexdump may misinterpret start offsets > 2 GiB.\n "
grep "we have" ${TEMPHEAD}_log.txt # show PX 'errorcount' nearer to end of screen output.
# Values up to a few tens are usually ok, but rare backward jumps in timestamps,
# which may truncate PX output, give much larger numbers.
#
echo
CMD="mv ${TEMPHEAD}_log.txt ${LOGDIR}/${OUTFILE}.PXcut$$.txt"
echo "running: "${CMD}""
${CMD}
rm bytelist$$
rm edlist$$
rm keylist$$
rm revedlist$$
rm pyscript$$
rm projx$$
rm temp$$.txt
rm log$$
rm -f $BN.png # old preview, which may not exist
cd ~
# mythpreviewgen isn't essential here so put it where failure won't cause other problems.
# Creates a blank frame from .mkv file.
CMD="ionice -c3 mythpreviewgen --chanid "$chanid" --starttime "$starttime" -q "
echo "running: "${CMD}""
${CMD}
echo -e "Preview created.\n"
exit 0
mythDVBcut_20130704.sh
#!/bin/bash
# Copyright (C) 2013 John Pilkington
# Uses ideas from scripts posted by Tino Keitel and Kees Cook in the Mythtv lists.
# The suggestion to convert HD files to .mkv, and code to do it, came from Michael Stucky.
# **** I have not tried running this script on an .mkv file that it has created *****
# Usage: "$0" <recording> ...or...
# ionice -c3 ./"$0" <recording> will reduce io priority and is recommended.
# <recording> is a file recorded by MythTV with a valid DB entry and seektable.
# e.g. 1234_20100405123400.mpg in one of the defined RECDIRs
# The output file replaces the input file which is renamed to <recording>.old
# This script is essentially a terminal-based replacement for the 'lossless' mpeg2 mythtranscode.
# It was developed from mythcutprojectx but will now cut some recording formats that defeat Project-X.
# All cuts are made at keyframes.
# Project-X is used here to apply a cutlist, and also to remove all but one video and one audio stream.
# If the audio is initially in .wav format it will be converted to .mp2; mp2 or ac3 will be unchanged.
# For non-mpeg2video all streams are passed through unchanged, but with cuts applied at the video keyframes.
# It then clears the cutlist, updates the filesize in the database, rebuilds the seek table and creates a new preview.
# If the script is edited to have MAKEMKV=false, the result should be acceptable as a recording within MythTV
# and perhaps as an input to MythArchive. After MAKEMKV=true the file must be treated as a Video.
# The logfile includes the positions in the new file at which deletions have been made.
# The script needs to be edited to define some local variables and folders.
# It will not recognise Storage Groups.
# A tarball of the most recent version of Project-X can be downloaded from the link at the bottom of this page:
# http://project-x.cvs.sourceforge.net/viewvc/project-x/Project-X/
# Some of the comments in this script have been left as reminders of things that might be useful.
####################
#. ~/.mythtv/mysql.txt # for DB access info
#PASSWD=`grep "^DBPassword" ~/.mythtv/mysql.txt | cut -d '=' -f 2-`
#PASSWD=mythtv
# mysql.txt is no longer referenced in 0.26+
# The values are now defined within ~/.mythtv/config.xml but it seems easier to
# get them into the script by editing. They are:
# DBUserName, DBPassword, DBLocalHostName, DBName
# Define the variables used for DB access
#
DBUserName="mythtv"
DBPassword="mythtv"
DBLocalHostName="localhost"
DBName="mythconverg"
BN="$1" # "Basename" of file as given in the command line but without special attributes.
MAKEMKV=false # after ProjectX use mplex to create a DVD profile file
# mplex fails with higher bitrates, but mkv can cope.
# MAKEMKV=true # after ProjectX create an .mkv file
# If TESTRUN is set to true, no changes will be made to the input file.
# TESTRUN=true # cutlists will be shown but the recording will be unchanged
TESTRUN=false # the recording will be processed
if [ $TESTRUN != "false" ] ; then TESTRUN=true
fi
#CUTMODE="FRAMECOUNT" # Not used here.
CUTMODE="BYTECOUNT"
USEPJX=true # Will be set to false if not mpeg2video (Main)
TIMEOUT=20 # Longest 'thinking time' in seconds allowed before adopting the automatically selected audio stream.
# Variables RECDIR1, TEMPDIR1, RECDIR2, TEMPDIR2, LOGDIR, PROJECTX, HOST1, HOST2 need to be customised.
# The TEMPDIRs are used to hold the demuxed recording, so should have a few GiB available.
HOST1="PresV5000" # laptop with single disk, SL6, MythTV 0.26-fixes FE/BE
HOST2="gateway12" # twin-disk box, Fedora 17, MythTV 0.26-fixes FE/BE
if [ $HOSTNAME = ${HOST1} ] ; then
RECDIR1=/home/john/Mythrecs
TEMPDIR1=/home/john/MythTmp
RECDIR2=${RECDIR1}
TEMPDIR2=${TEMPDIR1}
LOGDIR=/home/john/Logs
#PROJECTX=/path/to/ProjectX.jar (or to a link to it)
PROJECTX=~/projectx_link
elif [ $HOSTNAME = ${HOST2} ] ; then
# Section below for twin-disk setup using Project-X.
# RECDIR1 and TEMPDIR1 should if possible be on different drive spindles. Likewise RECDIR2 and TEMPDIR2.
RECDIR1=/mnt/f10store/myth/reca
TEMPDIR1=/mnt/sam1/tempb
RECDIR2=/mnt/sam1/recb
TEMPDIR2=/mnt/f10store/myth/tempa
LOGDIR=/home/John/Documents/PXcutlogs
#PROJECTX=/path/to/ProjectX.jar (or to a link to it)
PROJECTX=~/projectx_link
else
echo "Hostname $HOSTNAME not recognised."
exit 1
fi
if [ "$BN" = "-h" ] || [ "$BN" = "--help" ] ; then
echo "Usage: "$0" <recording>"
echo "<recording> is a file recorded by MythTV with a valid DB entry and seektable."
echo "e.g. 1234_20100405123400.mpg in one of the defined RECDIRs"
echo "The output file replaces the input file which is renamed to <recording>.old"
exit 0
fi
# exit if .old file exists
if [ -f ${RECDIR1}/"$BN".old ] ; then
echo " ${RECDIR1}/"$BN".old exists: giving up." ; exit 1
fi
if [ -f ${RECDIR2}/"$BN".old ] ; then
echo " ${RECDIR2}/"$BN".old exists: giving up." ; exit 1
fi
# Customize with paths to alternative recording and temp folders
cd ${RECDIR1}
RECDIR=${RECDIR1}
TEMP=${TEMPDIR1}
if [ ! -f "$BN" ] ; then
cd ${RECDIR2}
RECDIR=${RECDIR2}
TEMP=${TEMPDIR2}
if [ ! -f "$BN" ] ; then
echo " "$BN" not found. Giving up"
cd ~
exit 1
fi
fi
# Is it an mpeg-2 recording?
echo
echo
#mythffmpeg -i "$BN" 2>&1 | grep -C 4 Video | tee "temp$$.txt"
#mythffmpeg -i "$BN" 2>&1 | grep -B 2 -A 4 Video | tee "temp$$.txt"
mythffmpeg -i "$BN" 2>&1 | grep -B 2 -A 4 "mpeg2video (Main)" | tee "temp$$.txt"
mpeg=$(grep -c "mpeg2video (Main)" temp$$.txt)
echo "mpeg2video (Main) stream count: $mpeg"
if [ $mpeg != 1 ] ; then
echo "Not mpeg2video (Main), or no or multiple video streams"
if [ $mpeg = 0 ] ; then
USEPJX=false # really ought to see if it's a radio channel.
else
exit 1
fi
fi
if [ $# -lt 3 ]
then
echo "Needs one or three arguments."
echo
cat temp$$.txt
echo
# Examples
# Stream #0.0[0xe0]: Video: mpeg2video, yuv420p, 720x576 [PAR 64:45 DAR 16:9], 15000 kb/s, 25 fps, 25 tbr, 90k tbn, 50 tbc
# Stream #0.1[0xc0](deu): Audio: mp2, 48000 Hz, 2 channels, s16, 256 kb/s
# Thanks to Christopher Meredith for the basic parsing magic here.
VPID=`grep "mpeg2video (Main)" temp$$.txt | head -n1 | cut -f 1,1 -d']' | sed 's+.*\[++g'`
# It has to be tweaked for multiple audio streams. This (with head -n1 ) selects the first listed by ffmpeg.
# You may alternatively wish to select for language, format, etc. May be channel, programme, user dependent.
APID=`grep Audio temp$$.txt | head -n1 | cut -f 1,1 -d']' | sed 's+.*\[++g'`
echo -e "Choosing the first audio track listed by \" mythffmpeg -i \". It may not be the one you want."
echo -e "\nThe selected values would be "$VPID" and "$APID". The track info for these is \n"
grep "$VPID" temp$$.txt
grep "$APID" temp$$.txt
echo -e "\nTo accept these values press \"a\", or wait....\n"
echo "If you want to select other values, or quit to think about it, press another key within $TIMEOUT seconds."
echo -e "If the format is not mpeg2video all streams will be passed unchanged.\n"
read -t $TIMEOUT -n 1 RESP
if [ $? -gt 128 ] ; then
RESP="a"
fi
if [ "$RESP" != "a" ] ; then
echo -e "Quitting: if you want to select the PIDs from the command line its expected form is \n"
echo " "$0" 1234_20070927190000.mpg 0xvvv 0xaaa "
echo -e " filename_in_DB vPID aPID \n"
cd ~
exit 1
fi
echo -e "Going on: processing with suggested values $VPID $APID \n"
grep "$VPID" temp$$.txt
grep "$APID" temp$$.txt
echo
else
VPID="$2"
APID="$3"
fi
# Now do the actual processing
# chanid and starttime identify the recording in the DB
chanid=$(echo "select chanid from recorded where basename=\"$BN\";" |
mysql -N -u${DBUserName} -p${DBPassword} -h${DBLocalHostName} $DBName )
starttime=$(echo "select starttime from recorded where basename=\"$BN\";" |
mysql -N -u${DBUserName} -p${DBPassword} -h${DBLocalHostName} $DBName )
#exit
echo -e "\nLogfile listing:\n" > log$$
echo -e "Logfile is log$log$$ \n" | tee -a log$$
echo "chanid ${chanid} starttime ${starttime} " >> log$$
starttime=$(echo ${starttime} | tr -d ': -')
echo "Reformatted starttime ${starttime} " >> log$$
command="mythutil --getcutlist --chanid $chanid --starttime $starttime -q"
echo "Running: "${command}"" >> log$$
mythutilcutlist=$($(echo ${command}))
echo "${mythutilcutlist}" |tee -a log$$
shortlist=$(echo ${mythutilcutlist} | sed 's/Cutlist://' )
echo -e "\nIf you want to reset this cutlist, after restoring the .old file
and resetting the DB with mythsetsize, you could try: \n
mythutil --setcutlist ${shortlist} --chanid $chanid --starttime $starttime \n
-but seektables from different tools may differ!!" | tee -a log$$
#exit 0
if [ "${mythutilcutlist}" = "Cutlist: " ] ; then
echo "Cutlist was empty; inserting dummy EOF" >> log$$
mythutilcutlist=" 9999999 "
fi
echo -e "\nCutframe list from editor: " >> log$$
echo "${mythutilcutlist}" | tr -d [:alpha:] | tr [:punct:] " " | tee edlist$$ >> log$$
#cat edlist$$ | tee -a log$$
echo
#
#Reverse the sense of the cutlist
#
echo -n > revedlist$$
for i in $(cat edlist$$) ;
do
if [ $i -eq 0 ] ; then
for j in $(cat edlist$$) ;
do
if [ $j != 0 ] ; then echo -n "$j " >> revedlist$$
fi
done
else
echo -n " 0 " >> revedlist$$
for j in $(cat edlist$$) ;
do
echo -n "$j " >> revedlist$$
done
fi
break
done
echo >> revedlist$$
echo -e "\nPassframe (reversed cutframe) list from editor: " >> log$$
cat revedlist$$ | tee -a log$$
# For a byte-count cutlist, (PX CutMode=0)
# mark is the frame count in the seektable and is compared here with the editpoint
# mark type 9 is MARK_GOP_BYFRAME (see eg trac ticket #1088) with adjacent values typically separated by 12 or more,
# so that is presumably the cutpoint frame granularity in bytecount mode.
# Subjectively the granularity seems smaller, but this may not apply to locally encoded recordings;
# in dvb-t and similar systems the spacing of keyframes apparently depends on pre-transmission edits of content.
#
# Find the next keyframe (mark type 9) after a position slightly before the cutmark found by the editor.
# The original version, which I have been using for years, effectively had lag=0.
lag=4 # In frames. Best value might depend on recording source, MythTV version and seektable history.
scope=2000 # Sometimes h264 keyframes in the wild are much more widely spaced than expected.
for frame in $(cat revedlist$$)
do
i=$((${frame} - ${lag}))
j=$((${i}))
k=$((${i} + ${scope}))
echo "select offset, mark from recordedseek
where chanid=$chanid and starttime='$starttime' and type=9
and mark >= ${j} and mark < ${k} order by offset limit 3 ;" |
mysql -N -u${DBUserName} -p${DBPassword} -h${DBLocalHostName} ${DBName}
done > tmp0$$
echo "Full results of DB read:"
cat tmp0$$
cat tmp0$$ | sed -n '1,${p;n;n;}' > tmp1$$ # select lines 1,4,7...
cat tmp0$$ | sed -n '2,${p;n;n;}' > tmp2$$
cat tmp0$$ | sed -n '3,${p;n;n;}' > tmp3$$
rm tmp0$$
echo
# write the byte offset and frame number into one-line cutlists.
echo -e "\nActive keyframe passlist via DB. First is a cut-in:" >> log$$
cut -f2 tmp1$$ | tr "\n" " " | tee -a log$$ > keylist$$
echo >> keylist$$
echo >> log$$
echo -e "\n2nd keyframes, via DB. For info only." >> log$$
cut -f2 tmp2$$ | tr "\n" " " >> log$$
echo >> log$$
echo -e "\n3rd keyframes, via DB. For info only." >> log$$
cut -f2 tmp3$$ | tr "\n" " " >> log$$
echo >> log$$
echo -e "\nByte offsets of switchpoints in original file, via DB. First is a cut-in:" >> log$$
cut -f1 tmp1$$ | tr "\n" " " >> log$$
echo >> log$$
echo -e "\nByte offsets of 2nd keyframes, via DB. Compare with PjX log." >> log$$
cut -f1 tmp2$$ | tr "\n" " " >> log$$
echo >> log$$
mv tmp1$$ tmp$$ # for use
rm tmp2$$ tmp3$$
# With this one-packet-length adjustment in place, the byte-positions that PjX reports
# having used are exactly equal to those held in the myth DB for
# the keyframe that next follows the one selected:- i.e. it seems to report the
# end-of-GOP position when that GOP has been processed.
# Editing may be frame-accurate if edit-points are the first or last keyframes
# for which a wanted picture is displayed.
FILESIZE=$( du -bL "$BN" | cut -f 1 )
echo -e "\nCreating the cutlist for Project-X with 188 byte adjustment" >> log$$
cut -f1 tmp$$ > PXraw$$
for i in $( cat PXraw$$ ) ;
do
echo $(( $i + 188 ))
done > PXadj$$
rm tmp$$
echo "PXraw$$"
cat PXraw$$
echo "EOF"
echo "PXadj$$"
cat PXadj$$
echo "EOF"
# Add a limiting filesize value for use if EOF will be reached
J=$( cat PXraw$$ | wc -l )
echo "PXraw$$ has $J lines"
if [ $(( $J % 2 )) -eq 1 ] ; then
echo "Adding EOF endmark"
echo "$FILESIZE" >> PXraw$$
echo "$FILESIZE" >> PXadj$$
fi
echo "########### Start of recording " | tee -a log$$
hexdump -C -n 40 -s 0 ${1} | tee -a log$$
echo "########### PXraw$$ ### " | tee -a log$$
for i in $( cat PXraw$$ ) ;
do
echo ${i} | tee -a log$$
hexdump -C -n 40 -s $i ${1} | tee -a log$$
done
echo "########## PXadj$$ ###" | tee -a log$$
for i in $( cat PXadj$$ ) ;
do
echo ${i} | tee -a log$$
hexdump -C -n 40 -s $i ${1} | tee -a log$$
done
echo "CollectionPanel.CutMode=0" > projx$$
cat PXadj$$ >> projx$$
echo
echo >> log$$
cat projx$$ | tee -a log$$
cat PXadj$$ | tr "\n" " " > bytelist$$
echo >> bytelist$$
# Prepare pyscript$$, the script that would run pycut.py
echo -e " #!/bin/bash\n mv '$RECDIR/$BN' '$RECDIR/$BN.old' " > pyscript$$
echo -en " ionice -c3 ~/pycut.py '$RECDIR/$BN.old' '$RECDIR/$BN' " >> pyscript$$
cat bytelist$$ >> pyscript$$
rm PXraw$$ PXadj$$
echo -e "\nSwitchbyte positions in new file:" >> log$$
J=0
S=0 # 0 or 1 for cut or pass lists
for i in $(cat bytelist$$) ;
do
if [ $S -eq 0 ] ; then
J=$((J - i))
S=1
else
J=$((J + i))
S=0
echo -n "$J " >> log$$
fi
done
echo >> log$$
echo -e "\nSwitchframe positions in new file:" >> log$$
J=0
S=0 # 0 or 1 for cut or pass lists
for i in $(cat keylist$$) ;
do
if [ $S -eq 0 ] ; then
J=$((J - i))
S=1
else
J=$((J + i))
S=0
echo -n "$J " >> log$$
fi
done
echo >> log$$
echo
cat log$$
# create script to be run
echo -e "\nThis is pyscript$$, the script that will be run if TESTRUN is false
and you are not trying to use PjX instead:\n"
cat pyscript$$
chmod +x pyscript$$
echo -e "\nTo run this use ./pyscript$$. \n"
if $TESTRUN ; then
echo "Quitting because TESTRUN is ${TESTRUN}"
rm -f cutlist$$
rm -f temp$$
cd ~
exit 0
fi
# Now do the actual cutting and concatenation
TEMPHEAD=$TEMP/tempcut${$}
OUTFILE="$BN"
if ${USEPJX} ; then
#################
# For mpeg2 format only....
# use ProjectX to de-multiplex selected streams with the created cutlist
mv "$BN" "$BN".old
CMD="ionice -c3 java -jar "$PROJECTX" -name tempcut$$ -id ${VPID},${APID} \
-out $TEMP -cut projx$$ "$BN".old"
echo "running: "${CMD}""
${CMD}
#
# if demuxed audio is in .wav format (maybe direct camera output) convert it to mp2
if [ -f $TEMPHEAD.wav ] ; then
twolame $TEMPHEAD.wav
fi
if [ -f $TEMPHEAD.mp2 ] ; then
TEMPAUDIO=$TEMPHEAD.mp2
else
TEMPAUDIO=$TEMPHEAD.ac3
fi
if ! ${MAKEMKV} ; then
# Now remux. mplex -f 8, mplex -f 9 both use DVD profile.
# mplex -f 8 inserts blank DVD-nav sectors, -f 9 does not.
# Neither plays as a 'Recording' via uPnP on my Panasonic TV.
CMD="ionice -c3 mplex -o "$OUTFILE" -V -f 9 $TEMPHEAD.m2v $TEMPAUDIO"
echo "running: ${CMD}"
${CMD}
else
# Remux to .mkv
# This does play as a recording via uPnP but won't seek or skip in Mythfrontend.
OUTFILE=$( echo $BN | sed 's/mpg/mkv/')
echo "Newfile name will be: $OUTFILE "
CMD="ionice -c3 mythffmpeg -fflags +genpts -i $TEMPHEAD.m2v -i $TEMPAUDIO -vcodec copy -acodec copy $OUTFILE "
echo "running: ${CMD}"
${CMD}
# tell mythDB about new filename
echo "update recorded set basename='${OUTFILE}' where chanid=$chanid and starttime='$starttime';" |
mysql -N -u${DBUserName} -p${DBPassword} -h${DBLocalHostName} ${DBName}
fi
CMD="rm $TEMPHEAD.m2v $TEMPAUDIO " # Large; usually best to remove these.
echo "running: "${CMD}""
${CMD}
################
else
################
# Use pyscript, which isn't restricted to mpeg2; but PjX can repair some mpeg2 defects.
echo "Running : "
cat pyscript$$
echo
echo
./pyscript$$
echo
###############
fi
# Cutting completed. Now clean up.
CMD="ionice -c3 mythutil --clearcutlist --chanid "$chanid" --starttime "$starttime" -q "
echo "running: "${CMD}""
${CMD}
echo -e "Cutlist has been cleared.\n"
#rebuild seek table; this now also resets filesize in DB
CMD="ionice -c3 mythcommflag -q --rebuild --file $OUTFILE "
echo "running: "${CMD}""
${CMD}
#echo -e "Seek table has been rebuilt.\n"
#echo "The cutlist was applied in ** "$CUTMODE" ** mode."
echo -e "Output file is $OUTFILE. \n"
# Get tech details of output file into the log.
echo -e "\nRunning: mythffmpeg -i "$OUTFILE" 2>&1 | grep -C 4 Video" | tee -a log$$
echo
mythffmpeg -i "$OUTFILE" 2>&1 | grep -C 4 Video | tee -a log$$
echo
CMD="grep -A 2 Switch log$$ " # put outfile switchpoint info onto terminal
echo "running: "${CMD}""
${CMD}
# The PjX-listed switchpoints are usually 2 frames further apart than those calculated above.
# The editor often sees them about 1 GOP earlier. Dropped frames and/or seektable problems?
echo "outfile switchpoints listed by PjX"
grep 'cut-' ${TEMPHEAD}_log.txt
cat log$$ >> ${TEMPHEAD}_log.txt
echo -e "\nWhile the .old file still exists you can examine it with\n
hexdump -C -n 40 -s {byteoffset} ${RECDIR}/$BN.old\n
On my 32-bit installation, hexdump misinterprets start offsets > 2 GiB.\n "
CMD="mv ${TEMPHEAD}_log.txt ${LOGDIR}/$OUTFILE.PXcut$$.txt"
echo "running: "${CMD}""
${CMD}
rm bytelist$$
rm edlist$$
rm keylist$$
rm revedlist$$
rm pyscript$$
rm projx$$
rm temp$$.txt
rm log$$
cd ~
# mythpreviewgen isn't essential here so put it where failure won't cause other problems.
# Creates a blank frame from .mkv file.
CMD="ionice -c3 mythpreviewgen --chanid "$chanid" --starttime "$starttime" -q "
echo "running: "${CMD}""
${CMD}
echo -e "Preview created.\n"
exit 0
pycut_20200626.py
#! /usr/bin/env python
# -*- coding: utf-8 -*-
## This version for python3 was converted under Fedora 31 by
## "2to3 -w pycut.py" from the version I had been using in Fedora 30.
# Cut out and concatenate sections of a file
# access as pycut.py from mythDVBcut.sh
import sys, os
#print sys.argv
######################
## For tests
##
## echo "0123456789A123456789B123456789C123456789D123456789E123456789F123456789" > ~/test.txt
##
## fn1 = './test.txt'
## fn2 = './temp.txt'
## chunks = [ 3, 12, 35, 47, 53, 68 ]
## buflen = 5
## Doesn't recognise '~/test.txt' but this, or full path, seems ok
## python pycut.py './test.txt' './temp.txt' 3 12 35 47 53 68
# Zambesi HD
#./printcutlist /home/john/Mythrecs/1054_20120328222600.mpg
# Generates byte-mode cutlist for use with Project-X - and here
#CollectionPanel.CutMode=0
#fn1 = '/mnt/f10store/myth/reca/1054_20120323002600old.mpg'
#fn2 = '/mnt/sam1/recb/1054_20120323002600.mpg'
#chunks = [ 390284804, 4556742872 ]
#buflen = 1024*1024
#
########################
fn1 = sys.argv[1] # input file
fn2 = sys.argv[2] # output file
chunks = list(map( int, sys.argv [ 3 : ] )) # start and end bytes of chunks in infile
buflen = 1024*1024
#bignum = 10000000000 # for use as EOF if needed
# less likely to be surprised if we use the actual filesize here
print("infile ", fn1)
print("outfile ", fn2)
print("switchpoints ", chunks)
#######################
# sanity checks
chunklen = len(chunks)
if chunklen != 2 * ( chunklen / 2 ) :
# chunks.append(bignum)
chunks.append( 1 + os.path.getsize(fn1))
chunklen = len(chunks)
# adjust chunk-endpoints in the hope of keeping chain linkage in the data intact
n = 1
while n < chunklen :
chunks[n] += -1
n += 2
n=0
while n < chunklen - 2 :
if chunks[n] > chunks[n+1] :
print("Quitting: switchpoints out of order")
sys.exit(98)
n += 1
print("Adjusted switchpoints ", chunks)
n = 0
m = 0
offset = [ 0 ]
while n < chunklen - 1 :
m += 1 + chunks[ n+1 ] - chunks[ n ]
offset.append( m )
n += 2
print()
print("Byte offsets of cutpoints in output file: ", offset)
##print "DB table is recordedseek, mark (framecount) is type 9."
##################################
# Don't touch stuff below here
## byte numbering starts at 0 and output includes both chunk-endpoints
i=0
j=0
imax = 32 # buffers per star
jmax = 32 # stars per line
print() # for progress display
chnklim = len(chunks) - 1
nchnk = 0
chstart=chunks[nchnk]
chend=chunks[nchnk + 1]
bufstart = 0
f1 = open(fn1, 'rb')
f2 = open(fn2, 'wb')
while True :
data = f1.read(buflen)
lendat = len(data)
if lendat == 0 :
break
bufend = bufstart + lendat
while chstart < bufend :
if chend < bufend :
f2.write(data[chstart - bufstart : chend - bufstart + 1 ])
nchnk += 2
if nchnk > chnklim : # job done
chstart = bufend + buflen*2 # kill further looping
break
chstart = chunks[nchnk]
chend = chunks[nchnk + 1]
else :
f2.write(data[chstart - bufstart : ])
chstart = bufend
bufstart += lendat
i += 1 # progress display
if i > imax :
sys.stdout.write("*")
sys.stdout.flush()
i = 0
j += 1
if j > jmax :
print()
j = 0
f1.close()
f2.close()
print()
pycut_20130305.py
#! /usr/bin/env python
# -*- coding: utf-8 -*-
# Cut out and concatenate sections of a file
# access as pycut.py from mythDVBcut.sh
import sys, os
#print sys.argv
######################
## For tests
##
## echo "0123456789A123456789B123456789C123456789D123456789E123456789F123456789" > ~/test.txt
##
## fn1 = './test.txt'
## fn2 = './temp.txt'
## chunks = [ 3, 12, 35, 47, 53, 68 ]
## buflen = 5
## Doesn't recognise '~/test.txt' but this, or full path, seems ok
## python pycut.py './test.txt' './temp.txt' 3 12 35 47 53 68
# Zambesi HD
#./printcutlist /home/john/Mythrecs/1054_20120328222600.mpg
# Generates byte-mode cutlist for use with Project-X - and here
#CollectionPanel.CutMode=0
#fn1 = '/mnt/f10store/myth/reca/1054_20120323002600old.mpg'
#fn2 = '/mnt/sam1/recb/1054_20120323002600.mpg'
#chunks = [ 390284804, 4556742872 ]
#buflen = 1024*1024
#
########################
fn1 = sys.argv[1] # input file
fn2 = sys.argv[2] # output file
chunks = map( int, sys.argv [ 3 : ] ) # start and end bytes of chunks in infile
buflen = 1024*1024
#bignum = 10000000000 # for use as EOF if needed
# less likely to be surprised if we use the actual filesize here
print "infile ", fn1
print "outfile ", fn2
print "switchpoints ", chunks
#######################
# sanity checks
chunklen = len(chunks)
if chunklen != 2 * ( chunklen / 2 ) :
# chunks.append(bignum)
chunks.append( 1 + os.path.getsize(fn1))
chunklen = len(chunks)
# adjust chunk-endpoints in the hope of keeping chain linkage in the data intact
n = 1
while n < chunklen :
chunks[n] += -1
n += 2
n=0
while n < chunklen - 2 :
if chunks[n] > chunks[n+1] :
print "Quitting: switchpoints out of order"
sys.exit(98)
n += 1
print "Adjusted switchpoints ", chunks
n = 0
m = 0
offset = [ 0 ]
while n < chunklen - 1 :
m += 1 + chunks[ n+1 ] - chunks[ n ]
offset.append( m )
n += 2
print
print "Byte offsets of cutpoints in output file: ", offset
print "DB table is recordedseek, mark (framecount) is type 9."
##################################
# Don't touch stuff below here
## byte numbering starts at 0 and output includes both chunk-endpoints
i=0
j=0
imax = 40 # buffers per star
jmax = 25 # stars per line
print # for progress display
chnklim = len(chunks) - 1
nchnk = 0
chstart=chunks[nchnk]
chend=chunks[nchnk + 1]
bufstart = 0
f1 = open(fn1, 'rb')
f2 = open(fn2, 'wb')
while True :
data = f1.read(buflen)
lendat = len(data)
if lendat == 0 :
break
bufend = bufstart + lendat
while chstart < bufend :
if chend < bufend :
f2.write(data[chstart - bufstart : chend - bufstart + 1 ])
nchnk += 2
if nchnk > chnklim : # job done
chstart = bufend + buflen*2 # kill further looping
break
chstart = chunks[nchnk]
chend = chunks[nchnk + 1]
else :
f2.write(data[chstart - bufstart : ])
chstart = bufend
bufstart += lendat
i += 1 # progress display
if i > imax :
sys.stdout.write("*")
sys.stdout.flush()
i = 0
j += 1
if j > jmax :
print
j = 0
f1.close()
f2.close()
print
mythTScut_20180414.sh
#!/bin/bash
set -e # exit on error. I have only become aware of these options recently, and they have not been used.
# set -x # display command lines
# set -ex # both of the above
# by John Pilkington, developed using ideas from
# https://www.mythtv.org/wiki/H264_commercial_remover_and_remuxer
# by Ian Thiele (icthiele@gmail.com) and from my own
# https://www.mythtv.org/wiki/MythDVBcut
#
# This script is offered as work-in-progress. No guarantees given and no redress or support implied.
#
# Working directories and database access must be edited (in the first page) to suit your installation.
# Other parameters scattered through the script may not be optimum.
#
# This version uses tools from mythffmpeg, which is usually part of a MythTV installation.
# It was originally intended for cutting h264-encoded recordings but may also work with other TS formats.
#
# It does not use mkvmerge or Project-X. The output can either:
#
# replace the original recording, which is renamed. Cutlist and bookmarks are cleared and the seektable rebuilt
# or
# be written to the specified 'video' directory. For playback by mythfrontend it will need to be 'scanned in' and preferably given a seektable.
#
# Suggested work sequence is:
# Preliminary test: ~/mythTScut.sh /path/to/infile T 1
# which will read the database and process a cutlist but should not touch the recording.
#
# ionice -c3 ~/mythTScut.sh /path/to/infile R 1 NC
# It's best to establish the cutlist here, or at least to confirm its accuracy.
# ionice -c3 ~/mythTScut.sh /path/to/infile V 1
# scan for new videos from the mythfrontend 'Watch Videos' screen
# ionice -c3 mythutil --rebuild --video /path/to/the/video/outfile
#
# and you should then have before-and-after versions available for comparison. The logfile has the cut positions.
#
# Cuts are made at video keyframes near to positions taken from a list typically created or adjusted by the interactive cutlist editor.
# In the UK the spacing of keyframes is usually around 1 second, but their positions often coincide with edit-points at source.
# I have the impression that the keyframe positions used by MythTV do not exactly coincide with those found and used by ffmpeg tools:
# (The editor often finds a keyframe around 7 frames after a scene change)
# I usually place edits at the first or last keyframe for which the editor shows a wanted image.
# You may prefer to move ends-of-cut a few seconds early to allow for better lipsync recovery.
# Usage: /path/$0 /path/to/recording_known_to_MythTV_database (R or V or T)outmode (1 or 2)streamreadmode (NC don't cut)
# Use $HOSTNAME (as given by running 'hostname') to find out which machine is being used and assign working directories.
HOST1="HP_Fed" # f26
HOST2="HP_Box" # sl7
if [ $HOSTNAME = ${HOST1} ] ; then
VIDDIR="/mnt/sdb1/Vids"
WORKDIR="/mnt/sdb1/TScut_work"
LOGDIR="/mnt/sdb1/TScut_logs"
elif [ $HOSTNAME = ${HOST2} ] ; then
VIDDIR="/mnt/dat1/VSG2"
WORKDIR="/mnt/dat1/TScut_work"
LOGDIR="/mnt/dat1/TScut_logs"
else
echo "Hostname $HOSTNAME not recognised."
exit 1
fi
OUTDIR=${VIDDIR}
chunkdir=${WORKDIR}
#RECDIR is obtained here from the commandline
FFCALL="ionice -c3 mythffmpeg -hide_banner -ignore_unknown "
# keyline for connection to DB; copy entries from /etc/mysql.xml
#mysqlconnect="mysql -N -h$DBHostName -u$DBUserName -p$DBPassword $DBName"
mysqlconnect="mysql -N -hlocalhost -umythtv -pmythtv mythconverg"
export mysqlconnect
TIMEOUT=20 # sec: pause for interactive confirmation
TESTRUN=false
echo "WORKDIR is ${WORKDIR}"
echo "LOGDIR is ${LOGDIR}"
cd ${WORKDIR}
INFILE=$1
if [ $2 = R ] ; then
tovideo=false
elif [ $2 = V ] ; then
tovideo=true
elif [ $2 = T ] ; then
TESTRUN=true
else
echo "Second parameter needed, must be R, V, T"
echo "R will move the infile to $1.old, replace it and clear its cutlist, bookmarks etc."
echo "V will just create a new file in $VIDDIR"
echo "T will Test and exit without making changes."
exit 1
fi
# Third parameter on the command line:
# Either read video and audio streams 'independently'
# Or extend the duration of the input segment instead
if [ $3 -eq 1 ] ; then
TWINREAD=false
skewms=0 # empirically based on a few tests. Is there a standard value?
elif [ $3 -eq 2 ] ; then
TWINREAD=true
skewms=0 # 1600 Not currently used in the code?
else
echo " Third parameter needed, must be 1 or 2; read video and audio as 1 or 2 streams."
echo " 1 seems more robust and usually allows repeat runs; 2 has given bad sync on repeats."
exit 1
fi
nocut=false
if [ $# -ge 4 ] ; then
if [ $4 = NC ] ; then
nocut=true
else
echo " Fourth parameter NC will ignore an existing cutlist and work on the entire file."
exit 1
fi
fi
#let's make sure we have a sane environment
if [ -z "`which mythffmpeg`" ]; then
echo "mythffmpeg not present in the path. Adjust environment or install mythffmpeg"
exit 1
fi
if [ -z "`ls ${WORKDIR}`" ]; then
mkdir -p ${WORKDIR}
fi
if [ ! -f ${INFILE} ]; then
echo " ${INFILE} doesn't exist, aborting."
exit 1
fi
if [ -f ${INFILE}.old ] ; then
echo " ${INFILE}.old exists: giving up."
exit 1
fi
echo -e "\n Working on ${INFILE} \n"
#determine directory and filename
RECDIR=`dirname ${INFILE}`
BASENAME=`basename ${INFILE}`
bnhead=$(basename $BASENAME .ts )
logfile=${LOGDIR}/${bnhead}_$$.log
>${logfile}
echo "Processing ${RECDIR}/${BASENAME} run number $$ " | tee -a ${logfile}
# recordedid is the prime search key in recent versions of MythTV
query="select recordedid from recorded where basename=\"$BASENAME\";"
echo ${query} | tee -a ${logfile}
recordedid=$(echo ${query} | $mysqlconnect)
echo -e "\n recordedid: ${recordedid} \n " | tee -a ${logfile}
# but some tables and utils still need chanid and starttime.
query="select chanid from recorded where recordedid = '$recordedid' ;"
echo ${query} | tee -a ${logfile}
chanid=$(echo ${query} | $mysqlconnect)
echo -e "\n chanid: ${chanid} \n " | tee -a ${logfile}
query="select starttime from recorded where recordedid = '$recordedid' ;"
echo ${query} | tee -a ${logfile}
starttime=$(echo ${query} | $mysqlconnect)
echo -e "\n starttime: ${starttime} \n " | tee -a ${logfile}
if [ -z "$chanid" ] || [ -z "$starttime" ]
then
echo "Recording not found in MythTV database, script aborted."
exit 1
fi
# some myth tools don't like the original format of starttime
starttime=$(echo ${starttime} | tr -d ': -')
echo -e "\n reformatted starttime: ${starttime} \n " | tee -a ${logfile}
echo -e "\n chanid ${chanid} starttime ${starttime} recordedid ${recordedid} \n" | tee -a ${logfile}
#query="select data from recordedmarkup where chanid=$chanid and starttime='$starttime' and type=33 ; "
#echo ${query} | tee -a ${logfile}
#totaldurationms=$(echo ${query} | $mysqlconnect)
# use a nominal value rather than expecting this to be set
totaldurationms=36000000
echo -e "\n totaldurationms: ${totaldurationms} \n " | tee -a ${logfile}
#query="select data from recordedmarkup where chanid=$chanid and starttime='$starttime' and type=34 ; "
#echo ${query} | tee -a ${logfile}
#totalframes=$(echo ${query} | $mysqlconnect)
totalframes=$(( totaldurationms / 40 ))
echo -e "\n totalframes: ${totalframes} \n " | tee -a ${logfile}
CMD="mythutil --getcutlist --chanid $chanid --starttime $starttime -q"
echo " Running: "${CMD}"" | tee -a ${logfile}
mythutilcutlist=$($(echo ${CMD}))
echo "${mythutilcutlist}" | tee -a ${logfile}
#exit
shortlist=$(echo ${mythutilcutlist} | sed 's/Cutlist://' )
echo "${shortlist}" | tee -a ${logfile}
echo -e "\n If you want to reset this cutlist, after restoring the .old file
and rebuilding its seektable, you could try: \n
mythutil --setcutlist ${shortlist} --chanid $chanid --starttime $starttime \n
-but seektables from different tools may differ!!\n" | tee -a ${logfile}
if $nocut ; then
echo -e "\n You have elected to IGNORE the cutlist. \n" | tee -a ${logfile}
fi
echo -e "\nTo continue, press \"a\", or wait....\n"
echo "If you want to select other values, or to quit and think about it, press another key within $TIMEOUT seconds."
echo -e "If you go on, one video and one audio stream will be retained.\n"
read -t $TIMEOUT -n 1 RESP
if [ $? -gt 128 ] ; then
RESP="a"
fi
if [ "$RESP" != "a" ] ; then
echo -e "Quitting"
cd ~
exit 1
fi
if [ "${shortlist}" = "" ] ; then
echo -e "\n Cutlist was empty: working on the complete file \n" | tee -a ${logfile}
nocut=true
fi
if $nocut ; then
shortlist=""
echo -e " 0 $totaldurationms " | tee switchpointsms$$ | tee -a ${logfile}
echo -e " 0 $totalframes " | tee keylist$$ | tee - ${logfile} # dummies
else
echo -e "\n Cutframe list as supplied: " | tee -a ${logfile}
echo "${shortlist}" | tr -d [:alpha:] | tr [:punct:] " " | tee edlist$$ | tee -a ${logfile}
echo
#exit
#Reverse the sense of the cutlist; carried over from mythcutprojectX
#
echo -n > revedlist$$
for i in $(cat edlist$$) ;
do
if [ $i -eq 0 ] ; then
for j in $(cat edlist$$) ;
do
if [ $j != 0 ] ; then echo -n "$j " >> revedlist$$
fi
done
else
echo -n " 0 " >> revedlist$$
for j in $(cat edlist$$) ;
do
echo -n "$j " >> revedlist$$
done
fi
break
done
echo >> revedlist$$
echo -e "\n Passframe (reversed cutframe) list from editor: " | tee -a ${logfile}
cat revedlist$$ | tee -a ${logfile}
# For a byte-count cutlist, (ProjectX CutMode=0)
# mark is the frame count in the seektable and is compared here with the frame count defined by the cutlist editor.
# mark type 9 is MARK_GOP_BYFRAME (see eg trac ticket #1088) with adjacent values typically separated by 12 or more,
# so that is presumably the cutpoint frame granularity in bytecount mode.
# Subjectively the granularity seems smaller, but this may not apply to locally encoded recordings;
# In dvb-t and similar systems the position of keyframes often appears to depend on pre-transmission edits of content.
#
# Find the next keyframe (mark type 9) after a position slightly before the frame identified by the cutlist editor.
# The original version, which I used originally, for several years, effectively had lag=0.
# The 'recordedseek' table doesn't know about recordedid.
#### This version, for use with ffmpeg tools, uses 'type=33' to get times from the start of recording in ms ####
# In HD recordings from UK DVB-T2 the MythTV editor finds a typical keyframe separation of about 24 frames
# A change of scene is usually followed by an editor keyframe about 7 frames later
lead=44 # In frames. Best value might depend on recording source, MythTV version and seektable history.
scope=200 # Sometimes h264 keyframes in the wild (or from oldversiond of mythcommflag) are much more widely spaced than expected.
#
# This might only have been true for buggy 'rebuilt' seektables, now fixed, but the large value should do no harm.
go=0 # go=0 indicates the start of a playing segment. go=1 indicates a segment endpoint
for frame in $(cat revedlist$$)
do
i=$((${frame} ))
if [ $i = 0 ] ; then
echo -e "0 \t0 "
else
j=$((${i} + ${lead} ))
k=$((${j} - ${scope}))
if [ $k -lt 0 ]; then k=0
fi
query="select offset, mark from recordedseek where chanid=$chanid and starttime='$starttime' and type=33
and mark >= ${k} and mark < ${j} order by offset desc limit ${go}, 1 ;"
echo ${query} | $mysqlconnect
fi
if [ ${go} = 1 ]; then
go=0
else
go=1
fi
done > tmp1$$
echo "Full results of DB read:" | tee -a ${logfile}
cat tmp1$$ | tee -a ${logfile}
echo | tee -a ${logfile}
# write the byte offset and frame number into one-line cutlists.
echo -e "\n Active keyframe passlist via DB. First is a cut-in:\n" | tee -a ${logfile}
cut -f2 tmp1$$ | tr "\n" " " | tee keylist$$ | tee -a ${logfile}
echo -e "\n Millisecond switchpoints in original file, via DB. First is a cut-in:\n" | tee -a ${logfile}
cut -f1 tmp1$$ | tr "\n" " " | tee mslist$$ | tee -a ${logfile}
# echo -e "\nByte offsets of 2nd keyframes, via DB. Compare with PjX log." >> log$$
# cut -f1 tmp2$$ | tr "\n" " " | tee -a ${logfile}
mv tmp1$$ tmp$$ # for use. The discarded values were collected during development.
echo -e "\n\n Creating the cutlist" | tee -a ${logfile}
cut -f1 tmp$$ | tee -a ${logfile} > PXraw$$
rm tmp$$
# The PXraw and EOF below are intended to reveal unexpected extra lines from non-printing field separators
echo "PXraw$$"
cat PXraw$$
echo "EOF"
# Add a limiting filesize value for use if EOF will be reached
J=$( cat PXraw$$ | wc -l )
echo -e "\nPXraw$$ has $J lines" | tee -a ${logfile}
if [ $(( $J % 2 )) -eq 1 ] ; then
echo " an odd number: Adding EOF endmark" | tee -a ${logfile}
echo "$totaldurationms" >> PXraw$$ | tee -a ${logfile}
fi
echo -e "\n ########### hexdump of start of recording \n " | tee -a ${logfile}
CMD="hexdump -C -n 80 -s 0 ${1} "
echo ${CMD} | tee -a ${logfile}
# ${CMD}
echo -e "\n getting byte positions of switchpoints in original file - for possible examination only\n" | tee -a ${logfile}
for frame in $(cat keylist$$)
do
query="select offset from recordedseek where chanid=$chanid and starttime='$starttime' and mark = $frame and type=9 ; "
# echo ${query} | tee -a ${logfile}
# echo
bytepos=$(echo ${query} | $mysqlconnect )
# echo $bytepos
CMD="hexdump -C -n 40 -s $bytepos ${1} "
echo ${CMD} | tee -a ${logfile}
echo
# ${CMD}
done
echo -e "\n Millisecond switchpoints to be used \n" | tee -a ${logfile}
cat PXraw$$ | tr "\n" " " | tee switchpointsms$$ | tee -a ${logfile}
rm PXraw$$
# reentry point if thecutlist was blank
fi
echo
echo "SWPTSMS"
cat switchpointsms$$
echo
echo "KEYLIST"
cat keylist$$
ms2HMSf()
{ # convert $1, time interval in ms to hh:mm:ss.xxxx HMSf is global
local s=$(( $1 / 1000 )) ; local h=$(( ( $s / 3600 ) % 24 )) ; h=$(printf "%02d" $h )
local m=$(( ( $s / 60 ) % 60 )) ; m=$(printf "%02d" $m )
s=$( echo "scale=4; $(( $1 % 60000 )) / 1000.0 " | bc -l ) ; s=$(printf "%07.4f\n" $s )
HMSf=" $h":"$m":"$s "
}
#exit
# These calculations work only if no frames have been dropped
echo -e "\n\n Millisecond positions of joins in new file:\n" | tee -a ${logfile}
J=0
S=0 # 0 or 1 for cut or pass lists
for i in $(cat switchpointsms$$) ;
do
if [ $S -eq 0 ] ; then
J=$((J - i))
S=1
else
J=$((J + i))
S=0
ms2HMSf $J ; echo " $HMSf $J ms" | tee -a ${logfile}
fi
done
echo | tee -a ${logfile}
echo -e "\n Frame positions of joins in new file:\n" | tee -a ${logfile}
J=0
S=0 # 0 or 1 for cut or pass lists
for i in $(cat keylist$$) ;
do
if [ $S -eq 0 ] ; then
J=$((J - i))
S=1
else
J=$((J + i))
S=0
echo -n "$J " | tee -a ${logfile}
fi
done
echo | tee -a ${logfile}
switchlist="$(cat switchpointsms$$)"
echo -e "\n Millisec switchlist for action: ${switchlist} \n" | tee -a ${logfile}
if $TESTRUN ; then
echo -e "\n Quitting because TESTRUN is ${TESTRUN}"
rm -f cutlist$$
rm -f temp$$
cd ~
exit 0
fi
# save the input file and replace it as a 'recording' with the product of the merge
# It seems best not to meddle with filename extensions, in case ffmpeg uses them internally
if [ ! $tovideo ] ; then
OUTDIR=${RECDIR}
else
OUTDIR=${VIDDIR}
fi
OUTFILE=${OUTDIR}/${bnhead}_$$.ts
swarray=( ${switchlist} )
switchcount=${#swarray[@]}
echo -e "\n Number of switchpoints passed to cutter: $switchcount \n" | tee -a ${logfile}
if [ $switchcount -lt 3 ] ; then # only one segment, no need to concat
doconcat=false
chunkdir=${OUTDIR}
else
doconcat=true
chunkdir=${WORKDIR}
fi
chunkhead=${chunkdir}/${bnhead}_$$
clipcount=0 # initialise tag for file segments
#exit
ms2sf()
{ # input $1 millisec
# output sf, seconds sssss.xxxx formatted with leading zero for ffmpeg; global
local s
s=$(echo "scale=4; $1 / 1000.0 " | bc -l )
sf=$(printf "%.4f\n" $s )
}
i=0
until [ $i -ge $switchcount ]
do
#Real keyframe is around 7 frames before the editor value; go back abit further. Make duration half a frame longer.
startms="${swarray[$i]}" ; startms=$(( $startms - 20 )) # don't land on the fence
endms="${swarray[$i + 1]}" ; endms=$(( $endms + 20 )) # 6 frames early
ms2sf $(( $startms )) ; startsecf=$sf
ms2sf $(( $endms - $startms )) ; duration2f=$sf
ms2sf $(( $endms - $startms + $skewms )) ; duration1f=$sf
ms2sf $(( $skewms )) ; skewsecf=$sf
clipcount=$(( ++clipcount ))
###### https://trac.ffmpeg.org/wiki/Seeking
# CMD="ffmpeg -hide_banner-ss $startsecf -i ${RECDIR}/${bnhead} -t $durationf -map 0:0 -map 0:1 -c:v copy -c:a copy \
# -avoid_negative_ts 1 ${chunkhead}_${clipcount}.ts"
# It's not clear what options might be best here. Non-obvious interactions are common.
# https://superuser.com/questions/702645/ffmpeg-sync-when-extracting-parts-of-video
if ! ${TWINREAD} ; then
# selected by $3 -eq 1
##### duration extended here to allow for skew between video and audio streams
echo -e "\n skew applied in non-TWINREAD mode: $skewms ms \n" | tee -a ${logfile}
# read using video-frame timestamps only (?)
CMD="$FFCALL \
-ss $startsecf -noaccurate_seek -t $duration1f -i ${RECDIR}/${BASENAME} \
-map 0:0 -c:v copy \
-map 0:1 -c:a copy \
-avoid_negative_ts 1 ${chunkhead}_${clipcount}.ts"
else
# selected by $3 -eq 2
echo -e "\n skew applied once in TWINREAD mode: $skewms ms \n" | tee -a ${logfile}
# read each stream using its own timestamps (?). -ignore_unknown here is a per-etream repeat.
CMD="$FFCALL -ss $startsecf -t $duration2f -i ${RECDIR}/${BASENAME} \
-ignore_unknown -ss $startsecf -t $duration2f -i ${RECDIR}/${BASENAME} \
-map 0:0 -c:v copy \
-map 1:1 -c:a copy -copyts \
-avoid_negative_ts 1 ${chunkhead}_${clipcount}.ts"
fi
echo -e "${CMD} \n " | tee -a ${logfile}
${CMD} | tee -a ${logfile}
i=$(( $i + 2 ))
done
#exit
# ##### file segments have been created. Now prepare to merge them ########
###### https://trac.ffmpeg.org/wiki/Concatenate
if $doconcat ; then
mergestr=""
for i in $(ls ${chunkhead}_* | sort)
do
if [ -z "$mergestr" ]; then
mergestr="concat:$i"
continue
fi
mergestr="$mergestr|$i"
done
CMD="$FFCALL -i $mergestr -c copy ${OUTFILE}"
echo -e "${CMD} \n " | tee -a ${logfile}
${CMD} | tee -a ${logfile}
else
mv ${OUTDIR}/${bnhead}_$$_1.ts ${OUTFILE}
fi
# Here the outfile name is $bnhead_$$.ts, while the input was $bnhead.ts
#if [ $tovideo ] ; then
# This needs myth to be aware of the new video. Skip for now
# CMD="ionice -c3 mythcommflag -q --video ${OUTFILE}"
#fi
if ! $tovideo ; then
# move the original out of the way, replace it, and update the DB
mv ${INFILE} ${INFILE}_$$in.ts
mv ${OUTFILE} ${INFILE}
OUTFILE=${INFILE}
# Rebuilding the seek table also resets filesize in the DB
CMD="ionice -c3 mythcommflag -q --rebuild --file ${OUTFILE} "
echo "running: "${CMD}"" | tee -a ${logfile}
${CMD} # | tee -a ${logfile}
CMD="ionice -c3 mythutil --clearcutlist --chanid "$chanid" --starttime "$starttime" -q "
echo "running: "${CMD}"" | tee -a ${logfile}
${CMD} | tee -a ${logfile}
echo -e "Cutlist has been cleared.\n"
# This command was introduced at 0.28-pre-3055. See Ticket #11713
CMD="ionice -c3 mythutil --clearbookmarks --chanid "$chanid" --starttime "$starttime" -q "
echo "running: "${CMD}"" | tee -a ${logfile}
${CMD} | tee -a ${logfile}
echo -e "Bookmarks have been cleared.\n"
CMD="ionice -c3 mythpreviewgen --chanid "$chanid" --starttime "$starttime" -q "
echo "running: "${CMD}"" | tee -a ${logfile}
${CMD}
echo -e "Preview created.\n"
fi
echo -e "Output file is $OUTFILE. \n"
echo -e "\nWhile the .old file still exists you can examine it with\n
hexdump -C -n 40 -s {byteoffset} ${INFILE}_$$in.ts\n
32-bit hexdump may misinterpret start offsets > 2 GiB.\n "
# Get tech details of output file into the log.
CMD=" mythffprobe -hide_banner -v error -show_streams -show_format -of json $OUTFILE "
echo "${CMD}" | tee -a ${logfile}
${CMD} 2>&1 | tee -a ${logfile}
echo
rm edlist$$
rm keylist$$
rm revedlist$$
rm mslist$$
rm switchpointsms$$
# and if it all worked these may no longer be needed
# rm ${WORKDIR}/$bnhead_$$*
# rm ${INFILE}.old
cd ~
exit 0
mythTScut_20180622.sh
#!/bin/bash
set -e # exit on error. I have only become aware of these options recently, and they have not been used.
# set -x # display command lines
# set -ex # both of the above
# by John Pilkington, developed using ideas from
# https://www.mythtv.org/wiki/H264_commercial_remover_and_remuxer
# by Ian Thiele (icthiele@gmail.com) and from my own
# https://www.mythtv.org/wiki/MythDVBcut
#
# This script is offered as work-in-progress. No guarantees given and no redress or support implied.
#
# Working directories and database access must be edited (in the first page) to suit your installation.
# Other parameters scattered through the script may not be optimum.
#
# This version uses tools from mythffmpeg, which is usually part of a MythTV installation.
# It was originally intended for cutting h264-encoded recordings but may also work with other TS formats.
#
# It does not use mkvmerge or Project-X. The output can either:
#
# replace the original recording, which is renamed. Cutlist and bookmarks are cleared and the seektable rebuilt
# or
# be written to the specified 'video' directory. For playback by mythfrontend it will need to be 'scanned in' and preferably given a seektable.
#
# Suggested work sequence is:
# Preliminary test: ~/mythTScut.sh /path/to/infile T 1
# which will read the database and process a cutlist but should not touch the recording.
#
# ionice -c3 ~/mythTScut.sh /path/to/infile R 1 NC
# It's best to establish the cutlist here, or at least to confirm its accuracy.
# ionice -c3 ~/mythTScut.sh /path/to/infile V 1
# scan for new videos from the mythfrontend 'Watch Videos' screen
# ionice -c3 mythutil --rebuild --video /path/to/the/video/outfile
#
# and you should then have before-and-after versions available for comparison. The logfile has the cut positions.
#
# Cuts are made at video keyframes near to positions taken from a list typically created or adjusted by the interactive cutlist editor.
# In the UK the spacing of keyframes is usually around 1 second, but their positions often coincide with edit-points at source.
# I have the impression that the keyframe positions used by MythTV do not exactly coincide with those found and used by ffmpeg tools:
# (The editor often finds a keyframe around 7 frames after a scene change)
# I usually place edits at the first or last keyframe for which the editor shows a wanted image.
# You may prefer to move ends-of-cut a few seconds early to allow for better lipsync recovery.
# Usage: /path/$0 /path/to/recording_known_to_MythTV_database (R or V or T)outmode (1 or 2)streamreadmode (NC don't cut)
# Use $HOSTNAME (as given by running 'hostname') to find out which machine is being used and assign working directories.
HOST1="HP_Fed" # f26
HOST2="HP_Box" # sl7
if [ $HOSTNAME = ${HOST1} ] ; then
VIDDIR="/mnt/sdb1/Vids"
WORKDIR="/mnt/sdb1/TScut_work"
LOGDIR="/mnt/sdb1/TScut_logs"
elif [ $HOSTNAME = ${HOST2} ] ; then
VIDDIR="/mnt/dat1/VSG2"
WORKDIR="/mnt/dat1/TScut_work"
LOGDIR="/mnt/dat1/TScut_logs"
else
echo "Hostname $HOSTNAME not recognised."
exit 1
fi
OUTDIR=${VIDDIR}
chunkdir=${WORKDIR}
#RECDIR is obtained here from the commandline
FFCALL="ionice -c3 mythffmpeg -hide_banner -ignore_unknown "
#FFCALL="ionice -c3 mythffmpeg -hide_banner -ignore_unknown "
# keyline for connection to DB; copy entries from /etc/mysql.xml
#mysqlconnect="mysql -N -h$DBHostName -u$DBUserName -p$DBPassword $DBName"
mysqlconnect="mysql -N -hlocalhost -umythtv -pmythtv mythconverg"
export mysqlconnect
TIMEOUT=20 # sec: pause for interactive confirmation
TESTRUN=false
echo "WORKDIR is ${WORKDIR}"
echo "LOGDIR is ${LOGDIR}"
cd ${WORKDIR}
INFILE=$1
if [ $2 = R ] ; then
tovideo=false
elif [ $2 = V ] ; then
tovideo=true
elif [ $2 = T ] ; then
TESTRUN=true
else
echo "Second parameter needed, must be R, V, T"
echo "R will move the infile to $1.old, replace it and clear its cutlist, bookmarks etc."
echo "V will just create a new file in $VIDDIR"
echo "T will Test and exit without making changes."
exit 1
fi
# Third parameter on the command line:
# Either read video and audio streams 'independently'
# Or extend the duration of the input segment instead
if [ $3 -eq 1 ] ; then
TWINREAD=false
skewms=0 # empirically based on a few tests. Is there a standard value?
elif [ $3 -eq 2 ] ; then
TWINREAD=true
skewms=600 # kludge hoping to reduce a/v transient error after a cut # 1600 Not currently used in the code?
else
echo " Third parameter needed, must be 1 or 2; read video and audio as 1 or 2 streams."
echo " 1 seems more robust and usually allows repeat runs; 2 has given bad sync on repeats."
exit 1
fi
nocut=false
if [ $# -ge 4 ] ; then
if [ $4 = NC ] ; then
nocut=true
else
echo " Fourth parameter NC will ignore an existing cutlist and work on the entire file."
exit 1
fi
fi
#let's make sure we have a sane environment
if [ -z "`which mythffmpeg`" ]; then
echo "mythffmpeg not present in the path. Adjust environment or install mythffmpeg"
exit 1
fi
if [ -z "`ls ${WORKDIR}`" ]; then
mkdir -p ${WORKDIR}
fi
if [ ! -f ${INFILE} ]; then
echo " ${INFILE} doesn't exist, aborting."
exit 1
fi
if [ -f ${INFILE}.old ] ; then
echo " ${INFILE}.old exists: giving up."
exit 1
fi
echo -e "\n Working on ${INFILE} \n"
#determine directory and filename
RECDIR=`dirname ${INFILE}`
BASENAME=`basename ${INFILE}`
bnhead=$(basename $BASENAME .ts )
logfile=${LOGDIR}/${bnhead}_$$.log
>${logfile}
echo "Processing ${RECDIR}/${BASENAME} run number $$ " | tee -a ${logfile}
# recordedid is the prime search key in recent versions of MythTV
query="select recordedid from recorded where basename=\"$BASENAME\";"
echo ${query} | tee -a ${logfile}
recordedid=$(echo ${query} | $mysqlconnect)
echo -e "\n recordedid: ${recordedid} \n " | tee -a ${logfile}
# but some tables and utils still need chanid and starttime.
query="select chanid from recorded where recordedid = '$recordedid' ;"
echo ${query} | tee -a ${logfile}
chanid=$(echo ${query} | $mysqlconnect)
echo -e "\n chanid: ${chanid} \n " | tee -a ${logfile}
query="select starttime from recorded where recordedid = '$recordedid' ;"
echo ${query} | tee -a ${logfile}
starttime=$(echo ${query} | $mysqlconnect)
echo -e "\n starttime: ${starttime} \n " | tee -a ${logfile}
if [ -z "$chanid" ] || [ -z "$starttime" ]
then
echo "Recording not found in MythTV database, script aborted."
exit 1
fi
# some myth tools don't like the original format of starttime
starttime=$(echo ${starttime} | tr -d ': -')
echo -e "\n reformatted starttime: ${starttime} \n " | tee -a ${logfile}
echo -e "\n chanid ${chanid} starttime ${starttime} recordedid ${recordedid} \n" | tee -a ${logfile}
query="select data from recordedmarkup where chanid=$chanid and starttime='$starttime' and type=33 ; "
echo ${query} | tee -a ${logfile}
totaldurationms=$(echo ${query} | $mysqlconnect)
# use a nominal value rather than expecting this to be set
#totaldurationms=36000000
echo -e "\n totaldurationms: ${totaldurationms} \n " | tee -a ${logfile}
query="select data from recordedmarkup where chanid=$chanid and starttime='$starttime' and type=34 ; "
echo ${query} | tee -a ${logfile}
totalframes=$(echo ${query} | $mysqlconnect)
#totalframes=$(( totaldurationms / 40 ))
echo -e "\n totalframes: ${totalframes} \n " | tee -a ${logfile}
CMD="mythutil --getcutlist --chanid $chanid --starttime $starttime -q"
echo " Running: "${CMD}"" | tee -a ${logfile}
mythutilcutlist=$($(echo ${CMD}))
echo "${mythutilcutlist}" | tee -a ${logfile}
#exit
shortlist=$(echo ${mythutilcutlist} | sed 's/Cutlist://' )
echo "${shortlist}" | tee -a ${logfile}
echo -e "\n If you want to reset this cutlist, after restoring the .old file
and rebuilding its seektable, you could try: \n
mythutil --setcutlist ${shortlist} --chanid $chanid --starttime $starttime \n
-but seektables from different tools may differ!!\n" | tee -a ${logfile}
if $nocut ; then
echo -e "\n You have elected to IGNORE the cutlist. \n" | tee -a ${logfile}
fi
echo -e "\nTo continue, press \"a\", or wait....\n"
echo "If you want to select other values, or to quit and think about it, press another key within $TIMEOUT seconds."
echo -e "If you go on, one video and one audio stream will be retained.\n"
read -t $TIMEOUT -n 1 RESP
if [ $? -gt 128 ] ; then
RESP="a"
fi
if [ "$RESP" != "a" ] ; then
echo -e "Quitting"
cd ~
exit 1
fi
if [ "${shortlist}" = "" ] ; then
echo -e "\n Cutlist was empty: working on the complete file \n" | tee -a ${logfile}
nocut=true
fi
if $nocut ; then
shortlist=""
echo -e " 0 $totaldurationms " | tee switchpointsms$$ | tee -a ${logfile}
echo -e " 0 $totalframes " | tee keylist$$ | tee - ${logfile} # dummies
else
echo -e "\n Cutframe list as supplied: " | tee -a ${logfile}
echo "${shortlist}" | tr -d [:alpha:] | tr [:punct:] " " | tee edlist$$ | tee -a ${logfile}
echo
#exit
#Reverse the sense of the cutlist; carried over from mythcutprojectX
#
echo -n > revedlist$$
for i in $(cat edlist$$) ;
do
if [ $i -eq 0 ] ; then
for j in $(cat edlist$$) ;
do
if [ $j != 0 ] ; then echo -n "$j " >> revedlist$$
fi
done
else
echo -n " 0 " >> revedlist$$
for j in $(cat edlist$$) ;
do
echo -n "$j " >> revedlist$$
done
fi
break
done
echo >> revedlist$$
echo -e "\n Passframe (reversed cutframe) list from editor: " | tee -a ${logfile}
cat revedlist$$ | tee -a ${logfile}
# For a byte-count cutlist, (ProjectX CutMode=0)
# mark is the frame count in the seektable and is compared here with the frame count defined by the cutlist editor.
# mark type 9 is MARK_GOP_BYFRAME (see eg trac ticket #1088) with adjacent values typically separated by 12 or more,
# so that is presumably the cutpoint frame granularity in bytecount mode.
# Subjectively the granularity seems smaller, but this may not apply to locally encoded recordings;
# In dvb-t and similar systems the position of keyframes often appears to depend on pre-transmission edits of content.
#
# Find the next keyframe (mark type 9) after a position slightly before the frame identified by the cutlist editor.
# The original version, which I used originally, for several years, effectively had lag=0.
# The 'recordedseek' table doesn't know about recordedid.
#### This version, for use with ffmpeg tools, uses 'type=33' to get times from the start of recording in ms ####
# In HD recordings from UK DVB-T2 the MythTV editor finds a typical keyframe separation of about 24 frames
# A change of scene is usually followed by an editor keyframe about 7 frames later
lead=44 # In frames. Best value might depend on recording source, MythTV version and seektable history.
scope=200 # Sometimes h264 keyframes in the wild (or from oldversiond of mythcommflag) are much more widely spaced than expected.
#
# This might only have been true for buggy 'rebuilt' seektables, now fixed, but the large value should do no harm.
go=0 # go=0 indicates the start of a playing segment. go=1 indicates a segment endpoint
for frame in $(cat revedlist$$)
do
i=$((${frame} ))
if [ $i = 0 ] ; then
echo -e "0 \t0 "
else
j=$((${i} + ${lead} ))
k=$((${j} - ${scope}))
if [ $k -lt 0 ]; then k=0
fi
query="select offset, mark from recordedseek where chanid=$chanid and starttime='$starttime' and type=33
and mark >= ${k} and mark < ${j} order by offset desc limit ${go}, 1 ;"
echo ${query} | $mysqlconnect
fi
if [ ${go} = 2 ]; then
go=0
else
go=2
fi
done > tmp1$$
echo "Full results of DB read:" | tee -a ${logfile}
cat tmp1$$ | tee -a ${logfile}
echo | tee -a ${logfile}
# write the byte offset and frame number into one-line cutlists.
echo -e "\n Active keyframe passlist via DB. First is a cut-in:\n" | tee -a ${logfile}
cut -f2 tmp1$$ | tr "\n" " " | tee keylist$$ | tee -a ${logfile}
echo -e "\n Millisecond switchpoints in original file, via DB. First is a cut-in:\n" | tee -a ${logfile}
cut -f1 tmp1$$ | tr "\n" " " | tee mslist$$ | tee -a ${logfile}
# echo -e "\nByte offsets of 2nd keyframes, via DB. Compare with PjX log." >> log$$
# cut -f1 tmp2$$ | tr "\n" " " | tee -a ${logfile}
mv tmp1$$ tmp$$ # for use. The discarded values were collected during development.
echo -e "\n\n Creating the cutlist" | tee -a ${logfile}
cut -f1 tmp$$ | tee -a ${logfile} > PXraw$$
rm tmp$$
# The PXraw and EOF below are intended to reveal unexpected extra lines from non-printing field separators
echo "PXraw$$"
cat PXraw$$
echo "EOF"
# Add a limiting filesize value for use if EOF will be reached
J=$( cat PXraw$$ | wc -l )
echo -e "\nPXraw$$ has $J lines" | tee -a ${logfile}
if [ $(( $J % 2 )) -eq 1 ] ; then
echo " an odd number: Adding EOF endmark" | tee -a ${logfile}
echo "$totaldurationms" >> PXraw$$ | tee -a ${logfile}
fi
echo -e "\n ########### hexdump of start of recording \n " | tee -a ${logfile}
CMD="hexdump -C -n 80 -s 0 ${1} "
echo ${CMD} | tee -a ${logfile}
# ${CMD}
echo -e "\n getting byte positions of switchpoints in original file - for possible examination only\n" | tee -a ${logfile}
for frame in $(cat keylist$$)
do
query="select offset from recordedseek where chanid=$chanid and starttime='$starttime' and mark = $frame and type=9 ; "
# echo ${query} | tee -a ${logfile}
# echo
bytepos=$(echo ${query} | $mysqlconnect )
# echo $bytepos
CMD="hexdump -C -n 40 -s $bytepos ${1} "
echo ${CMD} | tee -a ${logfile}
echo
# ${CMD}
done
echo -e "\n Millisecond switchpoints to be used \n" | tee -a ${logfile}
cat PXraw$$ | tr "\n" " " | tee switchpointsms$$ | tee -a ${logfile}
rm PXraw$$
# reentry point if thecutlist was blank
fi
echo
echo "SWPTSMS"
cat switchpointsms$$
echo
echo "KEYLIST"
cat keylist$$
ms2HMSf()
{ # convert $1, time interval in ms to hh:mm:ss.xxxx HMSf is global
local s=$(( $1 / 1000 )) ; local h=$(( ( $s / 3600 ) % 24 )) ; h=$(printf "%02d" $h )
local m=$(( ( $s / 60 ) % 60 )) ; m=$(printf "%02d" $m )
s=$( echo "scale=4; $(( $1 % 60000 )) / 1000.0 " | bc -l ) ; s=$(printf "%07.4f\n" $s )
HMSf=" $h":"$m":"$s "
}
#exit
# These calculations work only if no frames have been dropped
echo -e "\n\n Millisecond positions of joins in new file:\n" | tee -a ${logfile}
J=0
S=0 # 0 or 1 for cut or pass lists
for i in $(cat switchpointsms$$) ;
do
if [ $S -eq 0 ] ; then
J=$((J - i))
S=1
else
J=$((J + i))
S=0
ms2HMSf $J ; echo " $HMSf $J ms" | tee -a ${logfile}
fi
done
echo | tee -a ${logfile}
echo -e "\n Frame positions of joins in new file:\n" | tee -a ${logfile}
J=0
S=0 # 0 or 1 for cut or pass lists
for i in $(cat keylist$$) ;
do
if [ $S -eq 0 ] ; then
J=$((J - i))
S=1
else
J=$((J + i))
S=0
echo -n "$J " | tee -a ${logfile}
fi
done
echo | tee -a ${logfile}
switchlist="$(cat switchpointsms$$)"
echo -e "\n Millisec switchlist for action: ${switchlist} \n" | tee -a ${logfile}
if $TESTRUN ; then
echo -e "\n Quitting because TESTRUN is ${TESTRUN}"
rm -f cutlist$$
rm -f temp$$
cd ~
exit 0
fi
# save the input file and replace it as a 'recording' with the product of the merge
# It seems best not to meddle with filename extensions, in case ffmpeg uses them internally
if [ ! $tovideo ] ; then
OUTDIR=${RECDIR}
else
OUTDIR=${VIDDIR}
fi
OUTFILE=${OUTDIR}/${bnhead}_$$.ts
swarray=( ${switchlist} )
switchcount=${#swarray[@]}
echo -e "\n Number of switchpoints passed to cutter: $switchcount \n" | tee -a ${logfile}
chunkdir=${WORKDIR}
chunkhead=${chunkdir}/${bnhead}_$$
clipcount=0 # initialise tag for file segments
#exit
ms2sf()
{ # input $1 millisec
# output sf, seconds sssss.xxxx formatted with leading zero for ffmpeg; global
local s
s=$(echo "scale=4; $1 / 1000.0 " | bc -l )
sf=$(printf "%.4f\n" $s )
}
i=0
until [ $i -ge $switchcount ]
do
#Real keyframe is around 7 frames before the editor value; go back abit further. Make duration half a frame longer.
startms="${swarray[$i]}" # ; startms=$(( $startms )) # don't land on the fence
endms="${swarray[$i + 1]}" # ; endms=$(( $endms )) # 6 frames early
if [ $(( $startms + 2000 )) -gt $totaldurationms ] ; then # kludge: ignore call for a short trailing segment
break # startms=$totlqdurationms
fi
if [ $endms -gt $totaldurationms ] ; then
endms=$totaldurationms
fi
ms2sf $(( $startms + $skewms )) ; startsecf=$sf
ms2sf $(( $endms - $startms )) ; duration2f=$sf
ms2sf $(( $endms - $startms + $skewms )) ; duration1f=$sf
ms2sf $(( $skewms )) ; skewsecf=$sf
clipcount=$(( $clipcount + 1 )) ; clipcountf=$(printf "%03d" ${clipcount} )
###### https://trac.ffmpeg.org/wiki/Seeking
# CMD="ffmpeg -hide_banner-ss $startsecf -i ${RECDIR}/${bnhead} -t $durationf -map 0:0 -map 0:1 -c:v copy -c:a copy \
# -avoid_negative_ts 1 ${chunkhead}_${clipcount}.ts"
# It's not clear what options might be best here. Non-obvious interactions are common.
# https://superuser.com/questions/702645/ffmpeg-sync-when-extracting-parts-of-video
if ! ${TWINREAD} ; then
# selected by $3 -eq 1
##### duration extended here to allow for skew between video and audio streams
echo -e "\n skew applied in non-TWINREAD mode: $skewms ms \n" | tee -a ${logfile}
# read using video-frame timestamps only (?)
CMD="$FFCALL \
-ss $startsecf -noaccurate_seek -t $duration1f -i ${RECDIR}/${BASENAME} \
-map 0:0 -c:v copy \
-map 0:1 -c:a copy \
-avoid_negative_ts make_zero ${chunkhead}_${clipcountf}.ts"
else
# selected by $3 -eq 2
echo -e "\n skew applied once in TWINREAD mode: $skewms ms \n" | tee -a ${logfile}
# read each stream using its own timestamps (?). -ignore_unknown here is a per-etream repeat.
CMD="$FFCALL -ss $startsecf -noaccurate_seek -t $duration2f -i ${RECDIR}/${BASENAME} \
-ignore_unknown -ss $startsecf -t $duration2f -i ${RECDIR}/${BASENAME} \
-map 0:0 -c:v copy \
-map 1:1 -c:a copy \
-avoid_negative_ts make_zero ${chunkhead}_${clipcountf}.ts"
fi
echo -e "${CMD} \n " | tee -a ${logfile}
${CMD} | tee -a ${logfile}
i=$(( $i + 2 ))
done
#exit
# ##### file segments have been created. Now prepare to merge them ########
###### https://trac.ffmpeg.org/wiki/Concatenate
if [ $clipcount -ge 2 ] ; then
mergestr=""
for i in $( ls ${chunkhead}_* )
do
if [ -z "$mergestr" ]; then
mergestr="concat:$i"
continue
fi
mergestr="$mergestr|$i"
done
CMD="$FFCALL -i $mergestr -c copy ${OUTFILE}"
echo -e "${CMD} \n " | tee -a ${logfile}
${CMD} | tee -a ${logfile}
else
mv ${chunkhead}_${clipcountf}.ts ${OUTFILE}
fi
# Here the outfile name is $bnhead_$$.ts, while the input was $bnhead.ts
#if [ $tovideo ] ; then
# build a seektable for the video
# # The build works even if the video has not been 'scanned in' to mythtv
# # but $OUTFILE isn't found in the SGs on my box, which demands
# # the different but equivqlent /home/john/SGs/VideosSG0/ instaed,
# # so the seektable is built but ignored. Also the log is obtrusive
# # so skip and do it externally Could also include mythutil --scanvideos here.
#
# CMD="ionice -c3 mythcommflag -q --video ${OUTFILE}"
# echo "running: "${CMD}"" | tee -a ${logfile}
# ${CMD} | tee -a ${logfile}
#fi
if ! $tovideo ; then
# move the original out of the way, replace it, and update the DB
mv ${INFILE} ${INFILE}_$$in.ts
mv ${OUTFILE} ${INFILE}
OUTFILE=${INFILE}
# Rebuilding the seek table also resets filesize in the DB
CMD="ionice -c3 mythcommflag -q --rebuild --file ${OUTFILE} "
echo "running: "${CMD}"" | tee -a ${logfile}
${CMD} # | tee -a ${logfile}
CMD="ionice -c3 mythutil --clearcutlist --chanid "$chanid" --starttime "$starttime" -q "
echo "running: "${CMD}"" | tee -a ${logfile}
${CMD} | tee -a ${logfile}
echo -e "Cutlist has been cleared.\n"
# This command was introduced at 0.28-pre-3055. See Ticket #11713
CMD="ionice -c3 mythutil --clearbookmarks --chanid "$chanid" --starttime "$starttime" -q "
echo "running: "${CMD}"" | tee -a ${logfile}
${CMD} | tee -a ${logfile}
echo -e "Bookmarks have been cleared.\n"
CMD="ionice -c3 mythpreviewgen --chanid "$chanid" --starttime "$starttime" -q "
echo "running: "${CMD}"" | tee -a ${logfile}
${CMD}
echo -e "Preview created.\n"
fi
echo -e "Output file is $OUTFILE. \n"
echo -e "\nWhile the .old file still exists you can examine it with\n
hexdump -C -n 40 -s {byteoffset} ${INFILE}.old\n
32-bit hexdump may misinterpret start offsets > 2 GiB.\n "
# Get tech details of output file into the log.
CMD=" mythffprobe -hide_banner -v error -show_streams -show_format -of json $OUTFILE "
echo "${CMD}" | tee -a ${logfile}
${CMD} 2>&1 | tee -a ${logfile}
echo
rm edlist$$
rm keylist$$
rm revedlist$$
rm mslist$$
rm switchpointsms$$
# and if it all worked these may no longer be needed
# rm ${WORKDIR}/$bnhead_$$*
# rm ${INFILE}.old
cd ~
exit 0
