Difference between revisions of "Talk:Script - RemoveCommercials"

Latest revision as of 16:15, 22 September 2016

I'm getting an error on mythtranscode that says Invalid video codec. The -m switch appears to fix the problem. -- jpn627

• This appears to be due to NULL values in the videocodec / audiocodec columns in the recordingprofiles table. A bypass for this problem is to use mythfrontend to update the recording profiles (under TV Settings) and edit each profile under the Transcoders group.--Jbrusey 11:55, 22 October 2006 (UTC)

Is there a way to allow this script to transcode from RTJpeg to MPEG4 while cutting out commercials? Nuvexport can do this but I don't think it updates the database. I have my transcoding defaults set to transcode to mpeg4, and this script will throw an error if the input video is RTJpeg. The error is "Transcoding aborted, NuppelVideoPlayer is not in raw audio mode." I've also tried recording in both raw audio and mp3. I'll post more info on my profile page. Napsilan 17:25, 27 June 2007 (UTC)

honorcutlist and mpeg2 lossless transcoding

Update:

Mythtranscode cannot load the cutlist from the filename alone when using the mpeg2 lossless transcoding. Another way to do this is to call mythtranscode with the chanid and starttime parameters. So the user job is modified as such.

• 0.23 - StartTime should be passed to mythcommflag and mythtranscode without modification. remove the sed filter and just pass the $4 parm directly. Fedora 14. eg.

  declare STARTTIME=$4

removecommercials %DIR% %FILE% %CHANID% %STARTTIME%

Then the script can be modified as such.

Note: This change has been incorporated into the #Updated script

CHANID=$3
STARTTIME=`echo $4 | sed -e 's/\([0-9]\{4\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)/\1-\2-\3-\4-\5/'`
....
mythtranscode --mpeg2 --honorcutlist -c $CHANID -s $STARTTIME -o $VIDEODIR/$FILENAME.tmp

Original: I had to modify the mythtranscode line to place --honorcutlist after the -i {file} option when doing mpeg2-to-mpeg2 lossless transcoding. I do not know if this change is necessary for any other transcode type, but it should be safe to make the change for any type.

Here is a link to that part of the source. It will not fully process this option unless found_infile, which is set true when processing option i.

P.S. It turns out that alone was not the problem. For some reason the cutlist is not loading upon transcode. So, I added:

CUTLIST=$(mythcommflag --getcutlist -f $VIDEODIR/$FILENAME | tail -n 1 | awk '{print $2}' | sed 's/,/ /g')
ERROR=$?
if [ $ERROR -ne 0 ]; then
        echo "Copying cutlist failed for ${FILENAME} with error $ERROR"
        exit $ERROR
fi

Then I supply that to mythtranscode:

mythtranscode --mpeg2 -i $VIDEODIR/$FILENAME --honorcutlist "$CUTLIST" -o $VIDEODIR/$FILENAME.tmp

Already Flagged Recordings

What possibilities exist for modifying the script to detect whether a recording has already been commflaged? For example, I set all my recordings to be automatically commflaged but I only want to remove commercials for certain recordings. Whether I decide to remove the commercials won't be decided at scheduling time, so disabling the commflag job and keeping only the removecommercial job won't do. As it stands, removecommercials reruns the commflag job. I see no switch on mythcommflag that would allow an ifcheck on whether a recording is already flagged.

One idea: mythcommflag --getskiplist -f filename.mpg

I'm waiting for an unflagged program to show up on my system to test what the result of that command on an unflagged program returns.

Here is a way you might be able to check for commercial flagging:

% mysql -u mythtv -pmythtv -B -N -e "select commflagged from recorded where basename = '${FILENAME}'" mythconverg

It will return 1 if the program is already flagged.

Note: This change has been incorporated into the #Updated script

Automatically delete old file

It would be nice, and I suspect fairly easy, to have the removecommercials script optionally delete the backup '.old' file. Perhaps a script option such as: removecommercials --deleteoriginal %DIR% %FILE%

Note: This #Updated script does this unconditionally

Alatar 06:55, 30 May 2007 (UTC)

enhanced script uses channelID and starttime throughout

Create two user jobs, if you like. add SKIP as the last parameter for the second user job. I call it "Remove Commercials, transcode only". Useful if you already have a cutlist. Seems to work better when mythtranscode and mythcommflag are given program parms instead of the file. I have only used it with --mpeg2 so YMMV. -Rick

Updated script

The following is an update from the previous script here (and the main page) that incorporates many of the changes suggested above. After others get some mileage on it, I think it should replace the script on the main page.

1. Looks up myth database password in your ~/.mythtv/config.xml
2. Auto detects whether mythcommflag has run.
3. Removes temporary files.
4. Clears auto skip list.
5. More honestly reports what program is generating the log messages.

#!/bin/bash

### removecommercials - for mythtv user job.
### $author Zack White - zwhite dash mythtv a t nospam darkstar deleteme frop dot org
### $Modified 20080330 Richard Hendershot - rshendershot a t nospam gmail deleteme dot youknowcom
### $Modified 20100112 Aaron Larson to get password from mythtv config file, clear autoskip list after transcoding, and detect pre-flagged files.

# Should be set as a mythtv user job with a command as:
#  removecommercials %DIR% %FILE% %CHANID% %STARTTIME% 
#
#   initialize;  all except SKIP are required for this to function correctly
declare VIDEODIR=$1
declare FILENAME=$2
declare CHANID=$3
declare STARTTIME=$(echo $4 | sed -e 's/\([0-9]\{4\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)\([0-9]\{2\}\)/\1-\2-\3-\4-\5-\6/')
declare SKIP=$5
# for lossless transcoding autodetect.  Set to empty string for rtJpeg/mpeg4.
declare MPEG2=--mpeg2
declare PROG=$(basename $0)

if [ -z "${VIDEODIR}" -o -z "${FILENAME}" -o -z "${CHANID}" -o -z "${STARTTIME}" ]; then
        cat - <<-EOF
	Usage: $PROG <VideoDirectory> <FileName> <ChannelID> <StartTime> [SKIP]

	Flag commercials (if they are not already flagged), do a lossless transcode
	to remove the commercials, and fixup the database as necessary.	 The net
	effect is that this script can be run as the *only* job after a recording,
	or as a job after a commercial flagging job (either way).  The optional 5th
	parameter 'SKIP', if specified as a non zero length string, will transcode
	using the existing cutlist.
	EOF
        exit 5
fi
if [ ! -f "${VIDEODIR}/${FILENAME}" ]; then
        echo "$PROG: File does not exist: ${VIDEODIR}/${FILENAME}"
        exit 6
fi
if [ ! -d "${VIDEODIR}" ]; then 
        echo "$PROG: <VideoDirectory> must be a directory"
        exit 7
fi
if [ ! -d "${VIDEODIR}/originals" ]; then 
        mkdir "${VIDEODIR}"/originals
fi
if [ ! -d "${VIDEODIR}/originals" ]; then 
        echo "$PROG: you must have write access to <VideoDirectory>"
        exit 8
fi

# mythtv stores the mysql configuration information in the following
# file.  Extract the DB user and password.
mythConfig=~/.mythtv/config.xml 
mysqlArgs=""
if [ -e "$mythConfig" ]; then
        mysqlUserOpt=$(sed $mythConfig -n -e '/<DBUserName/p')
        if [ -n "$mysqlUserOpt" ]; then
           mysqlUser=$(echo $mysqlUserOpt | sed 's: *</*DBUserName> *::g')
           mysqlArgs+=" -u $mysqlUser"
        fi
        mysqlPassOpt=$(sed $mythConfig -n -e '/<DBPassword/p')
        if [ -n "$mysqlPassOpt" ]; then
           mysqlPass=$(echo $mysqlPassOpt | sed 's: *</*DBPassword> *::g')
           if [ -n "$mysqlPass" ]; then 
               mysqlArgs+=" -p$mysqlPass"
           fi
        fi
fi

if [ -z "${SKIP}" ]; then
        #   if transcode was run after mythcommflag in the normal setup screens
        #   then the current file may not match the existing index, so rebuild
        echo "$PROG: Rebuilding seek list for ${FILENAME}"
        mythcommflag -c ${CHANID} -s ${STARTTIME} --quiet --rebuild
        ERROR=$?
        if [ $ERROR -ne 0 ]; then
                echo "$PROG: Rebuilding seek list failed for ${FILENAME} with error $ERROR"
                exit $ERROR
        else
                echo "$PROG: Rebuilding seek list successful for ${FILENAME}"
        fi

        #   flag commercials (generate skiplist)
        #   you can use mythcommflag -f ${VIDEODIR}/${FILENAME} --getskiplist
        #   to view results
	
	# has mythcommflag already run?
	alreadyFlagged=$(mysql $mysqlArgs -B -N -e "select commflagged from recorded where basename = '${FILENAME}'" mythconverg)
	if [ "$alreadyFlagged" == "1" ]; then
	    echo "$PROG: ${FILENAME} already flagged, skipping mythcommflag."
	else
	    echo "$PROG: Commercial flagging ${FILENAME}"
	    mythcommflag -c ${CHANID} -s ${STARTTIME} --quiet 
	    ERROR=$?
	    if [ $ERROR -gt 126 ]; then
		    echo "$PROG: Commercial flagging failed for ${FILENAME} with error $ERROR"
		    exit $ERROR
	    else
		    echo "$PROG: Commercial flagging successful for ${FILENAME}"
	    fi
	fi

	#   generate cutlist from skiplist
	#   you can use mythcommflag -f ${VIDEODIR}/${FILENAME} --getcutlist
	#   to view results
	echo "$PROG: Creating cutlist from skiplist for ${FILENAME}"
	mythcommflag -c ${CHANID} -s ${STARTTIME} --quiet --gencutlist
	ERROR=$?
	if [ $ERROR -ne 0 ]; then
		echo "$PROG: Creating cutlist from skiplist failed for ${FILENAME} with error $ERROR"
		exit $ERROR
	else
		echo "$PROG: Creating cutlist from skiplist successful for ${FILENAME}"
	fi
else
        echo "$PROG: skipping commercial detection due to parameter $SKIP"        
fi  #end skip

#   cut the commercials from the file.  creates a new file and a map file.
echo "$PROG: Transcoding commercials out of original file (${FILENAME})"
mythtranscode -c ${CHANID} -s ${STARTTIME} $MPEG2 --honorcutlist -o "${VIDEODIR}/${FILENAME}.mpeg"
ERROR=$?
if [ $ERROR -ne 0 ]; then
        echo "$PROG: Transcoding failed for ${FILENAME} with error $ERROR"
        exit $ERROR
else
        echo "$PROG: Transcoding successful for ${FILENAME}"
fi

echo "$PROG: Moving ${VIDEODIR}/${FILENAME}  to  ${VIDEODIR}/originals/${FILENAME}"
mv "${VIDEODIR}/${FILENAME}" "${VIDEODIR}/originals"
ERROR=$?
if [ $ERROR -ne 0 ]; then
        echo "$PROG: Moving failed with error $ERROR"
        exit $ERROR
else
        echo "$PROG: Moving successful"
fi

echo "$PROG: Moving ${VIDEODIR}/${FILENAME}.mpeg  to  ${VIDEODIR}/${FILENAME}"
if [ ! -f "${VIDEODIR}/${FILENAME}" ]; then
        mv "${VIDEODIR}/${FILENAME}.mpeg" "${VIDEODIR}/${FILENAME}"
        ERROR=$?
        if [ $ERROR -ne 0 ]; then
                echo "$PROG: Moving failed with error $ERROR"
                exit $ERROR
        else
                echo "$PROG: Moving successful"
        fi
else
        echo "$PROG: cannot replace original.  skipping file move. (${VIDEODIR}/${FILENAME})"
fi


echo "$PROG: removing map file: ${VIDEODIR}/${FILENAME}.mpeg.map"
if [ -f "${VIDEODIR}/${FILENAME}.mpeg.map" ]; then
        rm "${VIDEODIR}/${FILENAME}.mpeg.map"
        ERROR=$?
        if [ $ERROR -ne 0 ]; then
                echo "$PROG: unable to remove map file: ${VIDEODIR}/${FILENAME}.mpeg.map"
        else
                echo "$PROG: removed map file successfully"
        fi
fi    

#   file has changed, rebuild index
echo "$PROG: Rebuilding seek list for ${FILENAME}"
mythcommflag -c ${CHANID} -s ${STARTTIME} --quiet --rebuild
ERROR=$?
if [ $ERROR -ne 0 ]; then
        echo "$PROG: Rebuilding seek list failed for ${FILENAME} with error $ERROR"
        exit $ERROR
else
        echo "$PROG: Rebuilding seek list successful for ${FILENAME}"
fi

echo "$PROG: Clearing cutlist for ${FILENAME}"
mythcommflag -c ${CHANID} -s ${STARTTIME} --quiet --clearcutlist
ERROR=$?
if [ $ERROR -eq 0 ]; then
        echo "$PROG: Clearing cutlist successful for ${FILENAME}"
else
        echo "$PROG: Clearing cutlist failed for ${FILENAME} with error $ERROR"
        exit $ERROR
fi
        
# mythcommflag sets cutlist to zero, but doesn't update the filesize.
# Fix the database entry for the file
mysql $mysqlArgs mythconverg << EOF 
    UPDATE 
        recorded
    SET
        cutlist = 0,
        filesize = $(ls -l "${VIDEODIR}/${FILENAME}" | awk '{print $5}') 
    WHERE
        basename = '${FILENAME}';
EOF

echo "$PROG: Clearing autoskip list: ${VIDEODIR}/${FILENAME}"
mysql $mysqlArgs mythconverg << EOF 
    DELETE FROM 
	   recordedmarkup
    WHERE 
	   CONCAT( chanid, starttime ) IN (
		   SELECT 
			   CONCAT( chanid, starttime )
		   FROM 
			   recorded
		   WHERE 
			   basename = '$FILENAME'
	   );
EOF

# If you want to keep the originals, comment out this line.
echo "$PROG: removing saved copy of: ${VIDEODIR}/originals/${FILENAME}"
rm "${VIDEODIR}/originals/${FILENAME}"
ERROR=$?
if [ $ERROR -ne 0 ]; then
	echo "$PROG: failed to remove ${VIDEODIR}/originals/${FILENAME}"
	exit $ERROR
fi

STARTTIME With MythTv 0.26 and Later

To get the script to work with For MythTv 0.26 and later, pass %STARTTIMEUTC% as opposed to %STARTTIME% into the script.

DEBUG HINTS

I found the following useful debugging my customized script from the command line with MythTv 0.27.

1. To find a valid channel ID and start time corresponding to a recording, review the mythtv backend log /var/log/mythtv/mythtv.log to see the user job command line as output by MythTv.
2. To force MythTv to run the job on an existing recording, highlight the recording in the MythTv front end, then press M to get the menu and select the "Job Options" menu. Your user job should appear on this menu if you have configured the back end correctly.
3. The filename format relates to the channel ID and recording start time as follows:

Filename = CCCC_YYYYMMDDHHMMSS.mpg

where

     CCCC = Four digit channel ID. For US ATSC broadcast channels the number will be the channel number x 10 + 1000.

     YYYY = Four digit year.

     MM = Two digit month.

     DD = Two digit day of month.

     HH = Two digit start time hour, local time zone, 24 hour format.

     MM = Two digit start time minute.

     SS = Two digit start time second.

The required start time for the command line tools is the UTC time as noted above. Subtract your GMT offset (including daylight savings time) from the time in the filename to determine the appropriate STARTTIME.

Example:

     Filename = 1331_20140801222800.mpg

     ATSC channel = 33.1

     GMT offset = GMT-4

     STARTTIME = 20140802022800

.28 mythtranscode no longer honors the -o command-line parameter.

For this script, when it runs it means that when mythtranscode runs, it will delete/overwrite the original, not create a .tmp file.

You can change the behavior of mythtranscode in a tickbox in mythtv-setup / 1. General / Job Queue (global) about page 9 or so "save original files after transcoding", which will cause mythtranscode to create a *.old file (see https://code.mythtv.org/trac/ticket/12845)

For this script to work successfully, likely one should update the mythtv-setup settings, and then update the script appropriately (with the assumption that the .old file is being created by mythtranscode having been configured that way in the mythtv-setup)