Commercial detection with Comskip.exe and Wine

From MythTV Official Wiki
Revision as of 07:31, 22 May 2010 by Outbreak Monkey (talk | contribs) (Added: How-To for integrating comskip.exe with MythTV under Linux)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Comskip.exe under wine on MythBuntu/MythTV for Linux

For some time now (in Australia), I've had patchy success with commercial flagging (I suspect the TV stations are actively working to hide the commercial transitions - but that's a different soap box).

With some experimentation, I managed to find that Comskip has a better hit rate and less false positives than the built in mythcommflag. Mainly because I was able to better tune the selectivity using the comskip.ini file (there's plenty of advice for tuning comskip on www.kaashoek.com).

Running Comskip under Wine (windows emulator) works fine, the biggest challenge was getting the commercial data back in to MythTV's database.

Note: This How-To uses my PHP script to parse the comskip output, and do the database work. I'd be happier if there was a nice API for accessing the database, or better still some way of running something like "mythcommflag --import-edl /path/to/edl".

As it stands, if the MythTV database schema changes, OR paths change, things will get out of whack. (See warning below)

WARNING

FIRST - A WARNING! THIS DOES STUFF "BEHIND MYTHTV's BACK!" - IT COULD DESTROY YOUR WHOLE INSTALL

I'm pretty sure I don't accidentally do a "drop database mythconvrg;" - but maybe I do, so use at your own risk! :)

This how to is designed for advanced MythTV users, it's definitely beyond the scope of a standard installation. Only do this if you know what you're doing!

Note: It works fine on my install, but seriously, I highly recommend that unless you're comfortable with Linux/Databases/Scripting, don't do this. Backup your data before you start! I have had the misfortune of accidentally losing all my tv shows in the past, it's a horrible feeling to know that a 5 minute upgrade, resulted in days worth of pain trying to restore things.

Getting it going under MythBuntu / Linux Distros

I've chucked this into the Wiki because I found very little information, and no details about an APIs or interfaces for inserting commercial data in to the MythTV database. Perhaps others will inform me I've wasted my time, or that my scripts are ugly - don't care it works for me, and maybe it'll help someone :)

The basic process with comskip is as follows: MythTV records a program. Instead of calling mythcommflag, it runs a shell script that:

  1. Runs comskip (under wine).
  2. Runs a php script to insert the found commercials in to the database.

Pre-requisites, applications and required scripts

The scripts rely on the following other application:

  • wine (sudo apt-get install wine)
  • php-cli (sudo apt-get install php-cli)
  • comskip (wget http://www.kaashoek.com/files/comskip80_031.zip)
  • MySQL as the MythTV Database - the script needs to write the commercial flags somewhere and only supports MySQL
  • /etc/mythtv/mysql.txt (this is where the script gets the login details for the database)

You can get the PHP script and the shell script and php script I made here (I used php purely because it had it handy and was already working on something else):

Installing

I installed this in my home directory which is "/home/mythdesktopuser", you WILL need to adjust the scripts (I'll get to that):

First make your comskip dir, unzip the comskip files, and get the scripts into the same spot:

# mkdir ~/comskip
# cd ~/comskip
# unzip pathtocomskipthatyoudownloaded.zip
# tar -xvzf pathtocomskipsupport.tar.gz
# mv comskipsupport/* .
# touch executions.log

Basically, get all the files and chuck 'em in one spot, then make an empty file called executions.log (not necessary but helps to do that now if you get permissions problems or quirkiness).

This is what I have in my comskip directory:

# ls -1
comskip.dictionary
comskip.exe
comskipGUI.exe
comskip.ini
comskip.ini.mcebuddy
comskipmythdbinsert.php
comskip.txt
debugwindow.htm
executions.log
HideRun.exe
manual.htm
mcomflag.sh
PostProcessing.bat
readme.txt
tuning.htm

Configuring Files and Scripts

Next, edit mcomflag.sh & change the line to match your config: NOTE YOU MUST DO THESE STEPS

COM_SKIP_HOME=/home/mythdesktopuser/comskip

Your file "/home/mythtv/mysql.txt" should contain the following lines:

DBUserName=mythtv
DBPassword=R4Nd0M5TrinG
DBName=mythconverg

If you don't have the mysql.txt file in this location:

1. change comskipmythdbinsert.php (look for $mythconfig=file("/etc/mythtv/mysql.txt");).

OR

2. You can also /* comment the whole block out */ and manually set the variables in comskipmythdbinsert.php as follows:

$dbname=mythconverg;
$dbuser=mythtv;
$dbpass=RanD0mSTR1ng;

OR

3. Just make and put the file where the script wants it!

Set permissions

Make the directory and and executions.log writable by mythtv:

# sudo chown -R mythdesktopuser:mythtv .
# chmod g+r -R .
# chmod ug+w executions.log
# chmod ug+x mcomflag.sh
# chmod ug+x comskipmythdbinsert.php

i.e. Make sure the mythtv user can write to the log file, and can execute the scripts mcomflag.sh and comskipmythdbinsert.php

Manually testing the scripts

'Warning - read the previous warning, backup your databases and lock up your pets'

Test that wine and comskip run

First you should be able to run comskip.exe! If this doesn't work nothing will:

# wine comskip.exe -h

Comskip 0.80.030, made using:
mpeg2dec-0.4.0 - by Michel Lespinasse <walken@zoy.org> and Aaron Holtzman
The commandline used was:
Z:\home\mythdesktopuser\comskip\comskip.exe -h
...

i.e. This command will spray a whole lot of usage information up if it's working.

Test the mcomflag.sh and php scripts

Finally test the actual script - I like to do this AS the mythtv user so I can see that they have full read/write access as required:

# sudo su (enter your password)
# sudo - mythtv (become the mythtv user, change this if your myth user is a different name)
# /path/to/mcomflag.sh /PATH/TO/RECORDINGS A_RECORDED_PROGRAM.mpg
(eg: /home/mythdesktopuser/comskip/mcomflag.sh /var/lib/mythtv/recordings 1088_20100522141700.mpg)

Note: There's no real error checking here, make sure you give mcomflag.sh two parameters: a recording directory, and a real .mpg file.

It's probably a good idea to record a short TV show or use a small .mpg file so it doesn't take too long to test!

If it's working you'll see the following on screen:

The commandline used was:
Z:\home\mythdesktopuser\comskip\comskip.exe /media/myth/recordings/1022_20100522141700.mpg

Opening /var/lib/mythtv/recordings/1088_20100522141700.mpg
Using Z:\home\mythdesktopuser\comskip\comskip.ini for initiation values.
 5:23:40 - 28670 frames in 38.04 sec (753.68 fps), 4.33 sec (65.59 fps), 99%%
28694 frames decoded in 38.07 seconds (753.72 fps)
Commercials were found.

Then if you look at executions.log you'll see if it succeeded in sticking stuff in the database: EXAMPLE:

#cat /home/mythdesktopuser/comskip/executions.log
=============================
Running comskip under wine with command:
wine comskip.exe /var/lib/mythtv/recordings/1088_20100522141700.mpg
-----------------------------
Now running database insert script with command:
/home/mythdesktopuser/comskip/comskipmythdbinsert.php /var/lib/mythtv/recordings/1088_20100522141700.mpg


ComskipMythTVInserter - Version 0.1a - using commfile: 1088_20100522141700.txt
Deleting Marks
Adding new marks: XXXXXX (Total = 6)
Checking marks
Event mark: 1 Type = 4 - CommStart
Event mark: 2974 Type = 5 - CommEnd
Event mark: 17940 Type = 4 - CommStart
Event mark: 18750 Type = 5 - CommEnd
Event mark: 28682 Type = 4 - CommStart
Event mark: 28683 Type = 5 - CommEnd
Done!
=============================

The important thing to see is the "Event mark" lines after Checking marks - these are what has been read BACK from the mythconverg database. Each mark is frame number, and a type, these are inserted directly in to the mythconverg database, existing COMM_START and COMM_END marks for the recording are deleted.

With any luck, you're still with me at this point.

Making MythTV use the script

In the MythTV Backend Setup, under "1. General", flip through until you get to "Job Queue (Global)":

Uncheck 'Start Auto-Commercial Flagging jobs when the recording starts' (Because this script only works "after the fact")

Set Commercial Flagger command:

/home/mythdesktopuser/comskip/mcomflag.sh %DIR% %FILE%

i.e. This is the path to mcomflag.sh, it's passed %DIR% and %FILE% which is the same thing we simulated in the step above: passed in a directory for the recordings, and a filename.mpg for a recording.

OR - You could also 'nice' it up so it doesn't suck too much resources:

/usr/bin/nice -n 19 /home/mythdesktopuser/comskip/mcomflag.sh %DIR% %FILE% 

Step through to the finish, and exit mythtv setup!

That should be about it.

Testing a real recording

Probably a good time to test a real recording, select something short and see how you go.

A final word on auto vs manual skipping

As even comskip.exe isn't perfect I don't like to have autoskip turned on. I have Myth set to "notify but don't skip". Then I use my remote OR the 'Z' & 'Q' keys to jump between marks.

I find this to be the best method, sometimes Australian TV stations chuck two quick commercials together, and you end up 'out of sync', if you're automatically skipping commercials then you miss parts of your show. If you use the keys, it doesn't matter whether the START/END marks for the commercials are the right way around, 'Z' simply jumps to the next mark whether it's in or out!.

I also like to have it set to notify me 5 seconds before a commercial (Commercial SKip Notify Amount = 5) so I can grab the remote, and it's nice if it comes back 1 second (Commercial Skip Auto-Rewind Amount = 1) so you can be sure you caught the start of the show / end of the commercials. These settings are in Myth's front end: Utilities/Setup -> Setup -> TV Settings -> Playback -> Commercial Skip (8/8)