Difference between revisions of "Commercial detection with Comskip.exe and Wine"

From MythTV Official Wiki
Jump to: navigation, search
(Added: How-To for integrating comskip.exe with MythTV under Linux)
 
(New ComSkip Runner (Link Note))
 
(14 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 +
{{Note box|I have a update to this script that works with 0.26.
 +
 +
http://outbreakmonkey.org/mythtv/csr0.3.php
 +
 +
When I get time I'll move he instructions from in to the wiki
 +
}}
 +
 +
 +
{{Warning box|This script parses the database login from mysql.txt, which has been removed from the upcoming 0.26.}}
 +
 
= Comskip.exe under wine on MythBuntu/MythTV for Linux =
 
= Comskip.exe under wine on MythBuntu/MythTV for Linux =
  
Line 11: Line 21:
 
As it stands, if the MythTV database schema changes, OR paths change, things will get out of whack. (See warning below)''  
 
As it stands, if the MythTV database schema changes, OR paths change, things will get out of whack. (See warning below)''  
  
== WARNING ==
+
{{Warning box|'''FIRST - A WARNING! THIS DOES STUFF "BEHIND MYTHTV's BACK!" - IT COULD DESTROY YOUR WHOLE INSTALL'''
'''FIRST - A WARNING! THIS DOES STUFF "BEHIND MYTHTV's BACK!" - IT COULD DESTROY YOUR WHOLE INSTALL'''
 
  
 
''I'm pretty sure I don't accidentally do a "drop database mythconvrg;" - but maybe I do, so use at your own risk! :)''
 
''I'm pretty sure I don't accidentally do a "drop database mythconvrg;" - but maybe I do, so use at your own risk! :)''
  
This how to is designed for advanced MythTV users, it's definitely beyond the scope of a standard installation. Only do this if you know what you're doing!
+
This how-to is designed for advanced MythTV users, it's definitely beyond the scope of a standard installation. Only do this if you know what you're doing!
  
Note: It works fine on my install, but seriously, I highly recommend that unless you're comfortable with Linux/Databases/Scripting, don't do this. Backup your data before you start! I have had the misfortune of accidentally losing all my tv shows in the past, it's a horrible feeling to know that a 5 minute upgrade, resulted in days worth of pain trying to restore things.
+
Note: It works fine on my install, but seriously, I highly recommend that unless you're comfortable with Linux/Databases/Scripting, don't do this. Backup your data before you start! I have had the misfortune of accidentally losing all my tv shows in the past, it's a horrible feeling to know that a 5 minute upgrade, resulted in days worth of pain trying to restore things.}}
  
 
== Getting it going under MythBuntu / Linux Distros ==
 
== Getting it going under MythBuntu / Linux Distros ==
Line 36: Line 45:
 
* /etc/mythtv/mysql.txt (this is where the script gets the login details for the database)
 
* /etc/mythtv/mysql.txt (this is where the script gets the login details for the database)
  
You can get the PHP script and the shell script and php script I made here (I used php purely because it had it handy and was already working on something else):
+
=== comskipmythdbinsert.php ===
 +
You can get the PHP script and the shell script and php script I made here:
 
* comskipsupport files (wget [http://outbreakmonkey.org/mythtv/comskipsupport.tar.gz http://outbreakmonkey.org/mythtv/comskipsupport.tar.gz])
 
* comskipsupport files (wget [http://outbreakmonkey.org/mythtv/comskipsupport.tar.gz http://outbreakmonkey.org/mythtv/comskipsupport.tar.gz])
  
Line 46: Line 56:
 
# mkdir ~/comskip
 
# mkdir ~/comskip
 
# cd ~/comskip
 
# cd ~/comskip
# unzip pathtocomskipthatyoudownloaded.zip
+
# unzip <pathtocomskipthatyoudownloaded.zip>
# tar -xvzf pathtocomskipsupport.tar.gz
+
# tar -xvzf <pathtocomskipsupport.tar.gz>
 
# mv comskipsupport/* .
 
# mv comskipsupport/* .
 
# touch executions.log
 
# touch executions.log
Line 74: Line 84:
  
 
== Configuring Files and Scripts ==
 
== Configuring Files and Scripts ==
Next, edit mcomflag.sh & change the line to match your config:
 
 
'''NOTE YOU MUST DO THESE STEPS'''
 
'''NOTE YOU MUST DO THESE STEPS'''
 +
 +
Make a note of the directory that you've put all these files in. As stated, in my case it's in comskip within the mythdesktopuser's directory:
 +
<pre>
 +
# pwd
 +
/home/mythdesktop/comskip
 +
</pre>
 +
''i.e. If you haven't changed out of the comskip directory you made, you can run 'pwd' to confirm the exact path to the directory you made, this is what you need to set in the following edit''
 +
 +
Edit mcomflag.sh & change the following line to match your configuration:
 
<pre>
 
<pre>
 
COM_SKIP_HOME=/home/mythdesktopuser/comskip
 
COM_SKIP_HOME=/home/mythdesktopuser/comskip
 
</pre>
 
</pre>
  
Your file "/home/mythtv/mysql.txt" should contain the following lines:
+
Additionally, your file "/etc/mythtv/mysql.txt" should contain something like the following lines:
 
<pre>
 
<pre>
 
DBUserName=mythtv
 
DBUserName=mythtv
Line 108: Line 126:
 
<pre>
 
<pre>
 
# sudo chown -R mythdesktopuser:mythtv .
 
# sudo chown -R mythdesktopuser:mythtv .
# chmod g+r -R .
+
# chmod ug+r -R .
 +
# chmod ug+w .
 
# chmod ug+w executions.log
 
# chmod ug+w executions.log
 
# chmod ug+x mcomflag.sh
 
# chmod ug+x mcomflag.sh
Line 136: Line 155:
 
<pre>
 
<pre>
 
# sudo su (enter your password)
 
# sudo su (enter your password)
# sudo - mythtv (become the mythtv user, change this if your myth user is a different name)
+
# su - mythtv (become the mythtv user, change this if your myth user is a different name)
# /path/to/mcomflag.sh /PATH/TO/RECORDINGS A_RECORDED_PROGRAM.mpg
+
# /path/to/mcomflag.sh </PATH/TO/RECORDINGS> <A_RECORDED_PROGRAM.mpg>
 
(eg: /home/mythdesktopuser/comskip/mcomflag.sh /var/lib/mythtv/recordings 1088_20100522141700.mpg)
 
(eg: /home/mythdesktopuser/comskip/mcomflag.sh /var/lib/mythtv/recordings 1088_20100522141700.mpg)
 
</pre>
 
</pre>
Note: There's no real error checking here, make sure you give mcomflag.sh two parameters: a recording directory, and a real .mpg file.
+
''Note: There's no real error checking here, make sure you give mcomflag.sh two parameters: a recording directory, and a real .mpg file.''
  
 
It's probably a good idea to record a short TV show or use a small .mpg file so it doesn't take too long to test!
 
It's probably a good idea to record a short TV show or use a small .mpg file so it doesn't take too long to test!
Line 147: Line 166:
 
<pre>
 
<pre>
 
The commandline used was:
 
The commandline used was:
Z:\home\mythdesktopuser\comskip\comskip.exe /media/myth/recordings/1022_20100522141700.mpg
+
Z:\home\mythdesktopuser\comskip\comskip.exe /media/myth/recordings/1088_20100522141700.mpg
  
 
Opening /var/lib/mythtv/recordings/1088_20100522141700.mpg
 
Opening /var/lib/mythtv/recordings/1088_20100522141700.mpg
Line 155: Line 174:
 
Commercials were found.
 
Commercials were found.
 
</pre>
 
</pre>
 +
''Note: You should get Commercials were found, if not, make sure you have a file that contains commercials, OR you may need to tweak comskip.ini. If no commercials were found, then well there won't be any commercials to insert!''
  
 
Then if you look at executions.log you'll see if it succeeded in sticking stuff in the database:
 
Then if you look at executions.log you'll see if it succeeded in sticking stuff in the database:
Line 207: Line 227:
  
 
== Testing a real recording ==
 
== Testing a real recording ==
Probably a good time to test a real recording, select something short and see how you go.
+
Probably a good time to test a real recording. Record a short show that's likely to contain commercials, and see how you go.
 +
 
 +
After the commercial flagging has been run, you can <code>tail /home/mythdesktopuser/comskip/executions.log</code> - you should see the 'Event mark' lines.
  
 
== A final word on auto vs manual skipping ==
 
== A final word on auto vs manual skipping ==
Line 214: Line 236:
 
I find this to be the best method, sometimes Australian TV stations chuck two quick commercials together, and you end up 'out of sync', if you're automatically skipping commercials then you miss parts of your show. If you use the keys, it doesn't matter whether the START/END marks for the commercials are the right way around, 'Z' simply jumps to the next mark whether it's in or out!.
 
I find this to be the best method, sometimes Australian TV stations chuck two quick commercials together, and you end up 'out of sync', if you're automatically skipping commercials then you miss parts of your show. If you use the keys, it doesn't matter whether the START/END marks for the commercials are the right way around, 'Z' simply jumps to the next mark whether it's in or out!.
  
I also like to have it set to notify me 5 seconds before a commercial (Commercial SKip Notify Amount = 5) so I can grab the remote, and it's nice if it comes back 1 second (Commercial Skip Auto-Rewind Amount = 1) so you can be sure you caught the start of the show / end of the commercials.
+
Also, while sometimes Comskip.exe doesn't pick up the commercial at the very start, but it does seem to be pretty good at picking up the transition back in to the TV show - which means a couple of skips forward, and you've back in to your show! Very handy! :)
 +
 
 +
I also like to have it set to notify me 5 seconds before a commercial, and come back 1 second before the end of a commercial.
 
''These settings are in Myth's front end: Utilities/Setup -> Setup -> TV Settings -> Playback -> Commercial Skip (8/8)''
 
''These settings are in Myth's front end: Utilities/Setup -> Setup -> TV Settings -> Playback -> Commercial Skip (8/8)''
 +
<pre>
 +
Commercial Skip Notify Amount = 5
 +
Commercial Skip Auto-Rewind Amount = 1
 +
</pre>
 +
This gives me enough time to grab the remote before a commercial, and the 1 second rewind allows me to confirm that I'm not missing any of my show.
 +
 +
[[Category:User_Job_Scripts]]

Latest revision as of 05:51, 29 December 2012

Important.png Note: I have a update to this script that works with 0.26.

http://outbreakmonkey.org/mythtv/csr0.3.php

When I get time I'll move he instructions from in to the wiki


Warning.png Warning: This script parses the database login from mysql.txt, which has been removed from the upcoming 0.26.

Comskip.exe under wine on MythBuntu/MythTV for Linux

For some time now (in Australia), I've had patchy success with commercial flagging (I suspect the TV stations are actively working to hide the commercial transitions - but that's a different soap box).

With some experimentation, I managed to find that Comskip has a better hit rate and less false positives than the built in mythcommflag. Mainly because I was able to better tune the selectivity using the comskip.ini file (there's plenty of advice for tuning comskip on www.kaashoek.com).

Running Comskip under Wine (windows emulator) works fine, the biggest challenge was getting the commercial data back in to MythTV's database.

Note: This How-To uses my PHP script to parse the comskip output, and do the database work. I'd be happier if there was a nice API for accessing the database, or better still some way of running something like "mythcommflag --import-edl /path/to/edl".

As it stands, if the MythTV database schema changes, OR paths change, things will get out of whack. (See warning below)

Warning.png Warning: FIRST - A WARNING! THIS DOES STUFF "BEHIND MYTHTV's BACK!" - IT COULD DESTROY YOUR WHOLE INSTALL

I'm pretty sure I don't accidentally do a "drop database mythconvrg;" - but maybe I do, so use at your own risk! :)

This how-to is designed for advanced MythTV users, it's definitely beyond the scope of a standard installation. Only do this if you know what you're doing!

Note: It works fine on my install, but seriously, I highly recommend that unless you're comfortable with Linux/Databases/Scripting, don't do this. Backup your data before you start! I have had the misfortune of accidentally losing all my tv shows in the past, it's a horrible feeling to know that a 5 minute upgrade, resulted in days worth of pain trying to restore things.

Getting it going under MythBuntu / Linux Distros

I've chucked this into the Wiki because I found very little information, and no details about an APIs or interfaces for inserting commercial data in to the MythTV database. Perhaps others will inform me I've wasted my time, or that my scripts are ugly - don't care it works for me, and maybe it'll help someone :)

The basic process with comskip is as follows: MythTV records a program. Instead of calling mythcommflag, it runs a shell script that:

  1. Runs comskip (under wine).
  2. Runs a php script to insert the found commercials in to the database.

Pre-requisites, applications and required scripts

The scripts rely on the following other application:

  • wine (sudo apt-get install wine)
  • php-cli (sudo apt-get install php-cli)
  • comskip (wget http://www.kaashoek.com/files/comskip80_031.zip)
  • MySQL as the MythTV Database - the script needs to write the commercial flags somewhere and only supports MySQL
  • /etc/mythtv/mysql.txt (this is where the script gets the login details for the database)

comskipmythdbinsert.php

You can get the PHP script and the shell script and php script I made here:

Installing

I installed this in my home directory which is "/home/mythdesktopuser", you WILL need to adjust the scripts (I'll get to that):

First make your comskip dir, unzip the comskip files, and get the scripts into the same spot:

# mkdir ~/comskip
# cd ~/comskip
# unzip <pathtocomskipthatyoudownloaded.zip>
# tar -xvzf <pathtocomskipsupport.tar.gz>
# mv comskipsupport/* .
# touch executions.log

Basically, get all the files and chuck 'em in one spot, then make an empty file called executions.log (not necessary but helps to do that now if you get permissions problems or quirkiness).

This is what I have in my comskip directory:

# ls -1
comskip.dictionary
comskip.exe
comskipGUI.exe
comskip.ini
comskip.ini.mcebuddy
comskipmythdbinsert.php
comskip.txt
debugwindow.htm
executions.log
HideRun.exe
manual.htm
mcomflag.sh
PostProcessing.bat
readme.txt
tuning.htm

Configuring Files and Scripts

NOTE YOU MUST DO THESE STEPS

Make a note of the directory that you've put all these files in. As stated, in my case it's in comskip within the mythdesktopuser's directory:

# pwd
/home/mythdesktop/comskip

i.e. If you haven't changed out of the comskip directory you made, you can run 'pwd' to confirm the exact path to the directory you made, this is what you need to set in the following edit

Edit mcomflag.sh & change the following line to match your configuration:

COM_SKIP_HOME=/home/mythdesktopuser/comskip

Additionally, your file "/etc/mythtv/mysql.txt" should contain something like the following lines:

DBUserName=mythtv
DBPassword=R4Nd0M5TrinG
DBName=mythconverg

If you don't have the mysql.txt file in this location:

1. change comskipmythdbinsert.php (look for $mythconfig=file("/etc/mythtv/mysql.txt");).

OR

2. You can also /* comment the whole block out */ and manually set the variables in comskipmythdbinsert.php as follows:

$dbname=mythconverg;
$dbuser=mythtv;
$dbpass=RanD0mSTR1ng;

OR

3. Just make and put the file where the script wants it!

Set permissions

Make the directory and and executions.log writable by mythtv:

# sudo chown -R mythdesktopuser:mythtv .
# chmod ug+r -R .
# chmod ug+w .
# chmod ug+w executions.log
# chmod ug+x mcomflag.sh
# chmod ug+x comskipmythdbinsert.php

i.e. Make sure the mythtv user can write to the log file, and can execute the scripts mcomflag.sh and comskipmythdbinsert.php

Manually testing the scripts

'Warning - read the previous warning, backup your databases and lock up your pets'

Test that wine and comskip run

First you should be able to run comskip.exe! If this doesn't work nothing will:

# wine comskip.exe -h

Comskip 0.80.030, made using:
mpeg2dec-0.4.0 - by Michel Lespinasse <walken@zoy.org> and Aaron Holtzman
The commandline used was:
Z:\home\mythdesktopuser\comskip\comskip.exe -h
...

i.e. This command will spray a whole lot of usage information up if it's working.

Test the mcomflag.sh and php scripts

Finally test the actual script - I like to do this AS the mythtv user so I can see that they have full read/write access as required:

# sudo su (enter your password)
# su - mythtv (become the mythtv user, change this if your myth user is a different name)
# /path/to/mcomflag.sh </PATH/TO/RECORDINGS> <A_RECORDED_PROGRAM.mpg>
(eg: /home/mythdesktopuser/comskip/mcomflag.sh /var/lib/mythtv/recordings 1088_20100522141700.mpg)

Note: There's no real error checking here, make sure you give mcomflag.sh two parameters: a recording directory, and a real .mpg file.

It's probably a good idea to record a short TV show or use a small .mpg file so it doesn't take too long to test!

If it's working you'll see the following on screen:

The commandline used was:
Z:\home\mythdesktopuser\comskip\comskip.exe /media/myth/recordings/1088_20100522141700.mpg

Opening /var/lib/mythtv/recordings/1088_20100522141700.mpg
Using Z:\home\mythdesktopuser\comskip\comskip.ini for initiation values.
 5:23:40 - 28670 frames in 38.04 sec (753.68 fps), 4.33 sec (65.59 fps), 99%%
28694 frames decoded in 38.07 seconds (753.72 fps)
Commercials were found.

Note: You should get Commercials were found, if not, make sure you have a file that contains commercials, OR you may need to tweak comskip.ini. If no commercials were found, then well there won't be any commercials to insert!

Then if you look at executions.log you'll see if it succeeded in sticking stuff in the database: EXAMPLE:

#cat /home/mythdesktopuser/comskip/executions.log
=============================
Running comskip under wine with command:
wine comskip.exe /var/lib/mythtv/recordings/1088_20100522141700.mpg
-----------------------------
Now running database insert script with command:
/home/mythdesktopuser/comskip/comskipmythdbinsert.php /var/lib/mythtv/recordings/1088_20100522141700.mpg


ComskipMythTVInserter - Version 0.1a - using commfile: 1088_20100522141700.txt
Deleting Marks
Adding new marks: XXXXXX (Total = 6)
Checking marks
Event mark: 1 Type = 4 - CommStart
Event mark: 2974 Type = 5 - CommEnd
Event mark: 17940 Type = 4 - CommStart
Event mark: 18750 Type = 5 - CommEnd
Event mark: 28682 Type = 4 - CommStart
Event mark: 28683 Type = 5 - CommEnd
Done!
=============================

The important thing to see is the "Event mark" lines after Checking marks - these are what has been read BACK from the mythconverg database. Each mark is frame number, and a type, these are inserted directly in to the mythconverg database, existing COMM_START and COMM_END marks for the recording are deleted.

With any luck, you're still with me at this point.

Making MythTV use the script

In the MythTV Backend Setup, under "1. General", flip through until you get to "Job Queue (Global)":

Uncheck 'Start Auto-Commercial Flagging jobs when the recording starts' (Because this script only works "after the fact")

Set Commercial Flagger command:

/home/mythdesktopuser/comskip/mcomflag.sh %DIR% %FILE%

i.e. This is the path to mcomflag.sh, it's passed %DIR% and %FILE% which is the same thing we simulated in the step above: passed in a directory for the recordings, and a filename.mpg for a recording.

OR - You could also 'nice' it up so it doesn't suck too much resources:

/usr/bin/nice -n 19 /home/mythdesktopuser/comskip/mcomflag.sh %DIR% %FILE% 

Step through to the finish, and exit mythtv setup!

That should be about it.

Testing a real recording

Probably a good time to test a real recording. Record a short show that's likely to contain commercials, and see how you go.

After the commercial flagging has been run, you can tail /home/mythdesktopuser/comskip/executions.log - you should see the 'Event mark' lines.

A final word on auto vs manual skipping

As even comskip.exe isn't perfect I don't like to have autoskip turned on. I have Myth set to "notify but don't skip". Then I use my remote OR the 'Z' & 'Q' keys to jump between marks.

I find this to be the best method, sometimes Australian TV stations chuck two quick commercials together, and you end up 'out of sync', if you're automatically skipping commercials then you miss parts of your show. If you use the keys, it doesn't matter whether the START/END marks for the commercials are the right way around, 'Z' simply jumps to the next mark whether it's in or out!.

Also, while sometimes Comskip.exe doesn't pick up the commercial at the very start, but it does seem to be pretty good at picking up the transition back in to the TV show - which means a couple of skips forward, and you've back in to your show! Very handy! :)

I also like to have it set to notify me 5 seconds before a commercial, and come back 1 second before the end of a commercial. These settings are in Myth's front end: Utilities/Setup -> Setup -> TV Settings -> Playback -> Commercial Skip (8/8)

Commercial Skip Notify Amount = 5
Commercial Skip Auto-Rewind Amount = 1

This gives me enough time to grab the remote before a commercial, and the 1 second rewind allows me to confirm that I'm not missing any of my show.