Commercial detection in the UK

This page is to collect users' efforts at detecting commercials (adverts/ads in UK speak) on UK channels.

Ad breaks on UK channels don't follow the same pattern as those in the US, so existing commflagging programs frequently fail to detect breaks.

Ad break patterns

(Put any general info regarding ad break patterns here) Ad breaks on UK channels are usually shorter than in the US, and are usually separated by at least 10 mins of actual programme. (UK broadcasters don't have the same habit of showing a few "teaser" minutes at the start, followed by an ad break).

Ad breaks usually last around 4-5mins on the old "traditional" channels (ITV, Channel 4 and Channel 5). (These are sometimes known as "terrestrial" or "analogue" channels as they are the only commercial channels you get on the standard analogue terrestrial service). On "non-terrestrial"/"satellite"/"cable" channels the ad breaks can be much longer (and on some stations LOUDER![1]).

OFCOM has set general standards for how adverts should be separated from programmes [2] and how much advertising there should be [3] Particularly interesting are the rules in Table 1 which are enforced for channels 3-5 (i.e. ITV1,Channel4 and Five). Also paragraph 11:

Broadcasters must ensure that television advertising and teleshopping is readily 
recognisable and distinguishable from editorial content and kept distinct from other 
parts of the programme service. This shall be done by optical (including spatial) 
means; acoustic signals may also be used as well.

Approach

There doesn't seem to be any set pattern regarding fading to black, blank pictures, use of logos etc across all the channels. However, analysis of individual channels may reveal patterns that may allow commflagging of that individual channel. For example, Sky One is said succumb to Logo Detection as it doesn't show its logo in ad breaks.

Therefore, this page serves to collect user's successfull settings, observations of patterns, suggestions for potential detection methods ON A CHANNEL BY CHANNEL BASIS.

Commflagging Tools

There are a number of tools available which might be used to provide commflagging.

Mythcommflag

mythcommflag is MythTV's built in commercial flagging program.

The key advantage of mythcommflag is that it is integrated with MythTV and so can insert the detected breaks into the database automatically.

Comskip

Comskip is a Windows based program (originally based on mythcommflag?) used by a number of Windows based PVRs (e.g. GB-PVR) It includes a couple of extra detection methods not provided by mythcommflag. Experiences of Windows users can also be applied to Myth commflagging.

Comskip reportedly will work on Linux under Wine.

Interesting post regarding a custom comskip [4].

ShowAnalyzer

ShowAnalyzer or SA is a commercial/shareware Windows based program. Cost 28/3/08 is US$19.95. SA's author has made attempts at specifically producing UK detection methods and there are a number of user's config files posted on SAs support forum.

ShowAnalyzer reportedly will work on Linux under Wine.

Potential Methods for Ad Break Detection

Put any general methods that might be used here.

Silence Detection

Reportedly there is a UK requirement for a 120ms silence before and after each ad break.

This works surprisingly well: proof of concept (i.e., hack) silence-detect.sh. A user has developed this further into a practical solution called Mythcommflag-wrapper.

Another version which works with audio streams that are not MP3 (i.e.FreeviewHD) is Commercial_detection_with_silence_for_UK_freeviewHD. It seems to work on most of the Freeview and FreeviewHD channels.

Louder Adverts

According to [5] advertisers often increase the percieved volume of adverts by increasing the compression levels. So a 4-5 min loud section with low dynamic range would be an ad break indicator.

Change of aspect ratio

Some programmes are in 4:3 aspect ratio because they were produced before widescreen technology. Adverts are rarely anything but 16:9. 4:3 is thus a strong indicator of main programme material.

Position of subtitles

Subtitles invariably appear in the bottom 20% of the screen area during main programmes. During adverts, they are either absent or the advertisers 'shout' by placing them at the top of the screen, the middle, sometimes at the bottom. A section with subtitles in the top 80% of screen is therefore very likely to be advertising material.

Cue Dots

‘cue dots’ or ‘cue marks’ are used to indicate to regional broadcasters that a commercial break is about to occur. These are a rectangular block at top right of the screen with rolling black and white stripes. See http://en.wikipedia.org/wiki/Cue_mark#Cue_marks_in_television A former regional TV programme planner says that these are now mainly used in live broadcasts eg “Britain’s got talent” May 2009.

ITV Group

ITV 1

Observations of Patterns

With a couple of the ITV channels (can't remember which ones) they have a huge amount of programs sponsored by one or two sponsors (an Insurance Company with a Heavy Blue Logo and an Indian Bottled Beer spring to mind), these sponsors are segued as the entry and exit points into the adverts. If we can spot those specific ads they'd be good markers.

Live TV recordings (News, sporting events) provide a black & white box in the top right of the screen to indicate a upcoming break. This is accordingly for syncing regions for adverts. This icon disappears 5 seconds before the channel identity is shown.

ITV1's identity between commercial breaks are silent, Comskip's silence detection should pick up easily on these.

Configurations

Mythcommflag

Place config files here...

Comskip

Place config files here...

ShowAnalyzer

Place config files here...

ITV 2

Observations of Patterns

All ITV2 ITV3 and ITV4 remove their coloured logo during commercial breaks, however for promos they leave in place a greyscale semi transparent logo.

With a couple of the ITV channels (can't remember which ones) they have a huge amount of programs sponsored by one or two sponsors (an Insurance Company with a Heavy Blue Logo and an Indian Bottled Beer spring to mind), these sponsors are segued as the entry and exit points into the adverts. If we can spot those specific ads they'd be good markers.

Configurations

Mythcommflag

Place config files here...

Comskip

Place config files here...

ShowAnalyzer

From [6]:

Longest Possible Commercial Segment = 400
Longest Possible Individual Commercial = 120
Shortest Possible Commercial Segment = 44.5
Shortest Possible Show Segment = 35.5
[Score Modifiers]
Length Too Long = 0.1
Low Probability Length = 0.7
Length Too Short = 1
Medium Probability Length If Combined = 1.2
Medium Probability Length = 1.5
High Probability Length If Combined = 1.5
High Probability Length = 2
Edge Profile Positive Match = 0.1
Edge Profile Negative Match = 2
Edge Profile Unknown Match = 1
[Options]
Aggressive Segmentation = 1

ITV 3

Observations of Patterns

All ITV2 ITV3 and ITV4 remove their coloured logo during commercial breaks, however for promos they leave in place a greyscale semi transparent logo.

With a couple of the ITV channels (can't remember which ones) they have a huge amount of programs sponsored by one or two sponsors (an Insurance Company with a Heavy Blue Logo and an Indian Bottled Beer spring to mind), these sponsors are segwayed (spelling?) as the entry and exit points into the adverts. If we can spot those specific ads they'd be good markers.

Configurations

Mythcommflag

Place config files here...

Comskip

Place config files here...

ShowAnalyzer

From [7]:

Longest Possible Commercial Segment = 400
Longest Possible Individual Commercial = 120
Shortest Possible Commercial Segment = 44.5
Shortest Possible Show Segment = 35.5
[Score Modifiers]
Length Too Long = 0.1
Low Probability Length = 0.7
Length Too Short = 1
Medium Probability Length If Combined = 1.2
Medium Probability Length = 1.5
High Probability Length If Combined = 1.5
High Probability Length = 2
Edge Profile Positive Match = 0.1
Edge Profile Negative Match = 2
Edge Profile Unknown Match = 1
[Options]
Aggressive Segmentation = 1

ITV 4

Observations of Patterns

All ITV2 ITV3 and ITV4 remove their coloured logo during commercial breaks, however for promos they leave in place a greyscale semi transparent logo.

With a couple of the ITV channels (can't remember which ones) they have a huge amount of programs sponsored by one or two sponsors (an Insurance Company with a Heavy Blue Logo and an Indian Bottled Beer spring to mind), these sponsors are segwayed (spelling?) as the entry and exit points into the adverts. If we can spot those specific ads they'd be good markers.

Configurations

Mythcommflag

Place config files here...

Comskip

Place config files here...

ShowAnalyzer

Place config files here...

---

Channel 4 Group

Channel 4

Observations of Patterns

Channel ident at the beginning of commercial breaks is silent for at least a second. Adverts usually have a short silence between them. Promos are shown after the silent channel ident before returning to the program.

Methods that MIGHT work

Early morning programs e.g. Frasier at 8.30am contain a clock in the top left. Could this be used as a logo?

Methods that have some success

• After several runs with the various flagging methods available in MythTV, simply put, none of them work correctly. The best is Blank Scene detection which can usually pick out a few adverts out of the break.

Configurations

Mythcommflag

Change commmethod on the channel to 1

Comskip

1)Frasier Weekday Mornings (breakfast TV) Source: DVB-T Tried on Frasier (Weekday mornings has clock as icon). Good results with default comskip.ini file. However, the clock disappears before the end of the show segment and begins after the start.

Therefore

padding=6

is recommended.

ShowAnalyzer

The following settings were posted are from [8]

Longest Possible Commercial Segment = 420
Longest Possible Individual Commercial = 120
Shortest Possible Commercial Segment = 74
Shortest Possible Show Segment = 33
[Score Modifiers]
Length Too Long = 1
Low Probability Length = 1.4
Length Too Short = 1
Medium Probability Length If Combined = 1.4
Medium Probability Length = 1
High Probability Length If Combined = 1.5
High Probability Length = 1.5	
Edge Profile Positive Match = 0.5
Edge Profile Negative Match = 1
Edge Profile Unknown Match = 1
[Options]
Aggressive Segmentation = 1

"This doesnt mark any bits of show as commercial, but doesnt detect all commercial segments or all individual commercials."

E4

Methods that MIGHT work

Early morning programs e.g. Frasier at 8.30am contain a clock in the top left. Could this be used as a logo?

Configurations

Mythcommflag

Place config files here...

Comskip

Place config files here...

ShowAnalyzer

Place config files here...

More4

Observations of Patterns

Most shows have a More4 logo always displayed in the top left which disappears during the ad breaks. The logo is semi-transparent and unfortunately doesn't get detected :(

Methods that MIGHT work

Early morning programs e.g. Frasier at 8.30am contain a clock in the top left. Could this be used as a logo?

Configurations

Mythcommflag

Place config files here...

Comskip

Place config files here...

ShowAnalyzer

Place config files here...

FilmFour

Methods that MIGHT work

Early morning programs e.g. Frasier at 8.30am contain a clock in the top left. Could this be used as a logo?

Configurations

Mythcommflag

Place config files here...

Comskip

Place config files here...

ShowAnalyzer

Place config files here...

---

Channel 5 Group

Channel 5

Configurations

Mythcommflag

Place config files here...

Comskip

Place config files here...

ShowAnalyzer

From [9]

Longest Possible Commercial Segment = 400
Longest Possible Individual Commercial = 120
Shortest Possible Commercial Segment = 18
Shortest Possible Show Segment = 57
[Score Modifiers]
Length Too Long = 0.1
Low Probability Length = 1.8
Length Too Short = 1
Medium Probability Length If Combined = 1.2
Medium Probability Length = 1.5
High Probability Length If Combined = 2
High Probability Length = 2
Edge Profile Positive Match = 0.1
Edge Profile Negative Match = 1
Edge Profile Unknown Match = 1
[Options]
Aggressive Segmentation = 0

Generic Settings

The following settings are non-channel-specific:

Comskip

The following was taken from [10]:

detect_method=111         ;1=black frame, 2=logo, 4=scene change, 8=fuzzy logic, 16=closed captions, 32=aspect ration, 64=silence, 128=cutscenes, 255=all
validate_silence=1         ; Default, set to 0 to force using this clues if selected above.
validate_uniform=1         ; Default, set to 0 to force using this clues (like pure white frames) if blackframe is selected above.
validate_scenechange=1      ; Default, set to 0 to force using this clues if selected above.
verbose=0            ;show a lot of extra info, level 5 is also OK, set to 0 to disable
max_brightness=60               ;frame not black if any pixels checked are greater than this (scale 0 to 255)
test_brightness=40            ;frame not pure black if any pixels checked are greater than this, will check average brightness (scale 0 to 255)
max_avg_brightness=25         ;maximum average brightness for a dim frame to be considered black (scale 0 to 255) 0 means autosetting
max_commercialbreak=600       ;maximum length in seconds to consider a segment a commercial break
min_commercialbreak=25         ;minimum length in seconds to consider a segment a commercial break
max_commercial_size=155         ;maximum time in seconds for a single commercial or multiple commercials if no breaks in between
min_commercial_size=4         ;mimimum time in seconds for a single commercial
min_show_segment_length=155    ; any segment longer than this will be scored towards show.
non_uniformity=1500         ; Set to 0 to disable cutpoints based on uniform frames
max_volume=500            ; any frame with sound volume larger than this will not be regarded as black frame
min_silence=20            ; Any deep silence longer than this amount  of frames is a possible cutpoint
ticker_tape=0            ; Amount of pixels from bottom to ignore in all processing
logo_at_bottom=0         ; Set to 1 to search only for logo at the lower half of the video, do not combine with subtitle setting
punish=4               ; Compare to average for sum of 1=brightness, 2=uniform 4=volume, 8=silence, 16=schange, set to 0 to disable
punish_threshold=1.3      ; Multiply when amount is above average * punish_threshold
punish_modifier=2         ; When above average * threshold multiply score by this value
intelligent_brightness=0       ; Set to 1 to use a USA specific algorithm to tune some of the settings, not adviced outside the USA
logo_percentile=0.92         ; if more then this amount of logo is found then logo detection will be disabled
logo_threshold=0.75
punish_no_logo=1         ; Default, set to 0 to avoid show segments without logo to be scored towards commercial
aggressive_logo_rejection=0
connect_blocks_with_logo=1      ; set to 1 if you want successive blocks with logo on the transition to be regarded as connected, set to 0 to disable
cut_on_ar_change=0         ; set to 1 if you want to cut also on aspect ratio changes when logo is present, set to 2 to force cuts on aspect ratio changes. set to 0 to disable
delete_show_after_last_commercial=0   ; set to 1 if you want to delete the last block if its a show and after a commercial
delete_show_before_or_after_current=0   ; set to 1 if you want to delete the previous and the next show in the recording, this can lead to the deletion of trailers of next show
delete_block_after_commercial=0   ;set to max size of block in seconds to be discarded, set to 0 to disable
remove_before=4            ; amount of seconds of show to be removed before ALL commercials
remove_after=4            ; amount of seconds of show to be removed after ALL commercials
shrink_logo=5            ; Reduce the duration of the logo with this amount of seconds
after_logo=0      ; set to number of seconds after logo disappears comskip should start to search for silence to insert an additional cutpoint
padding=0
ms_audio_delay=5
max_repair_size=200         ; Will repair maximum 200 missing MPEG frames in the timeline, set to 0 to disable repairing for players that don't use PTS.
delete_logo_file=0         ; set to 1 if you want comskip to tidy up after finishing
output_framearray=0         ; create a big excel file for detailed analysis, set to 0 to disable
output_videoredo=0
output_womble=0
output_mls=0         ; set to 1 if you want MPeg Video Wizard bookmark file output
output_cuttermaran=0
output_mpeg2schnitt=0
output_mpgtx=0
output_dvrcut=0
output_zoomplayer_chapter=0
output_zoomplayer_cutlist=0
output_edl=0
output_edlx=0
output_vcf=0
output_bsplayer=0
output_btv=0            ; set to 1 if you want Beyond TV chapter cutlist output
output_projectx=0         ; set to 1 if you want ProjectX cutlist output (Xcl)
output_avisynth=0
output_demux=0            ; set to 1 if you want comskip to demux the mpeg file while scanning
sage_framenumber_bug=0
sage_minute_bug=0
live_tv=0               ; set to 1 if you use parallelprocessing and need the output while recording
live_tv_retries=1         ; change to 16 when using live_tv in BTV
standoff=0               ; change to 8000000 when using live_tv in BTV
cuttermaran_options="cut=\"true\" unattended=\"true\" muxResult=\"false\" snapToCutPoints=\"true\" closeApp=\"true\""
mpeg2schnitt_options="mpeg2schnitt.exe /S /E /R25  /Z %2 %1"
avisynth_options="LoadPlugin(\"MPEG2Dec3.dll\") \nMPEG2Source(\"%s\")\n"
dvrcut_options="dvrcut \"%s.dvr-ms\" \"%s_clean.dvr-ms\" "
windowtitle="Comskip - %s"

DVB Subtitles

a lot of ITV originated programming with subtitles can be seen to exhibit a full character white rectangle before and after each show segment. If these can somehow be detected it will become possible to mark such ad breaks with 100% accuracy where the rectangle appears. Other channels' programming may also have similar subtitle marks - as yet only ITV shows have been examined in detail.

Links

Commflagging - mythcommflag
[http://www.kaashoek.com/comskip/ ] - Comskip
[http://www.dragonglobal.org/showanalyzer.html ] - ShowAnalyzer