Split films
From MythTV Official Wiki
Contents
Aims
Some channels in the UK transmit films in 2 parts split by a short news item. It is frustrating to start watching the film at a later date only to find the second half missing! 'Record All' will not resolve this because ProgramId, Title and Subtitle are all the same for the two halves so the scheduler sees part 2 it as a duplicate. Programs such as this one using API calls to trigger recordings are not so constrained.
At the time of writing, the 'interesting' channels which do this are ITV2, ITV4, ITVBe, the 'GREAT' channels, Channel 5, 5ACTION, 5STAR, their +1 hours and their HD versions but callsigns may change in the future. 5SELECT also splits films but unfortunately wrongly categorises them as 'Social/Policical/Economics' (with the spelling error!) rather than 'Film' so is not a suitable channel for automation.
This perl script looks for these split films and triggers a separate recording for the second half automatically. It has been developed with xubuntu 20.04/22.04 and Mythtv 31 and 32.
If you run Mythtv v34 then be advised that there were a number of changes to the API interface. Version 2 of this program (see --help) should survive those changes.
How does it work?
The script is triggered by a system event either at the start of a recording or after the end.
If the channel is an 'interesting' one, it will read the first 6 programs (#0 to #5) from the guide, starting 'now' or at a time specified by --starttime.
If either of the first two entries is:
- a film which is recorded, recording or will record (a 'first half') AND
- the next but one entry is not being recorded and has the same Title and Description (a second half) AND
- other neighbours do not have the same Title and Description THEN
it will create a new recording rule for the second half.
The first and second entries are checked because we need to catch two situations. The start time may be shortly before the guide start time if pre-scheduled (second guide entry #1 will be part 1) or it could be manually triggered for recording after the guide start time and the film has already started (first guide entry #0 will be part 1).
Title and description are used for matching because this has proved more reliable than the more intuitive ProgramId.
Calling Parameters
--ChanId eg --ChanId=20025
--help or -h help text
--list or -l lists all visible channels showing channel id and CallSign.
--report or -r report only - does not trigger a recording.
--verbose or-v extra diagnostics
--Starttime Use if triggering with the recording ended event or for testing purposes.
eg --starttime=2022-03-15T21:00:00Z Default is 'now'.
--desc optional description which is reflected in the log. Typically %TITLE%.
--setup print out commands needed to create the initial log, lock and config files.
Setup
1. Put this perl script in checkfilm.pl somewhere in path (eg /usr/local/bin) and make it executable (chmod +x).
Check this with checkfilm.pl --help
2. Put the module scan_database.pm in perl path and make it everyone readable.
See: https://www.mythtv.org/wiki/Perl_API_examples
Check: You can list channels with checkfilm.pl --list
3. Create three files.
A log file /var/log/mythtv/checkfilm.log,
a lock file /home/mythtv/.mythtv/checkfilm.lock
the config file /home/mythtv/.mythtv/checkfilm.cfg which hold a list of 'interesting' channels.
the three files can be generated with:
checkfilm.pl --setup > setup.sh
chmod +x setup.sh
./setup.sh and supply sudo password.
4. Set up a system event 'Recording Started Writing':
sleep 20 && /usr/local/bin/checkfilm.pl -v --chanid=%CHANID% --desc=%TITLE% >> /var/log/mythtv/checkfilm.log 2>&1
If you feel that things are too busy for your backend at start of recording then you can set up the 'Recording ended' event instead, provided that post recording time is less than the duration of the 'news' item. You will need also to provide the starttime:
sleep 20 && /usr/local/bin/checkfilm.pl -v --chanid=%CHANID% --starttime=%STARTTIMEISOUTC% --desc=%TITLE% >> /var/log/mythtv/checkfilm.log 2>&1
You can drop the -v for a more concise log containing only triggered recordings.
Channel Filter
The config file /home/mythtv/.mythtv/checkfilm.cfg hold a list of interesting channels which need checking but if the file is missing then all channels are checked.
Setup will have inserted the channels known to be interesting as of December 2022 but after a retuning exercise you may need to re-populate the config file eg
checkfilm.pl -l | grep ITV4 > /home/mythtv/.mythtv/checkfilm.cfg checkfilm.pl -l | grep GREAT >> /home/mythtv/.mythtv/checkfilm.cfg checkfilm.pl -l | grep ITV2 >> /home/mythtv/.mythtv/checkfilm.cfg checkfilm.pl -l | grep 'Channel 5' >> /home/mythtv/.mythtv/checkfilm.cfg checkfilm.pl -l | grep 5ACTION >> /home/mythtv/.mythtv/checkfilm.cfg
The file will then have entries such as '20025 = ITV4'
Only the initial digits are used - the CallSign is comment.
Logging
Logging is maintained by a redirect in the system event line and only takes place if the channel is an 'interesting' one or --report is set. Log entries will consists of a single line saying that a recording had been triggered unless --verbose is set. eg
WillRecord Film 2022-03-20T22:01:00Z Hitman Redemption
Verbose logging will generate a more detailed log eg:
2022-11-18T20:15:20Z checkfilm -v --chanid=20041 --starttime=2022-11-18T19:11:00Z Reading config file /home/mythtv/.mythtv/checkfilm.cfg Found 20041 in config Callsign is GREAT! movies action # Status Category StartTime Title 0 Unknown Film 2022-11-18T18:11:00Z Hell River 1 Recorded Film 2022-11-18T19:15:00Z Miami Magma 2 Unknown Film 2022-11-18T20:10:00Z This Week Back Then 3 Unknown Film 2022-11-18T20:16:00Z Miami Magma 4 Unknown Film 2022-11-18T21:00:00Z Renegades 5 Unknown Film 2022-11-18T21:55:00Z This Week Back Then #1 is a film #1 is recording or recorded #3 matches #1 has no clashing neighbours #3 needs recording Triggering recording Change confirmed at try 1 Recording Film 2022-11-18T20:16:00Z Miami Magma
To inhibit logging completely just redirect output to /dev/null.
Locking
To prevent problems with two simultaneous invocations of the code a lock file is used: /home/mythtv/.mythtv/checkfilm.lock
Times
Note that all times are in UTC. This matches UK winter time but a summer recording at 9pm will show as 20:00:00.
Perl code
Copy the following code into path eg to /usr/local/bin/checkfilm.pl
#!/usr/bin/perl -w
use strict;
use Getopt::Long;
use Time::Local;
use Fcntl qw(:flock);
use lib '.';
#
# look for split films and record second half.
#12 April 22. Change log file to be NOt in /tmp (cannot write to it).
#Dec 2022 Revised strategy:
# Use fingerprint = title+description for matching
# Don't check category because unreliable. Many channels set it wrong.
#
# 4 Sept 2023. Changes made to sub getguide for v34and tested with port 6744.
# a) if program is NOT being recorded then Status is omitted from guide data
# b) Status now returned as integer - need to use StatusName instead.
# code should now work both pre and post change.
# See https://www.mythtv.org/wiki/Recording_Status for details.
# Also, included checks on module Scan_Database version.
#
my $lockfile='/home/mythtv/.mythtv/checkfilm.lock';
my $confile='/home/mythtv/.mythtv/checkfilm.cfg';
my %validchannels;
my $backend='http://127.0.0.1:6544';
my $content;
my %guide;
my $match='fingerprint'; #set to ProgramId or 'fingerprint' for Title + Description
my $logfile='/var/log/mythtv/checkfilm.log';
#Many film channels mis-categorise programs.
#Set this non zero to check all categories, not just film
my $AllowAllCategories=1;
#Get calling params
my $calling=join(' ',@ARGV);
my $ChanId = -1; my $StartTime = ''; my $listchannels=0; my $reporting=0; my $help=0; my $verbose=0;
my $developer=0; my $setup=0; my $desc='';
GetOptions ('ChanId=i' => \$ChanId, 'StartTime=s' =>\$StartTime,
'list'=>\$listchannels, 'report' => \$reporting,
'help' => \$help, 'verbose' => \$verbose,
'setup' => \$setup,
'developer' => \$developer,
'desc=s' => \$desc);
# $developer - unused
givehelp() if ($help);
generatesetup() if ($setup);
#we need a module:
my $sdv; #version no.
BEGIN {
unless (eval "require scan_database") {
print "couldn't load scan_database module\nSee https://www.mythtv.org/wiki/Perl_API_examples\n";
exit 0;
}
}
check_scan_database_version();
listchannels() if ($listchannels);
#Hash of recording status texts and whether they indicate records is being/will be made
my %recording=(
WillRecord => 1,
Pending => 1,
Tuning => 1,
Recording => 1,
Unknown => 0,
Failed => 0,
Recorded => 1,
"Don'tRecord" => 0
);
#open log file and lock it
open(LOCKFILE,'>',$lockfile) or die "Cannot open logfile $lockfile:\n$!\n";
flock(LOCKFILE, LOCK_EX); #wait til it's free
#vprint("\n$now log file open and locked");
#off we go:
my $now=TimeString(time());
prechecks(); #exit unless listing channels or have a valid channel
getguide($ChanId, $StartTime);
#show callsign and guide data
$content =~ m!<CallSign>([^<]*)</CallSign>!;
if ($verbose){
print "Callsign is $1\n";
showguide(0);
}
unless (exists $validchannels{$ChanId}){exit 0};
my $offset;
if (isolatedpart(1)){
vprint('#3 needs recording');
$offset=1;
}elsif (isolatedpart(0)){
vprint('#2 needs recording');
$offset=0;
}else{ vprint('nothing to do'); myexit('');
}
#sanity checks
if ($now ge $guide{2+$offset}{EndTime}){myexit("Too late to record this")};
if ($guide{$offset}{RecordId} == 0){myexit ("No recording rule for #$offset")};
$_=$offset +2;
vprint("Triggering recording");
if ($reporting){myexit('Recording suppressed as only reporting')};
#Get recording rule for part 1
my $url=$backend . "/Dvr/GetRecordSchedule?RecordId=$guide{$offset}{RecordId}";
scan_database::ReadBackend($url, $content);
my %recrule;
scan_database::GetAllFields(%recrule, $content, '>', '</RecRule>');
#modify it and trigger recording for part 2
$recrule{StartTime}=$guide{2+$offset}{StartTime};
$recrule{EndTime}=$guide{2+$offset}{EndTime};
$recrule{Station}=$recrule{CallSign};
scan_database::ValidatePost(%recrule, $backend .'/Dvr/AddRecordSchedule', 'raw', 12);
#confirm changed
my $found=0;
for my $try (1 .. 6){
sleep(3);
getguide($ChanId, $StartTime);
$_= $guide{$offset+2}{Status};
if ($recording{$_}==1){
vprint("Change confirmed at try $try");
$found=1;
last;
};
};
if ($found){
showguide(2+$offset)
}else{
print "Recording of $guide{$offset}{Title} triggered but not confirmed\n";
};
myexit('');
sub showguide{
my($start)=@_;
if ($start){ #final confirmation
printf "%-10s %-10s %22s $guide{$start}{Title}\n", $guide{$start}{Status}, $guide{$start}{Category}, $guide{$start}{StartTime};
return;
};
print "# Status Category StartTime Title\n";
for (0..5){
printf "$_ %-14s %-15s %22s $guide{$_}{Title}\n", $guide{$_}{Status}, $guide{$_}{Category}, $guide{$_}{StartTime};
#print "$guide{$_}{ProgramId}\n";
};
}
sub isolatedpart{
my($offset)=@_;
my $target=$offset+2;
#Check whether program '$offset' is a part 1 which needs a part 2 triggering
#first check if part 1 is film (or that all categories allowed)
vprint("#$offset category is $guide{$offset}{Category}");
if ($guide{0+$offset}{Category} ne 'Film'){
return 0 if ($AllowAllCategories==0);
}
#and that it is recording
$_= $guide{$offset}{Status};
if ($recording{$_}==0){vprint("#$offset is not recording"); return 0};
vprint("#$offset is recording or recorded");
#Check if part 2 matches
my $matchtext=$guide{0+$offset}{$match};
if ($guide{2+$offset}{$match} ne $matchtext){vprint("#$target does not match"); return 0};
vprint("#$target matches");
#Check neigbours have different ProgramId
for (1,3,4){
if ($guide{$_+$offset}{$match} eq $matchtext){ vprint("#$offset has clashing neighbour"); return 0};
}
vprint("#$offset has no clashing neighbours");
#Is part2 already scheduled or recorded?
$_= $guide{$offset+2}{Status};
if ($recording{$_}){vprint("#$target is being recorded already");return 0};
return 1; #this one can be recorded!
}
sub prechecks{
#Do checks before we open the scan_database module.
vprint("\n$now checkfilm $calling");
vprint("ScanDatabase version $sdv");
if ($ChanId==-1){myexit("Need --help or --list or --ChanId or --setup")};
$verbose=1 if $reporting;
#development aid -mythutil triggered invocation
if ($ChanId==0){$ChanId=20025; $verbose=1}; #if invoked by mythutil
if ($StartTime eq ''){$StartTime=$now}; #standard action
#Check the config file
unless (-r $confile){
vprint("no config file - using dummy channel entry");
$validchannels{$ChanId}='Unknown';
return;
}
#vprint("Reading config file $confile");
open(CONFIG,'<',$confile) or myexit("Cannot open config file $confile:\n$!");
while (<CONFIG>){
chomp;
#vprint($_);
next unless (/\=/);
s/^\s+//; #kill leading spaces
s/\s+$//; #trailing
next unless length;
my ($k,$v)=split(/\s*=\s*/,$_,2);
$validchannels{$k}=$v;
};
close CONFIG;
if (exists $validchannels{$ChanId}){
vprint("Found $ChanId in config");
return;
}else{
vprint("Not an interesting channel: $ChanId");
myexit('');
}
print "\nAt $now: checkfilm $calling\n" unless ($verbose);
}
sub getguide{
my ($chan, $start)=@_;
#Read the guide, show callsign, get 6 entries and show them
my $url="$backend/Guide/GetProgramList?StartTime=$StartTime&ChanId=$ChanId&Count=6&Details=true";
vprint($url);
unless (scan_database::ReadBackend($url,$content)){myexit("Could not get guide data")};
#Did we get any guide data?
$content =~ m!<Count>(\w+)</Count>!;
myexit('No guide data') if ($1==0);
#extract fields
scan_database::FillHashofHash(%guide, $content,'Program','#','StartTime','EndTime','Category','Title','ProgramId','Status',
'StatusName', 'RecordId','Description');
#check valid status values
for (0..5){
my $status=$guide{$_}{Status};
if ($status eq ''){
$status="Don'tRecord"
}elsif ($status =~ /^-?\d+$/){ # see https://www.mythtv.org/wiki/Recording_Status
$status=$guide{$_}{StatusName};
}
$guide{$_}{Status}=$status;
unless (exists $recording{$status}){
vprint("warning: Status not known: $status");
$recording{$status}=0; #Assume not a recording status
}
$guide{$_}{fingerprint}=$guide{$_}{Title} . $guide{$_}{Description};
}
}
sub vprint{
print "$_[0]\n" if ($verbose);
}
sub myexit{
if ($_[0] ne ''){print "$_[0]\n"};
close(LOCKFILE);
exit 0;
}
sub check_scan_database_version{
$sdv=$scan_database::VERSION;
$sdv||=0;
if ($sdv < 1.13){print "\nscan_database.pm need to be at least 1.13 to support version 2 of API interface.\n";
print "Please update it from https://www.mythtv.org/wiki/Perl_API_examples\n";
exit 0;
};
}
sub TimeString{
(my $epoch)=@_;
#return time as 2021-12-03T13:44:04
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = gmtime($epoch);
$year+=1900; $mon++;
return sprintf("%04d-%02d-%02dT%02d:%02d:%02dZ", $year, $mon, $mday, $hour, $min, $sec);
}
sub ZtoEpoch{
(my $Z)=@_;
#eg 2022-03-17T20:00:00Z to epoch seconds
$Z =~ /(\d\d\d\d)-(\d\d)-(\d\d)T(\d\d):(\d\d):(\d\d)Z/;
return timegm($6,$5,$4,$3,$2-1,$1-1900);
}
sub listchannels{
#list all channels - grep it to extract the interesting ones and put in %interestingchannels
my $temp; my %sources; my %ChanData;
#get sources
my $url=$backend. '/Channel/GetVideoSourceList';
scan_database::ReadBackend($url, $temp);
scan_database::FillHashofHash(%sources, $temp, 'VideoSource', 'Id', 'SourceName');
#get channels per source
for my $source (keys %sources){
scan_database::ReadBackend($backend . '/Channel/GetChannelInfoList?SourceID='.$source.
'&OnlyVisible=false&Details=true', $temp);
my %temphash;
scan_database::FillHashofHash(%temphash, $temp, 'ChannelInfo', 'ChanId', 'CallSign','Visible');
%ChanData = (%ChanData, %temphash);
}
for (sort keys %ChanData){
if ($ChanData{$_}{Visible} eq 'true'){print "$_ = $ChanData{$_}{CallSign}\n"};
}
exit 0;
}
sub generatesetup{
print my $setupfile="#!/bin/bash
#
# This script generates the necessary files for setting up checkfilm.
# Redirect to a file, chmod +x it then execute it.
#
# It sets up for UK channels which split films as of December 2022.
#
# You will also need to set up a system event.
#
echo 'Generating log file $logfile'
sudo touch $logfile
sudo chmod 666 $logfile
echo 'Generating lock file $lockfile'
sudo touch $lockfile
sudo chown mythtv:mythtv $lockfile
sudo chmod 666 $lockfile
echo 'Generating config file $confile'
sudo touch $confile
sudo chown mythtv:mythtv $confile
sudo chmod 666 $confile
checkfilm.pl -l | grep ITV4 > $confile
checkfilm.pl -l | grep GREAT >> $confile
checkfilm.pl -l | grep ITV2 >> $confile
checkfilm.pl -l | grep '5STAR' >> $confile
checkfilm.pl -l | grep '5ACTION' >> $confile
checkfilm.pl -l | grep '5SELECT' >> $confile
checkfilm.pl -l | grep ITVBe >> $confile
checkfilm.pl -l | grep 'Channel 5' >> $confile
";
exit;
}
sub givehelp{
my $location='/usr/local/bin';
print "
checkfilm.pl version 2
========================
Aims
----
Some channels in the UK transmit films in 2 parts split by a short news item. It is frustrating to start watching the film
at a later date only to find the second half missing! 'Record All' will not resolve this.
At the time of writing, the 'interesting' channels which do this are ITV2, ITV4, the 'GREAT' channels, Channel 5, 5ACTION, 5STAR,
5SELECT and their +1 and HD versions, but names may change in the future.
This perl script looks for such recordings and triggers the second half automatically.
Version 34 of Mythtv includes a number of changes affecting recording status as returned by guide APIs. Version 2 of this code should survive this change.
How does it work?
-----------------
The script is run by a system event either at the start of a recording or after the end.
If the channel is an 'interesting' one, it will read the first 6 programs (#0 to #5) from the guide,
starting 'now' or at a time specified by --starttime.
If either of the first two entries is:
- a film which is recorded, recording or will record (a 'first half') AND
- the next but one entry is not being recorded and has the same Title and Description (a second half) AND
- neighbours do not have the same Title and Description THEN
it will create a new recording rule for the second half.
The first and second entries are checked because we need to catch two situations. The start time may be:
- shortly before the guide start time if pre-scheduled (guide entry #1 will be part 1) OR
- after the guide start time if manually triggered after the film started (guide entry #0 will be part 1).
Title and description are used for matching because this has proved more reliable than the more intuitive ProgramId.
Parameters
----------
--ChanId eg --ChanId=20025
--help or -h this text
--list or -l lists all visible channels showing channel id and callsign.
--report or -r report only - does not trigger a recording.
--verbose or-v extra diagnostics
--Starttime Use if triggering with the recording ended event or for testing purposes.
eg --starttime=2022-03-15T21:00:00Z Default is 'now'.
--setup print out commands needed to create the log and lock files.
--desc optional data printed in log only (eg %TITLE%).
Setup
-----
1. Put this perl script in checkfilm.pl somewhere in path (eg $location)
and make it executable (chmod +x).
Check: display help with checkfilm.pl --help
2. Put the module scan_database.pm in perl path and make it everyone readable.
See: https://www.mythtv.org/wiki/Perl_API_examples
Check: You can list channels with checkfilm.pl --list
3. Create three files.
A log file $logfile,
a lock file $lockfile
the config file $confile which hold a list of 'interesting' channels.
Do this with:
checkfilm.pl --setup > setup.sh
chmod +x setup.sh
./setup.sh and supply sudo password.
4. Set up a system event Recording Started Writing:
sleep 20 && ${location}/checkfilm.pl -v --chanid=%CHANID% --desc=%TITLE%>> $logfile 2>&1
If you feel that things are too busy for your backend at start of recording then
you can set up the Recording ended event instead, provided that post recording
time is less than the duration of the 'news' item. You will need also to provide the starttime:
sleep 20 && ${location}/checkfilm.pl -v --chanid=%CHANID% --starttime=%STARTTIMEISOUTC% --desc=%TITLE%>> $logfile 2>&1
You can drop the -v for a log containing only triggered recordings.
Channel Filter
--------------
If the config file $confile is missing then all channels are checked, but setup will have inserted
the channels known to be interesting as of November 2022.
After a retuning you may need to re-populate the config file eg
checkfilm.pl -l | grep ITV4 > $confile
checkfilm.pl -l | grep GREAT >> $confile
checkfilm.pl -l | grep ITV2 >> $confile
checkfilm.pl -l | grep ITVBe >> $confile
The file will then have entries such as '20025 = ITV4'
Only the initial digits are used - the CallSign is comment.
Problem Channels
----------------
Some channels mis-categorise some programs and do not declare them as film.
To inhibit the 'is this a film' check then set \$AllowAllCategories near the head of the perl script to non zero.
The parameter is currently set to \$AllowAllCategories=$AllowAllCategories
Logging
-------
Logging is maintained by a redirect in the system event line and only takes place if the channel is an
'interesting' one or --report is set.
log entries will consists of a single line saying that a recording had been triggered unless --verbose is set.
eg
WillRecord Film 2022-03-20T22:01:00Z Hitman Redemption
To inhibit logging completely just redirect ouput to /dev/null.
Locking
-------
To prevent problems with two sumultaneous invocations of the code a lock file is used: $lockfile
Times
-----
Note that all times are in UTC. This matches UK winter time but a summer recording at 9pm will show as 20:00:00.
";
exit 0;
}
Phil Brady. 4 Sept 2023.
