Split films

From MythTV Official Wiki
Jump to: navigation, search

Aims

Some channels in the UK transmit films in 2 parts split by a short news item. It is frustrating to start watching the film at a later date only to find the second half missing! 'Record All' will not resolve this because ProgramId, Title and Subtitle are all the same for the two halves so the scheduler sees part 2 it as a duplicate. Programs such as this one using API calls to trigger recordings are not so constrained.

At the time of writing, the 'interesting' channels which do this are ITV2, ITV4, ITVBe, the 'GREAT' channels, Channel 5, 5ACTION, 5STAR, their +1 hours and their HD versions but callsigns may change in the future. 5SELECT also splits films but unfortunately wrongly categorises them as 'Social/Policical/Economics' (with the spelling error!) rather than 'Film' so is not a suitable channel for automation.

This perl script looks for these split films and triggers a separate recording for the second half automatically. It has been developed with xubuntu 20.04/22.04 and Mythtv 31 and 32.

If you run Mythtv v34 then be advised that there were a number of changes to the API interface. Version 2 of this program (see --help) should survive those changes.


How does it work?

The script is triggered by a system event either at the start of a recording or after the end.

If the channel is an 'interesting' one, it will read the first 6 programs (#0 to #5) from the guide, starting 'now' or at a time specified by --starttime.

If either of the first two entries is:

  • a film which is recorded, recording or will record (a 'first half') AND
  • the next but one entry is not being recorded and has the same Title and Description (a second half) AND
  • other neighbours do not have the same Title and Description THEN

it will create a new recording rule for the second half.

The first and second entries are checked because we need to catch two situations. The start time may be shortly before the guide start time if pre-scheduled (second guide entry #1 will be part 1) or it could be manually triggered for recording after the guide start time and the film has already started (first guide entry #0 will be part 1).

Title and description are used for matching because this has proved more reliable than the more intuitive ProgramId.


Calling Parameters

  --ChanId        eg --ChanId=20025
  --help or -h    help text
  --list or -l    lists all visible channels showing channel id and CallSign.
  --report or -r  report only - does not trigger a recording.
  --verbose or-v  extra diagnostics
  --Starttime     Use if triggering with the recording ended event or for testing purposes.
                   eg --starttime=2022-03-15T21:00:00Z   Default is 'now'.
  --desc          optional description which is reflected in the log.  Typically %TITLE%.
  --setup         print out commands needed to create the initial log, lock and config files.


Setup

1. Put this perl script in checkfilm.pl somewhere in path (eg /usr/local/bin) and make it executable (chmod +x).
Check this with checkfilm.pl --help

2. Put the module scan_database.pm in perl path and make it everyone readable.
See: https://www.mythtv.org/wiki/Perl_API_examples
Check: You can list channels with checkfilm.pl --list

3. Create three files.
A log file /var/log/mythtv/checkfilm.log,
a lock file /home/mythtv/.mythtv/checkfilm.lock
the config file /home/mythtv/.mythtv/checkfilm.cfg which hold a list of 'interesting' channels.
the three files can be generated with:

    checkfilm.pl --setup > setup.sh
    chmod +x setup.sh 
    ./setup.sh and supply sudo password.

4. Set up a system event 'Recording Started Writing':

sleep 20 && /usr/local/bin/checkfilm.pl -v --chanid=%CHANID% --desc=%TITLE% >> /var/log/mythtv/checkfilm.log 2>&1

If you feel that things are too busy for your backend at start of recording then you can set up the 'Recording ended' event instead, provided that post recording time is less than the duration of the 'news' item. You will need also to provide the starttime:

sleep 20 && /usr/local/bin/checkfilm.pl -v --chanid=%CHANID% --starttime=%STARTTIMEISOUTC% --desc=%TITLE% >> /var/log/mythtv/checkfilm.log 2>&1

You can drop the -v for a more concise log containing only triggered recordings.

Channel Filter

The config file /home/mythtv/.mythtv/checkfilm.cfg hold a list of interesting channels which need checking but if the file is missing then all channels are checked.
Setup will have inserted the channels known to be interesting as of December 2022 but after a retuning exercise you may need to re-populate the config file eg

checkfilm.pl -l | grep ITV4 > /home/mythtv/.mythtv/checkfilm.cfg
checkfilm.pl -l | grep GREAT >> /home/mythtv/.mythtv/checkfilm.cfg
checkfilm.pl -l | grep ITV2 >> /home/mythtv/.mythtv/checkfilm.cfg
checkfilm.pl -l | grep 'Channel 5' >> /home/mythtv/.mythtv/checkfilm.cfg
checkfilm.pl -l | grep 5ACTION >> /home/mythtv/.mythtv/checkfilm.cfg

The file will then have entries such as '20025 = ITV4'
Only the initial digits are used - the CallSign is comment.

Logging

Logging is maintained by a redirect in the system event line and only takes place if the channel is an 'interesting' one or --report is set. Log entries will consists of a single line saying that a recording had been triggered unless --verbose is set. eg

  WillRecord Film 2022-03-20T22:01:00Z  Hitman Redemption

Verbose logging will generate a more detailed log eg:

2022-11-18T20:15:20Z checkfilm -v --chanid=20041 --starttime=2022-11-18T19:11:00Z
Reading config file /home/mythtv/.mythtv/checkfilm.cfg
Found 20041 in config
Callsign is GREAT! movies action
# Status     Category          StartTime             Title
0 Unknown    Film              2022-11-18T18:11:00Z  Hell River
1 Recorded   Film              2022-11-18T19:15:00Z  Miami Magma
2 Unknown    Film              2022-11-18T20:10:00Z  This Week Back Then
3 Unknown    Film              2022-11-18T20:16:00Z  Miami Magma
4 Unknown    Film              2022-11-18T21:00:00Z  Renegades
5 Unknown    Film              2022-11-18T21:55:00Z  This Week Back Then
#1 is a film
#1 is recording or recorded
#3 matches
#1 has no clashing neighbours
#3 needs recording
Triggering recording
Change confirmed at try 1
Recording  Film         2022-11-18T20:16:00Z  Miami Magma

To inhibit logging completely just redirect output to /dev/null.

Locking

To prevent problems with two simultaneous invocations of the code a lock file is used: /home/mythtv/.mythtv/checkfilm.lock

Times

Note that all times are in UTC. This matches UK winter time but a summer recording at 9pm will show as 20:00:00.

Perl code

Copy the following code into path eg to /usr/local/bin/checkfilm.pl

#!/usr/bin/perl -w
use strict;
use Getopt::Long;
use Time::Local;
use Fcntl qw(:flock);
use lib '.';

#
# look for split films and record second half.
#12 April 22.  Change log file to be NOt in /tmp (cannot write to it). 
#Dec 2022   Revised strategy:  
#  Use fingerprint = title+description for matching
#  Don't check category because unreliable.  Many channels set it wrong.
#
# 4 Sept 2023.  Changes made to sub getguide for v34and tested with port 6744. 
#  a) if program is NOT being recorded then Status is omitted from guide data
#  b) Status now returned as integer - need to use StatusName instead.
#  code should now work both pre and post change.
#  See  https://www.mythtv.org/wiki/Recording_Status for details.
#  Also, included checks on module Scan_Database version.
#  

my $lockfile='/home/mythtv/.mythtv/checkfilm.lock';
my $confile='/home/mythtv/.mythtv/checkfilm.cfg';

my %validchannels;
my $backend='http://127.0.0.1:6544';
my $content;
my %guide;
my $match='fingerprint';   #set to ProgramId or 'fingerprint' for  Title + Description
my $logfile='/var/log/mythtv/checkfilm.log';

#Many film channels mis-categorise programs.
#Set this non zero to check all categories, not just film  
my $AllowAllCategories=1;

#Get calling params
my $calling=join(' ',@ARGV);
my $ChanId = -1; my $StartTime = ''; my $listchannels=0; my $reporting=0; my $help=0; my $verbose=0; 
my $developer=0; my $setup=0; my $desc='';
GetOptions ('ChanId=i' => \$ChanId, 'StartTime=s' =>\$StartTime, 
			'list'=>\$listchannels, 'report' => \$reporting,
			'help' => \$help, 'verbose' => \$verbose,
			'setup' => \$setup,
			'developer' => \$developer,
			'desc=s'   =>  \$desc);
			
# $developer - unused
 

givehelp() if ($help);
generatesetup() if ($setup);

#we need a module:
my $sdv;  #version no.
BEGIN {   
    unless (eval "require scan_database") {
        print "couldn't load scan_database module\nSee https://www.mythtv.org/wiki/Perl_API_examples\n";
        exit 0;
    }
}
check_scan_database_version();
listchannels() if ($listchannels);


#Hash of recording status texts and whether they indicate records is being/will be made
my %recording=(
    WillRecord => 1,
    Pending    => 1,
    Tuning     => 1,
    Recording  => 1,
    Unknown    => 0,
    Failed     => 0,
    Recorded   => 1,
    "Don'tRecord" => 0 
);

#open log file and lock it
open(LOCKFILE,'>',$lockfile) or die "Cannot open logfile $lockfile:\n$!\n";
flock(LOCKFILE, LOCK_EX);     #wait til it's free
#vprint("\n$now log file open and locked");

#off we go:
my $now=TimeString(time());
prechecks();    #exit unless listing channels or have a valid channel

getguide($ChanId, $StartTime);
	
#show callsign and guide data
$content =~ m!<CallSign>([^<]*)</CallSign>!;
if ($verbose){
	print "Callsign is $1\n";
	showguide(0);
}
unless (exists $validchannels{$ChanId}){exit 0};

my $offset;
if (isolatedpart(1)){
	vprint('#3 needs recording');
	$offset=1;
}elsif (isolatedpart(0)){
	vprint('#2 needs recording');
	$offset=0;
}else{ vprint('nothing to do'); myexit('');
}
	
#sanity checks
if ($now ge $guide{2+$offset}{EndTime}){myexit("Too late to record this")};
if ($guide{$offset}{RecordId} == 0){myexit ("No recording rule for #$offset")};
$_=$offset +2;

vprint("Triggering recording");
if ($reporting){myexit('Recording suppressed as only reporting')};

#Get recording rule for part 1 
my $url=$backend . "/Dvr/GetRecordSchedule?RecordId=$guide{$offset}{RecordId}";
scan_database::ReadBackend($url, $content);
my %recrule;
scan_database::GetAllFields(%recrule, $content, '>', '</RecRule>');

#modify it and trigger recording for part 2
$recrule{StartTime}=$guide{2+$offset}{StartTime};
$recrule{EndTime}=$guide{2+$offset}{EndTime};
$recrule{Station}=$recrule{CallSign};
scan_database::ValidatePost(%recrule, $backend .'/Dvr/AddRecordSchedule', 'raw', 12);

#confirm changed
my $found=0;

for my $try (1 .. 6){
	sleep(3);
	getguide($ChanId, $StartTime);
	$_= $guide{$offset+2}{Status};
	if ($recording{$_}==1){
		vprint("Change confirmed at try $try");
		$found=1;
		last;
	};
};
if ($found){
	showguide(2+$offset)
}else{
	print "Recording of $guide{$offset}{Title} triggered but not confirmed\n";
};
myexit('');

sub showguide{
	my($start)=@_;
	if ($start){  #final confirmation
		printf "%-10s %-10s %22s  $guide{$start}{Title}\n", $guide{$start}{Status}, $guide{$start}{Category}, $guide{$start}{StartTime};
		return;
	};
	print "#  Status      Category          StartTime             Title\n";
	for (0..5){
		printf "$_  %-14s %-15s %22s  $guide{$_}{Title}\n", $guide{$_}{Status}, $guide{$_}{Category}, $guide{$_}{StartTime};
		#print "$guide{$_}{ProgramId}\n";
	};
}

sub isolatedpart{
	my($offset)=@_;
	my $target=$offset+2;
	
	#Check whether program '$offset' is a part 1 which needs a part 2 triggering	
	
	#first check if part 1 is film (or that all categories allowed)
	vprint("#$offset category is $guide{$offset}{Category}");
	if ($guide{0+$offset}{Category} ne 'Film'){
		return 0 if ($AllowAllCategories==0);
	}
	
	#and that it is recording
	$_= $guide{$offset}{Status};
	if ($recording{$_}==0){vprint("#$offset is not recording"); return 0};
	vprint("#$offset is recording or recorded");
	
	#Check if part 2 matches
	my $matchtext=$guide{0+$offset}{$match};
	if ($guide{2+$offset}{$match} ne $matchtext){vprint("#$target does not match"); return 0};
	vprint("#$target matches");
	
	#Check neigbours have different ProgramId
	for (1,3,4){
		if ($guide{$_+$offset}{$match} eq $matchtext){ vprint("#$offset has clashing neighbour"); return 0};
	}
	vprint("#$offset has no clashing neighbours");

	#Is part2 already scheduled or recorded? 
	$_= $guide{$offset+2}{Status};
	if ($recording{$_}){vprint("#$target is being recorded already");return 0};
	return 1;   #this one can be recorded!
	
}
sub prechecks{
	#Do checks before we open the scan_database module.
	vprint("\n$now checkfilm $calling");
	vprint("ScanDatabase version $sdv");
	if ($ChanId==-1){myexit("Need --help or --list or --ChanId or --setup")};
    
    $verbose=1 if $reporting;
    
	#development aid -mythutil triggered invocation
	
	if ($ChanId==0){$ChanId=20025; $verbose=1};           #if invoked by mythutil
	if ($StartTime eq ''){$StartTime=$now};   #standard action 
	
	#Check the config file
	unless (-r $confile){
		vprint("no config file - using dummy channel entry");
		$validchannels{$ChanId}='Unknown';
		return;
	}
	
	#vprint("Reading config file $confile");
	open(CONFIG,'<',$confile) or myexit("Cannot open config file $confile:\n$!");
	
		
	while (<CONFIG>){
		chomp;
		#vprint($_);
		next unless (/\=/);
		s/^\s+//; #kill leading spaces
		s/\s+$//; #trailing
		next unless length;
		my ($k,$v)=split(/\s*=\s*/,$_,2);
		$validchannels{$k}=$v;
	};
	close CONFIG;
		
	if (exists $validchannels{$ChanId}){
		vprint("Found $ChanId in config");
		return;
	}else{
		vprint("Not an interesting channel: $ChanId");
		myexit('');
	}
	print "\nAt $now:    checkfilm $calling\n" unless ($verbose);	 
}

sub getguide{
	my ($chan, $start)=@_;

	#Read the guide, show callsign, get 6 entries and show them
	my $url="$backend/Guide/GetProgramList?StartTime=$StartTime&ChanId=$ChanId&Count=6&Details=true";
	vprint($url);
	unless (scan_database::ReadBackend($url,$content)){myexit("Could not get guide data")};
		
	#Did we get any guide data?
	$content =~ m!<Count>(\w+)</Count>!;
	myexit('No guide data') if ($1==0);
	#extract fields
	scan_database::FillHashofHash(%guide, $content,'Program','#','StartTime','EndTime','Category','Title','ProgramId','Status',
	               'StatusName', 'RecordId','Description');

	#check valid status values
	for (0..5){
		my $status=$guide{$_}{Status};
		if ($status eq ''){
			$status="Don'tRecord"
		}elsif ($status =~ /^-?\d+$/){   # see https://www.mythtv.org/wiki/Recording_Status
			$status=$guide{$_}{StatusName};
		}
		$guide{$_}{Status}=$status;
		
		unless (exists $recording{$status}){
			vprint("warning:  Status not known: $status"); 
			$recording{$status}=0;   #Assume not a recording status
		}
		$guide{$_}{fingerprint}=$guide{$_}{Title} . $guide{$_}{Description};
	}
}
	
sub vprint{
	print "$_[0]\n" if ($verbose);
}
	
sub myexit{
	if ($_[0] ne ''){print "$_[0]\n"};
	close(LOCKFILE);
	exit 0;
}

sub check_scan_database_version{
    $sdv=$scan_database::VERSION;
    $sdv||=0;
    if ($sdv < 1.13){print "\nscan_database.pm need to be at least 1.13 to support version 2 of API interface.\n";
        print "Please update it from   https://www.mythtv.org/wiki/Perl_API_examples\n";
        exit 0;
    };
    
}

sub TimeString{
	(my $epoch)=@_;
	#return time as 2021-12-03T13:44:04
	my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = gmtime($epoch);
	$year+=1900; $mon++;
	return sprintf("%04d-%02d-%02dT%02d:%02d:%02dZ", $year, $mon, $mday, $hour, $min, $sec);
}

sub ZtoEpoch{
	(my $Z)=@_;
	#eg 2022-03-17T20:00:00Z to epoch seconds
	$Z =~ /(\d\d\d\d)-(\d\d)-(\d\d)T(\d\d):(\d\d):(\d\d)Z/;
	return timegm($6,$5,$4,$3,$2-1,$1-1900);
}

		
sub listchannels{
	#list all channels - grep it to extract the interesting ones and put in %interestingchannels
	my $temp;  my %sources; my %ChanData;
	#get sources
	my $url=$backend. '/Channel/GetVideoSourceList';
	scan_database::ReadBackend($url, $temp);
	scan_database::FillHashofHash(%sources, $temp, 'VideoSource', 'Id', 'SourceName');
    #get channels per source
    for my $source (keys %sources){
		scan_database::ReadBackend($backend . '/Channel/GetChannelInfoList?SourceID='.$source.
					'&OnlyVisible=false&Details=true', $temp);
		my %temphash;
		scan_database::FillHashofHash(%temphash, $temp, 'ChannelInfo', 'ChanId', 'CallSign','Visible');
		%ChanData = (%ChanData, %temphash);
	}
    for (sort keys %ChanData){
		if ($ChanData{$_}{Visible} eq 'true'){print "$_ = $ChanData{$_}{CallSign}\n"};
	}
	exit 0; 
}
 
sub generatesetup{
	
print my $setupfile="#!/bin/bash
#
# This script generates the necessary files for setting up checkfilm.
# Redirect to a file, chmod +x it then execute it.
#
# It sets up for UK channels which split films as of December 2022.
#
# You will also need to set up a system event.
# 
echo 'Generating log file $logfile'
sudo touch $logfile
sudo chmod 666 $logfile

echo 'Generating lock file $lockfile'
sudo touch $lockfile
sudo chown mythtv:mythtv $lockfile
sudo chmod 666 $lockfile

echo 'Generating config file $confile'
sudo touch $confile
sudo chown mythtv:mythtv $confile
sudo chmod 666 $confile
checkfilm.pl -l | grep ITV4 > $confile
checkfilm.pl -l | grep GREAT >> $confile
checkfilm.pl -l | grep ITV2 >> $confile
checkfilm.pl -l | grep '5STAR' >> $confile
checkfilm.pl -l | grep '5ACTION' >> $confile
checkfilm.pl -l | grep '5SELECT' >> $confile
checkfilm.pl -l | grep ITVBe >> $confile
checkfilm.pl -l | grep 'Channel 5' >> $confile

";
exit;
}



sub givehelp{
my $location='/usr/local/bin';
print "
    
checkfilm.pl   version 2
========================
    
Aims
----
Some channels in the UK transmit films in 2 parts split by a short news item.  It is frustrating to start watching the film
at a later date only to find the second half missing!   'Record All' will not resolve this.

At the time of writing, the 'interesting' channels which do this are ITV2, ITV4, the 'GREAT' channels, Channel 5, 5ACTION, 5STAR,
5SELECT and their +1 and HD versions, but names may change in the future.

This perl script looks for such recordings and triggers the second half automatically.

Version 34 of Mythtv includes a number of changes affecting recording status as returned by guide APIs.  Version 2 of this code should survive this change.

How does it work?
-----------------
The script is run by a system event either at the start of a recording or after the end.

If the channel is an 'interesting' one, it will read the first 6 programs (#0 to #5) from the guide,
starting 'now' or at a time specified by --starttime.

If either of the first two entries is:
- a film which is recorded, recording or will record (a 'first half') AND
- the next but one entry is not being recorded and has the same Title and Description (a second half) AND
- neighbours do not have the same Title and Description THEN
it will create a new recording rule for the second half.

The first and second entries are checked because we need to catch two situations.  The start time may be:
- shortly before the guide start time if pre-scheduled (guide entry #1 will be part 1) OR
- after the guide start time if manually triggered after the film started (guide entry #0 will be part 1).

Title and description are used for matching because this has proved more reliable than the more intuitive ProgramId.


Parameters
----------
   --ChanId        eg --ChanId=20025
   --help or -h    this text
   --list or -l    lists all visible channels showing channel id and callsign.
   --report or -r  report only - does not trigger a recording.
   --verbose or-v  extra diagnostics
   --Starttime     Use if triggering with the recording ended event or for testing purposes.
                    eg --starttime=2022-03-15T21:00:00Z   Default is 'now'.
   --setup          print out commands needed to create the log and lock files.
   --desc          optional data printed in log only (eg %TITLE%).


Setup
-----
1.  Put this perl script in checkfilm.pl somewhere in path (eg  $location)  
	and make it executable (chmod +x).
    Check:  display help with checkfilm.pl --help 

2.  Put the module scan_database.pm in perl path and make it everyone readable.  
    See:  https://www.mythtv.org/wiki/Perl_API_examples
    Check:  You can list channels with checkfilm.pl --list
    
3.  Create three files.  
      A log file $logfile, 
      a lock file $lockfile 
      the config file $confile which hold a list of 'interesting' channels.
    Do this with:
		checkfilm.pl --setup > setup.sh
		chmod +x setup.sh
		./setup.sh and supply sudo password.
     
4.  Set up a system event Recording Started Writing:

        sleep 20 && ${location}/checkfilm.pl -v --chanid=%CHANID% --desc=%TITLE%>> $logfile 2>&1
        
    If you feel that things are too busy for your backend at start of recording then
    you can set up the Recording ended event instead, provided that post recording 
    time is less than the duration of the 'news' item.   You will need also to provide the starttime:
    
        sleep 20 && ${location}/checkfilm.pl -v --chanid=%CHANID% --starttime=%STARTTIMEISOUTC% --desc=%TITLE%>> $logfile 2>&1
    
    You can drop the -v for a log containing only triggered recordings.
    
Channel Filter
--------------
If the config file $confile is missing then all channels are checked, but setup will have inserted
the channels known to be interesting as of November 2022.
After a retuning you may need to re-populate the config file eg

     checkfilm.pl -l | grep ITV4 > $confile
     checkfilm.pl -l | grep GREAT >> $confile
     checkfilm.pl -l | grep ITV2 >> $confile
     checkfilm.pl -l | grep ITVBe >> $confile

The file will then have entries such as '20025 = ITV4'
Only the initial digits are used - the CallSign is comment.

Problem Channels
----------------
Some channels mis-categorise some programs and do not declare them as film.
To inhibit the 'is this a film' check then set \$AllowAllCategories near the head of the perl script to non zero.
The parameter is currently set to \$AllowAllCategories=$AllowAllCategories
  
Logging
-------
Logging is maintained by a redirect in the system event line and only takes place if the channel is an 
'interesting' one or --report is set. 
log entries will consists of a single line saying that a recording had been triggered unless --verbose is set.  
eg
  WillRecord Film 2022-03-20T22:01:00Z  Hitman Redemption

To inhibit logging completely just redirect ouput to /dev/null. 

Locking
-------
To prevent problems with two sumultaneous invocations of the code a lock file is used: $lockfile

Times
-----
Note that all times are in UTC.  This matches UK winter time but a summer recording at 9pm will show as 20:00:00.


";
exit 0;
}


Phil Brady. 4 Sept 2023.