Difference between revisions of "Swamp backend with recordings"

Latest revision as of 16:17, 16 April 2023

Purpose

This perl script was created to provoke a problem with failing recordings experienced with v32, an HD Homerun DBV-T/T2 tuner and the Crystal Palace transmitter in London. It generates batches of manual recordings via the API interface.

It includes facilities to get a list of multiplexes from a backend, to list the channels, to list upcoming recordings, to trigger 'manual' recordings and to delete all such recording rules and recordings. It can fire off 'batches' of recordings or simply single recordings depending on parameters which need setting by the user.

It was developed under xubuntu 22.04 with v32 Mythtv and v34, with both ports 6544 and 6744. It fails to delete recordings with 0.27 (an API limitation?) but might be more successful with version 0.28 onward. It has been tested with UK, Dutch and US data.

Options

./swamp.pl -h

   -a or -alternative         use alternative backend address.
   -b or --backend            ip address of backend and (optionally) port number.
   -c or --config <filename>  configuration file.  Default is swamp.cfg 
   -d or --diagnostics        Generate file swampdiags.txt for developer to diagnose problems.
   -f or --fullrun            to invoke full triggering of recordings or tidying                     
   -h or --help               this text
   -l or --listchannels       list channels
   -m or --multiplexes        to list sources & multiplexes
   -r or --recordings         list proposed recordings.                       Also needs --fullrun to trigger them. 
   -s or --spoof <filename>   Developer option - read diagnostic file rather than backend.
   -t or --tidy               List all upcoming and present swamp recordings. Also needs --fullrun to remove them.
   -u or --upcoming           list all upcoming recordings

Setup

Install the scan_database module. See https://www.mythtv.org/wiki/Perl_API_examples. This must be version 1.13 or later to cater for the version 2 of the API. Place it either in perl path or in your working directory. It needs read access.

The libwww-perl module needs loading - if using ubuntu:

sudo apt-get install libwww-perl

Put the code below in swamp.pl and set it executable.

Running the program

The parameters controlling how recordings are made are held in a configuration file (default name swamp.cfg). The help text includes an example of this with the defaults associated with each parameter as follows:

    The following parameters are optional and have these default values: 

    alternativebackend=127.0.0.1:6744  #used with -a
    backend           =127.0.0.1       #assumes 6544 unless specified otherwise.
    stagger           =0               #gap between start of recordings within a batch (in seconds)
    preroll           =2               #in mins     v32 does not seem to respect this value - it uses 4
    postroll          =2               #in mins     ditto  it uses 5.
    reclength         =1               #length of recordings in mins
    gap               =1               #gap in minutes between end of postroll and start of preroll for next batch.
    recordingsneeded  =5
    outputcontrol     =1               #Controls -r output to file. Set to 0 to suppress it, 1 if -f, 2 always

    You must however give one or more 'batch' lines if you record eg:
 
    batch=50883, 50819  #channels in first batch
    batch=50801         #second batch

You MUST specify at least one 'batch' directive to specify the channel number(s) you wish to record from if you use the -r option.

You can get lists of the multiplexes and of the channels on your backend:

./swamp.pl -m
Sources from 127.0.0.1:6544:
  5  HDHR4

Multiplexes:
Mpx    Source    Frequency    FreqId    VisibleChans    
 19         5    482000000        22       26           
 20         5    490000000        23       26           
 21         5    506000000        25       25           
 22         5    514000000        26       13           
 23         5    530000000        28       32           
 24         5    586000000        35        6           
 25         5    546000000        30       11   

./swamp.pl -l
Channels from 127.0.0.1:6544
50056  =  Mpx:25   Src:5  FqId:30   5SELECT
50066  =  Mpx:25   Src:5  FqId:30   TBN UK
50101  =  Mpx:25   Src:5  FqId:30   BBC ONE HD
50102  =  Mpx:25   Src:5  FqId:30   BBC TWO HD
50103  =  Mpx:25   Src:5  FqId:30   ITV1 HD
50104  =  Mpx:25   Src:5  FqId:30   Channel 4 HD
...

You can grep the output to select particular sources, multiplexes. frequencyids or callsigns.

In the UK, the developer finds FrequencyId to be a useful indicator of multiplex, but the concept does not exist in some TV cultures and may not always be available in all circumstances.

Recordings for all the channels in first batch will be scheduled to start at the same time, then those in the next batch etc. It will then cycle back to the first batch.

You can set the number of recordings, pre and post roll times, gap between recordings and length of recordings. You can 'stagger' the start of recordings within a batch; also set backend address and alternative address. Note that prior to v34 pre and post roll timings may not be respected and 4 and 5 minutes enforced.

Also note that unless recordings are staggered, they will be triggered 30 seconds after the minute in order to help examining /var/log/mythtv/mythbackend.log with a single grep.

You can list the proposed batches or recordings with their start and end times. These times will be extended by pre- and post-roll.

./swamp.pl -r

Proposed 5 recordings:
Start                   End                     Chan   Mpx   CallSign
2023-04-16T15:51:30Z    2023-04-16T15:52:30Z   50883    23   Yesterday
2023-04-16T15:51:30Z    2023-04-16T15:52:30Z   50819    20   BBC ONE Lon
2023-04-16T15:57:30Z    2023-04-16T15:58:30Z   50801    19   Film4+1
2023-04-16T16:03:30Z    2023-04-16T16:04:30Z   50883    23   Yesterday
2023-04-16T16:03:30Z    2023-04-16T16:04:30Z   50819    20   BBC ONE Lon

2023-04-16T16:06:30Z  = Sun Apr 16 17:06:30 2023    : All will be finished

To trigger these recordings, please re-run with --fullrun

./swamp.pl -r -f

Tidying up afterwards

When you have finished all your testing, you will be left with a bit of a mess with many recordings which you will no longer require and perhaps some in upcoming recordings. You can list these with:

./swamp.pl -t

If this looks sane (and you have a robust database strategy!!) then you can remove them with :

./swamp.pl --tidy --fullrun

This will stop any current 'swamp' recordings, will remove the recording rules for any upcoming 'swamp' recordings and then remove all 'swamp' recordings. Note that there is an issue with removing recordings with Myth version 0.27 and earlier which is believed to be a limitation in the API interface (but if you know better please say so!).

Diagnostics

Two diagnostic options are included in the code:

-d will generate a file swampdiags.txt holding API information from your backend .

-s will enable the author to read that file rather than a real backend.

Acknowledgements

I am indebted to kmdewaal for the inspiration and encouragement to produce this and to bill6502 for his debugging assistance.

Phil Brady 16th April 2023.

Code

#!/usr/bin/perl -w -C6
use strict;
use Getopt::Long;
use Time::Local;
use lib '.';    #add current dir to perl path if not already there

#swamp a backend with manual recordings  See --help

my $backend; 
my $version='2.16:  2023-04-13';

#Changes:
#11/3/23 Getchannels - GetVideoMultiplexList needs SourceID not SourceId for v0.27 compatibility.
#        deleteupcoming - allow PostOrGet==0 for unsupported
#        Tidyup needs RemoveRecorded not DeleteRecorded for 0.27 but it does not work.  Cannot delete recordings.
#20/3/23 Start time changed to 30 secs after minute.  FrequencyId added to -l option.  More useful than Mpx in UK. 
#22/3/23 Insert default mplexid of '?' in getchannels rather than in listchannels to prevent failures in setmax.
#        copy $params{postroll} and $params{preroll} to recording rule. 
#        global sed edit:  tab to '    ' for consistency in wiki.
#26/3/23 Bugfix:  Set default $Mplexinfo{$k}{FrequencyId} not frequencyId.  Also two instances need //= rather than ||=
#        because ||= incorrectly defaults zero values but //= only defaults undefined ones.  
#28/3/23 remove setting offsets in record TEMPORARY  - but reinstated 29/3/23
#29/3/23 added spoof to allow input of chedit2.pl diagnostics
#30/3/23 added -d to generate diagnostic report for author.
#31/3/23 fix for undefined FrequencyId with empty mpxs or only hidden channels - tested with Dutch and USA data.
#31/3/23 Sort recorded files by starttime in tidyup.  v1.14b
# 9/4/23 Add user defined data to diagnostics file.  v2.15
#13/3/23 User parameters in config files.  Values shown in -h are defaults.

#recording template (based on a rule extracted from Myth v32). 
my %template=(    
    AutoCommflag   => 'false',
    AutoExpire     => 'true',
    AutoMetaLookup => 'false',
    AutoTranscode  => 'false',
    AutoUserJob1   => 'false',
    AutoUserJob2   => 'false',
    AutoUserJob3   => 'false',
    AutoUserJob4   => 'false',
    AverageDelay   => '0',
    CallSign       => 'BBC TWO',
    Category       => '',
    ChanId         => '10002',
    Description    => 'Swamp test (Manual Record)',
    DupIn          => 'All Recordings',
    DupMethod      => 'None',
    EndOffset      => '2',
    EndTime        => '2023-02-03T12:55:00Z',
    Episode        => '0',
    Filter         => '0',
    FindDay        => '6',
    FindTime       => '12:45:00.000',
    Inactive       => 'false',
    Inetref        => '',
    LastRecorded   => '2023-02-03T12:43:48Z',
    MaxEpisodes    =>'0',
    MaxNewest      => 'false',
    NewEpisOnly    => 'false',
    ParentId       => '0',
    PlayGroup      => 'Default',
    PreferredInput => '0',
    ProgramId      => '',
    RecGroup       => 'Default',
    RecPriority    => '2',
    RecProfile     => 'Default',
    SearchType     => 'Manual Search',
    Season         => '0',
    SeriesId       => '',
    StartOffset    => '2',
    StartTime      => '2023-02-03T12:45:00Z',
    StorageGroup   => 'Default',
    SubTitle       => '',
    Title          => 'Swamp (Manual Record)',
    Transcoder     => '0',
    Type           => 'Single Record'
);

#default config
my $defaultconfig="
    The following parameters are optional and have these default values: 

    alternativebackend=127.0.0.1:6744  #used with -a
    backend           =127.0.0.1       #assumes 6544 unless specified otherwise.
    stagger           =0               #gap between start of recordings within a batch (in seconds)
    preroll           =2               #in mins     v32 does not seem to respect this value - it uses 4
    postroll          =2               #in mins     ditto  it uses 5.
    reclength         =1               #length of recordings in mins
    gap               =1               #gap in minutes between end of postroll and start of preroll for next batch.
    recordingsneeded  =5
    outputcontrol     =1               #Controls -r output to file. Set to 0 to suppress it, 1 if -f, 2 always

    You must however give one or more 'batch' lines if you record eg:
 
    batch=10948,10951,10999   #channels in first batch
    batch=11001,11039,11044   #second batch .. add others as necessary
";

my %params=();
my @batch=();

my $content;
my %Channelinfo;   #key is ChanId
my %Mplexinfo;     #key is MplexId
my %sources;
my %width;         #to hold column widths
my $tidycount=0;
my $fmt;  my $filewanted=0;   #need output file from --record.  Forced if --fullrun

#check calling parameters
my $optf=0; my $optl=0; my $optt=0; my $help=0;my $optm=0;my $optu=0;my $opta=0;my $optr=0; my $spoof;
my $optd=0;my $optb;my $config='swamp.cfg';

GetOptions ('listchannels'=>\$optl, 'fullrun'=>\$optf, 'tidy' => \$optt, 
            'backend:s' => \$optb, 'help' => \$help,'multiplexes' => \$optm,
            'upcoming' => \$optu, 'alternative' => \$opta, 'recordings' => \$optr,
            'spoof:s' => \$spoof, 'diagnostics' => \$optd, 'config:s' => \$config);

if ($help){GiveHelp()};
if ($optl + $optt + $optm + $optu +$optr +$optd !=1){
    print "Bad options.  Please do $0 --help\n";
    myexit();
}

#require scan_database;    #See https://www.mythtv.org/wiki/Perl_API_examples
BEGIN {   
    unless (eval "require scan_database") {
        print "couldn't load scan_database module\nSee https://www.mythtv.org/wiki/Perl_API_examples\n";
        exit 0;
    }
};

check_scan_database_version();

if (defined $spoof){$optf=0; $config=$spoof};
readconfig();

#decide backend address.  Hierarchy:  spoof file, -a, -b, config file, defaults.
$backend=$params{backend};
if (defined $optb){$backend = $optb};
if ($opta){$backend=$params{alternativebackend}};
unless ($backend =~ m!:!){$backend .= ':6544'};
#$backend="http://$backend";
if (defined $spoof){$backend = "file $spoof"};

if ($optd){Diagnostics()};   #generate info for author

#get channel details from backend
getchannels();

# What to do?
if ($optl){listchannels(); myexit()};
if ($optm){showmultiplexes()};
if ($optr){TriggerRecordings()};
if (defined $spoof){print "Cannot do -t or -u with \$spoof\n"; myexit()};
if ($optu){getupcoming(0); myexit()};
if ($optt){TidyUp()};
myexit();

sub parseconfig{
    (my $txt)=@_;
    my $k; my $v;
    for (split("\n",$txt)){
         chomp;
         if (/#/){($_)=split(/#/,$_)};
         $_ =~ s/\s//g;
         if (/=/){
            ($k, $v)=split(/=/,$_);
            if (lc $k eq 'batch'){
                push @batch,$v;
            }else{
                $params{lc $k}=$v;
            }
         }

    }
}
sub readconfig{
    #decide operating parameters - %params and @batch
    
    parseconfig($defaultconfig);  #default params
    @batch=();
    #now the real config file
    unless (open CFG, $config){print "Cannot open $config - assuming defaults\n"; return};
    my $temp='';
    while (<CFG>){
        last if /^##sources/;
        $temp.=$_;
    }
    close CFG;
    parseconfig($temp);
}

         
         
sub check_scan_database_version{
    my $sdv=$scan_database::VERSION;
    $sdv||=0;
    if ($sdv < 1.13){print "\nscan_database.pm need to be at least 1.13 to support version 2 of API interface.\n";
        print "Please update it from   https://www.mythtv.org/wiki/Perl_API_examples\n";
        exit 0;
    };
}

sub ReadDiagnostics{
     # author's debugging aid
    return '' unless (defined $spoof);
    (my $section)=@_;
    my $out='';  my $interesting=0;
    unless (open FN, "<$spoof"){print "Cannot open $spoof\n";exit 0};
    while (<FN>){
        $interesting=0 if (/^##/);
        if ($interesting){
            chomp;
            $out .= $_;
        }
        $interesting=1 if (/##$section/);
    }
    close FH;
    $out =~ s!\r!!g;                    #kill off any linefeeds
    $out =~ s!<([^>]*)>\n</!<$1></!g;   #kill off newlines in blank fields
    return $out;
}
sub TriggerRecordings{
    
    if (scalar @batch ==0){die "No 'batch' directives found in config\n"};

    #initial timekeeping
    my $epoch=time() + 60;
    $epoch -= $epoch%60;        #rounded down
    $epoch += 30+60*$params{preroll};   #starttime of next batch.  Align to 30 secs past the minute.

    #list proposed recordings - generate them if --fullrun 

    setmax('MplexId','Mpx','ProposerecsA');
    print "\nProposed $params{recordingsneeded} recordings:\n";

    print "Start                   End                    ";
    $fmt= '%' . $width{ChanId} . 's   %' . $width{MplexId} ."s   %s\n";
    printf $fmt, 'Chan', 'Mpx', 'CallSign';
    
    $filewanted =($optf+$params{outputcontrol}>1?1:0);   #Decide whether file output required
    if ($filewanted){
        open(FH, ">", "swamp.out")
        or die "Can't open swamp.out: $!";
        for (qw/stagger preroll reclength postroll gap/){
            print FH "$_=$params{$_}  ";
        }
        print FH "\nStart                   End                    ";
        printf FH $fmt, 'Chan', 'Mpx', 'CallSign';
    }
    my $lastEndTs=0;
    while ($params{recordingsneeded}){
        for my $set (@batch){    
            my $delay=0;   #stagger start of recording within a batch
            my @chlist = split /,/,$set;
            
            for my $chan (@chlist){
                $chan =~s/\s//g;    #remove spurious spaces
                if ($params{recordingsneeded}){
                    unless (exists $Channelinfo{$chan}){die "channel $chan not known"};
                    record($chan, $epoch+$delay);
                    $lastEndTs=$epoch + $delay + 60*($params{reclength} + $params{postroll});
                    $delay+=$params{stagger};
                    $params{recordingsneeded}--;
                }
            }
            if ($params{recordingsneeded}){
                #print "-----\n";
                $epoch += 60*($params{gap}+$params{preroll}+$params{reclength}+$params{postroll}); #starttime of next batch
            }
        }
    }

    print "\n", ZTime($lastEndTs), "  = ", scalar localtime($lastEndTs), "    : All will be finished\n";

    if ($optf){
        #list upcoming recordings
        sleep(5);
        getupcoming(0);
    }else{
        print "\nTo trigger these recordings, please re-run with --fullrun\n";
    }
    myexit();
}

sub myexit{
    close FH if ($filewanted);        
    exit 0;
}

sub showmultiplexes{
    print "\nSources from $backend:\n";
    for (sort keys %sources){
        printf "%3s  %s\n", , $_, $sources{$_}{SourceName};
    }
    print "\nMultiplexes:\n";
    my @heading=('Mpx,MultiplexId,1',
                 'Source,SourceId,1',
                 'Frequency,Frequency,1',
                 'FreqId,FrequencyId,1',
                 'VisibleChans,count,-1');
    my $format=doheading(@heading);
    
    #items are  Mpx   Source  Frequency   FreqId  Channel_count\n";
    my $count;
    for (sort {$a <=>$b} keys %Mplexinfo){
        $count=sprintf('%5d',$Mplexinfo{$_}{count});
        printf $format, $_ ,$Mplexinfo{$_}{SourceId},$Mplexinfo{$_}{Frequency},
                        $Mplexinfo{$_}{FrequencyId}, $count;
    }
    myexit();
}
sub doheading{
    my $finalfmt='';    
    for my $column (@_){
        (my $aka,my $tag,my $align)=split /,/,$column;
        setmax($tag,$aka,'doheading');
        $fmt='%' . $width{$tag}*$align . 's    ';
        printf $fmt, $aka;
        $finalfmt.=$fmt;
    }
    print "\n";
    return $finalfmt."\n";;
}

sub record{   #list recording and trigger it if --fullrun
    (my $chan, my $epochstart)=@_;
    my $starttime=ZTime($epochstart);
    my $endtime=ZTime($epochstart + 60*$params{reclength});
    print "$starttime    $endtime   ";
    printf $fmt, $chan,$Channelinfo{$chan}{Mpx}, $Channelinfo{$chan}{Name};
    if ($filewanted){
        print FH "$starttime    $endtime   ";
        printf FH $fmt, $chan,$Channelinfo{$chan}{Mpx}, $Channelinfo{$chan}{Name};
    }
    return unless ($optf);
    
    #get template recording rule
    my %recrule=%template;
    #modify it
    $recrule{StartTime}=$starttime;
    $recrule{EndTime}=$endtime;
    $recrule{CallSign}=$Channelinfo{$chan}{Name};
    $recrule{Station}=$recrule{CallSign};
    $recrule{ChanId}=$chan;
    $recrule{StartOffset}=$params{preroll};
    $recrule{EndOffset}=$params{postroll};

    #Trigger the recording:
    my $url=$backend .'/Dvr/AddRecordSchedule';
    scan_database::ValidatePost(%recrule, $url, 'raw', 12);
    print "Recording triggered\n";
    sleep(1);
}

sub getchannels{
    
    #query backend and get all channel names and multiplexes
    my $temp;   my %ChanData;
    
    #get sources
    my $url=$backend. '/Channel/GetVideoSourceList';
    $temp=&ReadDiagnostics('sources');
    scan_database::ReadBackend($url, $temp)unless (defined $spoof);;
    scan_database::FillHashofHash(%sources, $temp, 'VideoSource', 'Id', 'SourceName');

    #get mpx and channel info per source
    
    for my $source (keys %sources){
    
        setmax('source',$source,'getchannelsA');
        
        # Get source id and frequency for each source/multiplex
        
        my %Mpxtemp;
        $url=$backend. "/Channel/GetVideoMultiplexList?SourceID=$source";   #WAS SourceId
        $temp=&ReadDiagnostics("multiplex$source");
        scan_database::ReadBackend($url, $temp) unless (defined $spoof);
        scan_database::FillHashofHash(%Mpxtemp, $temp, 'VideoMultiplex', 'MplexId', 'SourceId', 'Frequency','FrequencyId');
               # FrequencyId is currently not available (v34 master) but this sets a blank default.
        %Mplexinfo = (%Mplexinfo,%Mpxtemp);
                
        #Now get channel info
        $temp=&ReadDiagnostics("channels$source");
        scan_database::ReadBackend($backend . '/Channel/GetChannelInfoList?SourceID='.$source.
                    '&OnlyVisible=false&Details=true', $temp) unless (defined $spoof);
        my %temphash;
        scan_database::FillHashofHash(%temphash, $temp, 'ChannelInfo', 'ChanId', 'CallSign','Visible','MplexId','FrequencyId','SourceId');
        %ChanData = (%ChanData, %temphash);
    }

    #insert channel data in  %Channelinfos and %Mplexinfo
    
    for (keys %Mplexinfo){$Mplexinfo{$_}{count}=0};  #initialise counts of channels per mpx first
    
    for (keys %ChanData){
        my $mpx = $ChanData{$_}{MplexId};
        $Mplexinfo{$mpx}{FrequencyId}||=$ChanData{$_}{FrequencyId};  
        if ($ChanData{$_}{Visible} eq 'true'){
            $Channelinfo{$_}{Mpx}=$mpx;
            $Channelinfo{$_}{Name}= $ChanData{$_}{CallSign};
            $Channelinfo{$_}{SourceId}= $ChanData{$_}{SourceId};
            
            $Mplexinfo{$mpx}{count}++;    
        }
    }
    
    #now note max widths of the data
    for my $k (keys %Channelinfo){  #key is ChanId
        setmax('ChanId',$k,'GetchannelsB');
        setmax('Mpx', $Channelinfo{$k}{Mpx}, 'getchannelsC');
        setmax('SourceId', $Channelinfo{$k}{SourceId}, 'getchannelsD');
    }
    for my $k (keys %Mplexinfo){  #key is MplexId
        $Mplexinfo{$k}{FrequencyId}//='?';
        setmax('MplexId',$k,'GetchannelsE');
        for (qw/Frequency FrequencyId SourceId/){
            setmax($_,$Mplexinfo{$k}{$_},"getchannelsF MplexId=$k");
        }
    }
}


sub setmax{ 
    #note max width of items so we can generate formats during printing.
    (my $item, my $text,my $trace)=@_;
    $width{$item}||=0;
    my $old=$width{$item};
    my $new=length($text);
    my $ok=0;
    if (defined $old){$ok++};
    if (defined $text){$ok++};
    if ($ok<2){
        print "bad data in setmax\n";
        print "item=$item txt=$text old=$old new=$new trace=$trace\n";
        die;
    }
    
    if ($old<$new){
        $width{$item}=$new;
    };
}
    

sub listchannels{    #called if --list
    
    #list channels
    print "Channels from $backend\n";
    my $format='%'.$width{ChanId}.'s' .
    '  =  Mpx:%-'. $width{Mpx}.'s   Src:%-' . $width{SourceId}. 's  FqId:%-'. $width{FrequencyId}."s   %s\n";
    my $mpx, my $fqid;
    for (sort {$a <=>$b} keys %Channelinfo){
        $mpx=$Channelinfo{$_}{Mpx}; $fqid=$Mplexinfo{$mpx}{FrequencyId};
        printf $format, $_, $mpx, $Channelinfo{$_}{SourceId}, $fqid, $Channelinfo{$_}{Name};
    }
}

sub ZTime{    #convert epoch time to 2021-12-03T13:44:04Z form
    (my $epoch)=@_;
    my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = gmtime($epoch);
    $year+=1900; $mon++;
    return sprintf("%04d-%02d-%02dT%02d:%02d:%02dZ", $year, $mon, $mday, $hour, $min, $sec);
}


sub getupcoming{
    (my $tidying)=@_;
    my $count=0;
    print "\nUpcoming recordings\n";
    my $url="$backend/Dvr/GetUpcomingList";
    unless (scan_database::ReadBackend($url,$content)){die "Could not get upcoming data"};
 
    #extract fields
    my %future;
    scan_database::FillHashofHash(%future, $content,'Program','#','StartTime','ChanId','Title','RecordId','RecordedId');
    
    $fmt='%' . $width{ChanId} . "s   %s\n";
    for (sort {$a <=> $b} keys %future){
        my $targettitle= ($future{$_}{Title} eq $template{Title})?1:0;
        if (1-$tidying + $targettitle >0){
            if ($count==0){  #print heading
                print "Start                    ";
                printf $fmt, 'Chan', 'Title';
            };
            print "$future{$_}{StartTime}     ";
            printf $fmt, $future{$_}{ChanId}, $future{$_}{Title};
            $count++;
            if ($tidying + $targettitle ==2){
                if ($optf){deleteupcoming($future{$_}{RecordedId},$future{$_}{RecordId})};
            }
          
        }
    }
    if ($tidying){print "$count found\n"};
    $tidycount+=$count;
}

sub deleteupcoming{
    (my $RecordedId, my $RecordId)=@_;
    my %tplate=('RecordedId' => $RecordedId, 'RecordId'=> $RecordId);
    my $url;my $resp;my $content;
    
    #checks following use with v0.27 (which is unsupported anyway!)
    my $bad=0;
    if ( ! defined $RecordedId){$bad=1};
    unless ($RecordedId =~ m!^\d+$!){$bad+=2};
    if ($bad>0){print "Bad =$bad  bad value for \$RecordedId \n"};
        
    #Stop a current recording 
    if ($RecordedId>0){
        my $PostOrGet=scan_database::APISupported("$backend/Dvr/StopRecording");
        if ($PostOrGet==2){    #POST it
            $url="$backend/Dvr/StopRecording";
            $resp=scan_database::ValidatePost(%tplate, $url, 'raw', 1);
            print "  Stopped (POST)";
            sleep(1);
        }elsif ($PostOrGet==1){  #GET
            $url="$backend/Dvr/StopRecording?RecordedId=$RecordedId";
            $resp=scan_database::ReadBackend($url,$content);
            print "  Stopped (GET).";
            sleep(1);
        }else{ 
            print 'Unsupported Stoprecording  API';
        }
        
    }
    #now remove the rule
    $url="$backend/Dvr/RemoveRecordSchedule";
    $resp=scan_database::ValidatePost(%tplate, $url, 'raw', 1);
    print "  Rule removed\n";
    sleep(1);
}

sub TidyUp{
    getupcoming(1);   # <- stops and removes upcoming recordings 

    # now delete any recordings made.
    my %delhash=('#ChanId' => '',
                '#StartTime' => '',
                'ForceDelete' => 'true',
                'AllowRerecord' => 'false'
                );
    
    print "\nLooking for swamp recordings\n";
    my $url="$backend/Dvr/GetRecordedList";
    unless (scan_database::ReadBackend($url,$content)){die "Could not get recorded list"};
    my %hash; my $resp;
    scan_database::FillHashofHash(%hash, $content,'Program','RecordedId','StartTime','ChanId','Title','CallSign','FileSize');
    
    #get widths
    
    $width{Title}=length $template{Title};
    $width{FileSize}=9;
    $width{StartTime}=20;
    for my $k (keys %hash){
        if ($hash{$k}{Title} eq $template{Title}){
            setmax('RecordedId',$k,'delrecordingA');
            for (qw/ChanId CallSign/){setmax($_,$hash{$k}{$_},'delrecordingsB')};
        }
    }

    #Show recordings and delete them
    my $count=0; my $format;
    for (sort {$hash{$a}{StartTime} cmp $hash{$b}{StartTime}} keys %hash){ 
        if ($hash{$_}{Title} eq $template{Title}){
            if ($count==0){
                #do heading and generate format
                $format=doheading('Chan,ChanId,1',
                    'StartTime,StartTime,-1',
                    'RecordedId,RecordedId,1',
                    'FileSize,FileSize,1',
                    'Title,Title,-1',
                    'CallSign,CallSign,-1');
            }
            my $chan=$hash{$_}{ChanId};
            my $fsize=$hash{$_}{FileSize}/1000000000;
            $fsize=sprintf ("%5.2f GB", $fsize);
            printf $format,$chan, $hash{$_}{StartTime}, $_, $fsize, $hash{$_}{Title}, $hash{$_}{CallSign};
            $count++;
            if ($optf){  #--fullrun
                if (scan_database::APISupported("$backend/Dvr/DeleteRecording") ==0){
                    print "   Sorry:  Dvr/DeleteRecording not supported by backend\n";
                    #0.27 and earlier versions of backend need GET RemoveRecorded
                    # Sadly, Phil can't get this working with 0.27
=pod
                    $url="$backend/Dvr/RemoveRecorded?StartTime=$hash{$_}{StartTime}&ChanId=$hash{$_}{ChanId}";
                    print "$url\n";
                    $resp=scan_database::ReadBackend($url,$content);
                    print "resp=$resp\n";
                    print "Content=$content\n";
=cut
                }else{
                    # newer versions need POST deleterecording
                    $delhash{RecordedId} = $_;
                    $url="$backend/Dvr/DeleteRecording";
                    scan_database::ValidatePost(%delhash, $url, 'raw', 3);
                    print "   Deleted\n";
                }    
                sleep(1);
            }    
        }
    }
    print "$count found\n";
    $tidycount+=$count;
    myexit() if ($tidycount ==0);
    if ($optf==0){print "\nto delete these do --tidy --fullrun\n";  print "You do have a good database backup?\n"};
    myexit();
}


sub GiveHelp{
    print "
    
    Swamp version $version    
    Swamp generates lots of manual recordings on a Mythtv backend for testing purposes.
    See:   https://www.mythtv.org/wiki/Swamp_backend_with_recordings
    
    Options:
    --------
    -a or -alternative         use alternative backend address.
    -b or --backend            ip address of backend and (optionally) port number.
    -c or --config <filename>  configuration file.  Default is swamp.cfg 
    -d or --diagnostics        Generate file swampdiags.txt for developer to diagnose problems.
    -f or --fullrun            to invoke full triggering of recordings or tidying                     
    -h or --help               this text
    -l or --listchannels       list channels
    -m or --multiplexes        to list sources & multiplexes
    -r or --recordings         list proposed recordings.                       Also needs --fullrun to trigger them. 
    -s or --spoof <filename>   Developer option - read diagnostic file rather than backend.
    -t or --tidy               List all upcoming and present swamp recordings. Also needs --fullrun to remove them.
    -u or --upcoming           list all upcoming recordings

    You need a config file to control runs.  see below.
    All recordings will align with 30 secs past the minute unless stagger is set.  This makes it easier to grep the backend log.
    
    Example Config File 
    -------------------$defaultconfig
    
    Files used:
    -----------
    swamp.out       list of recordings triggered when you --record --fullrun.
    swampdiags.txt  diagnostic output for author. 
    
    Pre-requisits:
    --------------
    The script requires the module scan_database to be place in perl path or working directory.  It needs version 1.13 or later.
    See https://www.mythtv.org/wiki/Perl_API_examples
";
    myexit();
}

sub Diagnostics{
    
    if (defined $spoof){print "Cannot use both -s and -d\n"; myexit};

    #Put interesting info to a diagnostic file for author.

    my $filename= 'swampdiags.txt';
    my $temp;
    my %sources;
    unless (open DI, ">$filename") {die  "Cannot open $filename $!"};
    
    #version and time at head
    print DI "\n# Swamp diagnostic file:\n";
    print DI "# Version:              $version\n";
    print DI "# Local time            ". scalar localtime ."\n";
    
    print DI "\n# Configuration:\n";
    
    for (sort keys %params){print DI "$_=$params{$_}\n"};
    for (@batch){print DI "batch=$_\n";};

    #sources
    scan_database::ReadBackend($backend. '/Channel/VideoSourceList', $temp);
    print DI "\n##sources\n";
    print DI "$temp\n";
    #my $reply=
    scan_database::FillHashofHash(%sources, $temp, 'VideoSource', 'Id', 'SourceName');
    for (sort keys %sources){
        print DI "\n##multiplex$_\n";
        scan_database::ReadBackend($backend. "/Channel/GetVideoMultiplexList?SourceID=$_", $temp);
        print DI "\n$temp\n";
        
        print DI "\n##channels$_\n";
        scan_database::ReadBackend($backend. "/Channel/GetChannelInfoList?SourceID=$_&OnlyVisible=false&Details=true",$temp);
        print DI "$temp\n";
    }

    print DI "##end\n";
    close DI;
    print "Diagnostics written to $filename\n";
    myexit();
    
}