Difference between revisions of "Autobackup.pl"

From MythTV Official Wiki
Jump to: navigation, search
m (Myth protocol link added)
(Spurious inconsistency report. Items start at 1 not 0 in services API.)
 
(5 intermediate revisions by the same user not shown)
Line 6: Line 6:
 
|category=Maintenance
 
|category=Maintenance
 
|file=autobackup.pl
 
|file=autobackup.pl
|S25=yes}}
+
|S25=yes|S27=yes}}
  
  
Line 28: Line 28:
 
::n=2 for stats,  
 
::n=2 for stats,  
 
::n=3 for a recordings list
 
::n=3 for a recordings list
::n=4 for protocol diagnostics
+
::n=4 for tracing
 
:::Default is 3.
 
:::Default is 3.
  
Line 116: Line 116:
 
===Development Regime and Protocol Versions===
 
===Development Regime and Protocol Versions===
  
The script has been developed and tested in the UK on a 64 bit Mythbuntu 12.04 (Mythtv 0.25) dedicated frontend/backend system with [[ACPI__Wakeup]], DVB-T receivers, a single directory for recordings, and a local time close to or equal to GMT.   
+
The script was originally developed and tested in the UK on a 64 bit Mythbuntu 12.04 (Mythtv 0.25) dedicated frontend/backend system with [[ACPI__Wakeup]], DVB-T receivers, a single directory for recordings, and a local time close to or equal to GMT.  It was modified to utilise the [[Services API]] interface in June 2014 and been validated against 0.27.  This interface should be Mythtv version independent for versions 0.25 and later and be more robust.
  
 
A 500Gb disk holds root, swap and backup recording partitions; a 2Tb drive holds emergency root and swap and the primary recording partition /var/lib/mythtv.
 
A 500Gb disk holds root, swap and backup recording partitions; a 2Tb drive holds emergency root and swap and the primary recording partition /var/lib/mythtv.
  
 
The intention is to reboot from the 2Tb and run a database restore should the 500Gb fail, and to edit fstab and run [[Find_orphans.py]] should the 2Tb fail..
 
The intention is to reboot from the 2Tb and run a database restore should the 500Gb fail, and to edit fstab and run [[Find_orphans.py]] should the 2Tb fail..
 
This system has been developed with version 72 of Myth Protocol and it will need validating carefully against other implementations.  See [[Myth Protocol/Guide]] and [[Myth Protocol]].  The author believes that there should be no problems with protocol versions 67 to 75 (myth 0.25 and 0.26) and the keys are included in the code but commented out.  To validate against one of these versions uncomment the key (see %MythProtocolKey) and test thoroughly with --loglevel 4 and then with --preview.  There were major changes to the program record at version 67 so earlier versions will need more radical changes.
 
  
  
Line 194: Line 192:
 
*set permissions to allow access by root and mythtv.
 
*set permissions to allow access by root and mythtv.
 
*Ensure you can access it too if you wish to test it.
 
*Ensure you can access it too if you wish to test it.
*Copy this code to /usr/bin/autobackup.pl (or elsewhere) and chmod + x it.
+
*Copy this code to /usr/bin/autobackup.pl (or elsewhere) and chmod +x it.
*Do test with --preview first!
+
*Test with --preview first!
*Research protocol versions and test thoroughly if necessary.
+
*If it fails to find the perl library 'LWP::UserAgent' then install it with 'sudo apt-get install libwww-perl'
 
*trigger it daily - see [[autobackup.sh]] as a possible mechanism.
 
*trigger it daily - see [[autobackup.sh]] as a possible mechanism.
 
  
 
{{perl|autobackup.pl|
 
{{perl|autobackup.pl|
Line 208: Line 205:
 
use Time::Local;
 
use Time::Local;
  
#
+
# if next line fails you need to do 'sudo apt-get install libwww-perl'
 +
use LWP::UserAgent;
 +
 
 +
 
 
# Select recordings for backup to a small partition and copy/delete as necessary.
 
# Select recordings for backup to a small partition and copy/delete as necessary.
 
# See --help for details
 
# See --help for details
Line 216: Line 216:
  
 
#program version
 
#program version
use constant VERSION  => '1.01 17 Dec 2012';
+
use constant VERSION  => '1.10 2 June 2014';
  
 
#change history
 
#change history
Line 226: Line 226:
 
   # threshold reduced to 82% to improve stability  
 
   # threshold reduced to 82% to improve stability  
 
   # cosmetic changes to reporting
 
   # cosmetic changes to reporting
 +
#version 1.02 23 March 2013
 +
  # skip livetv recordings
 +
#version 1.03, 18 Oct 2013
 +
  #format of recording file names has changed following retune in UK.
 +
  #  some are eg 17717_20131010 etc with 5 digits before the _ not 4.
 +
  # Minor changes to routines timestamp, scansources and showall to accommodate this.
 +
#version 1.04 3 Jan 2014
 +
  # Corrected wrong exit codes.  Minor edits
 +
#version 1.10 2 June 2014.
 +
  # switched to getting details from database via Service API.
 +
  #  This is simpler, more resilient and should need fewer changes with mythtv version changes.
 +
  # also removed tab stops from source - a cosmetic change only.
  
 +
 +
 
 
#control parameters
 
#control parameters
 
my %params =(
 
my %params =(
  "partition" => "/var/lib/mythtv2",
 
  "destdir" => "/var/lib/mythtv2/recordings",
 
  
   "sourcedir1" => "/var/lib/mythtv/recordings",  #first or only dir in recording group
+
#  destination
# "sourcedir2" => "/var/lib/mythtv/other", #add others like this. Note sourcedir1 to n, not 0 to n
+
  "partition"  => "/var/lib/mythtv2",
 +
  "destdir"    => "/var/lib/mythtv2/recordings",
 +
 
 +
#  source(s)
 +
   "sourcedir1" => "/var/lib/mythtv/recordings",  #first or only dir in recording group
 +
# "sourcedir2" => "/var/lib/mythtv/other", #add others like this. Note sourcedir1 to n, not 0 to n
  
   "percent" => 90,   # max used% of partition
+
#  How full?
   "threshold" => 82, # percentage for older files to be promoted.
+
   "percent"     => 90,     # max used% of partition
  "SeriesPolicy"=>  1, # if set, use priority of newest member of a series. ]  Use either, neither
+
   "threshold"   => 82,     # percentage for older files to be promoted.
  "SeriesBoost" => 0, # boost in days for series.  ]  or both of these boosts.
 
  "preview" =>  0, # command line parameter -p or --preview.
 
  "filelist" => "/var/lib/mythtv2/recordings/autobackup.list", #list recordings here so we can identify deleted ones next day
 
);
 
  
 +
# Series policies: use none, one or both:
 +
  "SeriesPolicy"=>  1,      # if set, use priority of newest member of a series.
 +
  "SeriesBoost" =>  0,      # boost in days for series.
  
# default --loglevel parameter
+
#Previewing default
my $loglevel=3; # 0=errors only, 1=calling params, 2=stats, 3=recordings details, 4=protocol diagnostic trace.
+
  "preview"    =0,     # command line parameter -p or --preview.
  
 +
# Keep a list of recordings here so we can identify deleted ones next day
 +
  "filelist"    =>  "/var/lib/mythtv2/recordings/autobackup.list",
 +
);
  
# backend parameters
 
use constant myipaddress => '127.0.0.1';
 
use constant backendip => '127.0.0.1';
 
my $sock;
 
  
 +
# default loglevel parameter
 +
my $loglevel=3;    # 0=errors only, 1=calling params, 2=stats,
 +
                    # 3=recordings details, 4=diagnostic trace.
  
# Myth protocol keys    see http://www.mythtv.org/wiki/Category:Myth_Protocol
 
#all these listed **should** be ok but only version 72 has been validated.
 
#uncomment the appropriate key(s) and do a sanity check with loglevel 4 before use. 
 
#version prior to 67 will need more radical changes - different program record definitions at least.
 
  
my %MythProtocolKey = (
+
# backend address
# '67' => '0G0G0G0',
+
use constant backendip      => '127.0.0.1';
# '68' => '90094EAD',
 
# '69' => '63835135',
 
# '70' => '53153836',
 
# '71' => '05e82186',
 
'72' => 'D78EFD6F',
 
# '73' => 'D7FE8D6F',
 
# '74' => 'SingingPotato',
 
# '75' => 'SweetRock',
 
);
 
  
  
#constants for extracting details from program record
+
# Bits in recording flags
use constant recstart        => 1;  #start of first record
 
use constant reclength      => 44; #length of each record
 
use constant recname        => 0;  #name of program
 
use constant recfilename    => 10;
 
use constant recsize        => 11;  #recording size
 
use constant recflags        => 27;  #flags
 
 
use constant    flagautoexp      => 4;
 
use constant    flagautoexp      => 4;
use constant    flagdeletepending => 128; #this bit utilised but never seen set!
+
use constant    flagdeletepending => 128;   #this bit utilised but never seen set!
 
use constant    flagpriority      => 1024;  #not used.
 
use constant    flagpriority      => 1024;  #not used.
use constant recgroup => 28; #more accurate indicator of pending delete
 
  
 
my $gig= 1024*1024*1024;
 
my $gig= 1024*1024*1024;
Line 292: Line 290:
 
                 # +4 =secure before, +8=secure after.
 
                 # +4 =secure before, +8=secure after.
 
my %fpriority;  #hold merit for saving the recording
 
my %fpriority;  #hold merit for saving the recording
my %newest; #time of most recent episode in series
+
my %newest;  #time of most recent episode in series
 
my @deleteq;    #list of files for deletion
 
my @deleteq;    #list of files for deletion
 
my @promoteq;  #ditto promotion
 
my @promoteq;  #ditto promotion
  
#statistics
+
 
my %stats;
+
my %stats;    #statistics
 +
my $url_buffer;     #buffer for reading url
  
 
#---------ok, let's start!---------
 
#---------ok, let's start!---------
Line 305: Line 304:
 
&check_partition;
 
&check_partition;
  
&scan_database; #to get list of recordings
+
&scan_database; #to get list of recordings
  
&mark_series; #identify series recordings and mark episodes
+
&mark_series;   #identify series recordings and mark episodes
  
#scan source directories and check consistency with database
 
&scansources;
 
  
#scan destination directory.  Identify which recordings are backed up and find obsolete ones.
+
&scansources;    #scan source directories and check consistency with database
&scandir($params{'destdir'},1);
 
  
&maintain_list('read'); #obsolete recordings are anonymous. Get names by checking yesterday's list
+
&scandir($params{'destdir'},1);   # Scan destination directory. 
 +
                                  # Identify which recordings are backed up
 +
                                  # and find obsolete ones.  
  
&decide_priorities; #based on expirability, obsolescence, series episode and recording time.
+
 
 +
&maintain_list('read');    #obsolete recordings are anonymous; 
 +
                            #Get names by checking yesterday's list
 +
 
 +
&decide_priorities;         #based on expirability, obsolescence, series episode and recording time.
  
 
#now sort them.   
 
#now sort them.   
my @sorted = sort {$fpriority{$b} <=> $fpriority{$a} or #priority
+
my @sorted = sort {$fpriority{$b} <=> $fpriority{$a} or         #priority
               $frecname{$b} cmp $frecname{$a} or   #keeps series episodes together
+
               $frecname{$b} cmp $frecname{$a} or               #keeps series episodes together
               substr($b,5,14) <=> substr($a,5,14)} keys %fstatus; #in date order
+
               substr($b,5,14) <=> substr($a,5,14)} keys %fstatus;   #in date order
 
my @sorted2=@sorted;
 
my @sorted2=@sorted;
  
&decide_action; #decide promotions/demotions
+
&decide_action;             #decide promotions/demotions
#note:  operation depends heavily on allocated priorities & sort order.
+
                            #note:  operation depends heavily on allocated priorities & sort order.
  
&do_action; #now do the copies/deletes
+
&do_action;                 #now do the copies/deletes
  
&showall if $loglevel==3; #list all known recordings
+
&showall if $loglevel==3;   #list all known recordings
  
&maintain_list('write'); #make new list so we can identify deleted recordings tomorrow
+
&maintain_list('write');   #make new list so we can identify deleted recordings tomorrow
  
 
#almost done!
 
#almost done!
Line 341: Line 343:
 
sub maintain_list{
 
sub maintain_list{
  
# Called with read or write.
+
#   Called with read or write.
  
# This routine writes a list of saved recording file names and program names to a file.
+
#   This routine writes a list of saved recording file names and program names to a file.
# It checks the list the following day to put a name to deleted recordings which  
+
#   It checks the list the following day to put a name to deleted recordings which  
# would otherwise be anonymous (named '??').
+
#   would otherwise be anonymous (named '??').
  
# If you are using the recordingdelete event to remove backup recordings then you will probably
+
#   If you are using the recordingdelete event to remove backup recordings then you will probably
# not see these anonymous recordings.   
+
#   not see these anonymous recordings.   
  
# To remove this functionality, uncomment the next line
+
#   To remove this functionality, uncomment the next line
# return;
+
#   return;
  
# Documentation/reminder:  Myth delete mechanism seems to be:
+
#   Documentation/reminder:  Myth delete mechanism seems to be:
# 1. changes group to 'Deleted'
+
#   1. changes group to 'Deleted'
# 2. Some while later, backend deletes file and removes entry from database.  
+
#   2. Some while later, backend deletes file and removes entry from database.  
# 3. Deletepending flag not seen in use.
+
#   3. Deletepending flag not seen in use.
  
if ($_[0] eq 'read'){
+
    if ($_[0] eq 'read'){
  
#check yesterday's list to name deleted recordings
+
        #check yesterday's list to name deleted recordings
my $nameyesterday;
+
        my $nameyesterday;
if ($loglevel != 3){return}; #nobody listening!
+
        if ($loglevel != 3){return};           #nobody listening!
if ($stats{'noname'}==0){return}; #no anonymous ones to check
+
        if ($stats{'noname'}==0){return};       #no anonymous ones to check
unless (open FH, "<$params{'filelist'}"){  
+
        unless (open FH, "<$params{'filelist'}"){  
print "Warning:  cannot read $params{'filelist'} $!\n";
+
            print "Warning:  cannot read $params{'filelist'} $!\n";
return;  #not fatal
+
            return;  #not fatal
}
+
        }
while (<FH>){
+
        while (<FH>){
chomp;
+
            chomp;    
($_,$nameyesterday)=split /\[\]:\[\]/,$_;
+
            ($_,$nameyesterday)=split /\[\]:\[\]/,$_;
if (exists $frecname{$_}){
+
            if (exists $frecname{$_}){
if ($frecname{$_} eq '??'){$frecname{$_} = ".$nameyesterday"};
+
                if ($frecname{$_} eq '??'){$frecname{$_} = ".$nameyesterday"};
}
+
            }
}
+
        }
close FH;
+
        close FH;
  
}elsif ($_[0] eq 'write'){
+
    }elsif ($_[0] eq 'write'){
if ($params{'preview'}) {return};
+
        if ($params{'preview'}) {return};
  
#write list of backed up recordings for use tomorrow
+
        #write list of backed up recordings for use tomorrow
unless (open FH, ">$params{'filelist'}"){
+
        unless (open FH, ">$params{'filelist'}"){
print "Cannot write $params{'filelist'} $!" if $loglevel >1;
+
            print "Cannot write $params{'filelist'} $!" if $loglevel >1;
return;
+
            return;
}  
+
        }  
for (keys %fstatus){
+
        for (keys %fstatus){
if (($fstatus{$_} & 11) > 8){
+
            if (($fstatus{$_} & 11) > 8){
#this is a backed up file so remember for tomorrow
+
                #this is a backed up file so remember for tomorrow
print FH $_ . '[]:[]' . $frecname{$_}."\n";
+
                print FH $_ . '[]:[]' . $frecname{$_}."\n";
}
+
            }
}  
+
        }  
close FH;
+
        close FH;
}else{
+
    }else{
print "bad maintain_list $_[0]\n";
+
        print "bad maintain_list $_[0]\n";
exit 1;
+
        exit 1;
}
+
    }
 
}
 
}
 
#-------------------
 
#-------------------
 
sub decide_priorities{
 
sub decide_priorities{
my $st;
+
    my $st;
foreach (keys %fstatus){
+
    foreach (keys %fstatus){
$st=($fstatus{$_} & 3);
+
        $st=($fstatus{$_} & 3);
if ($st==0){
+
        if ($st==0){
$fpriority{$_}=0;
+
            $fpriority{$_}=0;
}elsif ($st==3){
+
        }elsif ($st==3){
$fpriority{$_} +=7305; #20 years boost for vip=non expirable
+
            $fpriority{$_} +=7305;     #20 years boost for vip=non expirable
}elsif ($st==2){ #boost episodes of series
+
        }elsif ($st==2){               #boost episodes of series
if ($params{'SeriesPolicy'}){
+
            if ($params{'SeriesPolicy'}){
$fpriority{$_}=$newest{$frecname{$_}};
+
                $fpriority{$_}=$newest{$frecname{$_}};
}
+
            }
$fpriority{$_} += $params{'SeriesBoost'};
+
            $fpriority{$_} += $params{'SeriesBoost'};
}
+
        }
}
+
    }
 
}
 
}
 
#-------------------
 
#-------------------
 
sub timestamp{
 
sub timestamp{
#extract timestamp from filename - return epoch DAYS as a real.
+
    #extract timestamp from filename - return epoch DAYS as a real.
  
(my $in)=@_; #filename
+
    (my $in)=@_;   #filename
$in=substr($in,5,14);
+
    if ($in =~ /_(\d{14})\./) {
my @bits=($in =~ m/../g);
+
        $in=$1;
my $out = timelocal($bits[6],$bits[5],$bits[4],$bits[3],$bits[2]-1,$bits[0].$bits[1]);
+
    }else{
$out/86400;
+
        $in='20131010190000';
 +
    }
 +
    my @bits=($in =~ m/../g);
 +
    my $out = timelocal($bits[6],$bits[5],$bits[4],$bits[3],$bits[2]-1,$bits[0].$bits[1]);
 +
    $out/86400;
 
}
 
}
 
#-------------------
 
#-------------------
 
sub mark_series{
 
sub mark_series{
  
#identify series of recordings; mark the episodes
+
    #identify series of recordings; mark the episodes
  
my %namecount;
+
    my %namecount;
my $tmp;
+
    my $tmp;
my %oldest; # %newest is global as needed later in decide_priorities.
+
    my %oldest; # %newest is global as needed later in decide_priorities.
  
  
#count instances of names, keep newest/oldest
+
    #count instances of names, keep newest/oldest
foreach (keys %frecname){
+
    foreach (keys %frecname){
$tmp=&timestamp($_);
+
        $tmp=&timestamp($_);
$fpriority{$_}=$tmp;
+
        $fpriority{$_}=$tmp;      
$_=$frecname{$_};
+
        $_=$frecname{$_};
  
#keep count, oldest, newest for each name found
+
        #keep count, oldest, newest for each name found
$namecount{$_}++;
+
        $namecount{$_}++;
if (exists $newest{$_}) {
+
        if (exists $newest{$_}) {
if ($tmp < $oldest{$_}) {$oldest{$_}=$tmp};
+
            if ($tmp < $oldest{$_}) {$oldest{$_}=$tmp};
if ($tmp > $newest{$_}) {$newest{$_}=$tmp};
+
            if ($tmp > $newest{$_}) {$newest{$_}=$tmp};
}else{
+
        }else{
$oldest{$_}=$tmp;
+
            $oldest{$_}=$tmp;
$newest{$_}=$tmp;
+
            $newest{$_}=$tmp;
}
+
        }
}
+
    }
  
#mark as series if more than 2 episodes and average interval no more than 8 days.
+
    #mark as series if more than 2 episodes and average interval no more than 8 days.
my $count;
+
    my $count;
my $avinterval;
+
    my $avinterval;
foreach (keys %namecount){
+
    foreach (keys %namecount){
$count=$namecount{$_};
+
        $count=$namecount{$_};
$avinterval=0;
+
        $avinterval=0;
if ($count>1){$avinterval=($newest{$_}-$oldest{$_})/($count-1)};
+
        if ($count>1){$avinterval=($newest{$_}-$oldest{$_})/($count-1)};
if (($count>2) && ($avinterval<8)){
+
        if (($count>2) && ($avinterval<8)){
#this is a series
+
                #this is a series
$namecount{$_}=-1;
+
                $namecount{$_}=-1;
}
+
        }
}
+
    }
  
#mark the recordings as series episodes
+
    #mark the recordings as series episodes
foreach (keys %frecname){
+
    foreach (keys %frecname){
if ($namecount{$frecname{$_}} <0) {
+
        if ($namecount{$frecname{$_}} <0) {
if ($fstatus{$_}==1){$fstatus{$_}=2};  
+
            if ($fstatus{$_}==1){$fstatus{$_}=2};  
}
+
        }
}
+
    }
 
}
 
}
 
#--------------------
 
#--------------------
 
sub do_action{
 
sub do_action{
  
if ($params{'preview'}){return 0};
+
    if ($params{'preview'}){return 0};
my $command;
+
    my $command;
my $qsize;
+
    my $qsize;
my $fname;
+
    my $fname;
  
#do deletes first
+
    #do deletes first
$qsize=@deleteq;
+
    $qsize=@deleteq;
print "\nNow deleting $qsize files\n" if $loglevel>2;
+
    print "\nNow deleting $qsize files\n" if $loglevel>2;
if ($qsize){   
+
    if ($qsize){   
foreach $_ (@deleteq){
+
        foreach $_ (@deleteq){
$command= "rm -f $params{'destdir'}/$_";
+
            $command= "rm -f $params{'destdir'}/$_";
system $command;
+
            system $command;
}
+
        }
}
+
    }
  
#now do promotions
+
    #now do promotions
$qsize=@promoteq;
+
    $qsize=@promoteq;
print "Now promoting $qsize files\n" if $loglevel>2;
+
    print "Now promoting $qsize files\n" if $loglevel>2;
if ($qsize==0){return};
+
    if ($qsize==0){return};
  
#count the source directories
+
    #count the source directories
my $dirct=0;
+
    my $dirct=0;
while (exists $params{'sourcedir'.++$dirct}){
+
    while (exists $params{'sourcedir'.++$dirct}){
}
+
    }
$dirct--;
+
    $dirct--;
  
#do copies
+
    #do copies
for $fname (@promoteq){
+
    for $fname (@promoteq){
for (1..$dirct){
+
        for (1..$dirct){      
if ( -e "$params{'sourcedir'.$_}/$fname"){
+
            if ( -e "$params{'sourcedir'.$_}/$fname"){
$command="cp -p $params{'sourcedir'.$_}/$fname $params{'destdir'}";
+
                $command="cp -p $params{'sourcedir'.$_}/$fname $params{'destdir'}";
#print "$command\n";
+
                #print "$command\n";              
system $command;
+
                system $command;
last;
+
                last;
}
+
            }
}
+
        }
}
+
    }
+
   
 
}
 
}
 
#-------------------
 
#-------------------
 
sub decide_action{
 
sub decide_action{
  
$stats{'promote'}=0;
+
    $stats{'promote'}=0;
$stats{'promote size'}=0;
+
    $stats{'promote size'}=0;
$stats{'demote'}=0;
+
    $stats{'demote'}=0;
$stats{'demote size'}=0;
+
    $stats{'demote size'}=0;
  
#queue obsolete files for deletion
+
    #queue obsolete files for deletion
my $finished=0;
+
    my $finished=0;
while ($finished==0){
+
    while ($finished==0){
if (@sorted==0){    #array empty
+
        if (@sorted==0){    #array empty
$finished=1;
+
            $finished=1;
}else{
+
        }else{
$_=pop @sorted;
+
            $_=pop @sorted;
if ($fstatus{$_} & 3){        #not obsolete - we overshot!
+
            if ($fstatus{$_} & 3){        #not obsolete - we overshot!
push @sorted,$_;          #put it back
+
                push @sorted,$_;          #put it back
$finished=1;
+
                $finished=1;
}elsif ($fstatus{$_} ==12){  #obsolete file for deletion
+
            }elsif ($fstatus{$_} ==12){  #obsolete file for deletion
push @deleteq,$_;
+
                push @deleteq,$_;
$fstatus{$_} -=8;        #demoted
+
                $fstatus{$_} -=8;        #demoted
}
+
            }
}
+
        }
}
+
    }
  
$stats{'used size'} -=$stats{'obsolete recordings size'};
+
    $stats{'used size'} -=$stats{'obsolete recordings size'};
#report
+
    #report
$stats{'delete obsolete'}=$stats{'obsolete recordings'};    #cleaner name for user
+
    $stats{'delete obsolete'}=$stats{'obsolete recordings'};    #cleaner name for user
$stats{'delete obsolete size'}=$stats{'obsolete recordings size'};
+
    $stats{'delete obsolete size'}=$stats{'obsolete recordings size'};
  
#now decide promotions/deletion
+
    #now decide promotions/deletion
my $use_threshold=0;  #set once we find a secure normal recording
+
    my $use_threshold=0;  #set once we find a secure normal recording
while ($_ = shift @sorted){  #newest
+
    while ($_ = shift @sorted){  #newest
if ($fstatus{$_}==0){      #skip deletions
+
        if ($fstatus{$_}==0){      #skip deletions
}elsif ($fstatus{$_} <4){    #an unsaved recording - try to save it
+
        }elsif ($fstatus{$_} <4){    #an unsaved recording - try to save it
if (&freespace($fsize{$_},$use_threshold)) {
+
            if (&freespace($fsize{$_},$use_threshold)) {
#space made - mark for promotion
+
                #space made - mark for promotion
if ($use_threshold){ #put in order: oldest fresh, other fresh, newest stale, oldest stale
+
                #put in order: oldest fresh, other fresh, newest stale, oldest stale
push @promoteq,$_;
+
                if ($use_threshold){               
}else{
+
                    push @promoteq,$_;
unshift @promoteq,$_;
+
                }else{
}
+
                    unshift @promoteq,$_;
$fstatus{$_} +=8;
+
                }
$stats{'promote'}++;
+
                $fstatus{$_} +=8;
$stats{'promote size'} +=$fsize{$_};
+
                $stats{'promote'}++;
$stats{'used size'} +=$fsize{$_};
+
                $stats{'promote size'} +=$fsize{$_};
}
+
                $stats{'used size'} +=$fsize{$_};
}elsif ($fstatus{$_} == 13){
+
            }
        #non vip already backed up.
+
        }elsif ($fstatus{$_} == 13){
$use_threshold=1;
+
            #non vip already backed up.
}
+
            $use_threshold=1;
}
+
        }
 +
    }
  
&show_stats('Changes proposed:','delete obsolete','demote','promote');
+
    &show_stats('Changes proposed:','delete obsolete','demote','promote');
&show_stats('Proposed final state of partition:','used');
+
    &show_stats('Proposed final state of partition:','used');
 
}     
 
}     
  
 
#--------------------
 
#--------------------
 
sub freespace{
 
sub freespace{
#free up space by removing oldest secured files
+
    #free up space by removing oldest secured files
#return 1 if successful, 0 if not
+
    #return 1 if successful, 0 if not
my ($needed,$use_threshold)=@_;
+
    my ($needed,$use_threshold)=@_;
 
 
#buggy code removed 17 dec 12
 
#check against lower threshold first
 
#if ($needed < ($stats{'threshold size'} - $stats{'used size'})) {return 1};
 
#if ($use_threshold){return 0};
 
 
 
#decide appropriate limit
 
my $limit =  $stats{'limit size'};
 
if ($use_threshold) {$limit =  $stats{'threshold size'}};
 
  
 +
    #decide appropriate limit
 +
    my $limit =  $stats{'limit size'};
 +
    if ($use_threshold) {$limit =  $stats{'threshold size'}};
  
my $loopstate = -1;
+
    my $loopstate = -1;
while ($loopstate == -1){
+
    while ($loopstate == -1){
if ($needed < ($limit - $stats{'used size'})) {
+
        if ($needed < ($limit - $stats{'used size'})) {
$loopstate=1;  #will fit now
+
            $loopstate=1;  #will fit now
}else{
+
        }else{
if (@sorted){
+
            if (@sorted){
my $d = pop(@sorted);
+
                my $d = pop(@sorted);
if ($fstatus{$d} >12) {
+
                if ($fstatus{$d} >12) {
#demote this one
+
                    #demote this one
push @deleteq,$d;
+
                    push @deleteq,$d;
$fstatus{$d} -=8;
+
                    $fstatus{$d} -=8;
$stats{'demote'}++;
+
                    $stats{'demote'}++;
$stats{'demote size'} += $fsize{$d};
+
                    $stats{'demote size'} += $fsize{$d};
$stats{'used size'} -= $fsize{$d};
+
                    $stats{'used size'} -= $fsize{$d};
}
+
                }  
}else{
+
            }else{
#no files left to remove
+
                #no files left to remove
$loopstate=0;
+
                $loopstate=0;
}
+
            }
}
+
        }
}
+
    }
$loopstate;
+
    $loopstate;
}
 
#---------------------------
 
sub opensocket{
 
print "Opening socket to backend\n" if $loglevel>=4;
 
unless ($sock = IO::Socket::INET->new(PeerAddr => backendip,
 
PeerPort => 6543,
 
Proto => 'tcp',
 
timeout => 25) ) {
 
print "cannot open socket to backend.\n";
 
print "Wrong backend IP address?  Backend not running?\n";
 
exit 0;
 
}
 
 
}
 
}
 
#---------------------------
 
#---------------------------
 
sub scan_database {
 
sub scan_database {
  
#probe for protocol version
+
    #open the browser
&opensocket;
+
    my $browser = LWP::UserAgent->new;
print "Probing for protocol version\n" if $loglevel>=4;
+
    $browser->timeout(10);
my $r= &getreply("MYTH_PROTO_VERSION 1");
 
unless ($r =~ /^REJECT/){
 
die "Odd response from MYTH_PROTO_VERSION";
 
}
 
my $VersionFound=substr($r,11,);
 
close($sock);
 
unless (exists $MythProtocolKey{$VersionFound}){
 
die "Mythprotocol version not known or validated";
 
}
 
  
#now open properly
 
&opensocket;
 
$r= &getreply("MYTH_PROTO_VERSION ". $VersionFound .' '. $MythProtocolKey{$VersionFound});
 
unless (substr($r,0,6) eq 'ACCEPT') {
 
die "ERROR:  Wrong protocol version:  $r";
 
}
 
  
print "ANNouncing - if no reply, suspect myipaddress\n" if $loglevel>=4;
+
    my $item=-2;    #values to kick it off
$r= &getreply('ANN Monitor '.myipaddress . " 0");
+
    my $total=-1;
unless (substr($r,0,2) eq 'OK') {
+
    my $base;
die "ERROR:  ANN rejected:  $r";
+
    my $response;
}
+
    print "Getting list of recordings\n" if $loglevel>=4;
  
print "Getting list of recordings\n" if $loglevel>=4;
+
    while ($item < $total) {
$r= &getreply('QUERY_RECORDINGS Unsorted');
+
        $item=1 if $item<0;
my $sep = qr/\[\]:\[\]/;
+
 
my @recs = split($sep, $r);
+
        #get a batch of 100 recordings
 +
        my $response = $browser->get('http://' . backendip .
 +
              ":6544/Dvr/GetRecordedList?StartIndex=$item&Count=100&Descending=true");
 +
        unless($response->is_success){ die 'bad response to get'};
 +
        $url_buffer = $response -> content;
 +
 
 +
        if ($total <0) {$total=&getparameter('TotalAvailable',0)};
 +
 
 +
        #now process them
 +
        if ($total >0) {
 +
            # split this batch of recordings on <Program>.
 +
            $base=index($url_buffer, '<Program>');
 +
            while ($base>-1){
 +
                &ProcessRecording($base);
 +
                $base=index($url_buffer, '<Program>', $base+10);
 +
            }
 +
            $item=$item+100;
 +
        }
 +
    }
 +
    print "\nDatabase scan found $stats{'DBrecordings'} recordings\n" if $loglevel>1;
 +
    if ($loglevel == 4) { exit 0};
 +
}
 +
#------------------------------
 +
sub ProcessRecording{
  
$stats{'DBrecordings'}=$recs[0]; #no of recordings
+
    my $startat = $_[0];
print "\nDatabase scan found $recs[0] recordings\n" if $loglevel>1;
+
    my $fname= &getparameter('FileName',$startat);
 +
    $frecname{$fname} = &getparameter("Title", $startat);
 +
    $fsize{$fname}=&getparameter('FileSize',$startat);
  
#extract info from recordings
+
    my $flags=&getparameter('ProgramFlags',$startat);
my ($fname, $fstatus);
+
    my $recgroup=&getparameter('RecGroup',,$startat);
for (my $i= recstart; $i <= $#recs; $i += reclength) {
 
(undef, undef, undef, $fname)=split /\//, $recs[$i+recfilename];
 
$frecname{$fname} = $recs[$i+recname];
 
$fsize{$fname}=$recs[$i+recsize];
 
my $flags=$recs[$i+recflags];
 
if ($recs[$i+recgroup] eq 'Deleted'){    #just deleted
 
$fstatus{$fname}=0;
 
}elsif (($flags & flagdeletepending)>0){
 
$fstatus{$fname}=0;   #Delete pending flag (not used?)
 
}elsif (($flags & flagautoexp)==0){
 
$fstatus{$fname}=3;    #non expirable so vip
 
}else{
 
$fstatus{$fname}=1;    #expirable
 
}
 
if ($loglevel==4){
 
print "$fname $flags $frecname{$fname}\n";
 
}
 
}
 
  
close($sock);
+
    if ($recgroup eq 'Deleted'){      #only just deleted
if ($loglevel==4){
+
        $fstatus{$fname}=0;
print "Scan finishing because loglevel set to 4 for protocol diagnostics.\n";
+
    }elsif ($recgroup eq 'LiveTV'){    #ignore livetv recordings
exit 0;
+
        $fstatus{$fname}=0;
}
+
    }elsif (($flags & flagdeletepending)>0){
 +
        $fstatus{$fname}=0;            #Delete pending flag (not used?)
 +
    }elsif (($flags & flagautoexp)==0){
 +
        $fstatus{$fname}=3;            #non expirable so vip
 +
    }else{
 +
        $fstatus{$fname}=1;            #expirable
 +
    }
 +
    $stats{'DBrecordings'}++;
 +
    if ($loglevel==4){print "$fname $flags $frecname{$fname}\n"};
 
}
 
}
  
#------------------
+
#-------------------------
sub getreply {
+
sub getparameter{
my ($command) = @_;
+
 
my ($length, $ret, $bytes, $data);
+
my ($param, $base) = @_;
my $com = length($command). ' ' x (8 - length(length($command))). $command;
 
  
if ($loglevel==4){print "sending $com\n"};
+
        my $start= index($url_buffer, "<$param>",$base);
print $sock $com;
+
        if ($start<0){die "cannot find $_[0]"};
  
#read length of reply
+
        $start=$start + length($param) +2;
read($sock, $length, 8);  
+
       
$length = int($length);
+
        my $end=index($url_buffer, "</$param>", $start);
 +
        substr $url_buffer, $start, $end-$start;  
  
# Read and return any data that was returned
 
$ret = '';
 
while ($length > 0) {
 
$bytes = read($sock, $data, ($length < 8192 ? $length : 8192));
 
last if ($bytes < 1);  # EOF
 
# On to the next
 
$ret .= $data;
 
$length -= $bytes;
 
}
 
print "Reply starts:". substr($ret,0,200)."\n" if $loglevel >=4;
 
return $ret;
 
 
}
 
}
  
  
#-------------------------
+
#-----------------
 +
 
 
sub check_partition{
 
sub check_partition{
my $reply = `df $params{'partition'}`;
+
    my $reply = `df $params{'partition'}`;
if (length $reply == 0){die 'cannot find partition'};
+
    if (length $reply == 0){die 'cannot find partition'};
my @fields = split /\s+/,$reply;
+
    my @fields = split /\s+/,$reply;
$stats{'partition size'}=$fields[8]*1024;
+
    $stats{'partition size'}=$fields[8]*1024;
$stats{'used size'}=$fields[9]*1024;
+
    $stats{'used size'}=$fields[9]*1024;
$stats{'used'} = sprintf "%2d%%", $stats{'used size'} *100 / $stats{'partition size'};
+
    $stats{'used'} = sprintf "%2d%%", $stats{'used size'} *100 / $stats{'partition size'};
$stats{'limit size'} = ($stats{'partition size'} * $params{'percent'} /100);
+
    $stats{'limit size'} = ($stats{'partition size'} * $params{'percent'} /100);
$stats{'limit'}= $params{percent} . '%';
+
    $stats{'limit'}= $params{percent} . '%';
$stats{'size'} = '100%';
+
    $stats{'size'} = '100%';
$stats{'size size'}=$stats{'partition size'};  #same but more readable in output
+
    $stats{'size size'}=$stats{'partition size'};  #same but more readable in output
$stats{'threshold'}=$params{'threshold'} .'%';
+
    $stats{'threshold'}=$params{'threshold'} .'%';
$stats{'threshold size'}= $params{'threshold'} * $stats{'partition size'}/100;
+
    $stats{'threshold size'}= $params{'threshold'} * $stats{'partition size'}/100;
&show_stats ("Partition $params{'partition'} info", 'size', 'used', 'limit','threshold');
+
    &show_stats ("Partition $params{'partition'} info", 'size', 'used', 'limit','threshold');
 
}
 
}
  
Line 734: Line 713:
 
sub show_stats{
 
sub show_stats{
 
# print item, number & size
 
# print item, number & size
if ($loglevel<2) {return 0};
+
    if ($loglevel<2) {return 0};
my @items = @_;
+
    my @items = @_;
$_=shift @items;
+
    $_=shift @items;
if ($_) {print "\n$_\n"};    #optional heading
+
    if ($_) {print "\n$_\n"};    #optional heading
my $item_value;
+
    my $item_value;
while (@items){
+
    while (@items){
$_=shift @items;
+
        $_=shift @items;
if (exists $stats{$_}) {
+
        if (exists $stats{$_}) {
if ($_ eq 'used'){
+
            if ($_ eq 'used'){
#update used percentage
+
                #update used percentage
$stats{'used'} = sprintf "%2d%%", $stats{'used size'} *100 / $stats{'partition size'};
+
                $stats{'used'} = sprintf "%2d%%", $stats{'used size'} *100 / $stats{'partition size'};
}
+
            }
$item_value=$stats{$_};
+
            $item_value=$stats{$_};
#show item and number
+
            #show item and number
if ($item_value =~/%/){
+
            if ($item_value =~/%/){
printf "%7s",$item_value;
+
                printf "%7s",$item_value;
}else{
+
            }else{
printf "%7d",$item_value;
+
                printf "%7d",$item_value;
}
+
            }
}else{
+
        }else{
print "      ";
+
            print "      ";
}
+
        }
printf "  %-25s",$_;
+
        printf "  %-25s",$_;
  
#show size if present
+
        #show size if present
$_ .= ' size';
+
        $_ .= ' size';
if (exists $stats{"$_"}) {
+
        if (exists $stats{"$_"}) {
$item_value= $stats{$_};
+
            $item_value= $stats{$_};
printf "%7.2fG",$item_value/$gig;
+
            printf "%7.2fG",$item_value/$gig;
}
+
        }
print "\n";
+
        print "\n";
}
+
    }
 
}
 
}
 
#------------
 
#------------
 
sub showall{
 
sub showall{
#show all recordings
+
    #show all recordings
# print "\nfile status is Deleted, Secure, Insecure, Promote or Demote followed by vip (non expirable), series or deleted.\n";  
+
#   print "\nfile status is Deleted, Secure, Insecure, Promote or Demote followed by vip (non expirable), series or deleted.\n";  
print "\nAll files (priority order) Status    Size  Name (if known)\n";
+
    print "\nAll files (priority order) Status    Size  Name (if known)\n";
print "-----------------------------------------------------------\n";   
+
    print "-----------------------------------------------------------\n";   
foreach (@sorted2){
+
    foreach (@sorted2){
printf "$_  %9s  %3.1fGb  $frecname{$_}\n",&statusname($fstatus{$_}),$fsize{$_}/$gig;
+
        printf "%25s %9s  %3.1fGb  $frecname{$_}\n",$_, &statusname($fstatus{$_}),$fsize{$_}/$gig;
}
+
    }
 
}
 
}
 
#------------
 
#------------
 
sub scansources{
 
sub scansources{
#scan source directories, report only.
+
    #scan source directories, report only.
$stats{'missing'}=0;
+
    $stats{'missing'}=0;
$stats{'found'}=0;
+
    $stats{'found'}=0;
  
my $i=0;
+
    my $i=0;
while (exists $params{'sourcedir'.++$i}){
+
    while (exists $params{'sourcedir'.++$i}){
&scandir($params{'sourcedir'.$i},0);
+
        &scandir($params{'sourcedir'.$i},0);
}
+
    }
  
#counts consistent with d/base?
+
    #counts consistent with d/base?
$i=$stats{'DBrecordings'}-$stats{'found'};
+
    $i=$stats{'DBrecordings'}-$stats{'found'};
if ($i<0){$i=-$i};
+
    if ($i<0){$i=-$i};
$i += $stats{'missing'};
+
    $i += $stats{'missing'};  
if (($i>0) && ($loglevel>1)) {
+
    if (($i>0) && ($loglevel>1)) {
print "\nThere were $i database inconsistencies.\n";
+
        print "\nThere were $i database inconsistencies.\n";
print "Suggest using find_orphans.py\n";
+
        print "Suggest using find_orphans.py\n";
}
+
    }
 
}
 
}
 
#-------------
 
#-------------
 
sub scandir{
 
sub scandir{
my ($dir, $destination)=@_;
+
    my ($dir, $destination)=@_;
my ($size, $category);
+
    my ($size, $category);
#scan a folder, show stats for it
+
    #scan a folder, show stats for it
 
   
 
   
$stats{'vip recordings'}=0;
+
    $stats{'vip recordings'}=0;
$stats{'vip recordings size'}=0;
+
    $stats{'vip recordings size'}=0;
$stats{'expirable recordings'}=0;
+
    $stats{'expirable recordings'}=0;
$stats{'expirable recordings size'}=0;
+
    $stats{'expirable recordings size'}=0;
$stats{'obsolete recordings'}=0;
+
    $stats{'obsolete recordings'}=0;
$stats{'obsolete recordings size'}=0;
+
    $stats{'obsolete recordings size'}=0;
  
$stats{'series recordings'}=0;
+
    $stats{'series recordings'}=0;
$stats{'series recordings size'}=0;
+
    $stats{'series recordings size'}=0;
$stats{'noname'}=0;
+
    $stats{'noname'}=0;
  
opendir DH,$dir or die "Cannot open $dir: $!";
+
    opendir DH,$dir or die "Cannot open $dir: $!";
foreach $_ (readdir DH) {
+
    foreach $_ (readdir DH) {
if (/^\d{4}_\d{14}\.mpg$/) {
+
        if (/^\d+_\d{14}\.mpg$/) {
if (exists $fstatus{$_}) {
+
            if (exists $fstatus{$_}) {
$stats{'found'}++;
+
                $stats{'found'}++;
if ($destination){
+
                if ($destination){
$fstatus{$_} +=12;
+
                    $fstatus{$_} +=12;
}               
+
                }               
}else{
+
            }else{
$stats{'missing'}++;
+
                $stats{'missing'}++;
if ($destination){
+
                if ($destination){
$fsize{$_}= -s "$dir\/$_";
+
                    $fsize{$_}= -s "$dir\/$_";
$fstatus{$_} = 12;
+
                    $fstatus{$_} = 12;
$frecname{$_}='??';
+
                    $frecname{$_}='??';
$stats{'noname'}++;
+
                    $stats{'noname'}++;
}
+
                }
}
+
            }
+
           
#collect stats
+
            #collect stats
if (exists $fstatus{$_}){       
+
            if (exists $fstatus{$_}){       
$size=$fsize{$_};
+
                $size=$fsize{$_};
$_=($fstatus{$_} & 3);
+
                $_=($fstatus{$_} & 3);
$category=qw/obsolete expirable series vip/[$_];
+
                $category=qw/obsolete expirable series vip/[$_];
$stats{$category.' recordings'}++;
+
                $stats{$category.' recordings'}++;
$stats{$category.' recordings size'} += $size;
+
                $stats{$category.' recordings size'} += $size;
}
+
            }
}
+
        }
}
+
    }
closedir DH;
+
    closedir DH;
 
    
 
    
#summary
+
    #summary
$stats{'--- total ---'}=$stats{'series recordings'}+
+
    $stats{'--- total ---'}=$stats{'series recordings'}+
$stats{'expirable recordings'} + $stats{'vip recordings'} + $stats{'obsolete recordings'};
+
        $stats{'expirable recordings'} + $stats{'vip recordings'} + $stats{'obsolete recordings'};
$stats{'--- total --- size'}=$stats{'series recordings size'}
+
    $stats{'--- total --- size'}=$stats{'series recordings size'}
+$stats{'expirable recordings size'} + $stats{'vip recordings size'} + $stats{'obsolete recordings size'};
+
        +$stats{'expirable recordings size'} + $stats{'vip recordings size'} + $stats{'obsolete recordings size'};
$_=$destination?'Destination':'Source';
+
    $_=$destination?'Destination':'Source';
&show_stats("$_ directory $dir",'vip recordings', 'series recordings', 'expirable recordings','obsolete recordings','--- total ---');
+
    &show_stats("$_ directory $dir",'vip recordings', 'series recordings', 'expirable recordings','obsolete recordings','--- total ---');
 
}
 
}
 
    
 
    
 
#------------
 
#------------
 
sub statusname{
 
sub statusname{
#return text showing status  
+
    #return text showing status  
(my $st)=@_;
+
    (my $st)=@_;
my @ftype=('Del','  ','Ser','Vip');
+
    my @ftype=('Del','  ','Ser','Vip');
my $st2=($st & 12)/4;
+
    my $st2=($st & 12)/4;
qw/Ins Dem Pro Sec/[$st2] .' '. $ftype[$st & 3];
+
    qw/Ins Dem Pro Sec/[$st2] .' '. $ftype[$st & 3];
 
}
 
}
  
 
#------------------
 
#------------------
 
sub parse_command_line{
 
sub parse_command_line{
if (scalar @ARGV){
+
    if (scalar @ARGV){
my @argcopy=@ARGV;
+
        my @argcopy=@ARGV;
while (@argcopy){
+
        while (@argcopy){
$_ = shift @argcopy;
+
            $_ = shift @argcopy;
if (/^--preview$|^-p$/) {
+
            if (/^--preview$|^-p$/) {
$params{'preview'}=1;
+
                $params{'preview'}=1;
  
}elsif (/^--loglevel$/) {
+
            }elsif (/^--loglevel$/) {
my $bad=1;
+
                my $bad=1;
if (scalar @argcopy){
+
                if (scalar @argcopy){
$loglevel=shift @argcopy;
+
                    $loglevel=shift @argcopy;
if ($loglevel =~/\d/){
+
                    if ($loglevel =~/\d/){
if (($loglevel>=0) && ($loglevel<5)) {$bad=0};
+
                        if (($loglevel>=0) && ($loglevel<5)) {$bad=0};
}
+
                    }
}
+
                }
if ($bad){
+
                if ($bad){
print "bad loglevel needs 0 to 4\n";
+
                    print "bad loglevel needs 0 to 4\n";
exit 0;
+
                    exit 1;
}
+
                }
}elsif (/^--help$|^-h$/){
+
            }elsif (/^--help$|^-h$/){
&givehelp;
+
                &givehelp;
exit 0;
+
                exit 0;
}elsif (/^--delete$/){
+
            }elsif (/^--delete$/){
#recording delete event
+
                #recording delete event
$_ = "$params{'destdir'}/" . shift @argcopy;
+
                $_ = "$params{'destdir'}/" . shift @argcopy;
if ( -e $_){system "rm -f $_"};
+
                if ( -e $_){system "rm -f $_"};
exit 0;
+
                exit 0;
}else{
+
            }else{
print "parameter $_ not recognised.  Try $0 --help\n";
+
                print "parameter $_ not recognised.  Try $0 --help\n";
exit 0;
+
                exit 1;
}
+
            }
}
+
        }
}
+
    }
if ($loglevel){
+
    if ($loglevel){
$_= `date +%F" "%T`;
+
        $_= `date +%F" "%T`;
chomp;
+
        chomp;
print "\n$_ $0 @ARGV\n";
+
        print "\n$_ $0 @ARGV\n";
print "\nautobackup.pl version ". VERSION . ". running with loglevel=$loglevel, preview=". qw/no yes/[$params{'preview'}] ."\n";
+
        print   "\nautobackup.pl version ". VERSION . ". running with loglevel=$loglevel, preview=". qw/no yes/[$params{'preview'}] ."\n";
}
+
    }
  
 
}
 
}
Line 916: Line 895:
 
DESCRIPTION:
 
DESCRIPTION:
  
autobackup.pl selects a subset of mythtv recordings  
+
    autobackup.pl selects a subset of mythtv recordings  
and makes a backup of them to a destination partition which  
+
    and makes a backup of them to a destination partition which  
is typically smaller than the total size of all recordings
+
    is typically smaller than the total size of all recordings
and which is on a separate disk.  It allows recovery of  
+
    and which is on a separate disk.  It allows recovery of  
service with a 'useful' set of recordings in case of  
+
    service with a 'useful' set of recordings in case of  
primary disk failure.  
+
    primary disk failure.  
  
It should be augmented by appropriate database and system
+
    It should be augmented by appropriate database and system
backups.
+
    backups.
  
 
OPTIONS:
 
OPTIONS:
  
--preview or -p
+
    --preview or -p
to inhibit file copy/deletes.  This shows what  
+
        to inhibit file copy/deletes.  This shows what  
action would be taken with a full run.
+
        action would be taken with a full run.
  
--loglevel n to set logging levels.   
+
    --loglevel n   to set logging levels.   
1 for calling info,  
+
                1 for calling info,  
2 for stats,  
+
                2 for stats,  
3 for a recordings list
+
                3 for a recordings list
4 for protocol diagnostics
+
                4 to check extraction of recordings (depricated(
Default is 3.
+
                Default is 3.
  
--help or -h this text
+
    --help or -h       this text
  
--delete <filename>
+
    --delete <filename>
deletes file from destination directory.  For use with
+
        deletes file from destination directory.  For use with
recording delete system event, though use optional.
+
        recording delete system event, though use optional.
  
 
DETAILS:
 
DETAILS:
  
autobackup.pl is intended to be run as a regular job at system
+
    autobackup.pl is intended to be run as a regular job at system
closedown when the system is otherwise idle.  It checks  
+
    closedown when the system is otherwise idle.  It checks  
the destination partition to determine how full it is,  
+
    the destination partition to determine how full it is,  
then scans the database and the source and destination  
+
    then scans the database and the source and destination  
directories to check the state of all recordings.
+
    directories to check the state of all recordings.
It is then in a state to identify series recordings,  
+
    It is then in a state to identify series recordings,  
allocate priorities and determine the action for each file.
+
    allocate priorities and determine the action for each file.
  
Files are selected for backup in date/time order (newest  
+
    Files are selected for backup in date/time order (newest  
first) but:  
+
    first) but:  
  
- recordings marked as non-expirable have a 20 year  
+
    -   recordings marked as non-expirable have a 20 year  
priority boost
+
        priority boost
  
- recordings which are recognised as episodes of a series  
+
    -   recordings which are recognised as episodes of a series  
can be bunched together with the latest episode
+
        can be bunched together with the latest episode
and/or have a number of days added as a boost.
+
        and/or have a number of days added as a boost.
The author uses bunching with zero days boost.
+
        The author uses bunching with zero days boost.
  
Three or more recordings with the same name and an average  
+
    Three or more recordings with the same name and an average  
interval between them of less than 8 days are deemed to  
+
    interval between them of less than 8 days are deemed to  
be a series.
+
    be a series.
  
Files will be promoted from the head of the list, and files  
+
    Files will be promoted from the head of the list, and files  
demoted as necessary from the end in order to prevent  
+
    demoted as necessary from the end in order to prevent  
over-filling the partition.
+
    over-filling the partition.
  
 
PARTITION LIMITS
 
PARTITION LIMITS
  
New recordings will be promoted whilst observing the partition
+
    New recordings will be promoted whilst observing the partition
limit of $params{'percent'}%.  Older recordings will only be considered if the  
+
    limit of $params{'percent'}%.  Older recordings will only be considered if the  
partition is below a partition threshold of $params{'threshold'}% full.   
+
    partition is below a partition threshold of $params{'threshold'}% full.   
This is only likely happen after a spate of manual deletions.   
+
    This is only likely happen after a spate of manual deletions.   
  
The two limits prevent a recording from falling in and out
+
    The two limits prevent a recording from falling in and out
of favour as higher priority recordings are created and removed.
+
    of favour as higher priority recordings are created and removed.
Threshold should be a few days worth of recordings below limit.
+
    Threshold should be a few days worth of recordings below limit.
  
 
DELETED RECORDINGS
 
DELETED RECORDINGS
  
Recordings can optionally be deleted from the destination directory  
+
    Recordings can optionally be deleted from the destination directory  
with the RecordingDelete system event mechanism by calling this program  
+
    with the RecordingDelete system event mechanism by calling this program  
with --delete %FILE%.  This will remove the file more quickly.
+
    with --delete %FILE%.  This will remove the file more quickly.
  
However, such files will be routinely deleted by this program if the event
+
    However, such files will be routinely deleted by this program if the event
system is not used.
+
    system is not used.
 
   
 
   
Recordings which have been deleted from the database but still  
+
    Recordings which have been deleted from the database but still  
remain on the destination partition no longer have their name in
+
    remain on the destination partition no longer have their name in
the database.  In order to give these a name in listings, the script  
+
    the database.  In order to give these a name in listings, the script  
writes a list of today's recordings to the file to enable the names  
+
    writes a list of today's recordings to the file to enable the names  
to be established next day.  The file is  
+
    to be established next day.  The file is  
$params{'filelist'}  
+
    $params{'filelist'}  
+
   
 
WORKING PARAMETERS
 
WORKING PARAMETERS
  
See the parameters (\%params) at the head of the program defining  
+
    See the parameters (\%params) at the head of the program defining  
parameters such as the destination partition, percentage used,
+
    parameters such as the destination partition, percentage used,
source and destination directories, default logging level, series  
+
    source and destination directories, default logging level, series  
action, threshold setting and name of recordings list.
+
    action, threshold setting and name of recordings list.
  
 
LOGGING
 
LOGGING
  
If log files are required then they need to be redirected from STDOUT
+
    If log files are required then they need to be redirected from STDOUT
and STDERR and rotated as necessary by a calling script eg autobackup.sh
+
    and STDERR and rotated as necessary by a calling script eg autobackup.sh
  
 
   
 
   
 
DATABASE BACKUPS
 
DATABASE BACKUPS
  
The perl script has a sister routine autobackup.sh which
+
    The perl script has a sister routine autobackup.sh which
takes a database backup and triggers this script at last
+
    takes a database backup and triggers this script at last
closedown of the day.  See autobackup.sh.
+
    closedown of the day.  See autobackup.sh.
  
 
   
 
   
 
DEVELOPMENT REGIME
 
DEVELOPMENT REGIME
  
The script has been developed and tested in the UK on a  
+
    The script has been developed and tested in the UK on a  
Mythbuntu 12.04 dedicated frontend/backend system with  
+
    Mythbuntu 12.04 dedicated frontend/backend system with  
ACPI wakeup, DVB-T receivers, a single directory for  
+
    ACPI wakeup, DVB-T receivers, a single directory for  
recordings, and a local time close to or equal to GMT.   
+
    recordings, and a local time close to or equal to GMT.   
 
 
A 500Gb disk holds root, swap and backup recording partitions.
 
A 2Tb drive holds emergency root and swap and the primary
 
recording partition /var/lib/mythtv.
 
  
Assumptions may well have been inadvertently made which are
+
    A 500Gb disk holds root, swap and backup recording partitions.
not valid in all circumstances and the author can accept
+
    A 2Tb drive holds emergency root and swap and the primary
no responsibility for the results of this script.
+
    recording partition /var/lib/mythtv.
In particular, this system has been developed with version
 
72 of mythtv protocol and it will need validating carefully
 
against other implementations.
 
  
To use with other versions you will need to add a new version
+
    Assumptions may well have been inadvertently made which are
and its key in \%MythProtocolKey and test thoroughly
+
    not valid in all circumstances and the author can accept
with --loglevel 4 and then with --preview.
+
    no responsibility for the results of this script.
 +
   
 +
    The first versions of this script used protocol calls.
 +
    Versions starting with 1.10 use the services API introduced
 +
    with Mythtv 0.25 which should be more robust with Mythtv
 +
    version changes.
  
 +
    1.10 has been tested against Mythtv 0.25  and 0.27.
 
   
 
   
 
AUTHOR:
 
AUTHOR:
  
Written by Phil Brady, November 2012.
+
    Written by Phil Brady, November 2012, latest changes June 2014.
  
  
 
COPYRIGHT
 
COPYRIGHT
  
Copyright  ©  2012 Phil Brady.
+
    Copyright  ©  2012 ~ 2014 Phil Brady.
License  GPLv3+: GNU GPL version 3 or later
+
    License  GPLv3+: GNU GPL version 3 or later
See http://gnu.org/licenses/gpl.html.
+
    See http://gnu.org/licenses/gpl.html.
  
This is free software: you are free to change and redistribute  
+
    This is free software: you are free to change and redistribute  
it.  There is NO WARRANTY, to the extent permitted by law.
+
    it.  There is NO WARRANTY, to the extent permitted by law.
  
 
";
 
";

Latest revision as of 15:04, 20 July 2014

Important.png Note: The correct title of this article is autobackup.pl. It appears incorrectly here due to technical restrictions.


Author Phil Brady
Description Maintains a backup subset of recordings, taking into account age and whether they are part of a series.
Supports Version25.png  Version27.png 



Description

autobackup.pl selects a subset of mythtv recordings and maintains a backup of them on a destination partition which is typically smaller than the total size of all recordings and which is on a separate disk. It only copies or removes the files necessary to maintain the subset so minimising disk traffic and if run daily will take just a minute or two to complete.

If the destination partition has sufficient space, the script will effectively perform synchronisation of all recordings.

It allows recovery of service with a 'useful' set of recordings in case of primary disk failure and should be augmented by appropriate database and system backups.


Options

--preview or -p
to inhibit file copy/deletes. This shows what action would be taken with a full run but without changing the filestore.
--loglevel n to set logging levels.
n=0 for only errors
n=1 for calling info,
n=2 for stats,
n=3 for a recordings list
n=4 for tracing
Default is 3.
--help or -h Help text
--delete <filename>
deletes file from destination directory. For use with the recording delete system event, though use is optional.

Details

autobackup.pl is intended to be run as a regular job when the system is otherwise idle. It checks:

  • the destination partition to determine how full it is,
  • the database to extract information about recordings
  • the source directories as a consistency check
  • The destination directory to identify recordings which have alreay been backed up and others which are obsolete and need removing.

It is then in a state to identify series recordings, allocate priorities and determine the action for each file. Files are selected for backup in date/time order (newest first) but:

  • recordings marked as non-expirable have a 20 year priority boost
  • recordings which are recognised as episodes of a series can be bunched together with the latest episode and/or have a number of days added as a boost. The author finds bunching with zero days boost to be satisfactory.

Three or more recordings with the same name and an average interval between them of less than 8 days are deemed to be a series.

Files will be promoted from the head of the list, and files demoted as necessary from the end in order to prevent over-filling the partition.

The script should handle recording groups with more than one source directory, though there has only been limited testing of the code for this.

The script was designed to be run automatically at system closedown and to maintain a permanently fitted destination partition, though it should backup satisfactorily to a mounted external eSATA or USB drive if started manually.


Partition Limits

New recordings will be promoted and old ones demoted whilst observing the partition limit of 90%. Older recordings will only be considered if the partition is below a partition threshold of 85% full and this is only likely happen after a spate of manual deletions. The two limits prevent a recording from repeatedly falling in and out of favour as higher priority recordings are created and removed. Threshold should be a few days worth of recordings below limit for stability. The order in which files are copied over by the scipt is chosen for robustness of the algorithm even after an unexpected closedown during copying.


Deleted recordings

Recordings can optionally be deleted from the destination directory with the RecordingDelete system event mechanism by calling this script with --delete %FILE%. This will remove the file more quickly.

However, such files will be routinely deleted by this script even if the event system is not used. Recordings deleted routinely by the script have the advantage of appearing in the log file, complete with recording name, so giving a fuller picture of the change in the state of files.

Recordings which have been deleted from the database but still remain on the destination partition no longer have their name in the database. In order to give these a name in listings, the script writes a list of today's secure recordings to the file to enable the names to be established next day. The file is /var/lib/mythtv2/recordings/autobackup.list.


Working parameters

See the parameters (%params) at the head of the program defining parameters such as the destination partition, maximum percentage used, source directories, destination directories, default logging level, series action, threshold setting and name of recordings list file.

On the author's system, the script is placed in /usr/bin/autobackup.pl. Logging is directed by autobackup.sh to /var/log/mythtv/autobackup.log The source directory is /var/lib/mythtv/recordings and the destination /var/lib/mythtv2/recordings The destination partition is /var/lib/mythtv2; the limit is 90% and the threshold 82%.


Logging

If log files are required then they need to be redirected from STDOUT and STDERR and rotated as necessary by a calling script.


Database backups

The perl script has a sister routine autobackup.sh which takes a database backup at last closedown of the day. The sister routine will trigger a recordings backup if it finds this perl script present and manage logging.


Performance

The time taken by this script is dominated by file copying time. With SATA2 interfaces 6Gb/min is a rule of thumb. If the script is run daily, it will typically run in a minute or two.


Glossary

The listing includes these terms:

del deleted from database by the user but still present on the backup partition.

dem or demoted selected for removal from the backup partition in order to make space for promotion(s).

insecure or ins a file which has not been backed up and does not merit it.

promoted or pro selected for copying to the backup partition.

secure or sec a file which is already securely backed up and still merits that.

ser a recording deemed to be one of a series and given a priority boost.

vip a recording marked in the database as non expirable.

Development Regime and Protocol Versions

The script was originally developed and tested in the UK on a 64 bit Mythbuntu 12.04 (Mythtv 0.25) dedicated frontend/backend system with ACPI__Wakeup, DVB-T receivers, a single directory for recordings, and a local time close to or equal to GMT. It was modified to utilise the Services API interface in June 2014 and been validated against 0.27. This interface should be Mythtv version independent for versions 0.25 and later and be more robust.

A 500Gb disk holds root, swap and backup recording partitions; a 2Tb drive holds emergency root and swap and the primary recording partition /var/lib/mythtv.

The intention is to reboot from the 2Tb and run a database restore should the 500Gb fail, and to edit fstab and run Find_orphans.py should the 2Tb fail..


Example logging

This shows the statistics produced, and a (severely cut) list of recordings. The error has been deliberately introduced. See glossary for meaning of status.

2012-11-27 20:55:06 /usr/bin/autobackup.pl --preview
Version 1.0  26 Nov 2012. running with loglevel=3, preview=yes

Partition /var/lib/mythtv2 info
   100%   size                      452.50G
    88%   used                      401.03G
    90%   limit                     407.25G
    85%   threshold                 384.63G

Database scan found 240 recordings

Source directory /var/lib/mythtv/recordings
      3   vip recordings              4.67G
    100   series recordings         161.08G
    138   expirable recordings      246.29G
      0   obsolete recordings         0.00G
    241   --- total ---             412.04G

There were 1 database inconsistencies.
Suggest using find_orphans.py

Destination directory /var/lib/mythtv2/recordings
      3   vip recordings              4.67G
    100   series recordings         161.08G
    131   expirable recordings      233.50G
      0   obsolete recordings         0.00G
    234   --- total ---             399.26G

Changes proposed:
      0   delete obsolete             0.00G
      0   demote                      0.00G
      1   promote                     0.91G

Proposed final state of partition:
    88%   used                      401.94G

All files (priority order) Status    Size   Name (if known)
-----------------------------------------------------------
1704_20121113205700.mpg    Sec Vip   0.4Gb  All in the Mind
1003_20121127192600.mpg    Pro       0.9Gb  The Martin Lewis Money Show
1001_20121126223600.mpg    Sec       1.5Gb  Have I Got a Bit More News for You
1001_20121111205600.mpg    Sec Ser   2.3Gb  Andrew Marr's History of the World
1001_20121104205600.mpg    Sec Ser   2.1Gb  Andrew Marr's History of the World
1001_20121028205600.mpg    Sec Ser   2.2Gb  Andrew Marr's History of the World
1002_20121027212600.mpg    Sec       1.3Gb  The Thick of It
1704_20121026182600.mpg    Sec       0.4Gb  The News Quiz
[...]
1003_20121025223300.mpg    Sec       0.8Gb  Corfu
1003_20111028195600.mpg    Sec       0.8Gb  Wild Britain With Ray Mears
1009_20111024225600.mpg    Ins       2.4Gb  The Secret Life of Waves
1002_20110620205800.mpg    Ins       2.3Gb  Made in Britain
1009_20110322225600.mpg    Ins       2.2Gb  The Secret World of Whitehall

Finished: time taken    0 seconds

                                                                  

The Script

In order to use this script, you will need to:

  • create a destination partition and directory (I understand top level directories are not usable)
  • set permissions to allow access by root and mythtv.
  • Ensure you can access it too if you wish to test it.
  • Copy this code to /usr/bin/autobackup.pl (or elsewhere) and chmod +x it.
  • Test with --preview first!
  • If it fails to find the perl library 'LWP::UserAgent' then install it with 'sudo apt-get install libwww-perl'
  • trigger it daily - see autobackup.sh as a possible mechanism.


Application-x-perl.png autobackup.pl


#!/usr/bin/perl -w
use strict;
use IO::Socket;
use Time::Local;

#  if next line fails you need to do 'sudo apt-get install libwww-perl'
use LWP::UserAgent;


# Select recordings for backup to a small partition and copy/delete as necessary.
# See --help for details
#
# Written by Phil Brady  November 2012.
#

#program version
use constant VERSION  => '1.10  2 June 2014';

#change history
#version 1.0 26 Nov 2012 - first release
#version 1.01 17 Dec 2012
  # recording list put in destination directory - more logical
  # Protocol version keys included as comments
  # Bug fix in sub freespace.  Small recordings no longer overtake bigger ones if below threshold. 
  # threshold reduced to 82% to improve stability 
  # cosmetic changes to reporting
#version 1.02 23 March 2013
  # skip livetv recordings
#version 1.03, 18 Oct 2013
  #format of recording file names has changed following retune in UK.
  #   some are eg 17717_20131010 etc with 5 digits before the _ not 4. 
  # Minor changes to routines timestamp, scansources and showall to accommodate this.
#version 1.04 3 Jan 2014
  # Corrected wrong exit codes.  Minor edits
#version 1.10 2 June 2014.
  # switched to getting details from database via Service API. 
  #  This is simpler, more resilient and should need fewer changes with mythtv version changes.
  # also removed tab stops from source - a cosmetic change only.


   
#control parameters
my %params =(

#  destination
  "partition"   => "/var/lib/mythtv2",
  "destdir"     => "/var/lib/mythtv2/recordings",

#  source(s)
  "sourcedir1"  => "/var/lib/mythtv/recordings",  #first or only dir in recording group
# "sourcedir2"  => "/var/lib/mythtv/other", #add others like this. Note sourcedir1 to n, not 0 to n

#  How full?
  "percent"     => 90,      # max used% of partition
  "threshold"   => 82,      # percentage for older files to be promoted.

# Series policies: use none, one or both:
  "SeriesPolicy"=>  1,      # if set, use priority of newest member of a series.
  "SeriesBoost" =>  0,      # boost in days for series.

#Previewing default
  "preview"     =>  0,      # command line parameter -p or --preview.

# Keep a list of recordings here so we can identify deleted ones next day
  "filelist"    =>  "/var/lib/mythtv2/recordings/autobackup.list", 
);


# default loglevel parameter
my $loglevel=3;     # 0=errors only, 1=calling params, 2=stats, 
                    # 3=recordings details, 4=diagnostic trace.


# backend address
use constant backendip      => '127.0.0.1';


# Bits in recording flags
use constant    flagautoexp       => 4;
use constant    flagdeletepending => 128;   #this bit utilised but never seen set!
use constant    flagpriority      => 1024;  #not used.

my $gig= 1024*1024*1024;

#info on recorded programs. Hashes all use filename as key.
my %frecname;   #program name 
my %fsize;      #file size
my %fstatus;    #recording status: 0=obsolete/deletepending,1=normal,2=series record, 3=vip recording
                 # +4 =secure before, +8=secure after.
my %fpriority;  #hold merit for saving the recording
my %newest;  #time of most recent episode in series
my @deleteq;    #list of files for deletion
my @promoteq;   #ditto promotion


my %stats;    #statistics
my $url_buffer;      #buffer for reading url

#---------ok, let's start!---------

&parse_command_line;

&check_partition;

&scan_database; #to get list of recordings

&mark_series;   #identify series recordings and mark episodes


&scansources;    #scan source directories and check consistency with database

&scandir($params{'destdir'},1);    # Scan destination directory.  
                                   # Identify which recordings are backed up 
                                   # and find obsolete ones. 


&maintain_list('read');     #obsolete recordings are anonymous;  
                            #Get names by checking yesterday's list

&decide_priorities;         #based on expirability, obsolescence, series episode and recording time.

#now sort them.  
my @sorted = sort {$fpriority{$b} <=> $fpriority{$a} or         #priority
              $frecname{$b} cmp $frecname{$a} or                #keeps series episodes together
              substr($b,5,14) <=> substr($a,5,14)} keys %fstatus;   #in date order
my @sorted2=@sorted;

&decide_action;             #decide promotions/demotions
                            #note:  operation depends heavily on allocated priorities & sort order.

&do_action;                 #now do the copies/deletes

&showall if $loglevel==3;   #list all known recordings

&maintain_list('write');    #make new list so we can identify deleted recordings tomorrow

#almost done!
printf "\nFinished: time taken %4d seconds\n",time-$^T  if $loglevel;
exit 0;

#-------------------
sub maintain_list{

#   Called with read or write.

#   This routine writes a list of saved recording file names and program names to a file.
#   It checks the list the following day to put a name to deleted recordings which 
#   would otherwise be anonymous (named '??').

#   If you are using the recordingdelete event to remove backup recordings then you will probably
#   not see these anonymous recordings.  

#   To remove this functionality, uncomment the next line
#   return;

#   Documentation/reminder:  Myth delete mechanism seems to be:
#   1. changes group to 'Deleted'
#   2. Some while later, backend deletes file and removes entry from database. 
#   3. Deletepending flag not seen in use.

    if ($_[0] eq 'read'){

        #check yesterday's list to name deleted recordings
        my $nameyesterday;
        if ($loglevel != 3){return};            #nobody listening!
        if ($stats{'noname'}==0){return};       #no anonymous ones to check
        unless (open FH, "<$params{'filelist'}"){ 
            print "Warning:  cannot read $params{'filelist'} $!\n";
            return;   #not fatal
        }
        while (<FH>){
            chomp;      
            ($_,$nameyesterday)=split /\[\]:\[\]/,$_;
            if (exists $frecname{$_}){
                if ($frecname{$_} eq '??'){$frecname{$_} = ".$nameyesterday"};
            }
        }
        close FH;

    }elsif ($_[0] eq 'write'){
        if ($params{'preview'}) {return};

        #write list of backed up recordings for use tomorrow
        unless (open FH, ">$params{'filelist'}"){
            print "Cannot write $params{'filelist'} $!" if $loglevel >1;
            return;
        } 
        for (keys %fstatus){
            if (($fstatus{$_} & 11) > 8){
                #this is a backed up file so remember for tomorrow
                print FH $_ . '[]:[]' . $frecname{$_}."\n";
            }
        } 
        close FH;
    }else{
        print "bad maintain_list $_[0]\n";
        exit 1;
    }
}
#-------------------
sub decide_priorities{
    my $st;
    foreach (keys %fstatus){
        $st=($fstatus{$_} & 3);
        if ($st==0){
            $fpriority{$_}=0;
        }elsif ($st==3){
            $fpriority{$_} +=7305;      #20 years boost for vip=non expirable
        }elsif ($st==2){                #boost episodes of series
            if ($params{'SeriesPolicy'}){
                $fpriority{$_}=$newest{$frecname{$_}};
            }
            $fpriority{$_} += $params{'SeriesBoost'};
        }
    }
}
#-------------------
sub timestamp{
    #extract timestamp from filename - return epoch DAYS as a real.

    (my $in)=@_;    #filename
    if ($in =~ /_(\d{14})\./) {
        $in=$1;
    }else{
        $in='20131010190000';
    }
    my @bits=($in =~ m/../g);
    my $out = timelocal($bits[6],$bits[5],$bits[4],$bits[3],$bits[2]-1,$bits[0].$bits[1]);
    $out/86400;
}
#-------------------
sub mark_series{

    #identify series of recordings; mark the episodes

    my %namecount;
    my $tmp;
    my %oldest; # %newest is global as needed later in decide_priorities.


    #count instances of names, keep newest/oldest
    foreach (keys %frecname){
        $tmp=&timestamp($_);
        $fpriority{$_}=$tmp;        
        $_=$frecname{$_};

        #keep count, oldest, newest for each name found
        $namecount{$_}++;
        if (exists $newest{$_}) {
            if ($tmp < $oldest{$_}) {$oldest{$_}=$tmp};
            if ($tmp > $newest{$_}) {$newest{$_}=$tmp};
        }else{
            $oldest{$_}=$tmp;
            $newest{$_}=$tmp;
        }
    }

    #mark as series if more than 2 episodes and average interval no more than 8 days.
    my $count;
    my $avinterval;
    foreach (keys %namecount){
        $count=$namecount{$_};
        $avinterval=0;
        if ($count>1){$avinterval=($newest{$_}-$oldest{$_})/($count-1)};
        if (($count>2) && ($avinterval<8)){
                #this is a series
                $namecount{$_}=-1;
        }
    }

    #mark the recordings as series episodes
    foreach (keys %frecname){
        if ($namecount{$frecname{$_}} <0) {
            if ($fstatus{$_}==1){$fstatus{$_}=2}; 
        }
    }
}
#--------------------
sub do_action{

    if ($params{'preview'}){return 0};
    my $command;
    my $qsize;
    my $fname;

    #do deletes first
    $qsize=@deleteq;
    print "\nNow deleting $qsize files\n" if $loglevel>2;
    if ($qsize){   
        foreach $_ (@deleteq){
            $command= "rm -f $params{'destdir'}/$_";
            system $command;
        }
    }

    #now do promotions
    $qsize=@promoteq;
    print "Now promoting $qsize files\n" if $loglevel>2;
    if ($qsize==0){return};

    #count the source directories
    my $dirct=0;
    while (exists $params{'sourcedir'.++$dirct}){
    }
    $dirct--;

    #do copies
    for $fname (@promoteq){
        for (1..$dirct){        
            if ( -e "$params{'sourcedir'.$_}/$fname"){
                $command="cp -p $params{'sourcedir'.$_}/$fname $params{'destdir'}";
                #print "$command\n";                
                system $command;
                last;
            }
        }
    }
    
}
#-------------------
sub decide_action{

    $stats{'promote'}=0;
    $stats{'promote size'}=0;
    $stats{'demote'}=0;
    $stats{'demote size'}=0;

    #queue obsolete files for deletion
    my $finished=0;
    while ($finished==0){
        if (@sorted==0){     #array empty
            $finished=1;
        }else{
            $_=pop @sorted;
            if ($fstatus{$_} & 3){         #not obsolete - we overshot!
                push @sorted,$_;          #put it back
                $finished=1;
            }elsif ($fstatus{$_} ==12){   #obsolete file for deletion
                push @deleteq,$_;
                $fstatus{$_} -=8;         #demoted
            }
        }
    }

    $stats{'used size'} -=$stats{'obsolete recordings size'};
    #report
    $stats{'delete obsolete'}=$stats{'obsolete recordings'};    #cleaner name for user
    $stats{'delete obsolete size'}=$stats{'obsolete recordings size'};

    #now decide promotions/deletion
    my $use_threshold=0;   #set once we find a secure normal recording
    while ($_ = shift @sorted){   #newest
        if ($fstatus{$_}==0){       #skip deletions
        }elsif ($fstatus{$_} <4){    #an unsaved recording - try to save it
            if (&freespace($fsize{$_},$use_threshold)) {
                #space made - mark for promotion
                #put in order: oldest fresh, other fresh, newest stale, oldest stale
                if ($use_threshold){                
                    push @promoteq,$_;
                }else{
                    unshift @promoteq,$_;
                }
                $fstatus{$_} +=8;
                $stats{'promote'}++;
                $stats{'promote size'} +=$fsize{$_};
                $stats{'used size'} +=$fsize{$_};
            }
        }elsif ($fstatus{$_} == 13){
            #non vip already backed up.
            $use_threshold=1;
        }
    }

    &show_stats('Changes proposed:','delete obsolete','demote','promote');
    &show_stats('Proposed final state of partition:','used');
}    

#--------------------
sub freespace{
    #free up space by removing oldest secured files
    #return 1 if successful, 0 if not
    my ($needed,$use_threshold)=@_;

    #decide appropriate limit
    my $limit =  $stats{'limit size'};
    if ($use_threshold) {$limit =  $stats{'threshold size'}};

    my $loopstate = -1;
    while ($loopstate == -1){
        if ($needed < ($limit - $stats{'used size'})) {
            $loopstate=1;   #will fit now
        }else{
            if (@sorted){
                my $d = pop(@sorted);
                if ($fstatus{$d} >12) {
                    #demote this one
                    push @deleteq,$d;
                    $fstatus{$d} -=8;
                    $stats{'demote'}++;
                    $stats{'demote size'} += $fsize{$d};
                    $stats{'used size'} -= $fsize{$d};
                }   
            }else{
                #no files left to remove
                $loopstate=0;
            }
        }
    }
    $loopstate;
}
#---------------------------
sub scan_database {

    #open the browser
    my $browser = LWP::UserAgent->new;
    $browser->timeout(10);


    my $item=-2;    #values to kick it off
    my $total=-1;
    my $base;
    my $response;
    print "Getting list of recordings\n" if $loglevel>=4;

    while ($item < $total) {
        $item=1 if $item<0;

        #get a batch of 100 recordings
        my $response = $browser->get('http://' . backendip . 
              ":6544/Dvr/GetRecordedList?StartIndex=$item&Count=100&Descending=true");
        unless($response->is_success){ die 'bad response to get'};
        $url_buffer = $response -> content;

        if ($total <0) {$total=&getparameter('TotalAvailable',0)};

        #now process them
        if ($total >0) {
            # split this batch of recordings on <Program>.
            $base=index($url_buffer, '<Program>');
            while ($base>-1){
                &ProcessRecording($base);
                $base=index($url_buffer, '<Program>', $base+10);
            }
            $item=$item+100;
        }
    }
    print "\nDatabase scan found $stats{'DBrecordings'} recordings\n" if $loglevel>1;
    if ($loglevel == 4) { exit 0};
}
#------------------------------
sub ProcessRecording{

    my $startat = $_[0];
    my $fname= &getparameter('FileName',$startat);
    $frecname{$fname} = &getparameter("Title", $startat);
    $fsize{$fname}=&getparameter('FileSize',$startat);

    my $flags=&getparameter('ProgramFlags',$startat);
    my $recgroup=&getparameter('RecGroup',,$startat);

    if ($recgroup eq 'Deleted'){       #only just deleted
        $fstatus{$fname}=0;
    }elsif ($recgroup eq 'LiveTV'){    #ignore livetv recordings
        $fstatus{$fname}=0;
    }elsif (($flags & flagdeletepending)>0){
        $fstatus{$fname}=0;            #Delete pending flag (not used?)
    }elsif (($flags & flagautoexp)==0){
        $fstatus{$fname}=3;             #non expirable so vip
    }else{
        $fstatus{$fname}=1;             #expirable
    }
    $stats{'DBrecordings'}++;
    if ($loglevel==4){print "$fname $flags $frecname{$fname}\n"};
}

#-------------------------
sub getparameter{

my ($param, $base) = @_;

        my $start= index($url_buffer, "<$param>",$base);
        if ($start<0){die "cannot find $_[0]"};

        $start=$start + length($param) +2;
        
        my $end=index($url_buffer, "</$param>", $start);
        substr $url_buffer, $start, $end-$start; 

}


#-----------------

sub check_partition{
    my $reply = `df $params{'partition'}`;
    if (length $reply == 0){die 'cannot find partition'};
    my @fields = split /\s+/,$reply;
    $stats{'partition size'}=$fields[8]*1024;
    $stats{'used size'}=$fields[9]*1024;
    $stats{'used'} = sprintf "%2d%%", $stats{'used size'} *100 / $stats{'partition size'};
    $stats{'limit size'} = ($stats{'partition size'} * $params{'percent'} /100);
    $stats{'limit'}= $params{percent} . '%';
    $stats{'size'} = '100%';
    $stats{'size size'}=$stats{'partition size'};  #same but more readable in output
    $stats{'threshold'}=$params{'threshold'} .'%';
    $stats{'threshold size'}= $params{'threshold'} * $stats{'partition size'}/100;
    &show_stats ("Partition $params{'partition'} info", 'size', 'used', 'limit','threshold');
}

#----------------------------

sub show_stats{
# print item, number & size
    if ($loglevel<2) {return 0};
    my @items = @_;
    $_=shift @items;
    if ($_) {print "\n$_\n"};    #optional heading
    my $item_value;
    while (@items){
        $_=shift @items;
        if (exists $stats{$_}) {
            if ($_ eq 'used'){
                #update used percentage
                $stats{'used'} = sprintf "%2d%%", $stats{'used size'} *100 / $stats{'partition size'};
            }
            $item_value=$stats{$_};
            #show item and number
            if ($item_value =~/%/){
                printf "%7s",$item_value;
            }else{
                printf "%7d",$item_value;
            }
        }else{
            print "       ";
        }
        printf "   %-25s",$_;

        #show size if present
        $_ .= ' size';
        if (exists $stats{"$_"}) {
            $item_value= $stats{$_};
            printf "%7.2fG",$item_value/$gig;
        }
        print "\n";
    }
}
#------------
sub showall{
    #show all recordings
#   print "\nfile status is Deleted, Secure, Insecure, Promote or Demote followed by vip (non expirable), series or deleted.\n"; 
    print "\nAll files (priority order) Status    Size   Name (if known)\n";
    print "-----------------------------------------------------------\n";   
    foreach (@sorted2){
        printf "%25s %9s   %3.1fGb  $frecname{$_}\n",$_, &statusname($fstatus{$_}),$fsize{$_}/$gig;
    }
}
#------------
sub scansources{
    #scan source directories, report only.
    $stats{'missing'}=0;
    $stats{'found'}=0;

    my $i=0;
    while (exists $params{'sourcedir'.++$i}){
        &scandir($params{'sourcedir'.$i},0);
    }

    #counts consistent with d/base?
    $i=$stats{'DBrecordings'}-$stats{'found'};
    if ($i<0){$i=-$i};
    $i += $stats{'missing'};    
    if (($i>0) && ($loglevel>1)) {
        print "\nThere were $i database inconsistencies.\n";
        print "Suggest using find_orphans.py\n";
    }
}
#-------------
sub scandir{
    my ($dir, $destination)=@_;
    my ($size, $category);
    #scan a folder, show stats for it
 
    $stats{'vip recordings'}=0;
    $stats{'vip recordings size'}=0;
    $stats{'expirable recordings'}=0;
    $stats{'expirable recordings size'}=0;
    $stats{'obsolete recordings'}=0;
    $stats{'obsolete recordings size'}=0;

    $stats{'series recordings'}=0;
    $stats{'series recordings size'}=0;
    $stats{'noname'}=0;

    opendir DH,$dir or die "Cannot open $dir: $!";
    foreach $_ (readdir DH) {
        if (/^\d+_\d{14}\.mpg$/) {
            if (exists $fstatus{$_}) {
                $stats{'found'}++;
                if ($destination){
                    $fstatus{$_} +=12;
                }              
            }else{
                $stats{'missing'}++;
                if ($destination){
                    $fsize{$_}= -s "$dir\/$_";
                    $fstatus{$_} = 12;
                    $frecname{$_}='??';
                    $stats{'noname'}++;
                }
            }
            
            #collect stats
            if (exists $fstatus{$_}){      
                $size=$fsize{$_};
                $_=($fstatus{$_} & 3);
                $category=qw/obsolete expirable series vip/[$_];
                $stats{$category.' recordings'}++;
                $stats{$category.' recordings size'} += $size;
            }
        }
    }
    closedir DH;
  
    #summary
    $stats{'--- total ---'}=$stats{'series recordings'}+
        $stats{'expirable recordings'} + $stats{'vip recordings'} + $stats{'obsolete recordings'};
    $stats{'--- total --- size'}=$stats{'series recordings size'}
        +$stats{'expirable recordings size'} + $stats{'vip recordings size'} + $stats{'obsolete recordings size'};
    $_=$destination?'Destination':'Source';
    &show_stats("$_ directory $dir",'vip recordings', 'series recordings', 'expirable recordings','obsolete recordings','--- total ---');
}
   
#------------
sub statusname{
    #return text showing status 
    (my $st)=@_;
    my @ftype=('Del','   ','Ser','Vip');
    my $st2=($st & 12)/4;
    qw/Ins Dem Pro Sec/[$st2] .' '. $ftype[$st & 3];
}

#------------------
sub parse_command_line{
    if (scalar @ARGV){
        my @argcopy=@ARGV;
        while (@argcopy){
            $_ = shift @argcopy;
            if (/^--preview$|^-p$/) {
                $params{'preview'}=1;

            }elsif (/^--loglevel$/) {
                my $bad=1;
                if (scalar @argcopy){
                    $loglevel=shift @argcopy;
                    if ($loglevel =~/\d/){
                        if (($loglevel>=0) && ($loglevel<5)) {$bad=0};
                    }
                }
                if ($bad){
                    print "bad loglevel needs 0 to 4\n";
                    exit 1;
                }
            }elsif (/^--help$|^-h$/){
                &givehelp;
                exit 0;
            }elsif (/^--delete$/){
                #recording delete event
                $_ = "$params{'destdir'}/" . shift @argcopy;
                if ( -e $_){system "rm -f $_"};
                exit 0;
            }else{
                print "parameter $_ not recognised.  Try $0 --help\n";
                exit 1;
            }
        }
    }
    if ($loglevel){
        $_= `date +%F" "%T`;
        chomp;
        print "\n$_ $0 @ARGV\n";
        print   "\nautobackup.pl version ". VERSION . ". running with loglevel=$loglevel, preview=". qw/no yes/[$params{'preview'}] ."\n";
    }

}
#------------
sub givehelp{

print "

NAME:      autobackup.pl

SYNOPSIS:  autobackup.pl [options]

DESCRIPTION:

    autobackup.pl selects a subset of mythtv recordings 
    and makes a backup of them to a destination partition which 
    is typically smaller than the total size of all recordings
    and which is on a separate disk.  It allows recovery of 
    service with a 'useful' set of recordings in case of 
    primary disk failure. 

    It should be augmented by appropriate database and system
    backups.

OPTIONS:

    --preview or -p
        to inhibit file copy/deletes.  This shows what 
        action would be taken with a full run.

    --loglevel n    to set logging levels.  
                1 for calling info, 
                2 for stats, 
                3 for a recordings list
                4 to check extraction of recordings (depricated(
                Default is 3.

    --help or -h        this text

    --delete <filename>
        deletes file from destination directory.  For use with
        recording delete system event, though use optional.

DETAILS:

    autobackup.pl is intended to be run as a regular job at system
    closedown when the system is otherwise idle.  It checks 
    the destination partition to determine how full it is, 
    then scans the database and the source and destination 
    directories to check the state of all recordings.
    It is then in a state to identify series recordings, 
    allocate priorities and determine the action for each file.

    Files are selected for backup in date/time order (newest 
    first) but: 

    -   recordings marked as non-expirable have a 20 year 
        priority boost

    -   recordings which are recognised as episodes of a series 
        can be bunched together with the latest episode
        and/or have a number of days added as a boost.
        The author uses bunching with zero days boost.

    Three or more recordings with the same name and an average 
    interval between them of less than 8 days are deemed to 
    be a series.

    Files will be promoted from the head of the list, and files 
    demoted as necessary from the end in order to prevent 
    over-filling the partition.

PARTITION LIMITS

    New recordings will be promoted whilst observing the partition
    limit of $params{'percent'}%.  Older recordings will only be considered if the 
    partition is below a partition threshold of $params{'threshold'}% full.  
    This is only likely happen after a spate of manual deletions.  

    The two limits prevent a recording from falling in and out
    of favour as higher priority recordings are created and removed.
    Threshold should be a few days worth of recordings below limit.

DELETED RECORDINGS

    Recordings can optionally be deleted from the destination directory 
    with the RecordingDelete system event mechanism by calling this program 
    with --delete %FILE%.  This will remove the file more quickly.

    However, such files will be routinely deleted by this program if the event
    system is not used.
 
    Recordings which have been deleted from the database but still 
    remain on the destination partition no longer have their name in
    the database.  In order to give these a name in listings, the script 
    writes a list of today's recordings to the file to enable the names 
    to be established next day.  The file is 
    $params{'filelist'} 
     
WORKING PARAMETERS

    See the parameters (\%params) at the head of the program defining 
    parameters such as the destination partition, percentage used,
    source and destination directories, default logging level, series 
    action, threshold setting and name of recordings list.

LOGGING

    If log files are required then they need to be redirected from STDOUT
    and STDERR and rotated as necessary by a calling script eg autobackup.sh

 
DATABASE BACKUPS

    The perl script has a sister routine autobackup.sh which
    takes a database backup and triggers this script at last
    closedown of the day.  See autobackup.sh.

 
DEVELOPMENT REGIME

    The script has been developed and tested in the UK on a 
    Mythbuntu 12.04 dedicated frontend/backend system with 
    ACPI wakeup, DVB-T receivers, a single directory for 
    recordings, and a local time close to or equal to GMT.  

    A 500Gb disk holds root, swap and backup recording partitions.
    A 2Tb drive holds emergency root and swap and the primary 
    recording partition /var/lib/mythtv.

    Assumptions may well have been inadvertently made which are 
    not valid in all circumstances and the author can accept 
    no responsibility for the results of this script.
    
    The first versions of this script used protocol calls. 
    Versions starting with 1.10 use the services API introduced 
    with Mythtv 0.25 which should be more robust with Mythtv
    version changes.

    1.10 has been tested against Mythtv 0.25  and 0.27.
 
AUTHOR:

    Written by Phil Brady, November 2012, latest changes June 2014.


COPYRIGHT

    Copyright   ©   2012 ~ 2014 Phil Brady.
    License  GPLv3+: GNU GPL version 3 or later
    See http://gnu.org/licenses/gpl.html.

    This is free software: you are free to change and redistribute 
    it.  There is NO WARRANTY, to the extent permitted by law.

";

exit 0;
}