Difference between revisions of "Perl API Examples version 2"

Revision as of 17:27, 17 July 2015

Author	Phil Brady
Description	Interfacing with the Services API interface of Mythtv from a perl script.
Supports

WARNING

This page is in beta test stage. It will replace Perl API examples

Introduction

This page has been created to give examples of perl code for accessing the new API interface to Mythtv. It includes a module to assist with this.

Note that the API interface is still very new (still as at July 2015), is still under development and has certain inconsistencies which still are being ironed out. The documentation on it is necessarily sparse.

This page is itself very sparse but I hope it will be considered better than nothing and be a springboard for further contributions (and challenges) which are warmly welcomed! The original author does not claim to be a perl expert (or any other sort of expert!) and the code may well be capable of improvement.

Previous Version

This is version 2 of the Perl API page. The first version included a module with routines which both read information from the backend and parsed it in a single call. This new version separates those two functions as it is more flexible approach, it is more powerful and it makes module rewrites less likely as the API interface is developed. The original routines have been re-written to make them simple 'shims' which call the new routines but maintain the same user interface. Any code written to use them should still work with the new module.

The original pages can be seen here: Perl API examples

Existing sources of information

API wiki pages: Services API

Your own Mythtv server: eg http://192.168.1.67:6544/Channel/wsdl

The 'old' protocol version. Use of this mechanism is discouraged but its pages are a still a good source for (say) program data. Myth Protocol

The mythtv forum has a special API section – fast and helpful responses! https://forum.mythtv.org/

Source code (if desperate!):

https://code.mythtv.org/cgit/mythtv/tree/mythtv/programs/mythbackend/services/dvr.cpp#n146

Getting information from the backend

The backend can be read using a standard web interface and there are a number of ways of doing this..

Web browser

In its simplest form, a browser will show the data eg

http://192.168.1.67:6544/Dvr/GetRecordedList?StartIndex=2&Count=1&Descending=true

will show a single entry from the recorded list. You would of course need to supply your own backend IP address!

This is a useful development aid but note that the browser helpfully inserts newlines or tabs for readability but these will not be present if the same page is read with perl code.

It may also take liberties such as showing <XMLTVID></XMLTVID> as <XMLTV/>.

Module LWP::Simple

This CPAN module will get data from the backend.

#!/usr/bin/perl -w
use strict;
use LWP::Simple;
my $url='http://192.168.1.67:6544/Dvr/GetRecordedList';
my $content=get $url;

You may need to install libwww-perl for these CPAN routines to work. With Ubuntu do this:

 sudo apt-get install libwww-perl

Module LWP::UserAgent

If getting the data fails, you may want to know why. Do this instead:

#!/usr/bin/perl -w
use strict;
use LWP::UserAgent;
my $url='http://192.168.1.67:6544/Dvr/GetRecordedList';
my $browser = LWP::UserAgent->new;
$browser->timeout(10);
my $response = $browser->get($url);
unless($response->is_success){ 
        die $response->status_line;
}
my $content= $response -> content;

The scan_database Module

If you are using other routines in the scan_database module shown below then you may find it convenient to use the ReadBackend routine. This essentially runs the UserAgent code above.

#!/usr/bin/perl -w
use strict;
use scan_database;
my $url='192.168.1.67:6544/Dvr/GetRecordedList';
# if you omit the http:// - it will insert it for you.
my $content;
ReadBackend($url,$content);

Parsing the data

Having read the data from the backend using one of the methods above you will have a mass of XML code in a scalar variable. Before you can sensibly use this, you will need to extract from it the information you are interested in - this is called parsing.

There are many ways of parsing this - some memory intensive, some not.

Regex

A regex is an efficient way of extracting a limited amount of data. To extract protocol version and titles of all recordings for example:

#!/usr/bin/perl -w
use strict;
use scan_database;
my $url='192.168.1.67:6544/Dvr/GetRecordedList';
my $content;
ReadBackend($url,$content);
$content =~ m!<ProtoVer>(\w+)</ProtoVer>!;
print "Proto version is $1\n";
while ($buffer =~ m!<Title>([^<]*)<!g){
    print "$1\n";
}

Note that this will not convert the XML escape sequences such as & to &.

Parsers on the CPAN database

There may be others but you may like the look of SAX or XML::Parser

The parsers in scan_database module

There are two parsing routines in the scan_database module. Both take as input the content extracted by one of the backend reading mechanisms above and both write the information you say you are interested in to a hash or a hash of hashes. Both have been written to return the minimum requred information and to minimise memory usage (but if anyone can suggest improvement please do so!).

FillHashofHash

The first of these populates a hash of hashes. This would be useful if the data presented is a list of 'records' each with a set of 'fields'. eg a list of recordings with their respective fields describing filename, title, start time, filesize etc or a list of channels with their respective frequencies, call signs etc. The data would typically have been obtained with a GetSomethingList call.

eg to get a hash of hashes from GetRecordedList data with filename as a primary key and containing Title and filesize, firstly check the format of the data to determine the separator for the 'records'. You will find that the recordings are wrapped by <Program> ... </Program>, that the filename is in <FileName>..</FileName>, the title in <Title> ... </Title> and the size in <FileSize>..</FileSize>.

FillHashofHash takes parameters: hash, content, record separator, primary key, secondary key(s)

eg to print filename, size and title for all recordings:

#!/usr/bin/perl -w
use strict;
use scan_database;
my $url='192.168.1.67:6544/Dvr/GetRecordedList';
my $content;
ReadBackend($url,$content);
my %hash;
FillHashofHash(%hash, $content,'Program','FileName','Title','FileSize');
for (sort keys %hash){
    print "$_ $hash{$_}{FileSize} $hash{$_}{Title} \n";
}

Any of the 'key' fields can be replaced by '#' to give an incrementing (and unique) counter. The routine returns the number of records found.

This might be particularly useful for guaranteeing uniqueness of the primary key.

GetAllFields

Some of the APIs calls return a simple XML list of fields rather than a series or records. To extract ALL fields between particular start and end delimiters and put them in a hash, use

GetAllFields(%hash, $content, $start, $end)

The routine will only start looking for fields once it has encountered $start in the data and will stop if it finds $end.

A null value to start or end will indicate start and end of the whole string. eg to read the fields in the header of a GetRecordedList:

#!/usr/bin/perl -w
use strict;
use scan_database;
my $url='192.168.1.67:6544/Dvr/GetRecordedList?StartIndex=1&Count=1';
my $content;
ReadBackend($url,$content);
my %hash;
GetAllFields(%hash, $content, 'serializerVersion', '<Programs>');
for (keys %hash){
    print "$_ $hash{$_}\n";
}

The routine returns the number of fields extracted.

Getting data from the backend and parsing in one step (deprecated)

The first cut of the module code below only included these one step routines. The second cut introduced the three routines above which have a simpler interface, which are less mythtv specific and will probably need less maintenance.

The code for these one step routines has been re-written to exploit the three routines above but should maintain the same user interface. Note that hash references were needed (eg \%hash rather than %hash) and still do for compatibility. <blush> I didn't know about prototyping at the time </blush>. I suggest that you treat these routines as deprecated, but the details are here for completeness.

Readers will note that many of the API calls will return XML code which consists of

Generic 'header' information (eg Mythtv version) followed by:

Sets of data relating to an individual channel, video source, recording etc.

Code is presented here in module form which makes two routines available to calling perl scripts to reflect each type of data:

GetDBheader which populates a hash with the initial header information in the data returned by API calls.

Calling parameters are:

Reference to hash
IP of host
Name of API
Extra parameters to supply to the url (see below).

The hash gets populated with the header information and the return code is the number of entries found.

GetDBinfo which populates a hash with requested fields from the database. Calling parameters are:

reference to the hash
IP address of host
Name of API (eg getrecordedlist). This is case insensitive.
Extra information to supply to the url call to the backend (eg 'StartIndex=3&Count=4').
Primary key for the database. This must be unique in the database (eg 'FileName' would be unique but not 'ChanId')
One or more secondary keys to extract.

If extra parameters is left blank then the routines will normally return data on all recordings, video sources etc. Any initial '?' or '&' should be omitted.

If uniqueness of key is an issue then '#' can be used as a key and an incrementing counter will be returned as the value. The counter will increment at each new recording, channel etc.

The routine populates the hash and returns with the number of hash entries created.

A simple demo program should give a flavour of the routines:

testscan.pl

#!/usr/bin/perl -w
use strict;
use scan_database;


#Example of GetDBheader

print "\nGetDBheader\n";
my %header;
my $resp= GetDBheader(\%header, '192.168.1.67','VideoSourceList');
print "$resp entries returned\n";
for (sort keys %header){
    print "$_  $header{$_}\n";
}


#Example of GetDBinfo -  Note use of # to return a counter.
# If the extra param (StartIndex=2 &Count=10 in this case) is 
# null then all will be returned

print "\nGetDBinfo\n";
my %data;
$resp= GetDBinfo(\%data, '192.168.1.67', 'GetRecordedList', 
      'Startindex=2&Count=10',  'FileName','Title','FileSize','#');

#params are:  reference to hash, host, function, extra info to url call (if any),
#             primary key for hash data, secondary keys.

print "$resp entries returned\n";
print "FileName                  #  FileSize     Title\n";
for (sort keys %data){
    print "$_   $data{$_}{'#'}  $data{$_}{FileSize}   $data{$_}{Title}\n";
}

exit 0;

The above demo needs to be able to access the module code. The module currently only supports ChannelInfoList, GetRecordedList and VideoSourceList but other routines should be straightforward to include (hint: look at the hash %steer).

The Module

The module needs to be put in a file called scan_database.pm with read and execute permissions. It will work if placed either in the current working directory or one of the directories in the perl path. The perl path for your system will be held in the perl variable @INC.

Glitch: I had difficulty persuading this wiki to accept the code in KillEscapes. It insisted in translating 'ampersand amp;' to &, ampersand quot; to quote etc. Please do a sanity check that the four substitutions are not null.

scan_database.pm

#!/usr/bin/perl -w
package scan_database;

use strict;
use Exporter;
use LWP::UserAgent;  # needs 'sudo apt-get install libwww-perl'

our $VERSION     = 1.00;
our @ISA         = qw(Exporter);
our @EXPORT      = qw(ReadBackend FillHashofHash GetAllFields GetDBinfo GetDBheader);

#
#   Generic Routines
#   ----------------

# Read a url into buffer.
#   ReadBackend($url, $buff);


# Read the buffer and populate a hash of hashes.  Useful after reading a 'list' API.
#   FillHashofHash(%hash, $buffer, $separator, $key, $field1, $field2, etc)
#   where    
#       $separator is the xml tag which splits records
#          (eg Channel  for ChannelInfoList or Program for GetRecordedList)
#       $key is the primary key for the hash
#       $field1 etc are secondary.
#       A '#' can be used for $key or a $field in which case a counter is inserted.
#
#   The routine clears the hash before re-filling it
#   eg 
#       ReadBackend('192.168.1.67:6544/Dvr/GetRecordedList', $buffer);
#       FillHashofHashes(%hash, $buffer, 'Program', 'FileName', 'FileSize','Title');
#       print "$hash{'1001_20150525214100.mpg'}{Title}\n";
 
# Read the buffer and populate a hash with all tags found between delimiters.
#   GetAllFields(%hash, $buffer, $start_text, $end_text)
#       if $start_text or $end_text are empty strings, then start and end of buffer are assumed.
#       GetAllfields(%hash, $buffer, '', '<ChannelInfos>')
#       print "$hash{'ProtoVer'}\n";



#   The following two routines are more limited and are more vulnerable to changes in the API.
#   They are deprecated but have been updated to call the above routines.
#   note that a hash reference must be supplied (they are not protocol calls). 
   
#Extract data from database
#  Routine GetDBinfo
#  params are:
#   reference to hash to fill
#   IP of host
#   Catagory required eg GetRecordedList
#   extra infor to go in call (if any).
#   required hash key
#   list of things to extract
#   eg :-
#   GetDBinfo(\%mydata, '192.168.1.67', 'GetRecordedList', '',
#    'FileName','Title', 'LastModified');
#  If key or extracted name is # then this is replaced by a counter.

#routine GetDBheader
# params are:
#   reference to hash to fill
#   IP of host
#   Catagory required eg GetRecordedList
#   extra info to go in call (if any).
#  eg GetDBheader(\%mydata, '192.168.1.67','getrecordedlist','');



my $browseropen=0;
my $browser;
my $counter;


#steering file for GetDBHeader and GetDBinfo
my %steer = (
    getchannelinfolist  => ['Channel/GetChannelInfoList', 'ChannelInfo','ChannelInfos'],
    getrecordedlist     => ['Dvr/GetRecordedList?Descending=true', 'Program','Programs'],
    videosourcelist     => ['Channel/VideoSourceList', 'VideoSource','VideoSources'],
    
    #add more like this => [url, separator for getDBinfo, separator for getDBheader],  
);

#--------------------------
# published routines follow
#--------------------------

sub ReadBackend($\$) {
 
    my ($url, $buff)=@_;
    unless ($browseropen){
        $browser = LWP::UserAgent->new;
        $browser->timeout(10);
        $browseropen = 1;
    }
    my $response = $browser->get(tidyurl($url));
 
    unless($response->is_success){ 
        die $response->status_line;
    }
    $$buff = $response -> content;
}
#------------------------------
sub FillHashofHash(\%\$@){

    my ($hashref, $buff, $separator, $key_name, @params)=@_;
    %$hashref=();
    $counter=0;
    my $endrecord;
    my $base=index($$buff, "<$separator>");
    
    while ($base>-1){
        my $nextbase=index($$buff, "<$separator>", $base+10);
        $endrecord= ($nextbase<0)?length($$buff):$nextbase;
        &ProcessRecord($hashref, $buff, $base, $endrecord, $key_name, @params);
        $base=$nextbase;
        $counter++;
    }
    return $counter;
}
#-------------------------

sub GetAllFields(\%\$@){

    my ($hashref, $buff, $start_text,$end_text)=@_;

    my $start_index= $start_text?index($$buff,$start_text):0;
    if ($start_index<0) {die "cannot find start string $start_text"};

    my $end_index= $end_text? index($$buff,$end_text,$start_index):length($$buff);
    if ($end_index<0){die "cannot find end string $end_text"};

    %$hashref=();
    $counter=0;
    my @bits = split />/, substr($$buff,$start_index,$end_index-$start_index);
    foreach (@bits){
        if (/(.*)<\/(.*)/){
            $$hashref{$2}=&KillEscapes($1);
        }elsif (/<(.*)\//){
            $$hashref{$1}='';
        } 
        $counter++;
    }
    $counter;
}
#--------------
sub GetDBheader{
    #depricated.
    #expects hash reference as first item; returns count of header items. 

    my ($hashref, $ip_addr, $class,$extra)=@_;
    $class=lc $class; 
    unless (defined ($steer{$class})) {die "Sorry - $class not supported"};
    $counter=0;
    #read from backend 
    my $url= &FormURL($ip_addr, $steer{$class}[0], $extra);
    my $buffer;
    ReadBackend($url, $buffer);
    return GetAllFields(%$hashref, $buffer,'',$steer{$class}[2]);
}
#---------------------

sub GetDBinfo{

my ($hashref, $ip_addr, $class, $extra, $key_name, @params)=@_;

    $class=lc $class; 
    unless (defined ($steer{$class})) {die "Sorry - $class not supported"};
 
    #read from backend 
    my $url= &FormURL($ip_addr, $steer{$class}[0], $extra);
    my $buffer;
    ReadBackend($url, $buffer);
    #process buffer
    FillHashofHash(%$hashref, $buffer, $steer{$class}[1], $key_name, @params);
}


#-------------------------
#internal routines follow
#-------------------------

sub tidyurl{
    my ($url)=@_;
    $url='http://'. $url unless ($url =~ m!^http://!);
    $url;
}


sub FormURL{
    #used in 
    my ($ip, $class, $extra)=@_;
    my $addr="http://$ip:6544/$class";
    if ($extra){
        if ($addr =~ /\?/){
            $addr .= '&' . $extra;
        }else{
            $addr .= '?' . $extra;
        }
    }
    return $addr;
}


sub getparameter{
    my ($param, $buff, $base, $endat) = @_;

    my $start= index($$buff, "<$param>",$base);
   
    if (($start<0) or ($start>$endat)) {
        #oops!  look for <param/> instead
        $start= index($$buff, "<$param/>",$base);
        if (($start>0) and ($start < $endat)){return ''};
        die "cannot find $param in record";
    }
    $start=$start + length($param) +2;
    my $end=index($$buff, "</$param>", $start);
    my $data=substr $$buff, $start, $end-$start;
    &KillEscapes($data);
}


sub KillEscapes{
    my ($data)=@_;
    return $data unless ($data =~ /&\w+;/);
    $data =~ s/&quot;/"/g; 
    $data =~ s/&lt;/</g; 
    $data =~ s/&gt;/>/g; 
    $data =~ s/&apos;/'/g; 
    $data =~ s/&amp;/&/g; 
    
    $data;
}

sub ProcessRecord{
    #strip interesting data from record (where record=channel, recording, video source etc)
    my ($hashref, $buff, $start_at, $endat, $key_name, @params) = @_;

    my $key=$counter;
    if ($key_name ne '#'){ $key=&getparameter($key_name,$buff, $start_at, $endat)};

    foreach (@params){
        if (/^#$/){
            $$hashref{$key}{$_}=$counter;
        }else{
            $$hashref{$key}{$_}=&getparameter($_, $buff, $start_at, $endat);
        }
    }
}

1;

Writing to the database

Examples are hard to come by but this code should remove a recording from the database. Note that the StartTime parameter actually expects StartTS.

expire.pl

#!/usr/bin/perl -w 
use strict; 
use warnings FATAL => 'uninitialized'; 
use LWP::UserAgent; 

my $StartTS = '2012-01-18T21:26:00Z';	#extracted from GetRecordedList parameter <StartTS> 
my $ChanId = '1002';				# ditto but <ChanId> 

#open the browser 
my $browser = LWP::UserAgent->new; 
$browser->timeout(10); 

#issue post 
my $response = $browser -> post( 
	'http://192.168.1.67:6544/Dvr/RemoveRecorded', 
	[ 
	'StartTime' => $StartTS,          #NB StartTS not StartTime 
	'ChanId'	=> $ChanId, 
	], 
     ); 

unless($response->is_success){die "Error expiring recording"}; 
my $url_buffer = $response -> content; 
unless ($url_buffer =~ /<bool>true/) {print "Bad response expiring\n"}; 
exit 0;

Traps for the unwary

1. Many of the routines have parameters StartIndex and Count. If neither is supplied, then all entries are assumed. Note that there is an inconsistency with StartIndex.

With GetChannelList, StartIndex=0 must be supplied in order to get the first entry. That's logical.

With GetRecordedList, StartIndex=0 and StartIndex=1 return the same data. Thus code which grabs the data in chunks eg

StartIndex=0&Count=100

StartIndex=100&Count=100

will have a duplicate returned because he first call will return 1 to 100 inclusive, the second 100 to 199.

2. GetRecorded and RemoveRecorded both have a 'StartTime' parameter.

Note that the value supplied should be the time a recording started and not the time in the schedules. These will differ in cases where a recording is requested after the start of the recording or if all recordings are started early. If you extract the time from a GetSomethingList call then you need to extract 'StartTS' not 'StartTime'.

More information please!

Have fun and do help to expand this page!

Difference between revisions of "Perl API Examples version 2"

Revision as of 17:27, 17 July 2015

Contents

WARNING

Introduction

Previous Version

Existing sources of information

Getting information from the backend

Web browser

Module LWP::Simple

Module LWP::UserAgent

The scan_database Module

Parsing the data

Regex

Parsers on the CPAN database

The parsers in scan_database module

FillHashofHash

GetAllFields

Getting data from the backend and parsing in one step (deprecated)

The Module

Writing to the database

Traps for the unwary

More information please!

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Wiki Navigation

Official Resources

Tools

Print/export