Difference between revisions of "User:PhilB"

From MythTV Official Wiki
Jump to: navigation, search
(Merge)
(Page cleared.)
(17 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
+
Page cleared - contents moved to Channel editor.
 
 
 
 
  WARNING:  This page is under construction.
 
  Please Ignore for the moment.
 
 
 
 
 
 
 
==Introduction==
 
 
 
This page describes version 2 of a screen based MythTV Channel editor.
 
 
 
It is a perl script using the API interface which will allow channel data to be easily displayed, sorted and changed.  It can be run on the backend or from another computer with access to the backend.
 
 
 
A tutorial is included for user familiarisation.
 
 
 
Version 2 was developed following extensive user feedback from version one.  Not only has functionality been increased but the user interface has been improved.
 
 
 
==What does it do?==
 
 
 
It will:
 
 
 
- Extract channel data for all sources from a backend.
 
 
 
- Display channel data - ChannelId, Channel number, source, multiplex, Callsign and XMLTVID as well as the visible, CommFree and EIT flags.  A further parameter can be extracted, shown and modified.
 
 
 
- Allow channels to be sorted in various ways.
 
 
 
- Allow the data shown to be selected
 
 
- Allow channels to be edited, either singly or in bulk.
 
 
 
- Has an 'Undo' function.
 
 
 
- Export the state of channels to a file for re-import so the same settings can be re-applied after your next rescan.
 
 
 
- Allow import of XMLTVID data from UK Altas or Radio Times files.
 
 
 
- Allow running in demo mode with no changes to the database.
 
 
 
- Allow 'spoofing' with alien data from a file rather than reading the backend.  This was developed as a support tool but also supports the tutorial below.
 
 
 
The script uses the MythTV API interface and has been tested with both 0.27 and 0.28-pre with UK DVB-T and DVB-S data and with US cable and 'over the air' ATSC data.  All tests have been in a Linux environment, though there are successful reports of use with OSX.
 
 
 
Before trying it with your own precious database you will probably wish to try it out with artificial data and a safe step by step tutorial.  This shows the features of the editor with a subset of data 'pruned' and modified from a UK system.  It will not interact with your database!
 
 
 
==Setup==
 
 
 
A suggested setup for channel editor is:
 
 
 
Create a fresh directory to hold the files:
 
  mkdir chedit
 
  cd chedit
 
 
 
you will need three files;
 
 
 
*chedit2.pl          the perl script.
 
 
 
*scan_database.pm    the API interfacing module
 
 
 
Note that the suffixes are .pl and .pm respectively.
 
 
 
*tutorial.diags      if you wish to run the tutorial below.
 
 
 
 
 
The module scan_database.pm is fully described in [[Perl_API_examples]] and can be placed anywhere in the perl 'path' which you can list with:
 
 
 
<pre>perl -e 'print join ("\n", @INC), "\n"'</pre>
 
 
 
The three files can be found on github:  https://github.com/PBrady43/Channel-Editor
 
   
 
You can either extract these using git (if familiar) or copy and paste then into your favoured editor (hint:  Use 'raw' before copy). 
 
 
 
Please check the first lines of chedit2.pl and of scan_database.pm are:
 
 
 
    #!/usr/bin/perl -w
 
 
 
Then make them executable:
 
<pre>
 
chmod 755 chedit.pl
 
chmod 755 scan_database.pm
 
</pre>
 
 
 
If not running in a Linux environment please note the comments below.
 
 
 
==Command line options==
 
 
 
chedit.pl has four command-line options.
 
 
 
--backend  specify backend address:port number (but it may well find that itself!)
 
 
 
--nodemo or --demo      Demonstration mode.  Database will NOT be updated.  --demo is default.
 
 
 
--spoof <filename>  reads data from <filename> rather than your backend.  A support option and enables the tutorial.
 
 
 
--extra  <parameter>  specifies an additional field to extract from the database for display and editing.
 
 
 
 
 
==Viewing data==
 
 
 
===Normal view===
 
A list of channels should be displayed.  Note the scroll bar at the right.
 
The columns of data are:
 
Row number for referencing purposes
 
Channel Id - the database reference in MythTV
 
Channel Number
 
Source and multiplex.
 
A set of flags which are:
 
  D channel will be deleted
 
  V channel visible
 
  E  UseEIT flag
 
  C  Commflag
 
  d  duplicated callsign or channel number (if visible)
 
  Q  there is a match query following an import
 
  U  last change can be undone
 
  M  this channel has been modified.
 
CallSign
 
XMLTVID
 
 
 
===Extra View===
 
If the --extra parameter is specified, an additional menu item is shown. It provides a column setup which replaces XMLTVID with the values of the --extra parameter.
 
 
 
===Custom Columns===
 
If the items do not fit comfortably on the screen or a different set of columns is required then use of View-> Define Custom Columns will bring up a window allowing the desired columns to be selected.  Choose each in turn followed by 'ok' then finish with a final 'ok'.
 
 
 
The default view can be restored with View -> Show default columns, and the newly defined column selection again with View -> Show Custom Columns.
 
 
 
===Multiplexes===
 
View -> Multiplexes shows a listing of multiplexes.
 
 
 
===Show Log===
 
Shows the log.
 
 
 
 
 
==Sort menu==
 
The sort menu allows the order in which the channels are displayed to be changed.
 
 
 
Note that the channel shown on a particular row changes with the sort rule used.  This allows the power of the sort rules to be exploited in the bulk edit menus.
 
 
 
If none of the 14 pre-defined orders suit then a custom rule can be generated in much the same way as with the columns.
 
 
 
==Edit Menu==
 
 
 
The edit menu allows changing parameters of single channels, or changing parameters in bulk.
 
 
 
===Single Channel===
 
 
 
After selecting the row, this option puts up a window allowing fields for the particular channel to be altered.  Alter the fields and select ok or cancel.
 
 
 
===Bulk by row range===
 
This allows selection by a single row or a set of rows as displayed by the current sort.  Enter a single row as the number or a range as first:last.  eg 25:27 will select rows 25,26 and 27. 
 
 
 
 
 
With the three bulk edit options, boolean values can be set or cleared, text and numeric values can be 'set'. 
 
 
 
Numeric values can also be incremented by a fixed value.  eg to add 1000 to ChanNum put the value 1000 and 'add'.  Negative values will decrement.
 
 
 
===Bulk by Source:mpx===
 
 
 
This allows selection of a whole source (eg 1) or a whole multiplex (eg 1:6).
 
 
 
 
 
===Bulk by callsign===
 
 
 
This allows selection of channels by content of the CallSign.  eg bbc will pick out all CallSigns with 'bbc' in the name.
 
 
 
 
 
===Edit Undo===
 
 
 
Edit -> Undo will reverse the most recent change.  The channels which will be affected have the 'U' flag set.
 
 
 
Edit -> Undo all will clear all changes.  The channels which will be affected have the 'M' flag set.
 
 
 
Note that the 'undo' items are held in a stack which is cleared after a database save.
 
 
 
 
 
==File Menu==
 
 
 
===Exporting and Importing===
 
 
 
Having marked all the channels as desired the database can be updated (if not in demo mode!!), but before doing so it may be wise to 'export' the settings made to a file for later use.  The file is called channels.export.
 
 
 
The file can then be 'imported' after a channel re-scan to re-impose your settings on the new channel configuration.
 
 
 
The editor will read the channels.export file and attempt to match the channels - the matching rules are probably of little interest to the user (provided they work!) but are:
 
 
 
1. CallSign matches and is unique in the file and unique in the database.
 
2. Channel number and CallSign both match and the combination is unique in the file and also unique in the database.
 
3. Source, frequency and callsign all match and the combination is unique in the file and also unique in the database.
 
 
 
If callsign had been changed then three further rules are also applied but using the original callsign.
 
 
 
If any channel cannot be matched then it will leave it unchanged but put a Q (for query) state.  These can be identified by sorting by query.  Such queries will need to be resolved by hand.
 
 
 
When they are edited the 'Q' flag will clear.
 
 
 
As with all edits, you can 'undo' an import.
 
 
 
Notes:
 
 
 
1.  It is advisable to import on reading the database and export again before updating it.  This will keep the database and the export file in step.  This is important for two reasons:
 
 
 
a) A major matching parameter is CallSign and if this is changed, the 'old' name is retained and exported. On import the 'old' name is noted and used if necessary.
 
 
 
b).  If the --extra calling parameter is used then the --extra values are also written to the export file, and the file will support many --extra sets.  If it is necessary to alter different --extra parameters then they will keep in step with successive --extra runs provided imports and exports are coordinated.
 
 
2.  Even if you have no history of using the channel editor and are facing a channel rescan, then a run with no edits but an export BEFORE the rescan and a run with an import AFTER the rescan will recover many of the settings.
 
 
 
3. If you run the editor with a --extra parameter, the values of that parameter will be written to the export file for re-import later.  The values can be shown via a custom column selection.
 
 
 
4. If an export file is found when starting the editor, the header will be read to check which parameters were of interest last run.  These will then be extracted from the database for consistency.
 
 
 
===Updating Database===
 
 
 
This option will seek confirmation that a database backup is available then modify the database with the changes made.  Database changes will only be made in nodemo mode.  Export will be invited before update.
 
 
 
==XMLTV Menu==
 
 
 
There is a particular problem with XMLTVID data in the UK but the issue may be prevalent elsewhere.
 
The two main XMLTVID providers (Atlas and the now deprecated Radio Times) produce files which give a correlation between callsign and XMLTVID and it has been necessary to manually insert the IDs into the database either individually with a channel editor or by hand crafted sql statements.
 
 
 
Unfortunately, there are many regional variations and whilst a human may easy decide that BBC ONE LON is appropriate rather that BBC ONE Wales, Scotland etc, automation of this is tricky.
 
 
 
Comparing callsigns in the database with those in the file by 'folding' to lower case and dropping spaces does part of the job, but other 'hints' are useful.  eg 'BBC ONE Lon' in the database will match 'BBC ONE London' in the file with the hint: Try ''lon'' instead of ''london''.  Similarly, BBC TWO in the database will match BBC TWO ENGLAND in the file with the hint: Try <nothing> instead of ''england''.
 
 
 
The format of the hints file (for the above examples) is
 
  lon:london
 
  :england
 
 
 
Some characters will need 'escaping' by prefixing them with \.  The characters are:
 
\ | ( ) [{ ^ $ * + and .  eg
 
 
 
  :\[Freeview]
 
 
 
tries dropping [Freeview] from the callsign in the file to match that in the database
 
The XMLTV menu thus has four options:
 
 
 
====Import====
 
This allows selection of the MythTV source, the XMLTV file, its type and a hints file.  It attempts matching and displays the database callsigns alongside the matching file callsigns.  Further manual matching is then possible before finally 'committing' the XMLTVIDs to the image in memory of the database.
 
 
 
====Show Matches====
 
This option shows the database callsigns with the callsigns and XMLTVIDs from the file.  Matching ones are shown on the same row.
 
This is a sample of a mis-match:
 
 
 
<pre>
 
 
 
  CallSign              Row  File_CallSign                        XMLTVID
 
 
 
                        56  BBC London 94.9                      london.radio.bbc.co.uk
 
  BBC NEWS              57                                                         
 
                        58  BBC News Channel                    news.bbc.co.uk     
 
</pre>
 
 
 
===Merge===
 
If matching is incomplete, items can be manually matched with this option.
 
Any channels in the left column without a match can be linked to those on the right.
 
eg to match ''BBC NEWS'' in the database on row 57 with ''BBC News Channel'' on row 58 then entering 57:58 will do this and alter the display.
 
 
 
===Commit===
 
Once the matching is completed, the commit option will copy the XMLTVIDs to the main image of the database.  The main database in the backend will not be changed at this stage though, and the commit can be undone.
 
 
 
   
 
==Help==
 
 
 
The help menu gives access to this wiki.
 
 
 
The diagnostics option generates a file with a name of the form chedit<datestamp>.diags holding diagnostic information.  If not in 'spoof' mode it also includes a copy of API data from the backend.  The file should also be created on unexpected error.
 
 
 
 
 
==Changes from Version 1==
 
 
 
Version 2 has had a major rewrite to make improvements following user feedback.
 
 
 
Any field from the database can be accessed and changed, rather than just the ability to hide, show or delete a channel as in version 1.
 
Dynamic width columns and 'custom columns' have been introduced and the format of 'export' files changed as a result.  Also, View and Sort functions have been split.
 
 
 
More flags are introduced. 
 
 
 
Undo is now unlimited, rather than just allowing the last few edits to be reversed.
 
 
 
XMLTV data can now be imported, matched and committed but as yet only UK specific formats are catered for.
 
 
 
Cosmetic changed include the separation of view and sort, radio buttons showing the last sort and a 'prettier' custom sort mechanism. 
 
 
 
Help is now external - the help menu launches a browser to open the wiki page.  This avoids the previous clumsy display.
 
 
 
 
 
==Tutorial==
 
 
 
Start the channel editor in 'spoofing' mode with
 
<pre>./chedit2.pl -sp tutorial.diags</pre>
 
 
 
This will start the editor in demonstration mode and read the file tutorial.diags to show the (artificial) tutorial data rather than looking for data on your backend.
 
 
 
If it complains that the libwww-perl module needs loading and if using ubuntu try
 
<pre>
 
sudo apt-get install libwww-perl
 
</pre>
 
''Any other modules??''
 
 
 
Assuming that all required modules are available, it should show a full size screen with columns of channel data and a reminder that you are in demo mode.  There is also a demo reminder top right of the screen. Click on 'OK' in the reminder box. If prompted to read an export file, decline at this point.
 
 
 
A main channel view will be shown.  See above for the meaning of the columns and flags.
 
 
It should be possible to view the multiplexes, to create a custom set of columns, to sort in various ways and to create a custom sort rule via the View and Sort menus.
 
 
 
 
 
===Editing===
 
 
 
The crux of the software is the ability to perform edits quickly and efficiently.  A few examples are given here:
 
If an error is made then it can be undone with
 
 
 
  Edit -> undo
 
  Edit -> Undo all
 
 
 
Sort the channels by duplicate callsigns and note the duplicates sorted at the head of the screen.  Three of these are duplicated between multiplexes 1 and 7 and this data simulates a distant interfering transmitter.  Examination of View -> multiplexes and published data shows multiplex 7 to be the rogue (pretend here).  Eliminate all from multiplex 7.
 
  Edit -> Bulk by source:multiplex,  choose 1:7, delete, set.
 
Note the D , U and M flags appear and d flags clear.  The order is not changed as the channels would be lost 'visually'.
 
 
 
Sort again by duplicate callsigns and see that ITV4 is duplicated (channel numbers 24 and 792). 792 was introduced during a channel re-alignment and gave a 'please retune' reminder.  There are a multitude of ways of deleting or hiding this channel such as:
 
 
  Edit -> bulk by row range, 1, delete, set
 
  Edit -> bulk by row range, 1, visible, clear.
 
  Edit -> bulk by callsign, ITV4, delete, set
 
  Edit -> Single channel, row 1, change 'visible' to ''false'' or ''delete'', ok
 
 
 
Since BBC THREE no longer exists (except via internet) let's kill that off similarly.
 
 
 
BBC ONE recently changed name to BBC ONE Lon and invalidated all recording rules.  The callsign can be changed easily once we sort by  ChannelId (or ChannelNumber).
 
 
 
  Edit -> single channel -> row 0, change the name and ok.
 
 
 
Let's hide all 'adult' channels in one swoop.
 
  Edit -> bulk by callsign, adult, visible, clear.
 
 
 
.. and make all BBC channels CommFree
 
    Edit -> bulk by callsign, BBC, CommFree, set
 
 
 
Note with the single channel edit the format of CommFree changes between 0.27 (0 or 1) and 0.28 (false or true) but a reminder and checks are in place.
 
 
 
Finally, let's clear all EIT flags because we are about to set XMLTVIDs:
 
  Edit -> Bulk by source:multiplex,  choose 1, UseEIT, clear.
 
 
 
The ''bulk'' options can also be used to change callsign, the extra parameter, XMLTVID and channel number.  This powerful option needs to be used responsibly.  Numeric fields such as channel number can also be incremented (or decremented if a negative value is given). 
 
 
 
''UK satellite users seem to have a need to shuffle channel numbers and it could be that this simple addition does not fully meet their need.  Feedback please.''.
 
 
 
 
 
===XMLTVID===
 
Those in the UK who with to use a grabber have a problem inserting XMLTVIDs into the database and have resorted to raw sql comamnds.  This editor includes routines to help with this but uses the API interface.  The file /usr/share/xmltv/tv_grab_uk_rt/channel_ids seems to be included in a standard MythTV distribution (in 0.27 at least - despite UK radio times being deprecated) and can be used to demonstrate this feature.  Matching is done by reading a file giving CallSign against XMLTVID from the xmltvid supplier; also a'hints' file provided by th euser.
 
 
 
Now XMLTVID -> Import.  Choose Source 1, UK Radio Times, accept the suggested file, cancel the hints file.
 
Now scan down the list until you find a mis-match.  The first is Al-Jazeera callsign on row 19 which needs to match row 20.  Match these with XMLTVID -> merge, 19:20.
 
Similarly merge 40:41, 66:69, 237:289, 283:243, 343:344.  When all are matched then XMLTVID -> Commit to write to the main (in memory) database.
 
 
 
Alternatively, a 'hints' file like this will automate the matching (at least for London).
 
 
 
<pre>
 
 
 
#This is a hints file for matching XMLTV data with callsigns from MythTV
 
#It is UK specific
 
#
 
#  These are local - adjust to suit your own locale:
 
#
 
#format is database callsign text :  xml file text
 
lon:london
 
:london
 
 
 
#
 
#These may be more generic:
 
#
 
:england
 
eng:england
 
eng:english
 
e:east
 
w:west
 
westmid:westmidlands
 
emid:eastmidlands
 
scot:scotland
 
yorks:yorkshire
 
s:south
 
e:\(e\)
 
w:\(w\)
 
n:north
 
ni:northernireland
 
ci:channelislands
 
:\[freeview\]
 
wal:wales
 
star:\*
 
&:and
 
pick:picktv
 
</pre>
 
 
 
===Export Data and Update Database===
 
At this stage you may be ready to update your database (apart from the fact that we are in demo mode) but before doing so it will be helpful to ''export'' your hard work so it can be re-instated after a channel re-scan in Mythtv.
 
  File -> Export
 
This will write a file called channels.export.  Now do
 
  File -> Update Database
 
This will do nothing apart from making a few proud claims - the software does not even know where your database is!
 
 
 
and finally
 
  File => Exit
 
 
 
===Import===
 
Imagine that you had previously made changes to the database and you have lost them all with a channel scan.
 
In our tutorial this is the same as restarting the editor. Restart the editor as above but this time it will note that an export file is present.  Allow it to 'import' it.
 
 
 
All your changes should be re-applied and in a real-world situation, an update database should reinstate the database.
 
 
==Real World Use==
 
 
 
In order to run the editor against your own database in safe demo mode:
 
<pre>./chedit2.pl</pre>
 
 
 
 
 
To run it in 'live' mode
 
<pre>./chedit2.pl --nodemo</pre> 
 
Note that this WILL update the database if you choose the 'update database' option.
 
 
 
If the script cannot find the backend address (it searches for a config.xml file in all the likely places), then it will ask for the address or hostname to be specified with the --backend parameter eg:
 
 
 
<pre>./chedit2.pl --backend 192.168.1.65</pre>
 
 
 
If a port number other than 6544 is required then it can be added with:
 
<pre>./chedit2.pl --backend 192.168.1.65:1234</pre>
 
 
 
Please DO ensure that you have a robust backup strategy or are prepared to re-scan
 
channels before using the --nodemo option.
 
 
 
When running in 'real' modeitis advisable to read the export file when loading the editor and write it again BEFORE anddatabase updates.  This will keep the two in step. 
 
 
 
 
 
==Other platforms==
 
 
 
===OS X with MacPorts===
 
 
 
From Apple, OS X is lacking several of the software packages that this script depends on.  The [https://www.macports.org MacPorts] project can provide the necessary dependencies:
 
 
 
sudo port install xorg-server p5.22-tk
 
 
 
Edit the first line of the script so that it uses the MacPorts version of Perl.  Make it look so:
 
 
 
#!/opt/local/bin/perl5.22 -w
 
# was #!/usr/bin/perl -w
 
 
 
From Applications > MacPorts, start X11.app and then start the channel editor from Terminal, as described above.
 
 
 
==Problems==
 
 
 
Developing and testing this application with two sets of UK data and two sets of USA data (thank you all!!) has shown that TV cultures are both very rich and very diverse.  If the code does not behave sanely the help>diagnostics facility has been included to assist investigation.
 
 
 
This should generate a file 'chedit<timestamp.diags' which hopefully contains all the information necessary for the author to 'spoof' your
 
system and reproduce the fault.
 
 
 
The file should also be produced if the code throws an error.  Please submit the file with any reports of problems.
 
 
 
===Known Issues===
 
 
 
There are two known error messages, both thrown by the underlying Tk module.  It may be that these are errors induced by the way that Tk is called.
 
 
 
====Pane.pm line 301====
 
 
 
"Use of uninitialized value in string eq at /usr/lib/perl5/Tk/Pane.pm line 301"
 
 
 
This is triggered by single clicking on the right scroll bar and results in spurious messages to the terminal.  It may be a bug in the way Pane is called, but can be eliminated by a change to Pane.pm. 
 
 
 
Find the module with:
 
<pre>locate Pane.pm</pre>
 
 
Line 301 reads:-
 
 
 
<pre>if(defined($ch) && $ch->manager eq 'grid') {</pre>
 
 
 
Change it to:-
 
 
 
<pre>
 
my $fix=$ch->manager if defined($ch);
 
if(defined($fix) && $fix eq 'grid') {
 
</pre>
 
====focusPtr out of sync====
 
 
 
"TkFocusKeyEvent found dispPtr->focusPtr out of sync: expected ??, got .dialogbox2.bottom.button"
 
 
 
This error message to the termainal is not fully understood, but is believed to be caused by keyboard input whilst no input box has focus.  All screens expecting keyboard input set a suitable focus to minimise this, but any further insight would be appreciated!
 
 
 
 
 
 
 
[[Category:Services_API]]
 
[[Category:Perl Scripts]]
 
[[Category:Scripts]]
 

Revision as of 16:58, 28 May 2016

Page cleared - contents moved to Channel editor.