Difference between revisions of "User:Mrsdonovan"

From MythTV Official Wiki
Jump to: navigation, search
(Outstanding Issues: Updated)
m (Moving a DigitalOcean Droplet to Virtual Box: URLs amended. Mrsdonovan‎ hope this is OK.)
 
(115 intermediate revisions by one other user not shown)
Line 1: Line 1:
I've been using MythTV since 0.18 and most of [http://www.mythtv.org/wiki/Special:Contributions/Mrsdonovan my contributions] have been to the wiki in regards to modules I've worked with, like Firewire, Mythgame and playing recordings directly from Mythweb.
 
  
The setup below is used daily by the kids, parents and grandparents alike and has a high [http://en.wikipedia.org/wiki/Woman_acceptance_factor WAF factor].  The external DVD players rarely get used anymore.
+
With over a decade of continuous service, this MythTV system was torn down in November of 2016. Its demise started a couple years ago when I stopped watching TV directly and just maintained the system for the grandparents. I also stopped collecting movies or TV shows because they take up a lot of space and I never seem to watch them again, even the ones I like.  I will, over the next few years, whittle the movie collection down to ones that are important and I might want to watch in a few years.  Part of the reason the movie collection started was for my Brother's kids, but as teens now, they game on their phones or gaming console.  The added features, such as the video and picture section were never used and now that everyone has smartphones, the DID (Direct In Dial) phone number serviced by Asterisk (which displayed the callerID) has also been deleted.
 +
 
 +
This MythTV system was shut off largely because the ten year old tuners started having problems and the alternative was simpler.  As I never used the system, the maintenance had become onerous and I am just busy enough these days that I say no to more things then I say yes.  Although I had stopped the MythTV process on October 17th, as the statistics show, I made the decision yesterday, October 31st, 2016, to actually shut my MythTV down permanently.  It actually coincided with the Shaw technician coming by and replacing the mess of tuners with one, taking the old ones away. 
 +
 
 +
Having said that, the MythTV box will still live on as a NAS, run game servers when necessary, and still get hardware refreshes and software updates.  I am thankful to MythTV because it was how I learned about Linux and Open Source in general.
 +
 
 +
The entire MythTV system was replaced with a single six channel "Gateway HD PVR tuner" which even includes a smartphone app to search for and set up recordings.  With up to four tuners and two Haupauge HD PVRs now replaced with one tuner, at least my power bill will go down, which is great, given I just installed an 8kW(DC) 28 module solar system this summer (APS microinverters).
 +
 
 +
I am proud of this User's page, [https://www.mythtv.org/wiki?title=User:Mrsdonovan&action=history having nursed it for almost exactly ten years], and will be saving a PDF of it.  I hope you find it useful.
 +
 
 +
Everything below this line is now kept for historical sake:
 +
 
 +
----
 +
 
 +
I've been using MythTV since 0.18 and here are my paltry [http://www.mythtv.org/wiki/Special:Contributions/Mrsdonovan my contributions].
 +
 
 +
The setup below is used <del>daily</del> weekly by the grandparents and has a <s>low</s> medium [http://en.wikipedia.org/wiki/Woman_acceptance_factor WAF factor]<s>, yes low</s>.  <s>Sadly, the grandparents have reverted to the PVR that comes in the cable tuner using the MythTV as backup.</s>  After reducing the system to one machine it seems to have gotten more stableI have reduced the system to a single combined master backend / HD capable front-end.  Having two computer systems on 24/7 costs money for electricity, replacement HDs and UPS batteries.  I just don't watch enough TV to care anymore - it won't be a better PVR that kills MythTV, just cloud based options like NetFlix.
 
----
 
----
 
==Statistics==   
 
==Statistics==   
 
My current statistics are:<br>
 
My current statistics are:<br>
 
<blockquote>
 
<blockquote>
Number of shows: 4091<br>
+
Number of shows: 4993<br>
Number of episodes: 50017<br>
+
Number of episodes: 75753<br>
 
First recording: Friday May 6th, 2005<br>
 
First recording: Friday May 6th, 2005<br>
Last recording: Monday April 23rd, 2012<br>
+
Last recording: Monday October 17th, 2016<br>
Total Running Time: 6 years 11 months 18 days 3 hrs 37 mins<br>
+
Total Running Time: 11 years 5 months 12 days 23 hrs 58 mins<br>
Total Recorded: 5 years 4 months 18 days 17 hrs 26 mins<br>
+
Total Recorded: 9 years 2 months 26 days 22 hrs 53 mins<br>
 
<hr align="center" size="5" width="30%">
 
<hr align="center" size="5" width="30%">
System A Recording Space: 2,289,109 MB (RAID 5)<br>
+
System B Recording Space: 3TB RaidZ (Arbitrary quota controlled ZFS with SSD for ZIL and Cache)<br>
System B Recording Space: 2,623,048 MB (LVM)<br>
 
Total Recording Space Available: 4,912,157 MB (5 TB)<br>
 
 
<hr align="center" size="5" width="30%">
 
<hr align="center" size="5" width="30%">
System A Video space: 1,022,540 MB<br>
+
System B Video space: 10TB RaidZ<br>
System B Video space: 1,465,076 MB<br>
+
(The videos share the total size of the RaidZ ZFS filesystem with four 3TB Western Digital Red NAS drives)<br>
Total Space for Videos Available: 2,487,616 MB (2.5 TB, All RAID 5)<br>
 
 
<hr align="center" size="5" width="30%">
 
<hr align="center" size="5" width="30%">
 
</blockquote>
 
</blockquote>
Line 26: Line 38:
  
 
=== System A ===
 
=== System A ===
This system is used at my house and consists of the following:
+
This system was used at my house and consisted of the following:
  
==== '''A.a - MythTV Slave backend/ SD Frontend / Asterisk server''' ====
+
==== <s>A.a - MythTV Slave backend / SD Frontend / Asterisk server </s> (System retired 2015) ====
 
<gallery>
 
<gallery>
 
Image:Quantum_case.jpg
 
Image:Quantum_case.jpg
 
Image:Dct2500_bigview_2.jpg
 
Image:Dct2500_bigview_2.jpg
Image:Dct2500_bigview_2.jpg
 
Image:DCT6200.jpg
 
Image:Digium_DGM_TDM_card_side.jpg
 
 
Image:PVR150.jpg
 
Image:PVR150.jpg
 
Image:Pvr250_board.jpg
 
Image:Pvr250_board.jpg
Line 41: Line 50:
 
Image:Belkin_1500VA.jpg
 
Image:Belkin_1500VA.jpg
 
</gallery>
 
</gallery>
 +
<s>
 
* 3Ghz HT Intel with 2GB of dual channel RAM
 
* 3Ghz HT Intel with 2GB of dual channel RAM
* 1GB ethernet jack connected to a 10/100/1000Mbps Dlink 8 port router
+
* 1GB ethernet jack connected to a 10/100/1000Mbps Tredndnet 8 port router
 
* HD capable Geforce 6200 video card.
 
* HD capable Geforce 6200 video card.
 
** This system can play 720p but not 1080p as it stutters horribly.  The motherboard has an AGP port though and everything will have to upgraded to play 1080p.
 
** This system can play 720p but not 1080p as it stutters horribly.  The motherboard has an AGP port though and everything will have to upgraded to play 1080p.
 
* RAID 5 with four SATA harddrives, roughly about 900GB of space for recordings and 1TB for videos/music/pictures/ROMs.
 
* RAID 5 with four SATA harddrives, roughly about 900GB of space for recordings and 1TB for videos/music/pictures/ROMs.
 
* ASUS P4P800-E-Deluxe
 
* ASUS P4P800-E-Deluxe
* Haupauge PVR-150
+
* <del>Haupauge PVR-150</del> - Died
* Haupauge PVR-250 with remote
+
* Haupauge PVR-250 <del>with remote </del>
* Digium Wildcard TDM400P with three S100-M FXS expansion cards
+
* <s>Two One Motorola DCT2524 cable tuner controlled by IR blasters connected to a serial card.  (Custom built lirc_serial kernel module, ledxmit_serial)
* [http://www.freepbx.org/ Freepbx] with Asterisk 1.6
+
* DCT-6214 (Yes, it is a PVR, I know..) tuner connected via an Agere firewire card for HD - Moved it to the A.b MythFrontEndHD system
* Two Motorola DCT2524 cable tuners controlled by two IR blasters connected to a serial card.  (Custom built lirc_serial kernel module, ledxmit_serial)
 
* DCT-6214 (Yes, it is a PVR, I know..) tuner connected via an Agere firewire card for HD
 
 
* [http://www.mythbuntu.org/ MythBuntu 10.4] (32bit)
 
* [http://www.mythbuntu.org/ MythBuntu 10.4] (32bit)
 
* 22" LCD
 
* 22" LCD
* Belkin 1500VA UPS with Bulldog software
+
* Belkin 1500VA UPS <s>with Bulldog software (includes replacing the batteries on a roughly three year cycle.)</s>
 
<br>
 
<br>
  
==== '''A.b - MythTV HD Master Backend / Frontend''' ====
+
==== '''A.b - MythTV HD Master Backend & Frontend / Virtualbox / Asterisk / <s>Bitcoin Miner</s> / NAS ''' ====
 
<gallery>
 
<gallery>
 
Image:X-QPACK_apevia.jpg
 
Image:X-QPACK_apevia.jpg
 
Image:toshiba-regza-xf-lcd-tv.jpg   
 
Image:toshiba-regza-xf-lcd-tv.jpg   
 
Image:DCT6200.jpg
 
Image:DCT6200.jpg
 +
Image:Dct2500_bigview_2.jpg
 
Image:MCE-Remote-2-alt.jpg
 
Image:MCE-Remote-2-alt.jpg
Image:Rumblepad_2_wireless.jpg
 
Image:agere_firewire_card.jpg
 
 
Image:Hd_pvr_small.jpg
 
Image:Hd_pvr_small.jpg
 +
Image:Hd_pvr_small.jpg
 +
Image:Single-SC-ASIC-Butterfly-Labs.jpg
 +
Image:Digium_DGM_TDM_card_side.jpg
 
Image:Belkin_1200VA.jpg
 
Image:Belkin_1200VA.jpg
 
</gallery>
 
</gallery>
* AMD X2 5000+ with 4GB (32 bit)
+
* AMD Phenom X6 1090T with 16GB of Kingston ECC (required for ZFS because it is constantly doing checksums which need to be correct)
* Asus M4A79 Deluxe with Asus Geforce 210 1GB DDR3 (using VDPAU)
+
* Asus Sabertooth v1.0 AM3+ with Zotac GT430 (using VDPAU)
** With VDPAU, HD 1080p works flawlessly at 1920x1080.  The X2 2.6Ghz AMD could almost play 1080p by itself but stuttered.  The AMD X2/Geforce 210 replaced an Intel Dual Core 3Ghz (no VDPAU or XvMC) that made HD watchable, but it had infrequent issues with stuttering.
+
** With VDPAU, HD 1080p works flawlessly at 1920x1080.
* [[Hauppauge_HD-PVR]] and uses 6200ch to change channel through firewire connected to DCT-6200
+
* Two [[Hauppauge_HD-PVR]] running the latest firmware (v1.7.1.30058) and  
* Roughly 1.2TB in a Raid 5 setup and 1TB LVM for video backup (rsync'd each night with master server)
+
** One uses 6200ch to change channel through onboard firewire connected to a DCT-6200s (as an example, using "6200ch -n 1 -p 0" or "6200ch -n 2 -p 0")
* MythGame with SDLMame emulation and a wireless Logitech Rumblepad 2
+
** One Motorola DCT2524 cable tuner controlled by one of the HDPVR IR blasters via LIRC
* [http://www.mythbuntu.org/ MythBuntu 10.4]
+
* <s>3TB Raid 5 for recordings and video backup </s> - 10TB ZFS RaidZ setup with backups to old A.a MythTV Slave
* Toshiba Regza 42RV530U - 42" LCD driven at max resolution of 1900x1080 via an DMI cable.  We use the TV remote with an USB MCE receiver.
+
** Uses four 3TB drives (5400RPM Western Digital RED NAS drives) with a SSD providing a 2GB ZIL and 60GB Cache for read/write.
* <s>Belkin 1200VA UPS with Bulldog software</s> - Batteries dies and not worth replacing.
+
** Given the ZFS array is a single point of failure, important data is backed up every other week to separate alternating machines (part of the hardware is from A.a Humbleserver).  The two seperate backup machines are old socket 478 ASUS P4P800 with P4s, which while power hungry, are only on for <s>a few hours</s> tens of minutes a month while doing the backup.
 +
*** They are first turned on by a Wake-On-LAN (WOL) packet from A.b and boot via PXE Boot-on-LAN which loads a 12.04LTS kernel (from A.b of course) which kicks off a backup script to a 3TB drive.  Once the backup has completed, the computer shuts down.  This uses a minimum of power and as harddrives fail in the ZFS array, they are immediately replaced by one of these backup drives and another ordered without ever being reduced to a single point of failure.
 +
* <del>MythGame with SDLMame emulation and a wireless Logitech Rumblepad 2</del> - never used.
 +
* [http://www.ubuntu.com/ Ubuntu 12.04 LTS] running from an 80GB Intel SSD
 +
* Digium Wildcard TDM400P with <s>two</s> <s>one</s> three S100-M FXS expansion cards (two spares)
 +
* [http://www.freepbx.org/ Freepbx] with Asterisk v1.8
 +
* Toshiba Regza 42RV530U - 42" LCD driven at max resolution of 1900x1080 via an DMI cable.  We use the TV's remote with a USB MCE receiver (using Kernel drivers - no LIRC!).
 +
* Belkin 1200VA UPS <s>with Bulldog software - Batteries died and not worth replacing, Bulldog doesn't work on 10.4</s> - Replaced batteries for about $100.
 +
* <s>Butterfly Labs "Single" 25GH</s> - now junk, sigh
 +
* Running <s>four</s> one 24/7 VirtualBox Windows <s>7</s>10 guest for ARK Survivor <s>bots</s> server.
 +
** This is where the ZFS file system shines because it uses a SSD for ZIL/Cache to make read/writes super fast and deduplication means the four operating systems take up almost the space of only one.
 
<br>
 
<br>
  
==== '''<del>A.c - MythTV Frontend</del> (System not currently in use)''' ====
+
==== <del>A.c - MythTV Frontend </del> (System retired 2012) ====
 
<gallery>
 
<gallery>
 
Image:Asus_Pundit.jpg
 
Image:Asus_Pundit.jpg
 
Image:MCE-Remote-2-alt.jpg
 
Image:MCE-Remote-2-alt.jpg
 
</gallery>
 
</gallery>
 +
<strike>
 
* 2.4Ghz Celeron (Not for HD)
 
* 2.4Ghz Celeron (Not for HD)
 
* 80GB HD
 
* 80GB HD
Line 92: Line 113:
 
* MCE Remote with programmable buttons to control the TV.
 
* MCE Remote with programmable buttons to control the TV.
 
<br>
 
<br>
 +
</strike>
  
==== '''A.d - MythTV SD Frontend''' ====
+
==== <del> A.d - MythTV SD Frontend </del> (System retired 2011) ====
 
<gallery>
 
<gallery>
 
Image:Asus_Pundit.jpg
 
Image:Asus_Pundit.jpg
Line 99: Line 121:
 
Image:MCE-Remote-2-alt.jpg  
 
Image:MCE-Remote-2-alt.jpg  
 
</gallery>
 
</gallery>
 +
<strike>
 
* 2.4Ghz Celeron (Not for HD)
 
* 2.4Ghz Celeron (Not for HD)
 
* 80GB HD
 
* 80GB HD
Line 105: Line 128:
 
* [http://www.mythbuntu.org/ MythBuntu 10.4]
 
* [http://www.mythbuntu.org/ MythBuntu 10.4]
 
* Sony Trinitron 32" Flat Screen CRT
 
* Sony Trinitron 32" Flat Screen CRT
* Goes into hibernation after 3 hours using sleepd <s>(svideo doesn't recover correctly on resume!?)</s>
+
* Goes into hibernation after 3 hours using sleepd (svideo doesn't recover correctly on resume!?)
 
* MCE Remote with programmable buttons to control the TV.
 
* MCE Remote with programmable buttons to control the TV.
 
* This system connects through a hard-to-find Dlink DAP-1555 wireless bridge.
 
* This system connects through a hard-to-find Dlink DAP-1555 wireless bridge.
 
<br>
 
<br>
 +
</strike>
  
==== '''A.e - MythTV HD Frontend (Virtual)''' ====
+
==== '''A.e - MythTV HD Frontend (Virtual) - Primary Work Computer''' ====
[[Image:screenshot_of_MythTV_running_in_virtual_machine.jpg|none|thumb|650px|Screen shot of a six monitor setup, one running virtual MythBuntu]]
+
[[Image:screenshot_of_MythTV_running_in_virtual_machine.jpg|none|thumb|650px|Screen shot of a five monitor setup]]
* A virtual dual core VMware Workstation guest running on a host with a [http://products.amd.com/en-us/DesktopCPUDetail.aspx?id=640 six core 3.7Ghz AMD (overclocked 1090T)] on a [http://www.asus.com/Motherboards/AMD_AM3Plus/SABERTOOTH_990FX/ ASUS Sabertooth 990FX] with 16GB RAM (DDR3 1333Ghz) running stock Windows 7 off two 80GB SSD raided
+
* A virtual dual core <s>VMware Workstation</s> Virtualbox guest running on a host with an [http://products.amd.com/en-us/DesktopCPUDetail.aspx?id=809 eight core 4Ghz AMD FX-8350] (cooled with a Noctura NH-D15) on a [http://www.asus.com/Motherboards/SABERTOOTH_990FX_R20/ ASUS Sabertooth 990FX R2] with 32GB RAM (Kingston ECC DDR3 1333Ghz) running stock Windows 10<s>7</s> off a <s>80GB</s> 256GB SSD and the <s>VMware</s> Virtualbox guests off a Samsung 840 <s>240GB</s> 500GB SSD.
* virtual 6GB HD
+
** After using VMware for nearly a decade, I started using Virtualbox on my linux systems and found it fast and stable.  After upgrading to the 49" 4K LG panel and trying to spread the VMware Windows 7 guest across that and three other 24" 1920x1080 monitors in portrait mode, the guest started to stutter, making the operating system unusable.  Videos wouldn't play and the whole guest was awfully slow.  I tried the VirtualBox guest and it was an immediate improvement.  VB also has more bells and whistles, for example, you can pause guests (can't with VMware Player) which is very useful when putting the computer to sleep for the night.  Virtualbox also makes use of Nested Paging available in the latest AMD and i7 processors which makes the guests faster.  Virtualbox is also open source, updates faster, and has a deeper support community available online.  I have converted all my virtual machines to Virtualbox, will be uninstalling VMware and likely never looking back.  Windows 10 VirtualBox guests can play HD video flawlessly now. 
* virtual 500MB Ram
+
* Belkin 1500VA UPS <s>with Bulldog software</s> with Windows software for shutdown (includes replacing the batteries on a roughly three year cycle.)
* [http://www.mythbuntu.org/ MythBuntu 10.4]
+
* virtual 9GB HD
* 1900x1080 27" Monitor
+
* virtual 1GB Ram
* Both SD and HD 720p/1080i work great under VMware Workstation! <s>(still) stutters but it at least %90 watchable now.</s> (using extra audio buffering and libmpeg2 with xv-bit)
+
* [http://www.ubuntu.com/desktop Ubuntu Desktop 12.04]
* [http://www.desktopped.com/2010/08/four-monitor-portrait-array-that-emulates-a-52-display/ Here is a picture of the setup]
+
* <s>2560x1600 30" DELL Monitor</s> - 4160x2160 49" LG 49UB8200
 +
** Ok, so it runs at 30Hz and has a 4:2:0 chromakey so tiny text isn't perfect, but wow, does it display graphics well, and play games great.  <s>The GTX 760 can handle it fine at 30Hz in BF4 at full 4K resolution</s>. Err, no games are not great, even with a GTX <s>770</s> 780, it is laggy, so I play in window mode at a slightly lower resolution.
 +
*** I now do most of my reading on the 30" Dell in portrait mode.
 +
* Both SD and HD 720p/1080i work great under Virtualbox <s>VMware Player 6.0</s> <s>Workstation! (still) stutters but it at least %90 watchable now.</s>  
 +
* Here is a picture of the setup, there are
 +
** <s>Five</s> Four Acer V243H Ajbd 24" 1080x1920px portrait monitors on an Ergotron LX Triple Display (monitors numbered #3,5,6,4 in the picture below)
 +
*** Ergotron sent me a fourth mount that slid onto the aluminum bar.  In 2010, the setup was the highest pixel count for the lowest money.
 +
** 30" DELL 1600x2560px portrait mode on an Ergotron arm (#2)
 +
** LG UB8200 49" 4K 4096x2160px IPS LED monitor on a wall mount (#1)
 +
[[Image:six_monitor_setup_mrsdonovan_five_portrait.jpg|none|thumb|650px|Screen shot of a six monitor setup]]
 +
* Two of the Acers are connected to a GTX 645, another two to a GTX 460 <s>two Asus EN210 1GB passively cooled cards</s> and the Dell and LG monitors are connected to a EVGA <s>GTX760</s> GTX770.
 +
** The GT210s required old legacy drivers no longer being updated and it was time for an upgrade (the GTX645 cost me $25! )
 +
* All the monitors, speakers, battery chargers, shaver (yes it's a shaver) and headphone charger are controlled by a single switch DYI'd into the base of the Ergotron (it's in front of the green toy car in the photo). I put the computer to sleep, hit one button and have no phantom power draws.
 +
* I spend about $1K CND a year on the three systems in upgrades, although in 2015 it was closer to $3K (because of a testing computer for work and the 4K monitor).  Since 2007, on three separate systems, the testing machine, MythBackendHD (A.b and backup computers) and my primary, in total I have spent $8K, although most of it is a write-off with parts purchased off Kijiji.
 
<br>
 
<br>
  
Line 124: Line 161:
  
 
=== System B ===
 
=== System B ===
This system lives at my brother's house although I maintain it and it is a separate system.
+
This system <s>lives</s> died at my brother's house <s>although I maintain it</s>.
  
==== '''B.a - MythTV HD Frontend''' ====
+
==== <del> B.a - MythTV HD Frontend </del>  (System Died 2014) ====
 
<gallery>
 
<gallery>
 
Image:Silverstone_HTPC_Case_LC13BC.jpg
 
Image:Silverstone_HTPC_Case_LC13BC.jpg
Line 133: Line 170:
 
Image:Rumblepad_2_wireless.jpg  
 
Image:Rumblepad_2_wireless.jpg  
 
</gallery>
 
</gallery>
 +
<strike>
 
* INTEL 2.8Ghz P4
 
* INTEL 2.8Ghz P4
 
* 1.5GB of RAM
 
* 1.5GB of RAM
Line 140: Line 178:
 
* MythGame with SDLMame emulation and a wireless Logitech Rumblepad 2
 
* MythGame with SDLMame emulation and a wireless Logitech Rumblepad 2
 
* Goes into hibernation after 3 hours using sleepd
 
* Goes into hibernation after 3 hours using sleepd
 +
</strike>
 
<br>
 
<br>
 +
Computer died and they now use a Boxee to access a humungous disk on an Windows machine <s>the video RAID on backend server(B.B)</s>
  
==== '''B.b - MythTV HD Backend / Asterisk ''' ====
+
==== B.b - <del>MythTV HD Backend / Asterisk </del> (System Re-purposed 2015) ====
 
<gallery>
 
<gallery>
Image:Samsung_LT-P266W.jpg
 
Image:MCE-Remote-2-alt.jpg
 
Image:PVR150.jpg
 
Image:Rumblepad_2_wireless.jpg
 
 
Image:Digium_DGM_TDM_card_side.jpg
 
Image:Digium_DGM_TDM_card_side.jpg
 
Image:Belkin_F6C900.jpg
 
Image:Belkin_F6C900.jpg
 
</gallery>
 
</gallery>
* Quad Core 2.5Ghz Intel running on Mythbuntu 10.4 LTS (overkill, but wow, responsive)
 
* 700GB of RAID 5 for videos/pictures/music/ROMs
 
* 2.5TB LVM for recordings.
 
* 500GB for backups of other computers.
 
* Haupauge PVR-150 connects to a cheap ass Motorola tuner and is controlled via an IR blaster controlled by the USB MCE remote receiver.
 
* Digium Wildcard TDM400P with three FXS expansion cards
 
* [http://www.freepbx.org/ Freepbx] with Asterisk 1.6
 
* Samsung LT-P266W 26" HDtv flat-panel LCD
 
* Belkin 900VA UPS with Belkin Bulldog software
 
  
 +
* <deL>Quad Core 2.5Ghz Intel running on Mythbuntu 10.4 LTS (overkill, but wow, responsive)</del>
 +
* <del>1.3TB of RAID 5 for videos/pictures/music/ROMs</del>
 +
* <strike> 2.5TB LVM for recordings.</strike>
 +
* <strike>500GB for backups of other computers.</strike>
 +
* <del>Haupauge PVR-150 connects to a cheap ass Motorola tuner and is controlled via an IR blaster controlled by the USB MCE remote receiver.</del> Family uses Netflix now and has no cable provider.
 +
* <del>Digium Wildcard TDM400P with three two one (damn lightning) FXS expansion cards</del>
 +
* <del>[http://www.freepbx.org/ Freepbx] with Asterisk 1.6</del>
 +
* <strike>Samsung LT-P266W 26" HDtv flat-panel LCD</strike> Headless
 +
* <del>Belkin 900VA UPS with Belkin Bulldog software</del>
 +
 +
This system <del>is only used as an Asterisk box now.</del> is a gaming machine for the kids now.  Everyone uses their cell phones<s> and the home phone number goes straight to voicemail which is then forwarded via email.  The Asterisk system is actually not even in the same house, but System A.b above.</s>
  
 
=== Notes ===
 
=== Notes ===
  
 +
* Given three of the systems (A.a, A.b and B.b) are on 24/7, they each have a UPS rated at 900VA or better, which lasts an average of twenty minutes before cleanly shutting down (important for the RAIDed drives).
 +
* We currently use <del>link2voip.com</del> The Spout (.ca) as our DID and outbound provider for Asterisk.
 +
* Most of our machines use gigabyte Ethernet and the System A (A.a and A.b) route through a <s>$40 [http://www.amazon.com/BUFFALO-AirStation-HighPower-Wireless-WHR-300HP/dp/B0096239EC Buffalo WHR-300HP]</s> (proved buggy) $80 [http://www.asus.com/ca-en/Networking/RTN16/ Asus RT-N16] running the open source firmware, [http://www.dd-wrt.com/ DD-WRT] <del>[http://www.polarcloud.com/tomato Tomato]</del> (Tomato isn't updated enough and has some intermittent bugs).  There have been a lot of gateways in this house but the open source DD-WRT is by far the best.  It handles QoS properly for VoIP and has all the bells and whistles every other gateway lacks and a consistent interface across multiple devices.  We also use OpenDNS for all the DNS queries.
 +
* We use most of the default plugins including <del>MythGame</del> (Who am I kidding, kids game on iPhone/Pads these days), pictures (rarely), videos, <del>weather</del> and [[User:Mrsdonovan#Caller_ID |XYAC]] for CallerID.
 +
* <del> Both Asterisk boxes are connected to each other over an IAX2 link that works well through NAT and our mutual ISPs. </del>
 +
* <s>On System A, we also run two TrendNet (TV-IP851WIC and TV-IP422W) cameras for security that record any motion detected to A.a</s> - Giving up on remote cameras, they can't connect to new 802.11n routers, break and stop working much too frequently.
 
* <del>The link between the MythTV frontends [[User:Mrsdonovan#B_-_MythTV_HD_Frontend_.2F_Asterisk |B]], [[User:Mrsdonovan#D_-_MythTV_Frontend |D]] and the master and slave backends [[User:Mrsdonovan#A_-_MythTV_Master_backend.2F_HD_Frontend_.2F_Asterisk_server |A]], [[User:Mrsdonovan#C_-_MythTV_HD_Slave_Backend_.2F_Frontend |C]] is over a 5.8Ghz 802.11a wireless bridge using dual-band bridges from Trendnet (TEW-510apb) and one 21dbi and one 9dbi gain antenna.  Average throughput for 1080p HD is about 13mbps and during high motion scenes, tops out at 19mbps.  With the high gain antennas, the TEW-510apbs can operate at their maximum throughput of 21Mbps and HD is smooth. On rare occasions, the link is affected by thick ice on the roof and really heavy rain.</del>  (System B is now separate)
 
* <del>The link between the MythTV frontends [[User:Mrsdonovan#B_-_MythTV_HD_Frontend_.2F_Asterisk |B]], [[User:Mrsdonovan#D_-_MythTV_Frontend |D]] and the master and slave backends [[User:Mrsdonovan#A_-_MythTV_Master_backend.2F_HD_Frontend_.2F_Asterisk_server |A]], [[User:Mrsdonovan#C_-_MythTV_HD_Slave_Backend_.2F_Frontend |C]] is over a 5.8Ghz 802.11a wireless bridge using dual-band bridges from Trendnet (TEW-510apb) and one 21dbi and one 9dbi gain antenna.  Average throughput for 1080p HD is about 13mbps and during high motion scenes, tops out at 19mbps.  With the high gain antennas, the TEW-510apbs can operate at their maximum throughput of 21Mbps and HD is smooth. On rare occasions, the link is affected by thick ice on the roof and really heavy rain.</del>  (System B is now separate)
* Given three of the systems (A.a, A.b and B.b) are on 24/7, they each have a UPS rated at 900VA or better, which lasts an average of twenty minutes before cleanly shutting down (important for the RAIDed drives).
 
* We currently use link2voip.com as our DID and outbound provider for Asterisk with great ping times of 10 to 15ms.
 
* Most of our machines use gigabyte Ethernet and both systems use Linksys WRT54GL variants running the open source firmware, [http://www.polarcloud.com/tomato Tomato] or the mini DRT variant.  There have been a lot of gateways in this house but the open source Tomato is by far the best.  It handles QoS properly for VoIP and has all the bells and whistles every other gateway lacks.  We also use OpenDNS for all the DNS queries.
 
* We use most of the default plugins including MythGame, pictures, videos, weather and [[User:Mrsdonovan#Caller_ID |XYAC]] for CallerID.
 
* Both Asterisk boxes are connected two each other over an IAX2 link that works well through NAT and our mutual ISP.
 
* On System A, we also run two TrendNet (TV-IP252 and TV-IP422W) cameras for security that record any motion detected to A.a
 
  
 
== Tips from the HD Front Lines ==
 
== Tips from the HD Front Lines ==
I have been using MythTV with HD now for a couple years and wanted to share what has worked and what has not.  These comments could save you a lot of time and frustration:
+
[Updated June, 2015] I have been using MythTV with HD now for <s>a couple</s> a decade and wanted to share what has worked and what has not:
  
'''Firewire cards''' - get the best firewire cards that [[Firewire_Cablebox_Compatability |works with your tuner(s)]].  I have used NEC, VIA (onboard and PCI) cards and now finally, Agere cards.  Agere cards, the lastest chip revisions anyway, are by far the best with the Motorola DCT-62xx series boxes.  Ironically, "test-mpeg2" does not work making any priming script useless, but "broadcast" mode works pretty much flawlessly and fast!  You do still have to prime the firewire card, but I do it once at startup with the "firewire_tester -n 0 -B" script.
+
'''Firewire cards''' - <s>get the best firewire cards that [[Firewire_Cablebox_Compatability |works with your tuner(s)]].  I have used NEC, VIA (onboard and PCI) cards and now finally, Agere cards.  Agere cards, the lastest chip revisions anyway, are by far the best with the Motorola DCT-62xx series boxes.  Ironically, "test-mpeg2" does not work making any priming script useless, but "broadcast" mode works pretty much flawlessly and fast!  You do still have to prime the firewire card, but I do it once at startup with the "firewire_tester -n 0 -B" script.
  
VIA firewire chipsets will work, as they have for over a year with my system in P2P mode, but it sucks.  VIA chips require a priming script to run each time a channel change occurs and have two negative effects - first, because the priming is done at each channel change, channel surfing during Live TV times-out 2/3rds of the time.  Recordings seem to work %100 of the time.  Secondly, the priming script + VIA chipset + DCT-6200 crashed the DCT-6200 tuner when tuning to SD channels (seriously! Mine would hard lock and have to be unplugged).  I have no clue why because the same combination but with a DCT-6214 worked fine.  The tuner hard lock also doesn't happen on HD channels.
+
VIA firewire chipsets will work, as they have for over a year with my system in P2P mode, but it sucks.  VIA chips require a priming script to run each time a channel change occurs and have two negative effects - first, because the priming is done at each channel change, channel surfing during Live TV times-out 2/3rds of the time.  Recordings seem to work %100 of the time.  Secondly, the priming script + VIA chipset + DCT-6200 crashed the DCT-6200 tuner when tuning to SD channels (seriously! Mine would hard lock and have to be unplugged).  I have no clue why because the same combination but with a DCT-6214 worked fine.  The tuner hard lock also doesn't happen on HD channels.</s>  - Just use an onboard firewire for changing channels and get a:
 +
 
 +
'''Hauppage HD PVR''' - Gamers use this box to record their gaming without loading their computer down and you can find older versions for half price ($80CND) or better in second hand markets.  HD PVR boxes take in the analog signal (blue, red and green cables) and convert it to a USB which is recognized by the HDPVR driver in Linux.  Works great with MythTV and both our HD tuners now use an HD-PVR.  It is also immune to the "5C" encoding that Cable TV providers randomly flip on.
 
<blockquote>
 
<blockquote>
[[Image:Information.png|20px]] ''Conclusion'' - With DCT-62xx tuners, get Agere-5t FW323 v61 firewire chips, dump the priming script and use Broadcast at 200mbps.
+
[[Image:Information.png|20px]] ''Conclusion'' - With DCT-62xx tuners, use firewire to change channels with the 6200ch script and an Hauppauge HDPVR to capture the HD video.
 
</blockquote>
 
</blockquote>
  
'''Video cards''' - As XvMC support is going to be discontinued in MythTV 0.25 and replaced with VPDAU which supports a much large set of codecs.  NVIDIA has perhaps the best support and look for a card that uses "PureVideo HD" and perhaps the cheapest is the GeForce 210 which works great.
+
'''Video cards''' - As XvMC support is going to be discontinued in MythTV 0.25 and replaced with VPDAU which supports a much large set of codecs.  NVIDIA has perhaps the best support and look for a card that uses "PureVideo HD".
 
<blockquote>
 
<blockquote>
[[Image:Information.png|20px]] ''Conclusion'' - Get a Geforce 210 based video card and HD will work flawlessly with the OSD using VDPAU.
+
[[Image:Information.png|20px]] ''Conclusion'' - Get a Geforce 430 based video card and HD 1080p will work flawlessly with the OSD using VDPAU.
 
</blockquote>
 
</blockquote>
  
'''Processor''' - Get the largest and fastest one for HD because it makes a difference.  Everything is magnified by two with HD content, which means it takes more processor power to do playback, commercial detection and transcoding.  A faster processor also helps with database queries.  (As an aside, I also suggest looking into [[User:Yeffetn#Making_sure_mythconverg_.28mysql_db.29_is_in_memory_all_the_time | keeping the database in memory]])
+
'''Processor''' - Get the largest and fastest one for HD because it makes a difference.  Everything is magnified by two with HD content, which means it takes more processor power to do playback, commercial detection and transcoding.  A faster SSD <s>processor</s> also helps with database queries.  (As an aside, I also suggest looking into [[User:Yeffetn#Making_sure_mythconverg_.28mysql_db.29_is_in_memory_all_the_time | keeping the database in memory]])
 
<blockquote>
 
<blockquote>
 
[[Image:Information.png|20px]] ''Conclusion'' - Faster = better.
 
[[Image:Information.png|20px]] ''Conclusion'' - Faster = better.
Line 196: Line 236:
 
[[Image:Information.png|20px]] ''Conclusion'' - Get the Microsoft MCE remote and train it to control your TV as well as your MythTV box.  One remote to control them all.
 
[[Image:Information.png|20px]] ''Conclusion'' - Get the Microsoft MCE remote and train it to control your TV as well as your MythTV box.  One remote to control them all.
 
</blockquote>
 
</blockquote>
'''Hard drive Size''' - On the main backend ([[User:Mrsdonovan#A_-_MythTV_Master_backend.2F_HD_Frontend_.2F_Asterisk_server |A]]) I started out with an LVM which tied a bunch of disparately sized disks together, but I wanted more redundancy and speed.  I tried RAID 5 but ... [UPDATE] I have switched back to RAID 5 and it seems to work fine - best use the latest kernel to get all the latest updates. ([[User_talk:Mrsdonovan |Others have reported success with RAID 5]])  Having said that, the slave backend ([[User:Mrsdonovan#C_-_MythTV_HD_Slave_Backend_.2F_Frontend |C]]) [UPDATE] now has RAID 1 redundancy and records HD to two SATA II drives which seems plenty fast enough.
+
'''Hard drive Size''' - On the main backend ([[User:Mrsdonovan#A_-_MythTV_Master_backend.2F_HD_Frontend_.2F_Asterisk_server |A]]) I started out with an LVM which tied a bunch of disparately sized disks together, but I wanted more redundancy and speed.  I have switched to <s>RAID 5 using the mdadm</s> ZFS RaidZ and it seems to work fine.  
  
Hard drive size really only translates into one thing - longer time to watch something before it expires.  For just recordings, our system has a combined space of about a terabyte which means our recordings last about a month.  Our system records, on average, about eighteen one hour shows a day on weekdays and upwards of thirty or more on weekends with a mix of everything from sports to movies to the daily kid's shows.  Lots of it never gets watched. Of those recordings we want to keep longer term, the best solution has been to transcode and move them into our video section that exists on a separate harddrive (An LVM of all the free space left over from the RAID actually).  That harddrive is then RSYNC'd each night from the main server ([[User:Mrsdonovan#A_-_MythTV_Master_backend.2F_HD_Frontend_.2F_Asterisk_server |A]]) to extra space on the slave backend ([[User:Mrsdonovan#C_-_MythTV_HD_Slave_Backend_.2F_Frontend |C]]).
+
Hard drive size really only translates into one thing - longer time to watch something before it expires.  For recordings, our system has a combined space of about a three terabytes which means our recordings last about a month.  Our system records, on average, about eighteen one hour shows a day on weekdays and upwards of thirty or more on weekends with a mix of everything from sports <s>to movies to the daily kid's shows</s>.  Lots of it never gets watched. Of those recordings we want to keep longer term, the best solution has been to transcode and move them into our video section <s>that exists on a separate harddrive (An LVM of all the free space left over from the RAID actually)</s>The videos are backup (along with a bunch of other stuff) once every other week to alternating backup machines that are awoken via WOL, boot via PXE Boot-On-Lan and do the backup.
 
<blockquote>
 
<blockquote>
 
[[Image:Information.png|20px]] ''Conclusion'' - Larger = Longer lasting.  RAID is your friend.
 
[[Image:Information.png|20px]] ''Conclusion'' - Larger = Longer lasting.  RAID is your friend.
 
</blockquote>
 
</blockquote>
  
==== Keeping HD In Sync When Transcoding ====
 
One of the benefits of MythTV is that once the shows are recorded, they can be manipulated in many different ways, for example, the commercials stripped out and the rest dumped onto a DVD or transcoded with an open source codec like Xvid.  Transcoding HD recordings from either over-the-air or firewire captures presents significant difficulties, mostly related to keeping the audio in sync with the video.
 
  
I have found, through experience, that an MPEG-2 TS file fresh from being recorded over firewire from a DCT-62xx tuner, and edited with something like [http://avidemux.org/ Avidemux2], will result in an audio sync that will get worse as the recording playsBy the end of the recording, what started as a 250ms sync, is now out by a couple of seconds.
+
==== Other Random Tips ====
 +
* FreePBX includes PHPMyadmin which is useful for optimizing the database(s) which saves space and makes them run faster.  PHPMyadmin also comes in handy for editing the database directly, like getting rid of commercial detection jobs if mythcommflag hangs, or reordering the tuners.
 +
* Concerning Asterisk, I tried ADSI, the ability to program phones with custom scripts, but some phones just didn't work because they are "locked" by their manufacturer.  The comedian mail app for ADSI works, but it's clunky and slow and it was much simpler just to autologin and play the messages.
 +
* Monitoring of the Myth systems work great using [http://en.wikipedia.org/wiki/GKrellM Gkrellm] and I highly recommend it. Works on Linux and Windows and looks great with a large variety of skins. [Update <s>November 2013</s> June 2015 - This is still true ten years later which is rare especially since the windows client hasn't been updated in over five years but it works on Windows 7, the last Windows OS I plan to buy.]
 +
* Get ECC RAM - Your MythTV backend is on all the time, and it is exactly these conditions that the ECC ram (*unbuffered*) was created for.  AMD desktop processors and related motherboards support ECC out of the box unlike Intel which require server grade (expensive) CPUs.
 +
* If you run a disk array like RAID or ZFS RaidZ, a Uninterruptable Power Supply is good at keeping the array "clean" for those unexpected power outages.  Use a "True Sine Wave" UPS if you can afford it and your computer power supplies will appreciate it.
 +
 
 +
== Personal Setup Notes ==
 +
Notes for my own setup:
 +
 
 +
==== Moving a DigitalOcean Droplet to Virtual Box ====
 +
Moving a Digital Ocean droplet into a local virtual machine is pretty complicated and in the end, I combined two instruction sets together ([https://static.scriptthe.net/public/ConvertDigitalOceanDroplettoVMwareVM.pdf PDF here] and another [https://nickgiotis.com/2017/06/13/howto-export-a-digital-ocean-droplet-to-vmware-workstation/ here]) to get it working.  I wanted to document the process here, should I ever need it in the future (or I might need to do it in reverse):
 +
* Add block storage to your droplet of a size greater than your droplet, i.e the drop is 20GB, add a 22GB droplet as it needs to store the "dd" image.
 +
* Turn off the droplet and under "Recovery" flip it into recovery mode and boot the droplet back up
 +
* Because your droplet isn't running, you can't use SSH, which means you have to use the browser console link provided by Digital Ocean, which has some complications mentioned below.
 +
* Once the recovery OS is running, drop into the command line (menu selection #6)
 +
* First "fdisk -l" to show the partitions and you should have a /dev/vda which contains the droplet and unpartitioned /dev/sda with the block storage - the difference in sizes will make it clear which is which
 +
** Optional - run "fsck -vfy /dev/vda1" to remove any errors from the droplet
 +
* "fdisk /dev/sda" then create a partition ("n" for new, t" for type 83 and "w" to write the changes) and format it to ext4 "mkfs.ext4 /dev/sda1"
 +
* Now mount the new partition
 +
<pre>
 +
cd /mnt
 +
mkdir tor (or whatever)
 +
mount /dev/sda1 tor
 +
cd tor
 +
</pre>
 +
* Once on the new partition of the block storage, then
 +
** A problem that popped up was that the console did not recognize the pipe character (but yped ">"), so it was necessary to programatically create it via the command line:
 +
*** with <pre>pipe=$(nawk 'BEGIN {printf("%c",124);}')</pre> This assigns the variable "pipe" with the ASCII equivalent of "|"
 +
** then <pre> eval dd if=/dev/vda1 $pipe gzip > vda1.img.gz </pre> which creates the image file on the block storage
 +
** "md5sum vda1.img.gz" create a text file with the mda5sum which is used to test it after download it.
 +
* Now you have a copy of the vda1.img.gz file on block storage, take the droplet out of recovery and boot the droplet back up.
 +
* Then SSH back into the droplet and
 +
<pre>cd /mnt/
 +
mkdir tor
 +
mount /dev/sda1 tor
 +
</pre>
 +
and "ll /dev/sda1" which should have the two files on it, the vda1.img.gz and the md5sum file.
 +
* I use Windows 10 and putty to access my droplets via a shared key and as part of that package, I use
 +
** <pre>pscp root@<your ip>:/mnt/tor/vda1.img.gz .</pre> will download the image file
 +
** I opened another window and downloaded the md5sum too  with "pscp root@<your IP>:/mnt/tor/vda1.img.gz.md5"
 +
** The pscp program uses the private certificate provided by the running pageant program for the SSH tunnel.
 +
* Once you have the vda1.img.gz file local, then unzip and convert it with the "qemu-img" tool (apt-get install qemu-utils" or just type in qemu-img and it will suggest what package needs to be download)
 +
** "md5sum vda1.img.gz" and check the hash against the one downloaded in the md5 file to make sure you have a perfect copy (the md5 file is just a text file with a short hash in it)
 +
** <pre> gzip -d vda1.img.gz </pre>  (unzips it)
 +
<pre> gemu-img convert -O vmdk vda1.img <mydropletname.vmdk> </pre>(converts it to a vmdk file)
 +
* Now create a new Ubuntu 64bit image with a new VDI harddrive image and also connect the vmdk file to the same VM.
 +
** I had significant problems with an vmdk  that VB refused to connect the VMIt likely means it was corrupt.
 +
* Boot into a rescue CD and drop to the command line
 +
* Again, use fdisk to create a partition on the new VDI image (likely /dev/sdb1) and then use dd to transfer the image from the vmdk
 +
<pre> dd if=/dev/sda of=/dev/sdb1 conv=notrunc,noerror </pre>(copies the dd image from the VDMK to the VDI)
 +
<pre> fsck -t ext4 /dev/sdb1 </pre> (check it for errors)
 +
Now mount /dev/sdb1
 +
<pre>
 +
cd /mnt
 +
mkdir tor
 +
mount /dev/sdb1 tor</pre>
 +
* Now remove the old grub and networking setups:
 +
<pre>
 +
cd /mnt/tor/boot/grub
 +
rm -rf *
 +
cd /mnt/tor/etc/network/
 +
nano interfaces
 +
</pre>
 +
* Remove all the hardcoded DO IP addresses and add your own:
 +
<pre>
 +
auto eth0
 +
iface eth0 inet dhcp
 +
</pre>
 +
* Now shutdown the VM, disconnect the VMDK and restart it again with the rescue CD
 +
** Recreate the grub directory ([help.ubuntu.com/community/Boot-Repair I used the Ubuntu "Boot Repair" iso])
 +
* If it works, it should come up asking for a login (no desktop interface of course)
 +
* Now you should be able to ssh into the box just as you had when it was on DO!
 +
* Profit!
 +
 
 +
This method has a number of benefits:
 +
* The doplet is off and in a known state when the dd image is taken
 +
* the dd image is copied to a native VDI virtualbox harddrive image
 +
* the networking and grub are freshly scrubbed for Virtualbox
 +
* the ssh keys still work.
 +
 
 +
==== Biweekly Backup Script ====
 +
The A.b server has two backup servers that are woken up via Wake-On-Lan packets, which then boot via PXE.  The script the weekly backup cron job calls is this one on A.b:
 +
 
 +
<pre>
 +
/bin/bash
 +
#Now wake up the backup computer and exit, as the script there now  
 +
# takes over and does the backup and then shuts down.
 +
mark_file=/diskless/c1/mnt/right-side-computer-next
 +
 
 +
# check whether the job ran last week
 +
echo "Checking to see if the markfile exists"
 +
if [ -e $mark_file ] ; then
 +
rm -f $mark_file
 +
echo "Mark file found, backing up to right side computer"
 +
wakeonlan 00:EE:DD:FF:11:22 # The backup computer on the right
 +
else
 +
echo "Mark file not found, backing up to the left side computer"
 +
touch $mark_file
 +
wakeonlan 00:33:44:55:66:77 # The backup computer on the left
 +
fi
 +
echo "Done - Exiting"
 +
</pre>
 +
 
 +
It uses a file "right-side-computer-next" as a flag to alternate between the two backups, running every other backup once a week.  It uses the "wakeonlan" program available through Ubuntu.
 +
 
 +
===== Diskless Client Side Script =====
 +
On the diskless client side, once it is woken up, loads and runs the kernel from A.b (which was slightly tricky because the A.b NAS is an AMD CPU and the backup machines are both Intel P4s), it then runs this script:
 +
<pre>
 +
#/bin/bash
  
There are two problems related to keeping the audio and video in sync, the first are errors in the stream, which over time create small differences.  Unlike VLC which constantly adjusts the sync based on time codes, editors like Avidemux do not.  The other problem is commercials, which not only interrupt the MPEG-2 stream, but do other nasty things like change framerates and drop to lower resolutions.
+
set -o nounset
  
Looking over hundreds of forums, the consensus has been to pre-process the stream with ProjectX, an open source Java utility, which, for the longest time, worked.  I would edit out the commercials with ProjectX's slow interface, demux the final cut, load it into Avidemux, load the filters and transcode.  I like [http://avidemux.org/ Avidemux] because it is open source, has a slick interface and rips to a wide array of formats with lots of filters, like cropping, deinterlacing, framerate and resizing. Moreover, it's greatest feature is that it runs on both Windows and Linux which means the many hour transcodes can occur, as a background process, on our 24/7 Linux boxes. Unfortunately, Avidemux happens to be terrible at decoding raw MPEG-2 streams and isn't the most stable of programs.  
+
STARTTIME=$(date +%s)
 +
# Mount the drive
 +
mount /dev/sda1 /mnt/localdisk
 +
# Mount the partition to backup from
 +
mount 192.168.x.yyy:/bigtank /mnt/bigtankremote
  
However, ProjectX has recently not been able to deocode the MPEG-2 streams nor can repair utilities like mpeg2repair fix them.  I was stuck.
 
  
The great part is that [[User:Steveadeff#HR.HDTV_script |another MythTV user created a script to rip HD]], that, for some reason, keeps the audio in sync. Unfortunately, the script only works on raw MPEG-2 streams which means it doesn't run on the transcoded commercial free recordings that MythTV can create.  Besides doing the commercial detection automatically, MythTV has a fast interface for fixing the cut points and it is suppose to be able to do a straight lossless MPEG-2 to MPEG-2 copy.  Unfortunately, because [http://svn.mythtv.org/trac/ticket/2077 of a long outstanding bug with mythtranscode], it doesn't work. 
+
# Now start rsync
 +
TDATEIS=$(date +"%A-%B-%d-%Y")
 +
OUTPUT="Backup Slave Results for $TDATEIS \n----------------------------\n----------------------------\n "
 +
OUTPUT="$OUTPUT First backup all the videos if any have changed...\n-> But check to see if the mount is valid:"
  
If mythtranscodes's lossless copy did work, the whole process could be automated with a few clicks of the remote:
+
if [ "$(grep 192.168.x.yyy /etc/mtab)" ];
 +
then
 +
OUTPUT="$OUTPUT \nNow do the sync, deleting files from the backup not in the original"
 +
# -  Make sure to exclude the thumbcache on the destination generated
 +
        #    by MythTV or they will be deleted every night when sync'd.
 +
# Add "-vrun" to test the backup
  
#Manually adjust any cut points (if necessary) from what MythTV has detected
+
# First videos
#Tell MythTV to do a lossless MPEG-2 to MPEG-2 transcode which removes the commercials
+
RSYNC_RESULTS=`rsync -vru --delete  /mnt/bigtankremote/video /mnt/localdisk`
#Tell MythTV to run the user HD script to make an Xvid archival copy, dumping the result in a certain spot to be renamed and moved into an appropriate directory later.
+
OUTPUT="$OUTPUT \n Video Backup Results:\n---------------------------------------\n $RSYNC_RESULTS"
 +
# Then seansNAS
 +
RSYNC_RESULTS_TWO=`rsync -vru --delete --exclude-from /mnt/do_not_copy_list.txt /mnt/bigtankremote/seansNAS /mnt/localdisk`
 +
OUTPUT="$OUTPUT \n seansNAS Backup Results:\n-------------------------------------\n $RSYNC_RESULTS_TWO"
 +
else
 +
OUTPUT="$OUTPUT \nIt looks like the video/music/pictures are not mounted???"
 +
fi
  
Without step 2, you have to run the HD script on the raw recording with the commercials included, load the result into Avidemux and manually find and cut the commercials.  There is still an audio video sync problem, but I have found that whatever mencoder does, the audio delay is a straight 250ms delay, consistent from recording to recording and constant across the entire recording.  The other good news is that if you don't have to crop anything, the Xvid to Xvid copy is fast, taking less then a minute for a two hour long show.
+
OUTPUT="$OUTPUT \n-------------------------\nDone"
 +
DISKSPACE_LEFT=`df -h /dev/sda1`
 +
OUTPUT="$OUTPUT \nDisk Space Left:\n $DISKSPACE_LEFT"
 +
ENDTIME=$(date +%s)
 +
OUTPUT="$OUTPUT \nTotal backup time: $(($ENDTIME - $STARTTIME))s"
  
 +
echo -e "$OUTPUT" > /mnt/backup_status.txt
 +
 +
sleep 60
 +
shutdown -h now
 +
 +
</pre>
 +
 +
This script does the following:
 +
* Mounts the local harddrive (a GPT formated EXT4 3TB drive) and the A.b ZFS partition to be backed up
 +
* Checks to see if the mounts were successful
 +
** The IP address of A.b should be listed in mtab, otherwise, rsync would try to dump nearly 3TB of data onto A.b's 120GB SSD!
 +
* Does the backups using rsync
 +
** The do_not_copy_list.txt lists directories and file patterns that should be skipped, i.e. the .TIB files from Acronis, as backups of backups are not necessary.
 +
* Report how much diskspace is left on the local 3TB drive.
 +
* Then write the backup results to backup_status.txt
 +
** I couldn't get the diskless client to properly access the internet and postfix is a pain to setup: all just to send an email, so the script writes a file to the SSD and I have another cronjob on the A.b NAS which looks for the file and emails it is there.  Not elegant, but time effective.
 +
* The diskless client sleeps for a minute to give me time to stop it from the diskless client if I need to check something.
 +
* And shuts down the diskless client.
 +
 +
It works pretty elegantly so far and has significant benefits:
 +
* The backup computers are pretty much off which means they are:
 +
** less exposed to power surges
 +
** and being hacked
 +
** don't cause any wear on components (RAM and HDs specifically)
 +
** and consume a trickle of energy.
 +
* The backup harddrives can also be swapped out with the ones in A.b should any of those go down and the ZFS Raidz rebuilt while another HD is sourced (waiting for a sale if necessary).
 +
* Accidentally deleted files are recoverable up until the second HD synchronizes which is a two week window.
 +
* A diskless client means less maintenance because
 +
** it doesn't need a kernel running from a disk (which means no backups of said kernel)
 +
** nor does the kernel have to be updated (which breaks things)
 +
 +
If the system has a weakness, it is the system disk on A.b, however it is an SSD which is more reliable and imaged and stored separately.  None of the backups are offsite either, however the volume of data is much too large to warrant making that feasible.  The entirety of the ZFS array is also not backed up, however, it is made up of roughly two parts, changing data (a revolving cache of MythTV recordings) and static data (videos, pictures, NAS data including Windows backups, etc.)
 +
 +
==== Hauppauge HD PVR Audio Sync ====
 +
The 1445 model of the HD PVR has an audio sync problem (with 0x15 firmware version and up, [http://code.mythtv.org/trac/ticket/10007 bug documented here]) and the fix is to add:
 +
<pre>export FORCE_DTS_TIMESTAMPS=1</pre>
 +
to /etc/mythtv/session-settings ([http://ubuntuforums.org/archive/index.php/t-2054422.html thanks!]) on each Myth Frontend.
 +
 +
Updating the HD-PVR firmware likely works too if you have a newer kernel (3.3 or better)
 +
 +
===== Running two HDPVRs on one machine =====
 +
 +
Upon reboot, or having the HDPVRs power cycled, the /dev/video<x> designation for the two is completely random.  It is necessary to modify /etc/udev/rules.d/78-hdpvr.rules to include:
 +
<pre>
 +
#HDPVR - Hdpvr - Lowdef
 +
KERNEL=="video[0-9]",ATTRS{product}=="Hauppauge HD PVR",ATTRS{serial}=="00A8C18F",SYMLINK+="videoSD1"
 +
KERNEL=="video[0-9]",ATTRS{product}=="Hauppauge HD PVR",ATTRS{serial}=="00A4D9C0",SYMLINK+="videoHD1"
 +
</pre>
 +
 +
where the file name "78-hdpvr.rules" specifies the load time, "78" which is after "60" which is important and then the rest "-hdpvr.rules" is arbitrary.  The serial numbers are taken from a "dmesg" command in a section like this:
 +
 +
<pre>
 +
[ 6307.727781] usb 1-3: Product: Hauppauge HD PVR
 +
[ 6307.727783] usb 1-3: Manufacturer: AMBA
 +
[ 6307.727784] usb 1-3: SerialNumber: 00A49D0D
 +
</pre>
 +
 +
There is more information [https://www.mythtv.org/wiki/Device_Filenames_and_udev in the wiki] including a useful command that tests the udev rules:
 +
 +
 +
After getting the udev commands working I learned of another simpler way that likely works better:
 +
 +
<pre>
 +
Ubuntu 12lts now has explicit entries for each HDPVR under '/dev/v4l/by-id' and '/dev/v4l/by-path' and that's how I reference them in mythtv-setup </pre>
 +
 +
===== LIRC from two HDPVRS =====
 +
 +
And because you have two HDPVRs, you likely have two LIRC blasters, which show up when "modprobe lirc_zilog" is run like this (in dmesg):
 +
 +
<pre>
 +
lirc_zilog: Zilog/Hauppauge IR driver initializing
 +
lirc_zilog: probing IR Tx on Hauppage HD PVR I2C (i2c-3)
 +
i2c i2c-3: lirc_dev: driver lirc_zilog registered at minor = 1
 +
lirc_zilog: IR unit on Hauppage HD PVR I2C (i2c-3) registered as lirc1 and ready
 +
lirc_zilog: probe of IR Tx on Hauppage HD PVR I2C (i2c-3) done
 +
lirc_zilog: probing IR Tx on Hauppage HD PVR I2C (i2c-2)
 +
i2c i2c-2: lirc_dev: driver lirc_zilog registered at minor = 2
 +
lirc_zilog: IR unit on Hauppage HD PVR I2C (i2c-2) registered as lirc2 and ready
 +
</pre>
 +
 +
Again, you need a 78-hdpvr-lirc.rules like this:
 +
 +
<pre>KERNEL=="lirc[0-9]", ATTRS{product}=="Hauppauge HD PVR", ATTRS{serial}=="00A8C18F", SYMLINK+="lircSD1", MODE="0666</pre>
 +
 +
Useful commands for testing if your udev rules are working are:
 +
<pre>udevadm info -a -p $(udevadm info -q path -n /dev/lirc1)
 +
udevadm test /devices/pci0000:00/0000:00:13.2/usb2/2-5/i2c-3/lirc/lirc1
 +
</pre>
 +
(make sure the "i2c-3/lirc/lirc1" matches the output from the zilog in dmesg)
 +
 +
==== VNC server Xstartup ====
 +
 +
Getting a headless vnc server to run correctly took a bit of work and most of it came down to the /home/<user>/.vnc/xstartup which in my case was:
 +
 +
<pre>
 +
#!/bin/sh
 +
 +
# I don't think this line is used given it is for Windows Remote desktop software, XRBD which is not installed.
 +
xrbd $HOME/.Xresources
 +
xsetroot -solid grey
 +
 +
# Added this for some reason, now forgotten
 +
export XKL_XMODMAP_DISABLE=1
 +
 +
# Absolutely necessary for the dbus message bus to work when apps like update-manager,
 +
# etc. require elevated privileges  (Took me three hours to figure why pressing "install"
 +
# on the Ubuntu software Center did nothing!)
 +
# [Update April 2015] $%&@$ bloody thing - I spent another five hours trying to fix this
 +
# problem again on Unity and it turns out, it is by design!  WTF?
 +
# Remote RDP VNC sessions are not considered "active" by Ubuntu/Unity, only desktop sessions
 +
# and therefore you do not get the privilege elevation pop up.  More annoyingly, it fails silently,
 +
# as pressed buttons simply do nothing.  Even the log errors that update-manager, synaptic and software-center
 +
# poop out are about dbus introspection failing, which is a red herring.  The polkit-agent simply refuses to run!
 +
# And it doesn't have to be this way as I have a perfectly working system with xfce4 running the same linux core.
 +
# Works around are supper messy including hand-editing the policykit default XML files or PAM.
 +
# I end up running synaptic, the update-manager from the command line, which means I end up forgetting the
 +
# bloody name of the programs: Was it gnome-update-manager or update_manager or ishouldnothavetorememberthis.
 +
# Yes a bit annoyed.
 +
unset SESSION_MANAGER
 +
unset DBUS_SESSION_BUS_ADDRESS
 +
 +
# Kicks off the gnome 2d desktop which includes Unity
 +
# - need some packages installed for this to work, gnome-core??
 +
gnome-session --session=ubuntu-2d &
 +
</pre>
 +
 +
To control the size, start the server with "vncserver -geometry 2500x1500" which looks good on a 2560x1600 monitor.  Need "vncserver" (vnc4?) installed - avoiding Xtightvnc for now as consistently lost connect after a while.
 +
 +
==== Kernel devinput instead of LIRC ====
 +
With Ubuntu 12.04 64 bit, the drivers for the remotes are now in the kernel and LIRC, at least in my case, wasn't needed.
 +
 +
Here is how I moved from LIRC to devinput:
 
<blockquote>
 
<blockquote>
[[Image:Information.png|20px]] ''Conclusion'' - if you want to transcode an HD recording that you cannot edit with ProjectX use the following process:
+
1. apt-get install [http://manpages.ubuntu.com/manpages/natty/man1/ir-keytable.1.html ir-keytable], then run ir-keytable which returns a description of the receiver capabilities, i.e:
 +
<pre>
 +
Found /sys/class/rc/rc0 (/dev/input/event4) with:
 +
  Driver mcesub, table rc-rc6-mce
 +
  Supported protocols: NEC RC-5 RC-6 JVC SONY LIRC other
 +
  Enabled protocols: NEC
 +
  Repeat delay = 500ms, repeat_period = 125ms
 +
</pre>
 +
2. using 'ir-keytable -t' (replaces the old "irw") I found that my remote was being recognized by the MCE USB receiver, because for each keypress, it produced something like (pressing the Enter key for example):
 +
<pre><some long time stamp>: event MSC: scancode = 45bc44</pre>
 +
3. A grep of all the remotes for that scancode in /lib/udev/rc_keymaps returned no results which meant my remote had not been recorded.
 +
 
 +
4. I created a new keymap file under /etc/rc_keymaps with the following necessary format:
 +
<pre>
 +
==> NEC <==
 +
# Comments go here
 +
0x45bc44 KEY_ENTER
 +
...
 +
</pre>
 +
 
 +
5. The location of the file isn't important as you reference it directly, but the format of it is, i.e. having the NEC or RC6, etc.
 +
 
 +
6. Then configured it in rc.local with:
 +
<pre>ir-keytable -c --period=500 -p NEC -w /etc/rc_keymaps/<name of your remote></pre>
 +
where
 +
<pre>
 +
"-c" clears the previous keytable,
 +
"--period" (or "-P") sets the repeat period to slow down repeats
 +
"-p" enables just the NEC protocol (or RC-6, SONY, etc) and
 +
"-w" adds the keys from the keytable file.
 +
</pre>
 +
7. Pressing the key on the remote then gets passed to MythTV directly as if the remote was like a keyboard, sweet!  No .lirc/mythtv configuration or LIRCD daemon to crash, nor worrying about which /dev/input/eventxxx the receiver is plugged into.
 +
* To stop the remote repeating too quickly, I ran the "Keyboard" app under Unity to change the "Delay" from short to long and the "Speed" to slow.
 +
 
 +
 
 +
8. All this was figured out after a couple hours of googling and trial and error.  Lots of people still use the kernel drivers to send the key inputs to an LIRC daemon, but I couldn't get that to work and the kernel drivers weren't hard to setup.
 
</blockquote>
 
</blockquote>
#Process the raw HD MPEG-2 TS recording with the [[User:Steveadeff#HR.HDTV_script |HD script]]
 
#*I had to modify the script to accept larger files and create a smaller end result (640x360).  I can post the modified script here if anyone is interested.
 
#*This transcode takes a long time, upwards of six to eight hours on a Pentium 3Ghz machine.
 
#Load the resulting file into Avidemux, leave the audio and video as "copy", cut the commercials and set the audio delay to 250ms
 
#Save it and hit "yes" to smart copy.  Sixty seconds later you have a perfectly transcoded and in sync result.
 
  
==== Other Random Tips ====
+
==== Distinctive Ringing (no longer used) ====
* FreePBX includes PHPMyadmin which is useful for optimizing the database(s) which saves space and makes them run faster.  PHPMyadmin also comes in handy for editing the database directly, like getting rid of commercial detection jobs if mythcommflag hangs, or reordering the tuners.
 
* Concerning Asterisk, I tried ADSI, the ability to program phones with custom scripts, but some phones just didn't work because they are "locked" by their manufacturer.  The comedian mail app for ADSI works, but it's clunky and slow and it was much simpler just to autologin and play the messages.
 
* Monitoring of the Myth systems work great using [http://en.wikipedia.org/wiki/GKrellM Gkrellm] and I highly recommend it. Works on Linux and Windows and looks great with a large variety of skins.
 
 
 
== Personal Setup Notes ==
 
Notes for my own setup:
 
==== Distinctive Ringing ====
 
 
<blockquote>
 
<blockquote>
 
Setting up distinctive ringing for ring groups.<br>
 
Setting up distinctive ringing for ring groups.<br>
Line 253: Line 576:
  
 
</blockquote>
 
</blockquote>
*To avoid entering a password every time for voice mail, add the voice mail's extension "sXXX" into extensions.conf -> app-messagecenter instead of "default", i.e. s402
+
*Voicemail is emailed to us now and deleted.  <del>To avoid entering a password every time for voice mail, add the voice mail's extension "sXXX" into extensions.conf -> app-messagecenter instead of "default", i.e. s402</del>
 +
 
 
==== Caller ID ====
 
==== Caller ID ====
Code for sending caller ID from Asterisk to MythTV Frontends.  Requires [http://www.voip-info.org/wiki/view/Asterisk+tips+MythTV+integration XYAC].  Use the [[User:Mrsdonovan#Idle_Three_Hour_Sleepd_Daemon |startup script below]] to start up the XYAC daemon at login.   
+
The code for sending caller ID from Asterisk to MythTV Frontends.  Requires [http://www.voip-info.org/wiki/view/Asterisk+tips+MythTV+integration XYAC].  Use the [[User:Mrsdonovan#Idle_Three_Hour_Sleepd_Daemon |startup script below]] to start up the XYAC daemon at login.   
 
<br>
 
<br>
 
<blockquote>
 
<blockquote>
Line 264: Line 588:
 
Note:
 
Note:
 
* The "&" at the end runs the "nc" daemon in the background where it can fail silently and not cause delays in answering the calls.  Without the "&", the far side can intermittently time-out just as the phones start ringing.
 
* The "&" at the end runs the "nc" daemon in the background where it can fail silently and not cause delays in answering the calls.  Without the "&", the far side can intermittently time-out just as the phones start ringing.
 +
* This code lives in /etc/asterisk/extensions_custom.conf file
 +
* The Makefile for XYAC needs to have two changes to compile under Ubuntu 12.04:
 +
<pre>
 +
INCLUDES=-I/usr/include
 +
</pre>
 +
and
 +
<pre>
 +
xyac: xyac.o xosd.o
 +
  $(CC) -o xyac $(LIBS) xyac.o xosd.o $(LFLAGS)
 +
</pre>
 +
(the gcc compiler now uses the --as-needed flag instead of --no-as-needed which means it has to have the library after the compiling module, otherwise the linker doesn't think any of the functions in the library are needed)
 +
* The line to startup XYAC and increase the font size is:
 +
<pre> xyac -c yellow --align=center -f -*-*-*-*-*-*-72-*-*-*-*-*-*-* --shadow=4 --delay=45 </pre>
 +
The line after the "-f" is the font specifier in [https://wiki.archlinux.org/index.php/X_Logical_Font_Description XLFD format] and depends on what fonts are installed, but the generic one shown uses the first system font with a size of 72, which on a 1080p screen looks big.  A good way to find what fonts you have installed is to use "xfontsel" and a common system font is "Bitstream", which you would specify like this <pre>-bitstream-*-*-*-*-*-72-*-*-*-*-*-*-*</pre>
  
==== Idle Three Hour Sleepd Daemon ====
+
==== Idle Three Hour Sleepd Daemon (no longer used) ====
* Below is a method of using the sleepd daemon to shut off a MythTV frontend after three hours and to pop up a 30 second warning that the user can cancel.
+
<s> * Below is a method of using the sleepd daemon to shut off a MythTV frontend after three hours and to pop up a 30 second warning that the user can cancel. </s> - no longer used but left for others.
 
<pre>
 
<pre>
 
# This is called script is called from xfce's autostart manager under "Settings" -> "Sessions and Startup"
 
# This is called script is called from xfce's autostart manager under "Settings" -> "Sessions and Startup"
Line 279: Line 617:
  
 
</pre>
 
</pre>
==== Premptable Shutdown Script ====
+
 
* Here is the standby.sh script called by the sleepd daemon for shutdown:
+
==== Premptable Shutdown Script (no longer used) ====
 +
<s>* Here is the standby.sh script called by the sleepd daemon for shutdown:</s>
 +
Left here for history sake, but no longer used.
 +
 
 
'''Note: shutdown requires root access which means using [http://www.linuxhelp.net/guides/sudo/ sudo]'''
 
'''Note: shutdown requires root access which means using [http://www.linuxhelp.net/guides/sudo/ sudo]'''
 
<pre>
 
<pre>
Line 319: Line 660:
 
</pre>
 
</pre>
  
==== Backend Slave and Master MythTV Backend Process Reset ====
+
==== Backend Slave and Master MythTV Backend Process Reset (no longer used) ====
The WAF had been lower of late mostly because our master/save system seems to hang once in a while.  I added a method whereby the master, slave backends and the frontend are reset at the push of a remote button. A local script is run when the button is pushed that does the following (call it "restart_entire_myth.sh"):
+
<s> The WAF had been lower of late mostly because our master/save system seems to hang once in a while.  I added a method whereby the master, slave backends and the frontend are reset at the push of a remote button. </s> - no longer used.
 +
 
 +
A local script is run when the button is pushed that does the following (call it "restart_entire_myth.sh"):
 
<pre>
 
<pre>
 
#!/bin/bash
 
#!/bin/bash
Line 369: Line 712:
 
</pre>
 
</pre>
 
The "-H" is probably not necessary and just holds the window open, but the "-x" runs the "irexec" command.  You can also run this script and test to see if the button press from the remote is getting caught and running the restart script, the output of which appears in the terminal window.  I use xfce desktop obviously, but I'm sure the same can be done for the Gnome and KDE terminal programs.
 
The "-H" is probably not necessary and just holds the window open, but the "-x" runs the "irexec" command.  You can also run this script and test to see if the button press from the remote is getting caught and running the restart script, the output of which appears in the terminal window.  I use xfce desktop obviously, but I'm sure the same can be done for the Gnome and KDE terminal programs.
 +
 +
==== 6200ch Change Channel Script ====
 +
On reboot, the firewire driver randomly grabs a port which means calling the 6200ch change channel program with the previous setting has a fifty percent change of failing (or working if you are an optimist).  MythTV would then start missing all its recordings.
 +
 +
The work around was to try both ports and return 0 like this:
 +
<pre>
 +
#/bin/bash
 +
if /home/custom_scripts/6200ch -n 0 -p 0 $1
 +
then exit 0
 +
else
 +
/home/custom_scripts/6200ch -n 1 -p 0 $1
 +
fi
 +
 +
exit 0
 +
</pre>
 +
 +
(Instead of calling the 6200ch program directly, MythTV calls the change_chan.sh script above)
 +
 +
MythTV would then get back that the channel change worked and can go on its merry way.
  
 
== The Plus and Minus of MythTV ==
 
== The Plus and Minus of MythTV ==
Line 389: Line 751:
  
 
* <del>Not directly related to MythTV but, every couple of months, mdadm kicks all the HDs out of the RAID 5 array on System B.  Thankfully, a reboot brings the array right back up, but very strange.</del>  This was a hardware issue, which after the MOBO upgrade, is fixed.
 
* <del>Not directly related to MythTV but, every couple of months, mdadm kicks all the HDs out of the RAID 5 array on System B.  Thankfully, a reboot brings the array right back up, but very strange.</del>  This was a hardware issue, which after the MOBO upgrade, is fixed.
* Every once in a while, the front end will crash, and it seems that a reset of all the MythTV processes is required.  I have created a script, which when a "power" button is pressed on the remote, resets the local master backend and frontend and through a SSH tunnel it opens on the fly, resets the remote MythTV backend process (that took a bit of debugging to get a remote user to reset a different user's process)  Although likely unrelated, the samba/netbios daemon stops working too.
+
* <del>Every once in a while, the front end will crash, and it seems that a reset of all the MythTV processes is required.  I have created a script, which when a "power" button is pressed on the remote, resets the local master backend and frontend and through a SSH tunnel it opens on the fly, resets the remote MythTV backend process (that took a bit of debugging to get a remote user to reset a different user's process) </del> - The latest 0.25 version of MythTV now has a frontend restarter built in and my custom script was created multiple backend processes which completely messed things up.  Stability is still an issue over weeks and weeks.
 +
* <del>As of January, 2013, the HDPVR driver does not have the ID code, 4903, for my HDPVR unit which means /dev/video does not show up and fixing the problem means editing the source and recompiling. Because recompiling a driver is a bitch, I used a hex editor and changed the driver directly (Warning: This is the WRONG way to fix this problem. Don't ever do it. Ever. However, it did work for me in 10.04. Use a hex editor to edit hdpvr.ko, search for text 4901 and replace with 4903, search for hex 0149 and replace with 0349"  There is a great thread on this problem [http://ubuntuforums.org/showthread.php?t=1590146 here].)</del> Ubuntu 12.04 doesn't have this problem.
 
[[Category:MythPeople]][[Category:User_Systems]]
 
[[Category:MythPeople]][[Category:User_Systems]]
 +
 +
== Other Power Users ==
 +
 +
As of June 2015, I went through all the users listed in the categories below and found a few who are still updating their pages (within the last two years):
 +
* [[User:Blackoper]]
 +
* [[User:Northern_Man]]
 +
* [[User:Lwoggardner]]
 +
* [[User:Steveadeff]]
 +
* [[User:Michel]]
 +
* [[User:Cowbutt]]
 +
 +
A data point of note is that most of the other user accounts listed were created about 2006 to 2008 and then stopped being updated.  Was there a Peak MythTV event in 2007?

Latest revision as of 10:43, 11 June 2018

With over a decade of continuous service, this MythTV system was torn down in November of 2016. Its demise started a couple years ago when I stopped watching TV directly and just maintained the system for the grandparents. I also stopped collecting movies or TV shows because they take up a lot of space and I never seem to watch them again, even the ones I like. I will, over the next few years, whittle the movie collection down to ones that are important and I might want to watch in a few years. Part of the reason the movie collection started was for my Brother's kids, but as teens now, they game on their phones or gaming console. The added features, such as the video and picture section were never used and now that everyone has smartphones, the DID (Direct In Dial) phone number serviced by Asterisk (which displayed the callerID) has also been deleted.

This MythTV system was shut off largely because the ten year old tuners started having problems and the alternative was simpler. As I never used the system, the maintenance had become onerous and I am just busy enough these days that I say no to more things then I say yes. Although I had stopped the MythTV process on October 17th, as the statistics show, I made the decision yesterday, October 31st, 2016, to actually shut my MythTV down permanently. It actually coincided with the Shaw technician coming by and replacing the mess of tuners with one, taking the old ones away.

Having said that, the MythTV box will still live on as a NAS, run game servers when necessary, and still get hardware refreshes and software updates. I am thankful to MythTV because it was how I learned about Linux and Open Source in general.

The entire MythTV system was replaced with a single six channel "Gateway HD PVR tuner" which even includes a smartphone app to search for and set up recordings. With up to four tuners and two Haupauge HD PVRs now replaced with one tuner, at least my power bill will go down, which is great, given I just installed an 8kW(DC) 28 module solar system this summer (APS microinverters).

I am proud of this User's page, having nursed it for almost exactly ten years, and will be saving a PDF of it. I hope you find it useful.

Everything below this line is now kept for historical sake:


I've been using MythTV since 0.18 and here are my paltry my contributions.

The setup below is used daily weekly by the grandparents and has a low medium WAF factor, yes low. Sadly, the grandparents have reverted to the PVR that comes in the cable tuner using the MythTV as backup. After reducing the system to one machine it seems to have gotten more stable. I have reduced the system to a single combined master backend / HD capable front-end. Having two computer systems on 24/7 costs money for electricity, replacement HDs and UPS batteries. I just don't watch enough TV to care anymore - it won't be a better PVR that kills MythTV, just cloud based options like NetFlix.


Statistics

My current statistics are:

Number of shows: 4993
Number of episodes: 75753
First recording: Friday May 6th, 2005
Last recording: Monday October 17th, 2016
Total Running Time: 11 years 5 months 12 days 23 hrs 58 mins
Total Recorded: 9 years 2 months 26 days 22 hrs 53 mins


System B Recording Space: 3TB RaidZ (Arbitrary quota controlled ZFS with SSD for ZIL and Cache)


System B Video space: 10TB RaidZ
(The videos share the total size of the RaidZ ZFS filesystem with four 3TB Western Digital Red NAS drives)


Setup

System A

This system was used at my house and consisted of the following:

A.a - MythTV Slave backend / SD Frontend / Asterisk server (System retired 2015)

  • 3Ghz HT Intel with 2GB of dual channel RAM
  • 1GB ethernet jack connected to a 10/100/1000Mbps Tredndnet 8 port router
  • HD capable Geforce 6200 video card.
    • This system can play 720p but not 1080p as it stutters horribly. The motherboard has an AGP port though and everything will have to upgraded to play 1080p.
  • RAID 5 with four SATA harddrives, roughly about 900GB of space for recordings and 1TB for videos/music/pictures/ROMs.
  • ASUS P4P800-E-Deluxe
  • Haupauge PVR-150 - Died
  • Haupauge PVR-250 with remote
  • <s>Two One Motorola DCT2524 cable tuner controlled by IR blasters connected to a serial card. (Custom built lirc_serial kernel module, ledxmit_serial)
  • DCT-6214 (Yes, it is a PVR, I know..) tuner connected via an Agere firewire card for HD - Moved it to the A.b MythFrontEndHD system
  • MythBuntu 10.4 (32bit)
  • 22" LCD
  • Belkin 1500VA UPS <s>with Bulldog software (includes replacing the batteries on a roughly three year cycle.)


A.b - MythTV HD Master Backend & Frontend / Virtualbox / Asterisk / Bitcoin Miner / NAS

  • AMD Phenom X6 1090T with 16GB of Kingston ECC (required for ZFS because it is constantly doing checksums which need to be correct)
  • Asus Sabertooth v1.0 AM3+ with Zotac GT430 (using VDPAU)
    • With VDPAU, HD 1080p works flawlessly at 1920x1080.
  • Two Hauppauge_HD-PVR running the latest firmware (v1.7.1.30058) and
    • One uses 6200ch to change channel through onboard firewire connected to a DCT-6200s (as an example, using "6200ch -n 1 -p 0" or "6200ch -n 2 -p 0")
    • One Motorola DCT2524 cable tuner controlled by one of the HDPVR IR blasters via LIRC
  • 3TB Raid 5 for recordings and video backup - 10TB ZFS RaidZ setup with backups to old A.a MythTV Slave
    • Uses four 3TB drives (5400RPM Western Digital RED NAS drives) with a SSD providing a 2GB ZIL and 60GB Cache for read/write.
    • Given the ZFS array is a single point of failure, important data is backed up every other week to separate alternating machines (part of the hardware is from A.a Humbleserver). The two seperate backup machines are old socket 478 ASUS P4P800 with P4s, which while power hungry, are only on for a few hours tens of minutes a month while doing the backup.
      • They are first turned on by a Wake-On-LAN (WOL) packet from A.b and boot via PXE Boot-on-LAN which loads a 12.04LTS kernel (from A.b of course) which kicks off a backup script to a 3TB drive. Once the backup has completed, the computer shuts down. This uses a minimum of power and as harddrives fail in the ZFS array, they are immediately replaced by one of these backup drives and another ordered without ever being reduced to a single point of failure.
  • MythGame with SDLMame emulation and a wireless Logitech Rumblepad 2 - never used.
  • Ubuntu 12.04 LTS running from an 80GB Intel SSD
  • Digium Wildcard TDM400P with two one three S100-M FXS expansion cards (two spares)
  • Freepbx with Asterisk v1.8
  • Toshiba Regza 42RV530U - 42" LCD driven at max resolution of 1900x1080 via an DMI cable. We use the TV's remote with a USB MCE receiver (using Kernel drivers - no LIRC!).
  • Belkin 1200VA UPS with Bulldog software - Batteries died and not worth replacing, Bulldog doesn't work on 10.4 - Replaced batteries for about $100.
  • Butterfly Labs "Single" 25GH - now junk, sigh
  • Running four one 24/7 VirtualBox Windows 710 guest for ARK Survivor bots server.
    • This is where the ZFS file system shines because it uses a SSD for ZIL/Cache to make read/writes super fast and deduplication means the four operating systems take up almost the space of only one.


A.c - MythTV Frontend (System retired 2012)

  • 2.4Ghz Celeron (Not for HD)
  • 80GB HD
  • 768MB of RAM
  • Asus Pundit (The SIS version)
  • MythBuntu 10.4
  • MCE Remote with programmable buttons to control the TV.


A.d - MythTV SD Frontend (System retired 2011)

  • 2.4Ghz Celeron (Not for HD)
  • 80GB HD
  • 768MB of RAM
  • Asus Pundit (The SIS version)
  • MythBuntu 10.4
  • Sony Trinitron 32" Flat Screen CRT
  • Goes into hibernation after 3 hours using sleepd (svideo doesn't recover correctly on resume!?)
  • MCE Remote with programmable buttons to control the TV.
  • This system connects through a hard-to-find Dlink DAP-1555 wireless bridge.


A.e - MythTV HD Frontend (Virtual) - Primary Work Computer

Screen shot of a five monitor setup
  • A virtual dual core VMware Workstation Virtualbox guest running on a host with an eight core 4Ghz AMD FX-8350 (cooled with a Noctura NH-D15) on a ASUS Sabertooth 990FX R2 with 32GB RAM (Kingston ECC DDR3 1333Ghz) running stock Windows 107 off a 80GB 256GB SSD and the VMware Virtualbox guests off a Samsung 840 240GB 500GB SSD.
    • After using VMware for nearly a decade, I started using Virtualbox on my linux systems and found it fast and stable. After upgrading to the 49" 4K LG panel and trying to spread the VMware Windows 7 guest across that and three other 24" 1920x1080 monitors in portrait mode, the guest started to stutter, making the operating system unusable. Videos wouldn't play and the whole guest was awfully slow. I tried the VirtualBox guest and it was an immediate improvement. VB also has more bells and whistles, for example, you can pause guests (can't with VMware Player) which is very useful when putting the computer to sleep for the night. Virtualbox also makes use of Nested Paging available in the latest AMD and i7 processors which makes the guests faster. Virtualbox is also open source, updates faster, and has a deeper support community available online. I have converted all my virtual machines to Virtualbox, will be uninstalling VMware and likely never looking back. Windows 10 VirtualBox guests can play HD video flawlessly now.
  • Belkin 1500VA UPS with Bulldog software with Windows software for shutdown (includes replacing the batteries on a roughly three year cycle.)
  • virtual 9GB HD
  • virtual 1GB Ram
  • Ubuntu Desktop 12.04
  • 2560x1600 30" DELL Monitor - 4160x2160 49" LG 49UB8200
    • Ok, so it runs at 30Hz and has a 4:2:0 chromakey so tiny text isn't perfect, but wow, does it display graphics well, and play games great. The GTX 760 can handle it fine at 30Hz in BF4 at full 4K resolution. Err, no games are not great, even with a GTX 770 780, it is laggy, so I play in window mode at a slightly lower resolution.
      • I now do most of my reading on the 30" Dell in portrait mode.
  • Both SD and HD 720p/1080i work great under Virtualbox VMware Player 6.0 Workstation! (still) stutters but it at least %90 watchable now.
  • Here is a picture of the setup, there are
    • Five Four Acer V243H Ajbd 24" 1080x1920px portrait monitors on an Ergotron LX Triple Display (monitors numbered #3,5,6,4 in the picture below)
      • Ergotron sent me a fourth mount that slid onto the aluminum bar. In 2010, the setup was the highest pixel count for the lowest money.
    • 30" DELL 1600x2560px portrait mode on an Ergotron arm (#2)
    • LG UB8200 49" 4K 4096x2160px IPS LED monitor on a wall mount (#1)
Screen shot of a six monitor setup
  • Two of the Acers are connected to a GTX 645, another two to a GTX 460 two Asus EN210 1GB passively cooled cards and the Dell and LG monitors are connected to a EVGA GTX760 GTX770.
    • The GT210s required old legacy drivers no longer being updated and it was time for an upgrade (the GTX645 cost me $25! )
  • All the monitors, speakers, battery chargers, shaver (yes it's a shaver) and headphone charger are controlled by a single switch DYI'd into the base of the Ergotron (it's in front of the green toy car in the photo). I put the computer to sleep, hit one button and have no phantom power draws.
  • I spend about $1K CND a year on the three systems in upgrades, although in 2015 it was closer to $3K (because of a testing computer for work and the 4K monitor). Since 2007, on three separate systems, the testing machine, MythBackendHD (A.b and backup computers) and my primary, in total I have spent $8K, although most of it is a write-off with parts purchased off Kijiji.



System B

This system lives died at my brother's house although I maintain it.

B.a - MythTV HD Frontend (System Died 2014)

  • INTEL 2.8Ghz P4
  • 1.5GB of RAM
  • NVIDIA Geforce 8500
  • Samsung LN46B540P 46" LCD
  • Mythbuntu 9.10 (32bit)
  • MythGame with SDLMame emulation and a wireless Logitech Rumblepad 2
  • Goes into hibernation after 3 hours using sleepd


Computer died and they now use a Boxee to access a humungous disk on an Windows machine the video RAID on backend server(B.B)

B.b - MythTV HD Backend / Asterisk (System Re-purposed 2015)

  • Quad Core 2.5Ghz Intel running on Mythbuntu 10.4 LTS (overkill, but wow, responsive)
  • 1.3TB of RAID 5 for videos/pictures/music/ROMs
  • 2.5TB LVM for recordings.
  • 500GB for backups of other computers.
  • Haupauge PVR-150 connects to a cheap ass Motorola tuner and is controlled via an IR blaster controlled by the USB MCE remote receiver. Family uses Netflix now and has no cable provider.
  • Digium Wildcard TDM400P with three two one (damn lightning) FXS expansion cards
  • Freepbx with Asterisk 1.6
  • Samsung LT-P266W 26" HDtv flat-panel LCD Headless
  • Belkin 900VA UPS with Belkin Bulldog software

This system is only used as an Asterisk box now. is a gaming machine for the kids now. Everyone uses their cell phones and the home phone number goes straight to voicemail which is then forwarded via email. The Asterisk system is actually not even in the same house, but System A.b above.

Notes

  • Given three of the systems (A.a, A.b and B.b) are on 24/7, they each have a UPS rated at 900VA or better, which lasts an average of twenty minutes before cleanly shutting down (important for the RAIDed drives).
  • We currently use link2voip.com The Spout (.ca) as our DID and outbound provider for Asterisk.
  • Most of our machines use gigabyte Ethernet and the System A (A.a and A.b) route through a $40 Buffalo WHR-300HP (proved buggy) $80 Asus RT-N16 running the open source firmware, DD-WRT Tomato (Tomato isn't updated enough and has some intermittent bugs). There have been a lot of gateways in this house but the open source DD-WRT is by far the best. It handles QoS properly for VoIP and has all the bells and whistles every other gateway lacks and a consistent interface across multiple devices. We also use OpenDNS for all the DNS queries.
  • We use most of the default plugins including MythGame (Who am I kidding, kids game on iPhone/Pads these days), pictures (rarely), videos, weather and XYAC for CallerID.
  • Both Asterisk boxes are connected to each other over an IAX2 link that works well through NAT and our mutual ISPs.
  • On System A, we also run two TrendNet (TV-IP851WIC and TV-IP422W) cameras for security that record any motion detected to A.a - Giving up on remote cameras, they can't connect to new 802.11n routers, break and stop working much too frequently.
  • The link between the MythTV frontends B, D and the master and slave backends A, C is over a 5.8Ghz 802.11a wireless bridge using dual-band bridges from Trendnet (TEW-510apb) and one 21dbi and one 9dbi gain antenna. Average throughput for 1080p HD is about 13mbps and during high motion scenes, tops out at 19mbps. With the high gain antennas, the TEW-510apbs can operate at their maximum throughput of 21Mbps and HD is smooth. On rare occasions, the link is affected by thick ice on the roof and really heavy rain. (System B is now separate)

Tips from the HD Front Lines

[Updated June, 2015] I have been using MythTV with HD now for a couple a decade and wanted to share what has worked and what has not:

Firewire cards - get the best firewire cards that works with your tuner(s). I have used NEC, VIA (onboard and PCI) cards and now finally, Agere cards. Agere cards, the lastest chip revisions anyway, are by far the best with the Motorola DCT-62xx series boxes. Ironically, "test-mpeg2" does not work making any priming script useless, but "broadcast" mode works pretty much flawlessly and fast! You do still have to prime the firewire card, but I do it once at startup with the "firewire_tester -n 0 -B" script.

VIA firewire chipsets will work, as they have for over a year with my system in P2P mode, but it sucks. VIA chips require a priming script to run each time a channel change occurs and have two negative effects - first, because the priming is done at each channel change, channel surfing during Live TV times-out 2/3rds of the time. Recordings seem to work %100 of the time. Secondly, the priming script + VIA chipset + DCT-6200 crashed the DCT-6200 tuner when tuning to SD channels (seriously! Mine would hard lock and have to be unplugged). I have no clue why because the same combination but with a DCT-6214 worked fine. The tuner hard lock also doesn't happen on HD channels. - Just use an onboard firewire for changing channels and get a:

Hauppage HD PVR - Gamers use this box to record their gaming without loading their computer down and you can find older versions for half price ($80CND) or better in second hand markets. HD PVR boxes take in the analog signal (blue, red and green cables) and convert it to a USB which is recognized by the HDPVR driver in Linux. Works great with MythTV and both our HD tuners now use an HD-PVR. It is also immune to the "5C" encoding that Cable TV providers randomly flip on.

Information.png Conclusion - With DCT-62xx tuners, use firewire to change channels with the 6200ch script and an Hauppauge HDPVR to capture the HD video.

Video cards - As XvMC support is going to be discontinued in MythTV 0.25 and replaced with VPDAU which supports a much large set of codecs. NVIDIA has perhaps the best support and look for a card that uses "PureVideo HD".

Information.png Conclusion - Get a Geforce 430 based video card and HD 1080p will work flawlessly with the OSD using VDPAU.

Processor - Get the largest and fastest one for HD because it makes a difference. Everything is magnified by two with HD content, which means it takes more processor power to do playback, commercial detection and transcoding. A faster SSD processor also helps with database queries. (As an aside, I also suggest looking into keeping the database in memory)

Information.png Conclusion - Faster = better.

Remote Control - From a usability stand point, an excellent remote is important. I have settled on the Microsoft MCE remotes because they have lots of features that make it all come together - First, they can be trained with your existing TV remote to shut off the TV and control the volume of the TV. The volume control in MythTV frankly sucks because it only works when a recording is actually playing and doesn't go high enough (a fault of the hardware actually). Being able to turn off the TV and control it's volume directly mean you can chuck your TV's remote. If you are playing DVDs on your MythTV system, you can now have one remote to control everything and that rocks. Better, if you wreck the remote, they are inexpensive ($35 locally) and widely available. The receiver has wide reception, a quick response rate and the layout on the remote is excellent with important buttons in easy to reach places. The downside is that because it is back lit, it uses batteries quickly (every four months) but hopefully rechargeable ones in your case.

Information.png Conclusion - Get the Microsoft MCE remote and train it to control your TV as well as your MythTV box. One remote to control them all.

Hard drive Size - On the main backend (A) I started out with an LVM which tied a bunch of disparately sized disks together, but I wanted more redundancy and speed. I have switched to RAID 5 using the mdadm ZFS RaidZ and it seems to work fine.

Hard drive size really only translates into one thing - longer time to watch something before it expires. For recordings, our system has a combined space of about a three terabytes which means our recordings last about a month. Our system records, on average, about eighteen one hour shows a day on weekdays and upwards of thirty or more on weekends with a mix of everything from sports to movies to the daily kid's shows. Lots of it never gets watched. Of those recordings we want to keep longer term, the best solution has been to transcode and move them into our video section that exists on a separate harddrive (An LVM of all the free space left over from the RAID actually). The videos are backup (along with a bunch of other stuff) once every other week to alternating backup machines that are awoken via WOL, boot via PXE Boot-On-Lan and do the backup.

Information.png Conclusion - Larger = Longer lasting. RAID is your friend.


Other Random Tips

  • FreePBX includes PHPMyadmin which is useful for optimizing the database(s) which saves space and makes them run faster. PHPMyadmin also comes in handy for editing the database directly, like getting rid of commercial detection jobs if mythcommflag hangs, or reordering the tuners.
  • Concerning Asterisk, I tried ADSI, the ability to program phones with custom scripts, but some phones just didn't work because they are "locked" by their manufacturer. The comedian mail app for ADSI works, but it's clunky and slow and it was much simpler just to autologin and play the messages.
  • Monitoring of the Myth systems work great using Gkrellm and I highly recommend it. Works on Linux and Windows and looks great with a large variety of skins. [Update November 2013 June 2015 - This is still true ten years later which is rare especially since the windows client hasn't been updated in over five years but it works on Windows 7, the last Windows OS I plan to buy.]
  • Get ECC RAM - Your MythTV backend is on all the time, and it is exactly these conditions that the ECC ram (*unbuffered*) was created for. AMD desktop processors and related motherboards support ECC out of the box unlike Intel which require server grade (expensive) CPUs.
  • If you run a disk array like RAID or ZFS RaidZ, a Uninterruptable Power Supply is good at keeping the array "clean" for those unexpected power outages. Use a "True Sine Wave" UPS if you can afford it and your computer power supplies will appreciate it.

Personal Setup Notes

Notes for my own setup:

Moving a DigitalOcean Droplet to Virtual Box

Moving a Digital Ocean droplet into a local virtual machine is pretty complicated and in the end, I combined two instruction sets together (PDF here and another here) to get it working. I wanted to document the process here, should I ever need it in the future (or I might need to do it in reverse):

  • Add block storage to your droplet of a size greater than your droplet, i.e the drop is 20GB, add a 22GB droplet as it needs to store the "dd" image.
  • Turn off the droplet and under "Recovery" flip it into recovery mode and boot the droplet back up
  • Because your droplet isn't running, you can't use SSH, which means you have to use the browser console link provided by Digital Ocean, which has some complications mentioned below.
  • Once the recovery OS is running, drop into the command line (menu selection #6)
  • First "fdisk -l" to show the partitions and you should have a /dev/vda which contains the droplet and unpartitioned /dev/sda with the block storage - the difference in sizes will make it clear which is which
    • Optional - run "fsck -vfy /dev/vda1" to remove any errors from the droplet
  • "fdisk /dev/sda" then create a partition ("n" for new, t" for type 83 and "w" to write the changes) and format it to ext4 "mkfs.ext4 /dev/sda1"
  • Now mount the new partition
cd /mnt
mkdir tor (or whatever)
mount /dev/sda1 tor
cd tor
  • Once on the new partition of the block storage, then
    • A problem that popped up was that the console did not recognize the pipe character (but yped ">"), so it was necessary to programatically create it via the command line:
      • with
        pipe=$(nawk 'BEGIN {printf("%c",124);}')
        This assigns the variable "pipe" with the ASCII equivalent of "|"
    • then
       eval dd if=/dev/vda1 $pipe gzip > vda1.img.gz 
      which creates the image file on the block storage
    • "md5sum vda1.img.gz" create a text file with the mda5sum which is used to test it after download it.
  • Now you have a copy of the vda1.img.gz file on block storage, take the droplet out of recovery and boot the droplet back up.
  • Then SSH back into the droplet and
cd /mnt/
mkdir tor
mount /dev/sda1 tor

and "ll /dev/sda1" which should have the two files on it, the vda1.img.gz and the md5sum file.

  • I use Windows 10 and putty to access my droplets via a shared key and as part of that package, I use
    • pscp root@<your ip>:/mnt/tor/vda1.img.gz .
      will download the image file
    • I opened another window and downloaded the md5sum too with "pscp root@<your IP>:/mnt/tor/vda1.img.gz.md5"
    • The pscp program uses the private certificate provided by the running pageant program for the SSH tunnel.
  • Once you have the vda1.img.gz file local, then unzip and convert it with the "qemu-img" tool (apt-get install qemu-utils" or just type in qemu-img and it will suggest what package needs to be download)
    • "md5sum vda1.img.gz" and check the hash against the one downloaded in the md5 file to make sure you have a perfect copy (the md5 file is just a text file with a short hash in it)
    •  gzip -d vda1.img.gz 
      (unzips it)
 gemu-img convert -O vmdk vda1.img <mydropletname.vmdk> 
(converts it to a vmdk file)
  • Now create a new Ubuntu 64bit image with a new VDI harddrive image and also connect the vmdk file to the same VM.
    • I had significant problems with an vmdk that VB refused to connect the VM. It likely means it was corrupt.
  • Boot into a rescue CD and drop to the command line
  • Again, use fdisk to create a partition on the new VDI image (likely /dev/sdb1) and then use dd to transfer the image from the vmdk
 dd if=/dev/sda of=/dev/sdb1 conv=notrunc,noerror 
(copies the dd image from the VDMK to the VDI)
 fsck -t ext4 /dev/sdb1 
(check it for errors)

Now mount /dev/sdb1

cd /mnt
mkdir tor
mount /dev/sdb1 tor
  • Now remove the old grub and networking setups:
cd /mnt/tor/boot/grub
rm -rf *
cd /mnt/tor/etc/network/
nano interfaces
  • Remove all the hardcoded DO IP addresses and add your own:
auto eth0
iface eth0 inet dhcp
  • Now shutdown the VM, disconnect the VMDK and restart it again with the rescue CD
    • Recreate the grub directory ([help.ubuntu.com/community/Boot-Repair I used the Ubuntu "Boot Repair" iso])
  • If it works, it should come up asking for a login (no desktop interface of course)
  • Now you should be able to ssh into the box just as you had when it was on DO!
  • Profit!

This method has a number of benefits:

  • The doplet is off and in a known state when the dd image is taken
  • the dd image is copied to a native VDI virtualbox harddrive image
  • the networking and grub are freshly scrubbed for Virtualbox
  • the ssh keys still work.

Biweekly Backup Script

The A.b server has two backup servers that are woken up via Wake-On-Lan packets, which then boot via PXE. The script the weekly backup cron job calls is this one on A.b:

 /bin/bash
#Now wake up the backup computer and exit, as the script there now 
# takes over and does the backup and then shuts down.
mark_file=/diskless/c1/mnt/right-side-computer-next

# check whether the job ran last week
echo "Checking to see if the markfile exists"
if [ -e $mark_file ] ; then
	rm -f $mark_file
	echo "Mark file found, backing up to right side computer"
	wakeonlan 00:EE:DD:FF:11:22 # The backup computer on the right
else
	echo "Mark file not found, backing up to the left side computer"
	touch $mark_file
	wakeonlan 00:33:44:55:66:77 # The backup computer on the left
fi
echo "Done - Exiting"

It uses a file "right-side-computer-next" as a flag to alternate between the two backups, running every other backup once a week. It uses the "wakeonlan" program available through Ubuntu.

Diskless Client Side Script

On the diskless client side, once it is woken up, loads and runs the kernel from A.b (which was slightly tricky because the A.b NAS is an AMD CPU and the backup machines are both Intel P4s), it then runs this script:

#/bin/bash

set -o nounset

STARTTIME=$(date +%s)
# Mount the drive
mount /dev/sda1 /mnt/localdisk
# Mount the partition to backup from
mount 192.168.x.yyy:/bigtank /mnt/bigtankremote


# Now start rsync
TDATEIS=$(date +"%A-%B-%d-%Y")
OUTPUT="Backup Slave Results for $TDATEIS \n----------------------------\n----------------------------\n "
OUTPUT="$OUTPUT First backup all the videos if any have changed...\n-> But check to see if the mount is valid:"

if [ "$(grep 192.168.x.yyy /etc/mtab)" ];
then
	OUTPUT="$OUTPUT \nNow do the sync, deleting files from the backup not in the original"
	# -  Make sure to exclude the thumbcache on the destination generated
        #    by MythTV or they will be deleted every night when sync'd.
	# Add "-vrun" to test the backup

	# First videos
	RSYNC_RESULTS=`rsync -vru --delete  /mnt/bigtankremote/video /mnt/localdisk`
	OUTPUT="$OUTPUT \n Video Backup Results:\n---------------------------------------\n $RSYNC_RESULTS"
	# Then seansNAS
	RSYNC_RESULTS_TWO=`rsync -vru --delete --exclude-from /mnt/do_not_copy_list.txt /mnt/bigtankremote/seansNAS /mnt/localdisk`
	OUTPUT="$OUTPUT \n seansNAS Backup Results:\n-------------------------------------\n $RSYNC_RESULTS_TWO"
else
	OUTPUT="$OUTPUT \nIt looks like the video/music/pictures are not mounted???"
fi

OUTPUT="$OUTPUT \n-------------------------\nDone"
DISKSPACE_LEFT=`df -h /dev/sda1`
OUTPUT="$OUTPUT \nDisk Space Left:\n $DISKSPACE_LEFT"
ENDTIME=$(date +%s)
OUTPUT="$OUTPUT \nTotal backup time: $(($ENDTIME - $STARTTIME))s"

echo -e "$OUTPUT" > /mnt/backup_status.txt

sleep 60
shutdown -h now

This script does the following:

  • Mounts the local harddrive (a GPT formated EXT4 3TB drive) and the A.b ZFS partition to be backed up
  • Checks to see if the mounts were successful
    • The IP address of A.b should be listed in mtab, otherwise, rsync would try to dump nearly 3TB of data onto A.b's 120GB SSD!
  • Does the backups using rsync
    • The do_not_copy_list.txt lists directories and file patterns that should be skipped, i.e. the .TIB files from Acronis, as backups of backups are not necessary.
  • Report how much diskspace is left on the local 3TB drive.
  • Then write the backup results to backup_status.txt
    • I couldn't get the diskless client to properly access the internet and postfix is a pain to setup: all just to send an email, so the script writes a file to the SSD and I have another cronjob on the A.b NAS which looks for the file and emails it is there. Not elegant, but time effective.
  • The diskless client sleeps for a minute to give me time to stop it from the diskless client if I need to check something.
  • And shuts down the diskless client.

It works pretty elegantly so far and has significant benefits:

  • The backup computers are pretty much off which means they are:
    • less exposed to power surges
    • and being hacked
    • don't cause any wear on components (RAM and HDs specifically)
    • and consume a trickle of energy.
  • The backup harddrives can also be swapped out with the ones in A.b should any of those go down and the ZFS Raidz rebuilt while another HD is sourced (waiting for a sale if necessary).
  • Accidentally deleted files are recoverable up until the second HD synchronizes which is a two week window.
  • A diskless client means less maintenance because
    • it doesn't need a kernel running from a disk (which means no backups of said kernel)
    • nor does the kernel have to be updated (which breaks things)

If the system has a weakness, it is the system disk on A.b, however it is an SSD which is more reliable and imaged and stored separately. None of the backups are offsite either, however the volume of data is much too large to warrant making that feasible. The entirety of the ZFS array is also not backed up, however, it is made up of roughly two parts, changing data (a revolving cache of MythTV recordings) and static data (videos, pictures, NAS data including Windows backups, etc.)

Hauppauge HD PVR Audio Sync

The 1445 model of the HD PVR has an audio sync problem (with 0x15 firmware version and up, bug documented here) and the fix is to add:

export FORCE_DTS_TIMESTAMPS=1

to /etc/mythtv/session-settings (thanks!) on each Myth Frontend.

Updating the HD-PVR firmware likely works too if you have a newer kernel (3.3 or better)

Running two HDPVRs on one machine

Upon reboot, or having the HDPVRs power cycled, the /dev/video<x> designation for the two is completely random. It is necessary to modify /etc/udev/rules.d/78-hdpvr.rules to include:

#HDPVR - Hdpvr - Lowdef
KERNEL=="video[0-9]",ATTRS{product}=="Hauppauge HD PVR",ATTRS{serial}=="00A8C18F",SYMLINK+="videoSD1"
KERNEL=="video[0-9]",ATTRS{product}=="Hauppauge HD PVR",ATTRS{serial}=="00A4D9C0",SYMLINK+="videoHD1"

where the file name "78-hdpvr.rules" specifies the load time, "78" which is after "60" which is important and then the rest "-hdpvr.rules" is arbitrary. The serial numbers are taken from a "dmesg" command in a section like this:

[ 6307.727781] usb 1-3: Product: Hauppauge HD PVR
[ 6307.727783] usb 1-3: Manufacturer: AMBA
[ 6307.727784] usb 1-3: SerialNumber: 00A49D0D

There is more information in the wiki including a useful command that tests the udev rules:


After getting the udev commands working I learned of another simpler way that likely works better:

Ubuntu 12lts now has explicit entries for each HDPVR under '/dev/v4l/by-id' and '/dev/v4l/by-path' and that's how I reference them in mythtv-setup 
LIRC from two HDPVRS

And because you have two HDPVRs, you likely have two LIRC blasters, which show up when "modprobe lirc_zilog" is run like this (in dmesg):

lirc_zilog: Zilog/Hauppauge IR driver initializing
lirc_zilog: probing IR Tx on Hauppage HD PVR I2C (i2c-3)
i2c i2c-3: lirc_dev: driver lirc_zilog registered at minor = 1
lirc_zilog: IR unit on Hauppage HD PVR I2C (i2c-3) registered as lirc1 and ready
lirc_zilog: probe of IR Tx on Hauppage HD PVR I2C (i2c-3) done
lirc_zilog: probing IR Tx on Hauppage HD PVR I2C (i2c-2)
i2c i2c-2: lirc_dev: driver lirc_zilog registered at minor = 2
lirc_zilog: IR unit on Hauppage HD PVR I2C (i2c-2) registered as lirc2 and ready

Again, you need a 78-hdpvr-lirc.rules like this:

KERNEL=="lirc[0-9]", ATTRS{product}=="Hauppauge HD PVR", ATTRS{serial}=="00A8C18F", SYMLINK+="lircSD1", MODE="0666

Useful commands for testing if your udev rules are working are:

udevadm info -a -p $(udevadm info -q path -n /dev/lirc1)
udevadm test /devices/pci0000:00/0000:00:13.2/usb2/2-5/i2c-3/lirc/lirc1

(make sure the "i2c-3/lirc/lirc1" matches the output from the zilog in dmesg)

VNC server Xstartup

Getting a headless vnc server to run correctly took a bit of work and most of it came down to the /home/<user>/.vnc/xstartup which in my case was:

#!/bin/sh

# I don't think this line is used given it is for Windows Remote desktop software, XRBD which is not installed.
xrbd $HOME/.Xresources
xsetroot -solid grey

# Added this for some reason, now forgotten
export XKL_XMODMAP_DISABLE=1

# Absolutely necessary for the dbus message bus to work when apps like update-manager,
# etc. require elevated privileges  (Took me three hours to figure why pressing "install" 
# on the Ubuntu software Center did nothing!)
# [Update April 2015] $%&@$ bloody thing - I spent another five hours trying to fix this 
# problem again on Unity and it turns out, it is by design!  WTF?
# Remote RDP VNC sessions are not considered "active" by Ubuntu/Unity, only desktop sessions
# and therefore you do not get the privilege elevation pop up.  More annoyingly, it fails silently, 
# as pressed buttons simply do nothing.  Even the log errors that update-manager, synaptic and software-center
# poop out are about dbus introspection failing, which is a red herring.  The polkit-agent simply refuses to run!
# And it doesn't have to be this way as I have a perfectly working system with xfce4 running the same linux core.
# Works around are supper messy including hand-editing the policykit default XML files or PAM.
# I end up running synaptic, the update-manager from the command line, which means I end up forgetting the
# bloody name of the programs: Was it gnome-update-manager or update_manager or ishouldnothavetorememberthis. 
# Yes a bit annoyed.
unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS

# Kicks off the gnome 2d desktop which includes Unity 
# - need some packages installed for this to work, gnome-core??
gnome-session --session=ubuntu-2d &

To control the size, start the server with "vncserver -geometry 2500x1500" which looks good on a 2560x1600 monitor. Need "vncserver" (vnc4?) installed - avoiding Xtightvnc for now as consistently lost connect after a while.

Kernel devinput instead of LIRC

With Ubuntu 12.04 64 bit, the drivers for the remotes are now in the kernel and LIRC, at least in my case, wasn't needed.

Here is how I moved from LIRC to devinput:

1. apt-get install ir-keytable, then run ir-keytable which returns a description of the receiver capabilities, i.e:

Found /sys/class/rc/rc0 (/dev/input/event4) with:
  Driver mcesub, table rc-rc6-mce
  Supported protocols: NEC RC-5 RC-6 JVC SONY LIRC other
  Enabled protocols: NEC
  Repeat delay = 500ms, repeat_period = 125ms

2. using 'ir-keytable -t' (replaces the old "irw") I found that my remote was being recognized by the MCE USB receiver, because for each keypress, it produced something like (pressing the Enter key for example):

<some long time stamp>: event MSC: scancode = 45bc44

3. A grep of all the remotes for that scancode in /lib/udev/rc_keymaps returned no results which meant my remote had not been recorded.

4. I created a new keymap file under /etc/rc_keymaps with the following necessary format:

==> NEC <==
# Comments go here
0x45bc44 KEY_ENTER
...

5. The location of the file isn't important as you reference it directly, but the format of it is, i.e. having the NEC or RC6, etc.

6. Then configured it in rc.local with:

ir-keytable -c --period=500 -p NEC -w /etc/rc_keymaps/<name of your remote>

where

"-c" clears the previous keytable, 
"--period" (or "-P") sets the repeat period to slow down repeats
"-p" enables just the NEC protocol (or RC-6, SONY, etc) and 
"-w" adds the keys from the keytable file.

7. Pressing the key on the remote then gets passed to MythTV directly as if the remote was like a keyboard, sweet! No .lirc/mythtv configuration or LIRCD daemon to crash, nor worrying about which /dev/input/eventxxx the receiver is plugged into.

  • To stop the remote repeating too quickly, I ran the "Keyboard" app under Unity to change the "Delay" from short to long and the "Speed" to slow.


8. All this was figured out after a couple hours of googling and trial and error. Lots of people still use the kernel drivers to send the key inputs to an LIRC daemon, but I couldn't get that to work and the kernel drivers weren't hard to setup.

Distinctive Ringing (no longer used)

Setting up distinctive ringing for ring groups.
1. Via A@H or FreePBX, create two ZAP extensions for each ZAP channel,

a. The lower number one, i.e. ext 201 uses a "dial" command "ZAP/1r2", where 1 is the ZAP daughter card in slot 1 and "r2" stands for ring type 2 (or r3 or r4). This extension will ring the phone using the distinctive single/long ring.
b. The second ZAP extension, i.e 211, uses just "ZAP/1" as the dial command and will ring the same ZAP phone with a regular US ring tone.

2. Then set it up such that internal callers go to ring group 200 which rings all the distinctive ZAP extensions, i.e. x201, etc. When external callers call, their separate ring group calls x211, etc. with regular ring commands.

The great thing is that the modifications are done through A@H and don't need custom modification of any script.

  • Voicemail is emailed to us now and deleted. To avoid entering a password every time for voice mail, add the voice mail's extension "sXXX" into extensions.conf -> app-messagecenter instead of "default", i.e. s402

Caller ID

The code for sending caller ID from Asterisk to MythTV Frontends. Requires XYAC. Use the startup script below to start up the XYAC daemon at login.

exten => s,1,System(/bin/echo -n -e "@CALL${ARG1} ~${ARG2}" | nc -w 1 127.0.0.1 10629 &)

Note:

  • The "&" at the end runs the "nc" daemon in the background where it can fail silently and not cause delays in answering the calls. Without the "&", the far side can intermittently time-out just as the phones start ringing.
  • This code lives in /etc/asterisk/extensions_custom.conf file
  • The Makefile for XYAC needs to have two changes to compile under Ubuntu 12.04:
INCLUDES=-I/usr/include

and

xyac: xyac.o xosd.o
   $(CC) -o xyac $(LIBS) xyac.o xosd.o $(LFLAGS)

(the gcc compiler now uses the --as-needed flag instead of --no-as-needed which means it has to have the library after the compiling module, otherwise the linker doesn't think any of the functions in the library are needed)

  • The line to startup XYAC and increase the font size is:
 xyac -c yellow --align=center -f -*-*-*-*-*-*-72-*-*-*-*-*-*-* --shadow=4 --delay=45 
The line after the "-f" is the font specifier in XLFD format and depends on what fonts are installed, but the generic one shown uses the first system font with a size of 72, which on a 1080p screen looks big. A good way to find what fonts you have installed is to use "xfontsel" and a common system font is "Bitstream", which you would specify like this
-bitstream-*-*-*-*-*-72-*-*-*-*-*-*-*

Idle Three Hour Sleepd Daemon (no longer used)

* Below is a method of using the sleepd daemon to shut off a MythTV frontend after three hours and to pop up a 30 second warning that the user can cancel. - no longer used but left for others.

# This is called script is called from xfce's autostart manager under "Settings" -> "Sessions and Startup"
#!/bin/sh

# Start up the sleep daemon
# -U waits for 10800 seconds
# -i watches IRQ 4, the remote to reset the counter
# -a Also watches the mouse and keyboard
# -s then runs the standby.sh script
/usr/sbin/sleepd -U 10800 -i 4 -a -s /home/mythtv/standby.sh

Premptable Shutdown Script (no longer used)

* Here is the standby.sh script called by the sleepd daemon for shutdown: Left here for history sake, but no longer used.

Note: shutdown requires root access which means using sudo

#!/bin/sh

# Play the sounds
#cd /home/mythtv
#mplayer -softvol -softvol-max 20 cut_power.mp3 &

(
# First wait 30 seconds
echo "10" ; sleep 2
echo "# Oh, it's nothing, whew!  That was close! On with the shutdown" ; sleep 2
echo "20" ; sleep 2
echo "# Ok, on with the shutdown - Resetting the bubble blaster!!" ; sleep 2
echo "30" ; sleep 2
echo "# Bubble blaster reset, now we discharge the Nancy tubes!!" ; sleep 2
echo "40" ; sleep 2
echo "# Tubes discharge, don't forget to put out the teddy sparker!!" ; sleep 2
echo "50" ; sleep 2
echo "# Sparker stuffed! Now we can cancel the impending dog runs!!" ; sleep 2
echo "75" ; sleep 2
echo "# Dog runs cancelled - Now finally ready to stop all run-away princesses" ; sleep 3
echo "# Shutting dddoooowwww [NO CARRIER]" ; sleep 2
echo "100" ; sleep 2
# First kill the sleep daemon 
sleepctl off
# Hibernate
sudo pm-hibernate
# Upon resume, restart the sleep daemon
sleepctl on
) |
# Zenity is the program used for message boxes by Gnome/xfce
zenity --progress \
--title="Shutting Down MythTV FrontEnd Four" \
--text="Wait, there is something wrong!!!" \
--percentage=0 --width=400  --auto-close

Backend Slave and Master MythTV Backend Process Reset (no longer used)

The WAF had been lower of late mostly because our master/save system seems to hang once in a while. I added a method whereby the master, slave backends and the frontend are reset at the push of a remote button. - no longer used.

A local script is run when the button is pushed that does the following (call it "restart_entire_myth.sh"):

#!/bin/bash
echo "---> First Kill the Mythfront end..."
killall mythfrontend.real
echo "---> Next, restart the local mythtv server... "
sudo restart mythtv-backend
echo "---> Now wait a few seconds and then kill the remote MythTV backend server ...."
sleep 2
echo "---> Restarting remote process...."
ssh <username>@<hostname> -t "sudo restart mythtv-backend"

I'm not sure the "-t" is necessary now, as I was using it for debugging purposes.

To get that SSH command working, you need to have a properly working SSH link between the computers that doesn't require a password, but uses a public key authentication mechanism. There are lots of resources on the net to set this up and one useful one is here.

You also need to allow a regular MythTV user to restart the processes on both the master and slave backends. With root privileges on each, run visudo and add at the end of the file

<username> <hostname>= NOPASSWD: /sbin/restart mythtv-backend

Where <username> is the user your are either login into via SSH or under which you are running the MythTV frontend. You can test the NOPASSWD setting by exiting root privileges (#exit) and then using "sudo -k" to remove the root access. Then try restarting the mythtv-backend process ($restart mythtv-backend) If the process reset correctly without root access, then it will work for the local or SSH user.

Once you can then run the "restart_entire_myth.sh" script mentioned above to see if actually restarts all the appropriate processes. Once the restart script works, then it is necessary to tie it into the power button on the remote. The first step is to modify the .lircrc file that is used by MythTV to make sure it passes the command to run to "irexec"

begin
 remote = yourremotename
 prog = irexec
 button = Power
 config = restart_entire_myth.sh
 repeat = 0
 delay = 0
end

The directory where "restart_entire_myth.sh" lives is in the path used by the OS to check for files (can't remember how to set this up), but worst case, use the direct path (i.e. /home/my_custom_scripts/restart_entire_myth.sh). The next tricky bit is to get irexec to take input from the right screen. I also had issues with irexec not working unless it was run in a TTY capable terminal (no idea why). To get around these issues, I ran irexec via a script that run when the MythTV user logged in:

#
if [ ! "$(pidof irexec)" ]
then
	sleep 10
        echo "Irexec is not started. Starting Irexec..."
        xfce4-terminal -H -x "irexec"
else
        echo "Irexec is already started."
fi

The "-H" is probably not necessary and just holds the window open, but the "-x" runs the "irexec" command. You can also run this script and test to see if the button press from the remote is getting caught and running the restart script, the output of which appears in the terminal window. I use xfce desktop obviously, but I'm sure the same can be done for the Gnome and KDE terminal programs.

6200ch Change Channel Script

On reboot, the firewire driver randomly grabs a port which means calling the 6200ch change channel program with the previous setting has a fifty percent change of failing (or working if you are an optimist). MythTV would then start missing all its recordings.

The work around was to try both ports and return 0 like this:

 #/bin/bash
if /home/custom_scripts/6200ch -n 0 -p 0 $1 
	then exit 0
else
	 /home/custom_scripts/6200ch -n 1 -p 0 $1
fi

exit 0

(Instead of calling the 6200ch program directly, MythTV calls the change_chan.sh script above)

MythTV would then get back that the channel change worked and can go on its merry way.

The Plus and Minus of MythTV

  • Note this was a comment added to an article comparing TIVO 3 and MythTV.

I've been a MythTV user for a couple years now and there are a number of things the article missed or glossed over ("+" means a feature and "-" a problem).

(+) Plugins - MythTV has lots of plugins that let you do everything from surf the web, view local weather information, play old style ROM arcade games, watch recordings over the LAN on a windows machine, view a slide show of pictures, to watching XviD encoded videos, all of which we use. The video collection has been of great utility with kids. The usefulness of the plugins was a big reason I chose MythTV and have stayed with it. (Actually TIVO was never considered because of the subscription costs)

(-) Setup and Maintenance - It should not be underestimated how much time it can take getting MythTV to work and then maintaining it. As a MythTV user you will have to know the Linux command line, how to download and install drivers and fix things when they inevitably don't work. If you haven't installed a PCIe card before, and don't like googling stuff to figure out why it doesn't work, then MythTV is not for you. MythTV will probably not save money over a commercial system either because I've found that the money I would have spent on a subscription went into hardware upgrades.

(+) The flip side to the cost and time commitment is that MythTV has immense flexibility, for example, in the size and redundancy of storage capability, the type of peripherals and the split arrangement between backend, slave backend and frontends. Take our ever growing video collection (most ripped from DVDs, fyi), where any new videos are backed up onto a separate machine each night (rsync'd). Try doing that with a TIVO.

(+) The machines are also full blown general purpose computers that can do other things. For example, once you have sunk the money into the MythTV hardware, you can use the same machines for VoIP (via Asterisk) which really saves us money, about $100CND a month. The pay back period for the extra hardware to do VoIP was less then a year, and gets cheaper as time goes on. We use dedicated Digium TDM-400 cards that cost ~$500CND about a year ago. Again, our long distance costs are pennies per minute, each phone is a separate extension, we can make multiple outgoing or incoming calls, late-night callers are prompted by a menu first before the phones ring, we can control our own caller-ID, etc.

(+) DRM You can now buy the Hauppauge HD-PVR which takes the component analog outputs and converts it to a digital USB H.264 MPEG4 stream that works great with MythTV. "5C" encrypted channels can be recorded.For HD tuners using firewire and 5C flagged channels, there is no current work around (as of Feb. 21st, 2007).

Outstanding Issues

  • Not directly related to MythTV but, every couple of months, mdadm kicks all the HDs out of the RAID 5 array on System B. Thankfully, a reboot brings the array right back up, but very strange. This was a hardware issue, which after the MOBO upgrade, is fixed.
  • Every once in a while, the front end will crash, and it seems that a reset of all the MythTV processes is required. I have created a script, which when a "power" button is pressed on the remote, resets the local master backend and frontend and through a SSH tunnel it opens on the fly, resets the remote MythTV backend process (that took a bit of debugging to get a remote user to reset a different user's process) - The latest 0.25 version of MythTV now has a frontend restarter built in and my custom script was created multiple backend processes which completely messed things up. Stability is still an issue over weeks and weeks.
  • As of January, 2013, the HDPVR driver does not have the ID code, 4903, for my HDPVR unit which means /dev/video does not show up and fixing the problem means editing the source and recompiling. Because recompiling a driver is a bitch, I used a hex editor and changed the driver directly (Warning: This is the WRONG way to fix this problem. Don't ever do it. Ever. However, it did work for me in 10.04. Use a hex editor to edit hdpvr.ko, search for text 4901 and replace with 4903, search for hex 0149 and replace with 0349" There is a great thread on this problem here.) Ubuntu 12.04 doesn't have this problem.

Other Power Users

As of June 2015, I went through all the users listed in the categories below and found a few who are still updating their pages (within the last two years):

A data point of note is that most of the other user accounts listed were created about 2006 to 2008 and then stopped being updated. Was there a Peak MythTV event in 2007?