Difference between revisions of "User:Lwoggardner"

From MythTV Official Wiki
Jump to: navigation, search
(Proposed Upgrade approach using containers)
(override settings?)
(One intermediate revision by the same user not shown)
Line 611: Line 611:
  
 
Since the ip addresses and ports, both for connecting *and for listening* are stored in the central database, if there are any remote frontends or backends then these ports must be exposed publically.  Therefore the easiest thing to do is to launch the images with --net host and not worry about bridging.
 
Since the ip addresses and ports, both for connecting *and for listening* are stored in the central database, if there are any remote frontends or backends then these ports must be exposed publically.  Therefore the easiest thing to do is to launch the images with --net host and not worry about bridging.
 +
(Alternative could use "override-settings" ???)
  
 
This does means that to run a test version on the same hardware requires a second set of ports.
 
This does means that to run a test version on the same hardware requires a second set of ports.
Line 651: Line 652:
 
The whole point of doing docker is to enable this...
 
The whole point of doing docker is to enable this...
  
PROD "master" runs on default port set (3306, 65xx), with PROD_LATEST backup
+
PROD "master" runs on default port set (3306, 65xx),
  
Part the first - create a new working system, empty of data
+
 
# Create "new_master" backend - set SQL_PORT 9906)
+
 
# Restore from PROD_LATEST << starting point for configuration- skip this step for new hardware
+
Part the first - create a new temporary system from a snapshot of PROD
# Run mythtv-setup to upgrade schema
+
 
# Run SQL script to update ports 65xx -> 99xx
+
# Create and mount LVM snapshot of mythtv filesystem.
# SQL script to empty the partial restore tables (addition to mythconverg_restore.pl?)
+
# Create "new_master" backend, with volumes linked to snapshot - set SQL_PORT 9906, and optionally supply a previous CONFIG backup. The process to initialise a master backend is then...
# Start "new_frontend"(s) attached to "new_master"
+
## Database restored from latest backup in mythtv filesystem
 +
## Schema upgraded by mythtv-setup
 +
## If CONFIG backup is available this is merged in by...
 +
### taking a new UPGRADED backup
 +
### restoring from CONFIG
 +
### running a SQL script to empty partial restore tables (change to mythconverge_restore.pl?)
 +
### running mythconverg_restore.pl to perform a partial restore, with plugin-data, from UPGRADED
 +
## Run SQL script to update ports 65xx -> 99xx
 +
# Create "new_frontend"(s) attached to "new_master", ports etc automatically mapped to 99xx
 
# Configure/Test, recordings, guide data, front end playback, remote controls, music, images, dvds etc.
 
# Configure/Test, recordings, guide data, front end playback, remote controls, music, images, dvds etc.
# --- time passes, family keeps using PROD, I keep testing until all is working ---
+
# --- time passes, family keeps using PROD, I keep testing until all is working (or we run out of space in the snapshot)
# SQL script to empty the partial restore tables (addition to mythconverg_restore.pl?)
+
# Copy backups from this environment to somewhere outside of the snapshot to be used as CONFIG for subsequent iterations.
# Backup as NEW_READY
+
# Drop the snapshot, and containers
 +
# Repeat as necessary to refresh data from PROD
  
Part the second -- test restore of data
+
Part the second -- Cutover
# Recreate "new_master" backend - with prod storage group volumes linked in (as read-only if like me you don't have backups)
 
# Restore from PROD_LATEST
 
# Run mythtv-setup to upgrade schema
 
# Backup to create PROD_RESTORE
 
# Drop database and restore from NEW_READY
 
# Partial restore, with plugin data, from PROD_RESTORE
 
# Apply data fixes from previous test cycles (see below)
 
# Start backend, frontends
 
# Test restored data is working appropriately, including data expiry etc..
 
# if broken...
 
## Remember actions (sql scripts?) to fix restored data
 
## Fix configuration data, run SQL script to empty partial restore tables and recreate NEW_READY backup
 
## Repeat
 
  
 +
# Stop master backend, take a FINAL backup, stop all prod container(s)
 +
# Create an LVM snapshot of mythtv filesystem (readonly for rollback)
 +
# Create "new_prod" backend, with volumes linked to prod mythtv filesystem, set SQL_PORT 3306, and last good CONFIG backup from above.
 +
# Start backend,front-ends - and away we go.
  
Part the third -- Cutover
+
Part the third -- uh-oh ... rollback
# Stop master backend, take a final backup, shutdown prod container(s).
+
# Stop and discard the new prod containers.
# Repeat the restore of data process above (including tests!)
+
# Restore the LVM snapshot.
# Stop "new_master" and "new_frontend"(s)
+
# Restart the old prod containers.
# SQL script to update settings back to prod port set (3306, 65xx)
 
# Restart "new_master" and "new_frontends" - which are now the new prod.
 
# Voila!
 
  
Part the fourth -- Uh-oh.... rollback
+
Part the fourth -- point of no return
# Throw away "new_master" and "new_frontend"(s)
+
# Remove the LVM snapshot
# Restore files and directories (if you had a copy/backup) - eg recordings expired by broken new system
+
# Discard old prod containers
# Restart old master - no db restore required, restart frontends
 
# Remove orphaned files (recordings etc taken in new, broken system)
 
# Start again...
 

Revision as of 23:52, 27 August 2014

I was keeping track of where I was at with Myth on my home wiki, but then figured I might as well share it here!

My Setup

MythBuntu 12.04 with separated front-end/backend

  • Live TV/TV recording Free to air HD/SD (from Sydney, Australia)
  • Multi-rec
  • DTS/AC3 passthrough including 5,1 surround digital out
  • VDPAU
  • MythMusic
  • MythGallery
  • MythVideo
  • Frontend Suspend to RAM + Wake On USB
  • Backend Suspend to RAM

Backend

Hardware

  • HDDs totalling 1.2Tb
  • Intel S775 Core 2 Q6600 Quad-Core CPU
  • Gigabyte S775 GA-G33M-S2 Motherboard
  • 4GB RAM
  • PCI Hauppauge Nova-T-500 MCE(OEM) Dual DVBT tuner
  • PCI Hauppauge Nova-TD 400 Dual DVBT tuner

Software

Configuration

  • Channel 9, Gem, GO will not tune. The dtv_multiplex.network_id is null - set to 4114 seems to fix things.

Shepherd grabber

Standard install except cron job is disabled and allow mythtv to run mythfilldatabase. This is to allow mythtv to also be in control of suspend.

Suspend to RAM on idle

Why Suspend to RAM and not just a full shutdown that would "just work"? Partly due to boot speed and partly just because.

Overview:

  • mythbackend controls idle detection, supported by a custom script to check other reasons to stay up, and sudo rules to allow passwordless root activity
  • suspend needs to unload dvb/usb modules which requires stopping mythbackend which means triggering suspend via upstart
  • wake up for recordings triggered by /sys/class/rtc as per ACPI_Wakeup, hwlock-save update disabled via upstart override
  • wake up for daily processing (anacron) triggered by cron running on my (always on) DD-WRT enabled router issuing a WOL
  • wake up for serving adhoc web requests (mythweb et al) triggered by [custom router script]

Settings (set via mythweb as server is basically headless)

Name Setting Value Why
Pre ShutdownWakeUp Check Command preSDWUCheckCommand /home/mythtv/bin/check-idle.sh check if i'm logged in, or machine is in use for some other reason
Block ShutdownWakeUp without client blockSDWUwithoutClient 0 because this is a backend only machine
Set Wakeup time command SetWakeuptimeCommand sudo /usr/local/sbin/set-wakeup.sh $time
Server Halt command ServerHaltCommand sudo /usr/local/sbin/emit-suspend.sh can't suspend directly because we need to shutdown mythbackend during suspend
Wakeup time format WakeupTimeFormat time_t as per ACPI wakeup page to use /sys/class/rtc
/home/mythtv/bin/check_idle.sh
#!/bin/bash

ACCESS_LOG="/var/log/apache2/access.log"
LOG_AGE="5 minutes ago"

check_frontend() {
# Not this doesn't care about what the FE is doing, just that it is up
expect - $1 <<'END_EXPECT'
#exp_internal 1
set timeout 2
log_user 0
set prompt "# "
set cmd "query location\r"
set frontend [lindex $argv 0]
spawn netcat $frontend 6546
  expect {
        timeout { send_user "\nNo prompt\n"; exit 1 }
        eof { send_user "\nNo output\n"; exit 1 }
        $prompt
  }

  send $cmd
  expect  {
       timeout { send_user "\nNo response\n"; exit 1 } 
       eof { send_user "\nPremature EOF\n"; exit 1 }
       -re "query location\r\n(.*)\r\r\n# "
  }       
  set location $expect_out(1,string)
  send "exit\r"
  expect eof
  send_user "MythTVLocation:$location\n"
END_EXPECT

}

log() {
  logger -s -t "mythtv-checkidle"
}

# Regardless of WOL settings and going into standbymode the 0.25 front-end
# simply cannot cope with the backend dropping out unexpectedly so we
# keep the BE up until the FE suspends itself
check_frontend mythion && {
    echo "Frontend is still up" | log
    exit 1
}

# anacron is running daily, weekly or monthly jobs
pgrep 'anacron' && {
    echo "Anacron job is running" | log
    exit 1
}

# apache request in last 5 minutes
if [ -f $ACCESS_LOG -a `stat -c %Y $ACCESS_LOG` -gt `date --date "$LOG_AGE" "+%s"` ] ; then
    echo "Web server in use" | log
    exit 1
fi

# users are logged in
if [ `who | wc -l` -gt 0 ] ; then
    echo "Users logged in" | log
    who | log
    exit 1
fi

/usr/bin/mythshutdown -v --check | log

exit ${PIPESTATUS[0]}
/etc/sudoers.d/mythtv
#Enable mythtv to set the real time clock and emit the suspending event without a password prompt

%mythtv ALL = NOPASSWD: /usr/local/sbin/emit-suspend.sh, /usr/local/sbin/set-wakeup.sh
/usr/local/sbin/set-wakeup.sh
#!/bin/sh
#$1 is the first argument to the script. It is the time in seconds since 1970
#this is defined in mythtv-setup with the time_t argument

echo 0 > /sys/class/rtc/rtc0/wakealarm      #this clears your alarm.
echo $1 > /sys/class/rtc/rtc0/wakealarm     #this writes your alarm
/usr/local/sbin/emit-suspend.sh
#!/bin/sh
# Suspend via upstart
initctl emit suspending
/etc/init/suspending.conf
start on (suspending)

script
   logger -t "suspending" "Suspending via upstart" 
   pm-suspend
end script
/etc/pm/sleep.d/50_mythtv
#!/bin/sh
# The dvb modules need to be unloaded to allow suspend/resume to work
# To unload the modules mythbackend needs to be stopped
# 
# NOTE: Suspend cannot be executed in a child process of mythbackend itself (eg via server halt command)
# since it will simply abort at the point we stop mythbackend below. If this is required use upstart or some
# other means to decouple pm-suspend from the mythbackend process.
#!/bin/bash

LOG=/var/log/mythtv/mythsuspend.log
MODULE="dvb_usb_dib0700"

echo "Will now $1 mythtv" | tee -a $LOG

case $1 in
  suspend|hibernate)
        
        stop mythtv-backend 2>&1 | tee -a $LOG
      
        while [ `lsmod | grep "^${MODULE}" | wc -l` -gt 0 ]
        do 
            sleep 1
            modprobe -r $MODULE 2>&1 | tee -a $LOG 
        done
  ;;
  resume|thaw)
        
        modprobe $MODULE
        start mythtv-backend | tee -a $LOG
  ;;
esac

Frontend

Hardware

  • ZOTAC ionitx-d-e board, 2G memory
  • Szrealan E-2011 case (tiny!)
  • External DVD player
  • 1Tb external e-Sata drive
  • BenQ PE8700 DLP Projector
  • Sony Bravia TV over HDMI
  • CambridgeAudio AV receiver


Software

  • MythBuntu 12.04 64bit
  • Synergy to allow laptop to be the keyboard/mouse.

Configuration

Enable Wake on USB

We need to enable USB0 in /proc/acpi/wakeup

Note if USB2 is also enabled, then the board immediately wakes up regardless, unsure if this is due to another USB device

Seems to be enabled by default for Mythbuntu 12.04 but Current cheap as Chinese remote does not send the wakeup signal from the power button unfortunately

Auto-start X

  • disabled lightdm in via upstart override (echo 'manual' > /etc/init/lightdm.override)
  • Simple mingetty startup with a tty definition and a startx script in the console
  • user .xsession linked to /usr/share/mythbuntu/session.sh so we get the same xfce startup.
/etc/init/tty7.conf
# tty7 - mythtv
#
start on runlevel [23]
stop on runlevel [!23]

respawn
exec /sbin/mingetty --autologin=mythtv tty7
~mythtv/.profile

Appended to detect if we are on tty7, in which case we start-up X with an optional layout specified in ~/.mythrc

Note that when X is stopped we automatically logout and respawn

start-frontend() {
MYTHRC="$HOME/.mythrc"
if [ -r $MYTHRC ] ; then
   . $MYTHRC
fi

LAYOUT_SPEC=
[ ! -z "$LAYOUT" ] && LAYOUT_SPEC="-layout $LAYOUT"

startx --  $LAYOUT_SPEC -logverbose 5
logout
}

tty | grep tty7 && start-frontend
xorg.conf

For PE8700 Samsung LCD. See also the tips for the NVidiaProprietaryDriver

The FX5200 cards were the cause of my DVI problems on my projector. Apparently they freak out a PCB component causing the EDID data to become corrupt. (a Gigabyte card lasted from Aug 2006 to Dec 2007. Fortunately the new nvidia driver allow you to use a CustomEDID to load the data from a file.

Option ExactModeTimingsDVI is also important for the projector.

Option "UseEDIDDpi" "false" or some equivalent control of DPI is important for the Samsung television otherwise you get really small fonts (fixed in 0.21)

There are two layouts, the default uses outputs via VGA to a Samsung LCD television, the second uses DVI to the PE8700 projector.

Section "ServerLayout"
    Identifier     "Main" # Output to Samsung LCD television
#LCD[0] VGA, LCD[1] HDMI
    Screen         0 "LCD[1]" 0 0
    Option         "Xinerama" "off"
EndSection

Section "ServerLayout"
    Identifier     "Projector"
    Screen         0 "Projector[0]" 0 0
    Option         "Xinerama" "off"
EndSection
 
Section "Module"
        Load    "glx"
EndSection

Section "Device"
    Identifier     "ION-GPU[0]"
    Driver         "nvidia"
    BusID          "PCI:3:0:0"
    Screen         0
    #This prevents the cursor being displayed on resume from suspend
    Option         "HWCursor" "off"
EndSection

Section "Screen"
    Identifier     "LCD[1]"
    Device         "ION-GPU[0]"
    Monitor        "AnyMonitor"
    DefaultDepth    24
    Option         "UseDisplayDevice" "DFP-1" # HDMI
    Option         "ConnectedMonitor" "DFP-1"
EndSection

Section "Screen"
    Identifier     "LCD[0]"
    Device         "ION-GPU[0]"
    Monitor        "AnyMonitor"
    DefaultDepth    24
    Option         "ConnectedMonitor" "CRT-1"
    Option         "UseEDIDDpi" "FALSE" #Computed DPI values not so good for TV display
    Option         "UseDisplayDevice" "CRT-1" # VGA out
    SubSection     "Display"
        Depth       24
        Modes       "1360x768" "800x600"
    EndSubSection
EndSection

Section "Screen"
    Identifier     "Projector[0]"
    Device         "ION-GPU[0]"
    Monitor        "AnyMonitor"
    Option         "ConnectedMonitor" "DFP"
    Option         "UseDisplayDevice" "DFP" # DVI out
    Option         "CustomEDID" "DFP-0:/etc/X11/benq-c.bin"

    DefaultDepth    24
    SubSection     "Display"
        Depth       24
        Modes       "1280x720" "800x600"
    EndSubSection
EndSection

Section "Monitor"
    Identifier   "AnyMonitor"
    Option       "DPMS"
    Option       "ExactModeTimingsDVI" "true"
    Option       "RenderAccel" "true"
EndSection

~mythtv/.mythtv/session
#!/bin/sh
LAPTOP=

export | egrep "DBUS_SESSION_BUS_ADDRESS|DISPLAY" > ~/.xsession-export
#unclutter -root -idle 2 -display $DISPLAY &

# Start a synergy client
synergyc $LAPTOP

# Start an idle detection loop
idle_mythtv.sh & 
~mythtv/bin/idle_mythtv.sh
#!/bin/bash

WAIT_STARTUP=300
WAIT_CHECK=60

# Don't idle immediately!
sleep $WAIT_STARTUP 

while true
do
        if check_idle.sh ; then
	    suspend_mythtv.sh
            # we only do this because although irexec responds fine, mythtv's
            # lirc connection can't reset itself on resume from suspend
            kill_mythtv.sh

	    # if kill_mythtv above works then we'll be dead here
            sleep $WAIT_STARTUP
        else
	    sleep $WAIT_CHECK
        fi
done
~mythtv/bin/check_idle.sh
#!/bin/bash

check_frontend() {

expect - $1 <<'END_EXPECT'
#exp_internal 1
set timeout 2
log_user 0
set prompt "# "
set cmd "query location\r"
set frontend [lindex $argv 0]
spawn netcat $frontend 6546
  expect {
        timeout { send_user "\nNo prompt\n"; exit 1 }
        eof { send_user "\nNo output\n"; exit 1 }
        $prompt
  }

  send $cmd
  expect  {
       timeout { send_user "\nNo response\n"; exit 1 } 
       eof { send_user "\nPremature EOF\n"; exit 1 }
       -re "query location\r\n(.*)\r\r\n# "
  }       
  set location $expect_out(1,string)
  send "exit\r"
  expect eof
  send_user "MythTVLocation:$location\n"
END_EXPECT

}

# Is frontend in standbymode?
check_frontend localhost | grep "standbymode"
if [ ${PIPESTATUS[0]} -eq 0 -a ${PIPESTATUS[1]} -ne 0 ]; then
  echo "Frontend is up and not in standbymode"
  exit 1
fi

OTHER_USERS=$(who | grep -v "mythtv" |  wc -l)
if [ ${OTHER_USERS} -gt 0 ] ; then
  who | grep -v "mythtv"
  exit 1
fi

# Backups in progress
if [ -f /var/local/backups/mythion_daily.inprogress ] ; then
   echo "Jenix/Mythion backups in progress"
   exit 1
fi

# anacron is running daily, weekly or monthly jobs
pgrep 'anacron' && {
    echo "Anacron job is running"
    exit 1
}
exit 0
~mythtv/bin/suspend_mythtv.sh
#!/bin/sh

# We use dbus-send because it seems to block until resume which is what we want
echo "Suspending"

dbus-send --system --print-reply --dest=org.freedesktop.Hal /org/freedesktop/Hal/devices/computer \
        org.freedesktop.Hal.Device.SystemPowerManagement.Suspend int32:2

echo "Resumed"
~mythtv/bin/kill_mythtv.sh

Clean restart of the front end

Kill mythtv and logout of xfce (respawn of tty causes immediate restart)

#!/bin/sh

[ -f $HOME/.xsession-export ] && . $HOME/.xsession-export
echo "killing mythfrontend"
pkill mythfrontend
#pkill unclutter
echo "xfce logout $DISPLAY"
xfce4-session-logout --logout --fast

LIRC Remote Control

Standard mythbuntu setup for MCE remote.

power button is mapped vie irexec to kill_mythtv.sh

Sound

Onboard sound

Settings for using the receiver. When connected to TV just use HDMI audio, requires "SPDIF1" to be unmuted

~/.asoundrc

Could probably do better 48/44kHz switching on the analog output between music and video but if that is important we just go digital passthrough and let the receiver do the work.

In myth config, you can see that default sound device is ALSA:myth-video, the passthrough device is ALSA:myth-passthru, and mythmusic uses ALSA:myth-music. the AC3PassThru and DTSPassThru settings are managed by the ruby program

#First line comment

pcm.myth-music {
  type copy
  slave.pcm "default"
}

ctl.myth-music {
  type hw
  card 0
}

pcm.myth-video {
  type copy 
  slave.pcm "default"
}

ctl.myth-video {
  type hw
  card 0
}

pcm.myth-passthru {
  type copy
  slave.pcm = "spdif"
}

ctl.myth-passthru {
  type hw
  card 0
}

minimyth.conf

TODO - Works nicely through the WRT310N router etc, but we have a hard drive have decided to stick with Mythbuntu for now.

Upgrades

Mythbuntu 12.04

Reconfigured from scratch for no particular reason

Mythbuntu 10.10

100% clean on server

front end was missing non-versioned symlink for vdpau nvidia library

ln -s /etc/alternatives/libvdpau_nvidia.so /usr/lib/libvdpau_nvidia.so

Mythbuntu 9.10

100% clean on server

front end needed /etc/events.d/tty7 moved to /etc/init/tty7.conf

Mythbuntu 8.10

0.21-fixes

upgrade smooth

Docker

Thinking about how to run MythTV in Docker containers for portability, helping with testing and upgrades, alternate configurations. None of this exists yet.

Images

Based on ubuntu because that's the reference build environment

  • mythtv-common - includes mythbuntu PPA, and installs mythtv-common.
  • mythtv-backend - phusion/baseimage + install backend from PPA (results in 1G image), init, sshd, mysql, cron (for mythfilldb and backups), mythbackend,
  • mythtv-frontend - simple install from PPA
  • mythweb - install from PPA

Containers

Backend

Since the ip addresses and ports, both for connecting *and for listening* are stored in the central database, if there are any remote frontends or backends then these ports must be exposed publically. Therefore the easiest thing to do is to launch the images with --net host and not worry about bridging. (Alternative could use "override-settings" ???)

This does means that to run a test version on the same hardware requires a second set of ports.

  • WORKING
    • able to use dvb apps to scan in a container - uses -v /dev/dvb:/dev/dvb --privileged.
    • launch mysql in baseimage, with mythtv user
    • launch mythbackend as mythtv user (run second runit service as mythtv user, which then runs the backend)
    • use nsenter (via docker-bash from phusion baseimage) to run mythtv-setup and scan channels. Need to copy xauth from host into the container.

Frontend

Sound and Video gonna be tricky.

Is backend worth doing if frontend fails due to tight hardware integration? maybe - some intermediate things - new configs, xmltv, replacement tuners, data migrations etc are all backend only. but - a mythtv upgrade requires new environment for both FE and BE, and this is the main reason to try containers. oh well, lets see how we go.

Video: Run X display on host, just run the app in the container forwarding X over a Unix socket. OpenGL & VDPAU/VAAPI might be fun http://maci0.wordpress.com/2014/05/02/run-any-applications-on-rhel7-containerized-with-3d-acceleration-and-pulseaudio-steam-pidgin-vlc/ http://www.slideshare.net/dotCloud/dockerizing-your-appli

or Video: Pass devices and run X in the container - http://www.jonnor.com/2010/03/hardware-passthrough-in-lxc-or-running-a-desktop-in-a-cgroup/

ALSA: Pass /dev/snd, enabled devices in cgroups - lxc config (libcontainer?)

PulseAudio: run PA on host and enable networking over unix socket?

LIRC: run daemon in a containter, passing devices, exposing unix socket.

Proposed Upgrade approach using containers

The whole point of doing docker is to enable this...

PROD "master" runs on default port set (3306, 65xx),


Part the first - create a new temporary system from a snapshot of PROD

  1. Create and mount LVM snapshot of mythtv filesystem.
  2. Create "new_master" backend, with volumes linked to snapshot - set SQL_PORT 9906, and optionally supply a previous CONFIG backup. The process to initialise a master backend is then...
    1. Database restored from latest backup in mythtv filesystem
    2. Schema upgraded by mythtv-setup
    3. If CONFIG backup is available this is merged in by...
      1. taking a new UPGRADED backup
      2. restoring from CONFIG
      3. running a SQL script to empty partial restore tables (change to mythconverge_restore.pl?)
      4. running mythconverg_restore.pl to perform a partial restore, with plugin-data, from UPGRADED
    4. Run SQL script to update ports 65xx -> 99xx
  3. Create "new_frontend"(s) attached to "new_master", ports etc automatically mapped to 99xx
  4. Configure/Test, recordings, guide data, front end playback, remote controls, music, images, dvds etc.
  5. --- time passes, family keeps using PROD, I keep testing until all is working (or we run out of space in the snapshot)
  6. Copy backups from this environment to somewhere outside of the snapshot to be used as CONFIG for subsequent iterations.
  7. Drop the snapshot, and containers
  8. Repeat as necessary to refresh data from PROD

Part the second -- Cutover

  1. Stop master backend, take a FINAL backup, stop all prod container(s)
  2. Create an LVM snapshot of mythtv filesystem (readonly for rollback)
  3. Create "new_prod" backend, with volumes linked to prod mythtv filesystem, set SQL_PORT 3306, and last good CONFIG backup from above.
  4. Start backend,front-ends - and away we go.

Part the third -- uh-oh ... rollback

  1. Stop and discard the new prod containers.
  2. Restore the LVM snapshot.
  3. Restart the old prod containers.

Part the fourth -- point of no return

  1. Remove the LVM snapshot
  2. Discard old prod containers