MoveRecording.sh
| Author | Andy Boff |
| Description | Move a completed recording from local storage on a Slave backend, into primary recording storage on the Master backend. |
| Supports |
move_recording.sh is intended to be used as a "Recording Finished" script to copy the recording from local storage on the slave backend onto the master backend.
By Andy Boff
Rationale
One common configuration for "centrally shared storage" is to share out the central storage from the Master backend using NFS. Unfortunately, this introduces a whole new area of performance tuning and optimisations, as well as the potential for performance degradation over time.
This script allows an alternative approach, where slave backends record to local storage, followed by an automated move of the recording to the master backend when the recording completes. This actually improves resilience, as in the event of a failure of the master backend host, the recordings active on the slave will complete (and the script will fail gracefully and leave the recording available on the slave).
Setup
First of all, save the script from this wiki page into a text file. I recommend installing the file to /usr/local/bin/move_recording.sh but you can put it wherever you want. If you choose a different location, adjust the commands below to suit.
Give the script executable permission:
# sudo chmod +x /usr/local/bin/move_recording.sh
Next, edit the script and customise the following variables (near the top):
- SSH_IDENTITY
- SSH_USER
- MBE_HOSTNAME
- MBE_PATH
- RSYNC_BWLIMIT
- FAIL_LOG
You will need a user on the master backend, which can remotely authenticated from the slave. Ideally, the primary user group of this user will be the same as the user that runs mythbackend on your master (which will assist with keeping the permissions of the copied files correct). Information about how to create this user is dependent on your distribution, and instructions on setting up SSH public key passwordless authentication are widely available elsewhere.
You will need to ensure that you can SSH from the user that runs mythbackend on the slave, to the target user on the master - the first connection is likely to require confirmation that the ECDSA key of the master is acceptable. A good way to do this is to download a copy of Big Buck Bunny (http://www.bigbuckbunny.org/) to the SBE storage directory, and run the script on the file, confirming it correctly gets moved to the master.
Finally, in mythtv-setup on the slave backend, configure the storage groups to point to local storage (this will override the master backend storage group settings, which would otherwise be inherited by the slave). Then under System Events, find the "Recording finished" entry, and set the associated value to be:
/usr/local/bin/move_recording.sh "%DIR%/%FILE%"
move_recording.sh
#!/bin/bash
# This script is designed to copy a completed recording from local
# storage on a SBE, to the "main" recording storage on the MBE.
# Correct configuration requires the "Master Backend Override"
# option to be set, so the master backend can still find the moved
# file, despite it not existing on the host that recorded it.
# This script should be configured as a "Recording Finished"
# script, on the slave backend that the files originate.
#
# It should be called with a single parameter: the file to move
# (e.g. "/usr/local/bin/move_recording.sh %DIR%/%FILE%")
#
#
# The following are required:
# ssh-agent, ssh-add, ssh, rsync
#
# It is recommended to create a new user on the MBE for this
# script (e.g. mythtv-sbe), and make its primary group the
# same as the Mythbackend user.
# Before relying on this script to transfer recordings,
# try transferring a non-trivially sized file from SBE source
# dir to MBE destination. It is likely that the ECDSA key will
# require approval the first time this script is run. This should
# be run as the mythbackend user on the SBE.
#
# ===================================================================
#
# You will need to configure the following:
# SSH identity key to use to authenticate on the master backend:
SSH_IDENTITY="/mnt/mythtv/mythtv-sbe-id_rsa"
# User to attempt to authenticate as on the master backend (make sure
# that the user the MBE runs as has the authority to delete files
# owned by this user):
SSH_USER="mythtv-sbe"
# The hostname of the master backend:
MBE_HOSTNAME="masterbackend"
# The path on the master backend to copy into (with trailing /):
MBE_PATH="/mnt/mythtv/default/"
# Copy speed limit (in KBytes/sec)
RSYNC_BWLIMIT=10240
# Location of a failure log:
FAIL_LOG="/mnt/mythtv/default/move-recording-failures.log"
#
# ===================================================================
#
SRC=$1
# register the SSH identity
eval $(ssh-agent)
ssh-add "${SSH_IDENTITY}"
# try and copy the file
DEST="${SSH_USER}@${MBE_HOSTNAME}:${MBE_PATH}"
rsync --append --chmod=ug+w -e 'ssh' "${SRC}" "${DEST}"
RSYNC_RESULT=$?
# if the rsync was successful, unlink the file locally
# if the file is open by SBE for streaming, it will remain available
# for the SBE until it closes it. Any future access should work via
# the MBE, due to the MBE override flag.
if [ ${RSYNC_RESULT} -eq 0 ];
then
# rsync returned code 0, so we take that as a success
# TODO ideally update the DB so the recording is now listed against
# the MBE, meaning MBE override no longer needs to be set.
# Unfortunately at the moment the only way is direct DB access
unlink "${SRC}"
unlink "${SRC}.*" # thumbs may have been created - we can let them recreate
# done
exit 0 # signal success
else
# the rsync failed. Append to a failure log. The file will remain
# available on the SBE
echo "Moving recording ${SRC} from SBE to MBE failed (${RSYNC_RESULT})"
touch "${FAIL_LOG}"
DT=$(date)
echo "${DT}: rsync error ${RSYNC_RESULT} for ${SRC} -> ${DEST}" >> ${FAIL_LOG}
exit ${RSYNC_RESULT} # signal (with the same error as rsync)
fi
