Tuning Linux for MythTV: Using cgroups

From MythTV Official Wiki
Jump to: navigation, search

This article will describe how to tune or partition your system's resources to guarantee (or best) the performance of MythTV. It is geared toward a multi-core CPU that runs both backend and frontend(s). This allows me to record 6 HD Streams (4 HDHRuns, 1 HDPVR and 1 remote HDPVR writing to this disk over NFS) while commflagging them all realtime and watching a 1080 program *finally judder free* !!

If you are uncomfortable with anything outlined here, DO NOT attempt to configure it. At a minimum, make sure it is running properly before you enable the services at boot.

Targeted Audience

  • Users with ample hardware that still get jitter/judder and/or video/audio latency
  • Users who use their system for more than just MythTV
  • Users who need more that nice/ionice deliver
  • Users who like to watch glitch free TV!

Brief Description

cgroups have been in the kernel for a while, but not heavily used as of yet.

Wikipedia - Brief description of cgroups

HomePage - Project Site for Userspace Apps

KernelDocs - Straight from the horses mouth (kernel implementation)

With upcoming kernels, automatic cgroup(s) will become standard allowing for very fine grained control of systems. By default, user/group/process control can be applied to each of the following:

cat /proc/cgroups 
#subsys_name	hierarchy	num_cgroups	enabled
cpuset	1	3	1
ns	0	1	1
cpu	2	5	1
cpuacct	3	2	1
memory	4	4	1
devices	5	2	1
freezer	6	2	1
net_cls	7	2	1
blkio	8	5	1

This allows, well, finer grain control than I could ever desire!

Prerequisites

The cgroup Libraries

- libcgroup

  • While cgroups can be configured on any system that has the proper kernel options compiled in and binary tools installed, this document should apply to RHEL6, and recent Fedora and Ubuntu distros as they all implement cgconfig and cgred services*

Frequency Scalers (only if you are interested in saving power)

- cpufreq-utils (should support all modern x86 CPUs)

- cpuspeed (RedHat)

- cpufreqd (*buntu)

Judder Free Setup

- You should already have your system configured for JudderFreePlayback (Even with the most powerful CPUs, you can get judder due to audio and video hiccups - this minimizes them)

The CGROUP Setup

For this guide, I will only be using:

  • cpuset (assign processes to cores)
  • cpu (assign time to cores)
  • memory (set swappiness of processes)
  • blkio (set IO thresholds for processes)

System Defaults (higher values have priority)

These values are the default RedHat values - I am unsure what Ubuntu uses (if anything).

  • cpuset.cpus (all)
  • cpuset.mems [0] (always zero on non-NUMA machines (everyone reading this)
  • cpuset.cpu_exclusive [0] (whether to let other processes run on this cpuset.cpus group)
  • cpuset.sched_load_balance [1] (can the kernel load balance across cpus in this cpuset)
  • cpu.shares [1024]
  • memory.swappiness [60] (0 - do not allow unless critical)
  • blkio.weight [500] (0-1000)
  • blkio.weight_device [8:0 500] (any mythtv discs in raw device number)

Groups I Configured

mythtv_high (I will run mythfrontend under this group)

group mythtv_high {
    perm {
        task {
            uid = greg;
            gid = greg;
        }
        admin {
            uid = root;
            gid = root;
        }
    }
    blkio {
        blkio.weight = 800;
    }
    cpu {
        cpu.shares = 2048;
    }
    cpuset {
        cpuset.cpus = 6-7;
        cpuset.mems = 0;
        cpuset.cpu_exclusive = 1;
    }
    memory {
        memory.swappiness = 0;
    }
}

As you can see, It lets the user greg add/remove tasks from the cgroup, and root modify the group itself.

  • I bumped the blkio priority to 800, doubled the CPU timeshare, assigned Core4 (thread 6&7) on my Core i7 CPU and told it no swapping of memory.

mythtv_med (mythbackend and some plugins)

group mythtv_med {
    perm {
        task {
            uid = greg;
            gid = greg;
        }
        admin {
            uid = root;
            gid = root;
        }
    }
    blkio {
        blkio.weight = 750;
    }
    cpu {
        cpu.shares = 1768;
    }
    memory {
        memory.swappiness = 0;
    }
}

Same as mythtv_high except the CPU timeshare is lower (but higher than the system default), it runs on all available cores, but has higher than default blkio weights while lower than mythfronted. It also is told to only swap as a last resort before the OOM killer gets called.

mythtv_low (mythcommflag/mythfilldatabase and other background tasks)


group mythtv_low {
    perm {
        task {
            uid = greg;
            gid = greg;
        }
        admin {
            uid = root;
            gid = root;
        }
    }
    blkio {
        blkio.weight = 200;
    }
    cpu {
        cpu.shares = 512;
    }
}

You can see the values here are lower than the default system values.

Config Files

/etc/cgconfig.conf

  • Here is a complete example file
# cat /etc/cgconfig.conf

mount {
	cpuset	= /cgroup/cpuset;
	cpu	= /cgroup/cpu;
	cpuacct	= /cgroup/cpuacct;
	memory	= /cgroup/memory;
	devices	= /cgroup/devices;
	freezer	= /cgroup/freezer;
	net_cls	= /cgroup/net_cls;
	blkio	= /cgroup/blkio;
}

group mythtv_low {
    perm {
        task {
            uid = greg;
            gid = greg;
        }
        admin {
            uid = root;
            gid = root;
        }
    }
    blkio {
        blkio.weight = 200;
    }
    cpu {
        cpu.shares = 512;
    }
}

group mythtv_med {
    perm {
        task {
            uid = greg;
            gid = greg;
        }
        admin {
            uid = root;
            gid = root;
        }
    }
    blkio {
        blkio.weight = 750;
    }
    cpu {
        cpu.shares = 1768;
    }
    memory {
        memory.swappiness = 0;
    }
}

group mythtv_high {
    perm {
        task {
            uid = greg;
            gid = greg;
        }
        admin {
            uid = root;
            gid = root;
        }
    }
    blkio {
        blkio.weight = 1000;
    }
    cpu {
        cpu.shares = 2048;
    }
    cpuset {
        cpuset.cpus = 6-7;
        cpuset.mems = 0;
        cpuset.cpu_exclusive = 1;
    }
    memory {
        memory.swappiness = 0;
    }
}

group system_high {
    blkio {
        blkio.weight = 1000;
    }
    cpu {
        cpu.shares = 4096;
    }
}

/etc/cgrules.conf

  • This file is what associates users/groups/processes with the above controllers
# cat /etc/cgrules.conf
#Each line describes a rule for a user in the forms:
#
#<user>			<controllers>		<destination>
#<user>:<process name>	<controllers>		<destination>
#
#Where:
# <user> can be:
#        - an user name
#        - a group name, with @group syntax
#        - the wildcard *, for any user or group.
#        - The %, which is equivalent to "ditto". This is useful for
#          multiline rules where different cgroups need to be specified
#          for various hierarchies for a single user.
#
# <process name> is optional and it can be:
#	 - a process name
#	 - a full command path of a process
#
# <controller> can be:
# 	 - comma separated controller names (no spaces)
# 	 - * (for all mounted controllers)
#
# <destination> can be:
# 	 - path with-in the controller hierarchy (ex. pgrp1/gid1/uid1)
#
# Note:
# - It currently has rules based on uids, gids and process name.
#
# - Don't put overlapping rules. First rule which matches the criteria
#   will be executed.
#
# - Multiline rules can be specified for specifying different cgroups
#   for multiple hierarchies. In the example below, user "peter" has
#   specified 2 line rule. First line says put peter's task in test1/
#   dir for "cpu" controller and second line says put peter's tasks in
#   test2/ dir for memory controller. Make a note of "%" sign in second line.
#   This is an indication that it is continuation of previous rule.
#
#
#<user>  	<controllers>  	<destination>
#
#john          cpu		usergroup/faculty/john/
#john:cp       cpu		usergroup/faculty/john/cp
#@student      cpu,memory	usergroup/student/
#peter	       cpu		test1/
#%	       memory		test2/
#@root	    	*		admingroup/
#*		*		default/

# MythTV Rules
greg:mythfrontend     blkio,cpu,cpuset,memory   mythtv_high/
greg:mythbackend      blkio,cpu,memory          mythtv_med/
greg:mythfilldatabase blkio,cpu                 mythtv_low
greg:mythcommflag     blkio,cpu                 mythtv_low/
mysql:mysqld          *                         system_high/
root:lircd            blkio,cpu,memory          system_high/
root:sshd             *                         system_high/

# Catchall Rule
*                     *                         sysdefault/

You should be able to see from the commented examples and my uses what is possible as far as user/group/process definitions there.

My MythTV processes all run as user:group greg:greg in these examples

System Scripts

  • RedHat makes it easy to enable the default cgroup for all system process by enabling
/etc/sysconfig/cgconfig

CREATE_DEFAULT=yes

The default startup does not allow the cpuset.cpus_exclusive setting though, so I added it to rc.local

/etc/rc.d/rc.local

echo "0-5" > /cgroup/cpuset/sysdefault/cpuset.cpus

This allocates all current running processes in the sysdefault cgroup to cores 1-3.

The MythTV Setup

Mythtv System Events

Event Handler Script

Here is the script I use to throttle up my CPUs and isolate mythfrontend to them:

# cat /home/greg/.mythtv/bin/playback.sh

#!/bin/bash

CPUS="6 7"

if [ $1 = $(hostname) ]; then
    if [ $2 = "START" ] || [ $2 = "UNPAUSE" ]; then
        # If you do not want to dedicate a core when not watching mythtv, then you
        # can un-comment the following line to only set it when watching video
        #sudo sh -c 'echo "0-5" > /cgroup/cpuset/sysdefault/cpuset.cpus'
        for i in $CPUS; do
            sudo /usr/bin/cpufreq-set -c $i -g performance
        done
    elif [ $2 = "STOP" ] || [ $2 = "PAUSE" ]; then
        # Uncomment the following to restore all CPUs on playback stop/pause
        #sudo sh -c 'echo "0-7" > /cgroup/cpuset/sysdefault/cpuset.cpus'
        for i in $CPUS; do
            sudo /usr/bin/cpufreq-set -c $i -g ondemand
        done
    fi
fi

  • Obviously the script will need to be modified to match your CPU requirements

mythtv-setup

  • under System Events, I created entries for:
Playback Start - /home/greg/.mythtv/bin/playback.sh %SENDER% START
Playback Pause - /home/greg/.mythtv/bin/playback.sh %SENDER% PAUSE
Playback Stop - /home/greg/.mythtv/bin/playback.sh %SENDER% STOP
Playback Pause - /home/greg/.mythtv/bin/playback.sh %SENDER% PAUSE
  • I would have done LiveTV, but there is no *stop* event, so no way to put the system back to default state. It also would be very nice to get the calling Event, so one script could be used and parse the info to determine the action.
  • Obviously the script checks the hostname to make sure it is the local frontend.
  • I use only hostnames on my network, so if you use IP addresses in your mythtv setup and receive that from the MythTV event handler rather than a name, you will need to modify the script to accomodate.

Notes

  • By enabling the RedHat cgconfig sysconfig setting to enable sysdefault, the system moves all current processes into the sysdefault cgroup. I have no idea how Ubuntu works with this, but here is the jest of how RedHat accomplishes it:
cgclassify -g <controller>:<defaultgroup> `ps --no-headers -eL o tid` 2>&1>/dev/null | :
  • for each controller - those listed in /proc/cgroups

Post Configuration

  • enable the cgconfig and cgred services for boot time
  • reboot

You can check at any time what is happening by example:

  • Start a Video
# cat /cgroup/cpuset/mythtv_high/cpuset.cpus

6-7

# cat /cgroup/cpuset/sysdefault/cpuset.cpus

0-5

# grep -i mhz /proc/cpuinfo

cpu MHz		: 1197.000
cpu MHz		: 1197.000
cpu MHz		: 1197.000
cpu MHz		: 1197.000
cpu MHz		: 2794.000
cpu MHz		: 1197.000
cpu MHz		: 2794.000 <--- 6
cpu MHz		: 2794.000 <--- 7
  • Stop or Pause the video
# cat /cgroup/cpuset/mythtv_high/cpuset.cpus

6-7

# cat /cgroup/cpuset/sysdefault/cpuset.cpus

0-7

# grep -i mhz /proc/cpuinfo

cpu MHz		: 1197.000
cpu MHz		: 1197.000
cpu MHz		: 1197.000
cpu MHz		: 1197.000
cpu MHz		: 2794.000
cpu MHz		: 1197.000
cpu MHz		: 1197.000 <--- 6
cpu MHz		: 1197.000 <--- 7


That's it - You should be up and running