[mythtv-users] random livetv stalls

[Mike Thomas]

On Sun, 23 Feb 2014 06:01:22 -0500
"Stephen P. Villano" <stephen.p.villano at gmail.com> wrote:
> 
> On 2/23/14, 5:10 AM, Mike Thomas wrote:
> > On Sat, 22 Feb 2014 10:29:27 -0800
> > Monkey Pet <monkeypet at gmail.com> wrote:
> >> On Thu, Feb 20, 2014 at 10:47 AM, Monkey Pet <monkeypet at gmail.com>
> >> wrote:
> >>
> >>> On Wed, Feb 19, 2014 at 3:27 PM, Jean-Yves Avenard
> >>> <jyavenard at gmail.com>wrote:
> >>>
> >>>> On 20 February 2014 08:55, Doug Lytle <support at drdos.info> wrote:
> >>>>> What would this have caused?
> >>>>>
> >>>>> I put the patch into place on my mother's system this morning.
> >>> How often are people seeing these stalls in LiveTV? I applied the
> >>> patch on my backend and LiveTV seems smooth.  However, I didn't
> >>> test the before patch LiveTV, so I don't know if this is an
> >>> improvement or not.
> >>>
> >> I saw a stall after 30 mins using the patch.  It reproduced a few
> >> times. The frontend is remote using streaming not nfs.  On the
> >> backend side, I am suspecting the btrfs is slowing down the writes,
> >> so I am switch back to ext4.
> > Dear Monkey and others,
> >
> > I suggest you switch to xfs (defaults should be fine) for your
> > video storage directories. ext4 for video directories has always
> > caused stalling on my computers. ext3 is a joke. ext2 can't support
> > files large enough. I think trendy new filesystems like btrfs will
> > exhibit greater latency than xfs because in general they do more
> > work than xfs.
> >
> > Conversely, xfs is NBG for the database. ext4 seems best for me, but
> > some people use ext2. The claimed 'improvement' from ext2 comes
> > from it using the drive's write cache in a less than safe manner.
> >
> > Another show-stopper is RAID. Anyone using RAID 5/6 for either a
> > database or video storage (or anything) deserves it.
> >
> > Yours,
> >
> > Mike.
> > _______________________________________________
> >
> Interesting! My BE runs the SQL database, Zoneminder, a few other
> tasks and is running a RAID 5 array with ext4 fs. Of course, it's a
> Dell Poweredge 2850 with perc RAID card installed.
> It all depends on what you're running, how you run it and how you
> tune it.

If you're RAID card contains a battery it is likely it cheats by
reporting to the O/S that it's committed the data to disk when in fact
it's only committed it to battery-backed RAM. This is fine so long as
the O/S doesn't schedule too many commits per second, because in that
instance, the RAID card is constrained by the speed at which the HDDs
can commit. RAID cards with a GB or so of RAM will ride out almost all
the bursts you are likely to experience.

In general, a database insert will consist of a tail append of a MYISAM
data file and index file, corresponding updates to the filesystem inode
table, block bitmaps or free extent maps and a journal write. Although
it is true that the filesystem structures are only updated at
journal commit intervals, they will all hit you eventually. On a
streaming system like MythTV, it could easily cause a stutter if there
are lots of filesystem structure updates to perform at each commit
interval. As the saying goes, you don't get owt for nowt. For this
reason, I advise Monkey Pet to use a shorter commit interval. The
default of 5 seconds seems reasonable.

A further consideration for RAID 5/6 systems is the nature of each disk
write. Remember, I'm talking about database writes which will be mostly
single filesystem block writes. In a RAID 5/6 array, the disks are
presented to the O/S as stripes. Each stripe, consists of one or more
sectors from disk 1, the same from disk 2, ... and the same from a
parity disk (or two for RAID 6). To keep the parity correct, a write of
a single filesystem block (say 1kB), the RAID system must do this:

- Read those sectors from disk 1's share of the appropriate stripe.
- Read those sectors from disk 2's share of the appropriate stripe.
- Read those sectors from disk 3's share of the appropriate stripe.
...
- Overwrite the memory buffer with the changed data
- Re-compute the parity
- Write out the changed sectors to the disk containing the data.
- Write out the changed sectors to the disk containing the parity.

This is a slow process because it consists of four sequential stages.
Although a RAID controller will launch the reads in parallel, it will
be forced to wait for much longer than you might think because the
spindles of the drives are not synchronized. You should expect an
average of three disk rotations per read-modify-write sequence. At
5400rpm, this sure eats into your performance budget.

For sure it's not all that bad. With a large-enough cache many of these
writes can be coalesced into full-stripe writes, and it's also likely
that regular tail appends will still be in the RAID cache, but even so
it's very easy to hit pathological cases. Don't forget that RAID5/6's
read-modify-write process requires N times the cache memory of RAID 0
or 1. If your video is being written through the same controller (even
if it's on separate disks) this will turn over the cache rapidly.

If you've spent any time working with databases you'll know just how
much everyone relies on data being in cache, nicely indexed, subject to
only the shortest of locks and above all, with uniform data
distribution. A battery-backed RAID system is a nice adjunct to a
database, but only if it offers low-latency and sustained throughput.
RAID 5/6 will come and bite you in the arse at a critical time.

For a more complete review of the impact of RAID 5, I suggest reading
chapters 29-31 of Recovery Mechanisms in Database Systems by Kumar and
Hsu. The performance figures they quote bear out my experience.

I haven't been following this thread closely so forgive me if it's
already been asked, but what kind of disk systems do those people who
are experiencing problems have? I'll start the ball rolling by
describing mine:

- One cheap-jack old motherboard with some SATA ports and a 2.5GHz
  Dual-core Celeron.
- A couple of 1.5TB 7200rpm HDDs, both with video data on them and one
  with the database as well.
- LVM and dmcrypt.
- ext4 noatime,nosuid,barrier=1 for the database
- xfs nodev,noexec,nosuid,barrier for the video
- MySQL binlogging on
- MythTV 0.27 + my posted patch for bulk inserts to recordedseek
- Ring buffer set to 14000KB I think
- No kernel tweaks
- Not using NFS

I haven't had a chance to examine any later versions of MythTV.

I'm not saying it has anything to do with disks but I'd like to know if
problems are more likely when people move away from what might be
considered a bog-standard config.

Yours,

Mike.