[mythtv-users] File System Recommendation?
Brian Wood
beww at beww.org
Mon Apr 3 09:30:19 UTC 2006
On Apr 2, 2006, at 10:10 PM, f-myth-users at media.mit.edu wrote:
> Date: Mon, 3 Apr 2006 13:01:21 +1000
> From: "Richard Dale" <richard at interlink.com.au>
>
>> I'm using ext3 on a RAID-0 system. Who the heck cares if it takes a
>> second or two to delete a 1-hour show ??
>
> Here's an interesting solution - the recording could be flagged
> for deletion
> and then deleted in a background thread. So, even if it takes
> a while it's
> the background thread that has to wait on I/O, not the
> foreground user
> interface.
>
> The problem isn't making the -user- wait for the disk, it's making
> -Myth- wait for the disk.
>
> While ext3fs is deleting a file, all other writers block. If you're
> currently writing a stream (or several) to the same filesystem, -they-
> will block. Eventually your buffers will overflow, and you'll lose
> data, and your recordings in progress will have glitches in them.
>
> I verified this experimentally with ext3fs on an ordinary disk (no
> LVM or RAID), using all SD NTSC feeds from ivtv. Deleting a one-hour
> recording was typically okay, even if multiple streams were writing,
> but deleting (say) 10G (e.g., a 5hr recording) was not. I don't
> recall exactly where the breakpoint was (I could look it up in my
> notes), but it also varies based on how many other streams you're
> writing, and whether the database is doing anything, etc. Of course,
> the breakpoint might be different on your machine, and if something
> decides to delete too many 1-hour recordings at the same time (such as
> expiration), you'll be in the many-GB case and lose in the same way.
>
> This is simple to reproduce---just use dd or something to create files
> of arbitrary sizes, and time their deletion. That'll give you an idea
> of the work the filesystem is going through. Then (if you want) try
> recording n streams to that filesystem while doing a deletion. There
> will come a point in ext3fs where the amount of time it takes for the
> deletion to happen will exceed your RAM buffers, and you'll lose data.
> (In JFS, the deletions are in the tens of milliseconds or less no
> matter
> how large the file is, so that won't happen there. Ditto XFS, I
> believe.)
>
> If you queued all deletions until no writers were writing---or due to
> write in the next 5-10 seconds---this would be fine, but you might
> wind up waiting hours for a deletion to happen. You'd have to be
> pretty sure you wouldn't run out of space in the meantime.
>
>
Thank you for a good explanation, now everything I have read makes
sense, including the "bilgewater". I guess I have never noticed a
problem because I almost never record anything longer than a 1-hour
show, and thus never delete anything longer either. I also have a lot
of RAM.
It seems pretty stupid for a deletion to block all writes, but I'm
sure there is a reason this was done, I can see possible problems if
it were otherwise.
I guess every filesystem is a compromise based on performance,
safety, size of code, speed of code execution and other factors, and
what is good for storing characters from a newswire is different from
what's best for storing HD video streams.
Thanks again.
More information about the mythtv-users
mailing list