[mythtv-users] Howto prevent MythTV flushing/Syncing disk writes (LiveTV/Recordings) [ SOLVED ]

[Albert Graham]

David Rees wrote:
> On Wed, Mar 19, 2008 at 2:07 PM, Albert Graham <agraham at g-b.net> wrote:
>   
>>  Here are some of my /proc/sys/vm settings for reference, (I know they
>>  are extreme for now, but for testing only).
>>     
>
> Yeah, those are fairly extreme!
>
>   
>>  dirty_ratio = 99
>>  dirty_background_ratio = 99
>>  dirty_expire_centisecs = 1000000 <--- Basically wait a long long time...
>>     
Since then, I've found that setting dirty_xx to around 80% works better 
for me, basically the documentation (At least that I've read is a bit 
misleading, I found an explanation of this here: 
http://www.westnet.com/~gsmith/content/linux-pdflush.htm).

>
> Yeah, you are basically saying don't bother writing any data to disk
> until it is a couple hours old!
>
>   
>>  dirty_writeback_centisecs = 0
>>     
>
> This basically disables writeback, so the previous settings shouldn't
> even be used.
>   

Yes, but for some reason, this does not work as I expected.
>   
>>  When reviewing the source I could see why things are done the way they
>>  are, I'm thinking about writing a patch that would be called "The Night
>>  Time Patch", the idea is to combine my "no_sync patch" detailed below,
>>  but also track if any frontends are connected, if they are, everything
>>  operates as normal, but when they are all disconnected, switch to
>>  "no_sync mode", in other words when you stop watching TV, it records in
>>  silent mode until you start watching TV again, or until it finishes
>>  recording a program :) (in addition to a simple daemon script mentioned
>>  later).
>>     
>
> I don't see why you would need to bother with tracking frontends,
> unless you want to use your extreme VM settings. I can't see many
> people going as far as you have to reduce the frequency of writes, so
> I think doing so will have a very limited audience. But hey, if you
> want to... ;-) Just don't expect to get such a patch upstream.
>   
OK, in my case I am not interested in tracking frontends for this 
purpose, but the idea is to "quieten the disks at night time", when 
no_sync is disabled - so in other words, that would be an alternative 
configuration option - switch to no_sync mode when I'm not watching TV - 
i.e. (assuming people sleep at night time) :)


I think the patch  (no_sync)  would be a good "feature" for many, I know 
I'm using extreme settings, but that is at OS level and nothing  really 
to do with MythTV, on the other hand the "night time" thing was just an 
idea that could utilize the concept.

>   
>>  The myth admin would need to be warned that that this feature may cause
>>  data loss if the backend crashes or has power loss during the night
>>  time, however, the admin "should :)" be asleep so the backend should be
>>  safe :)
>>     
>
> Again, I think that by simply disabling the explicit syncs, you aren't
> exposing yourself to any significant amount of data loss - not any
> more than usual unless you are tweaking the VM writeout settings. At
> which point you should know what you are getting yourself in to.
>
>   
>>  +/**  just before the file is closed, I actually call Sync() **/
>>  +        if (no_sync == 1) {
>>  +          no_sync = 0;
>>  +           Sync();
>>  +        }
>>     
>
> I don't see any point in syncing when closing the file. This will
> cause a large writeout which could cause other activity to lag. It
> will be better to just let the OS flush the data out to disk as usual.
> The large writeout will be even worse when you have extreme VM
> settings like you do. Luckily, you have the hardware to support this
> type of activity which is probably why you haven't seen any ill
> effects.
>   

OK, I too am in two minds about this, the idea is that most shows are 
1/2 - 1 hour long, so this happens at the end of show
rather than when the OS wants it to happen - also if the machine did 
crash, you'll have the last finished recording, I'm using XFS (with 
extreme settings too :))

>   
>>  Thats all the changes I've made and it works a treat, however, I noticed
>>  when the Linux cache fills up, say 6G in my case, Linux slowly starts to
>>  trickle the data from the Linux cache to disk, which in a sense puts you
>>  back to square one :(, however it much less noticeable and you do get
>>  the initial 6G write free.
>>     
>
> Not really. The behavior you see once memory has filled up is about as
> good as you'd expect. You could reduce the frequency of these writes
> in a steady state writing case by re-enabling the
> dirty_writeback_centisecs setting and setting it to 5-10 seconds or
> so.
>   
OK, my box is recording 24/7 so after an hour or so Linux starts to slow 
flush/sync out to disk (in about 8MB chunks) the cache, and the goes on 
forever!
>   
>>  To solve this last problem, I have a little script which could run from
>>  cron or just an endless loop while the backend is runnig (I prefer the
>>  latter) that monitors the amount of cache memory used e.g. (free -m|grep
>>  "Mem"|awk {'print $7'}) or  (better still) use /proc etc.. and when it
>>  reaches a threshold (in my case 6G) do the following:
>>
>>  echo "1" > /proc/sys/vm/laptop_mode
>>  sync
>>  echo "3" > /proc/sys/vm/drop_caches
>>  sync
>>  tw_cli /c0 flush
>>  echo "0" > /proc/sys/vm/laptop_mode
>>     
>
> Eek! Why write out all that data to disk that you just worked so hard
> to buffer up? And then after syncing it all to disk, you throw it all
> away?
>
>   
Well, yes, but my goal is "quietness", and having the entire show flush 
in 1 large write is what I personally was looking for.
>>  This is a really nice use of the "laptop_mode" feature, because Linux
>>  will flush everything on the next sync, I then drop the Linux caches
>>  (which should always be preceded by a sync anyway - but at this point
>>  all out data is safely on disk, (thanks to laptop_mode), now, in my case
>>     
>
> The only reason a sync is required before dropping the caches is
> because dropping the caches will only drop data that has been synced
> to disk. So when you have aggressive VM settings which don't ever
> flush things to disk, trying to drop the caches will not actually drop
> anything. You won't lose any data by dropping caches without syncing.
>   
Well, I was really just going by some docs on this one, e.g.

"As this is a non-destructive operation, and dirty objects are not 
freeable, the user should run "sync" first in order to make sure all 
cached objects are freed."

ref: http://www.linuxinsight.com/proc_sys_vm_drop_caches.html

Albert.