[mythtv-users] Update - FIXED mythbackend not responding -and even more weirdness

Thu Dec 28 02:02:58 UTC 2006

> All of that loading typically happens right after 9PM or 10PM for
> obvious reasons. The load goes up on the backend, but it's not at
> 100%. The machine acts fine. It's response, etc., but mythbackend quits.

I think I misunderstood your problem then. Your problem is that ONLY mythbackend process is quitting/dying under load in SMP but is fine under uniprocessor mode. But the kernel settings that I mentioned may be still relevant because mythbackend typically is a hugely multi-threaded appilication (at idle it has like 15 threads) and under load, scheduling may become more pronounced. One thread getting killed is enough to bring the whole process down. Moreover, the kind of load you are putting it thru also means that the IO subsystem is overloaded as well.

One more thing you might wanna check is if there are OOM-killer messages in your /var/log/messages. There was a recent OOM-killer bug fixed in kernel ( Linux 2.6.19-rc4) where it incorrectly killed a process even when there was enough memory available. I have had firefox disappear suddenly in front of me in the past because of OOM killer. I have since then put

vm.overcommit_memory = 2
vm.overcommit_ratio = 90

in my /etc/sysctl.conf to effectively (still not completely) disable memory overcommit and resort to more closer to traditional Unix memory allocation where in malloc returns non-null only if there is enough memory available, and fork fails if there is not enough memory available etc.. This way you avoid random process kills when under memory pressure, but take the memory failure hits upfront. Which of the two is better is a trade-off and very subjective. I like apps to fail upfront with ENOMEM under memory pressure and let the already running processes keep running. Apparently, linux defaults to overcommit memory to begin with (so malloc may return non-null although all the needed vm may not be available) and later kill processes depending upon a heuristic based OOM score.

-devsk

>
> -devsk
>
>
> >
> > -devsk
> >
> >
> > ----- Original Message ----
> > From: Mark Knecht <markknecht at gmail.com>
> > To: hah at alumni.rice.edu; Discussion about mythtv <mythtv-users at mythtv.org>
> > Sent: Wednesday, December 27, 2006 9:37:05 AM
> > Subject: Re: [mythtv-users] **Update - FIXED** mythbackend not responding
> > -and even more weirdness
> >
> >
> > On 12/27/06, Henry A Harper III <hah at alumni.rice.edu> wrote:
> > >
> > >
> > > > -----Original Message-----
> > > > From: mythtv-users-bounces at mythtv.org
> > > > [mailto:mythtv-users-bounces at mythtv.org]On Behalf Of Mark Knecht
> > > > Sent: Wednesday, December 27, 2006 9:48 AM
> > > > To: Discussion about mythtv
> > > > Cc: dusteur at excite.com
> > > > Subject: Re: [mythtv-users] **Update - FIXED** mythbackend not
> > > > responding -and even more weirdness
> > > >
> > > >
> > > > On 12/27/06, Bill Arlofski <waa-mythtv at revpol.com> wrote:
> > > > <SNIP>
> > > > >
> > > > > Does any of the burden of this problem fall on the mythtv devs? Is
> it
> > a
> > > > > programming issue with Mythtv and SMP? Is it an AMD Athlon64 X2 SMP
> > > > > issue specific to the CPU I am using? Again, I do not know.
> > > > >
> > > > <SNIP>
> > > >
> > > > Interesting results. It might well be SMP related. I have a P4HT
> > > > machine as the backend server and it's having similar problems.
> > > >
> > > > We are having a lot of problems with mythbackend just shutting off,
> > > > but it seems to only shut off only when there are mythfrontend
> > > > machines that are just finishing watching programs. Over the Christmas
> > > > break we travelled for 4 days. mythbackend recorded everything
> > > > correctly for about 40 hours of material. Lots to watch, so we started
> > > > trying to do so. Yesterday mythbackend shut off 3 times. I'd restart
> > > > the backend and it would happen again maybe 1-2 hours later.
> > > >
> > > > Thanks for the SMP idea. I'll look at that and report back.
> > > >
> > >
> > > FYI, my 4600+ x2 w/ 2.6.17 SMP kernel in a FE/BE has been up 97 days
> with
> > no
> > > backend "weirdness" other than xbox360-caused upnp issues (just 100%
> core
> > > usage, not unresponsive) until I went to newer svn head last week.
> >
> > Good to know and glad it's working for you. We're Gentoo here. HEre's
> > what's installed on the backend machine:
> >
> > dragonfly ~ # eix -Ic mythtv
> > [I] media-tv/mythtv (0.20_p11626): Homebrew PVR project
> > [I] x11-themes/mythtv-themes (0.20): A collection of themes for the
> > MythTV project.
> > Found 2 matches.
> > dragonfly ~ # uname -a
> > Linux dragonfly 2.6.17-gentoo-r2 #4 SMP PREEMPT Fri Nov 24 14:20:30
> > PST 2006 i686 Intel(R) Pentium(R) 4 CPU 3.00GHz GenuineIntel GNU/Linux
> > dragonfly ~ #
> >
> > Note that this problem existed before on myth-0.18 and 0.19. It was
> > not changed much by going to gcc-4.1.1. Pretty much the same level of
> > problems.
> >
> > Here's an example of something I'm seeing in the log files this
> > morning. I do NOT know if this is a reoccuring problem. It's jsut at
> > the end of my log:
> >
> > [mpeg2video @ 0xb7345348]00 motion_type at 0 27
> > [mpeg2video @ 0xb7345348]00 motion_type at 0 28
> > [mpeg2video @ 0xb7345348]00 motion_type at 0 29
> > [mpeg2video @ 0xb7345348]Warning MVs not available
> > *** glibc detected *** /usr/bin/mythbackend: double free or corruption
> > (fasttop): 0xa8605138 ***
> > ======= Backtrace: =========
> > /lib/libc.so.6[0xb5ea9bab]
> > /lib/libc.so.6(__libc_free+0x79)[0xb5eab0e7]
> >
> /usr/lib/gcc/i686-pc-linux-gnu/4.1.1/libstdc++.so.6(_ZdlPv+0x21)[0xb603a929]
> >
> /usr/lib/libmythtv-0.20.so.0(_ZNSt6vectorIiSaIiEE13_M_insert_auxEN9__gnu_cxx17__normal_iteratorIPiS1_EERKi+0xfc)[0xb78a1da6]
> > ======= Memory map: ========
> > 08048000-0812a000 r-xp 00000000 03:08 296515     /usr/bin/mythbackend
> > 0812a000-0812b000 rw-p 000e2000 03:08 296515     /usr/bin/mythbackend
> > 0812b000-083cc000 rw-p 0812b000 00:00 0          [heap]
> > a8400000-a8460000 rw-p a8400000 00:00 0
> > a8460000-a8500000 ---p a8460000 00:00 0
> > a8600000-a8644000 rw-p a8600000 00:00 0
> > a8644000-a8700000 ---p a8644000 00:00 0
> > a9000000-a9100000 rw-p a9000000 00:00 0
> > a91f4000-a91f5000 ---p a91f4000 00:00 0
> > a91f5000-a99f5000 rw-p a91f5000 00:00 0
> > a99f5000-a99f6000 ---p a99f5000 00:00 0
> >
> > <SNIP>
> > b7ef7000-b7ef8000 rw-p 00000000 03:08 705168
> > /usr/lib/mythtv/filters/libbobdeint.so
> > b7ef8000-b7efa000 r-xp 00000000 03:08 705149
> > /usr/lib/mythtv/filters/libadjust.so
> > b7efa000-b7efb000 rw-p 00001000 03:08 705149
> > /usr/lib/mythtv/filters/libadjust.so
> > b7efb000-b7f03000 r-xp 00000000 03:08 753706     /lib/libnss_files-2.4.so
> > b7f03000-b7f05000 rw-p 00007000 03:08 753706     /lib/libnss_files-2.4.so
> > b7f05000-b7f0e000 r-xp 00000000 03:08 522754
> > /usr/qt/3/plugins/sqldrivers/libqsqlmysql.so
> > b7f0e000-b7f0f000 rw-p 00009000 03:08 522754
> > /usr/qt/3/plugins/sqldrivers/libqsqlmysql.so
> > b7f0f000-b7f10000 rw-p b7f0f000 00:00 0
> > b7f10000-b7f29000 r-xp 00000000 03:08 753640     /lib/ld-2.4.so
> > b7f29000-b7f2a000 r--p 00019000 03:08 753640     /lib/ld-2.4.so
> > b7f2a000-b7f2b000 rw-p 0001a000 03:08 753640     /lib/ld-2.4.so
> > bf9cd000-bf9e3000 rw-p bf9cd000 00:00 0          [stack]
> > ffffe000-fffff000 ---p 00000000 00:00 0          [vdso]
> > 0: start_time: 0.036 duration: 151.976
> > 1: start_time: 0.026 duration: 151.954
> > stream: start_time: 0.289 duration: 1688.731 bitrate=5193 kb/s
> > [mpeg2video @ 0xb7345348]ac-tex damaged at 22 6
> > 0: start_time: 0.036 duration: 232.471
> > 1: start_time: 0.026 duration: 232.444
> > stream: start_time: 0.289 duration: 2583.125 bitrate=7217 kb/s
> > Starting up as the master server.
> >
> >
> > I don't even know if the Myth developers read this list or want this
> > info. Maybe pass it upstream through Gentoo?
> >
> > Thanks,
> > Mark
> > _______________________________________________
> > mythtv-users mailing list
> > mythtv-users at mythtv.org
> > http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
> >
> >
> > __________________________________________________
> > Do You Yahoo!?
> > Tired of spam? Yahoo! Mail has the best spam protection around
> > http://mail.yahoo.com
> > _______________________________________________
> > mythtv-users mailing list
> > mythtv-users at mythtv.org
> > http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
> >
> >
> >
> _______________________________________________
> mythtv-users mailing list
> mythtv-users at mythtv.org
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
> _______________________________________________
> mythtv-users mailing list
> mythtv-users at mythtv.org
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
>
>
>
_______________________________________________
mythtv-users mailing list
mythtv-users at mythtv.org
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mythtv.org/pipermail/mythtv-users/attachments/20061227/625d6b83/attachment.htm 

[mythtv-users] **Update - FIXED** mythbackend not responding -and even more weirdness

[mythtv-users] Update - FIXED mythbackend not responding -and even more weirdness