600481 - firefox.exe unresponsiveness, severe memory handling problems, system load (red curve) on winxp, [@ mozalloc_handle_oom | js::gc::MarkObject]

Status: RESOLVED WORKSFORME

(Core :: General, defect)

Description

[abittner]

User-Agent:       Mozilla/5.0 (Windows NT 5.1; rv:2.0b7pre) Gecko/20100928 Firefox/4.0b7pre
Build Identifier: Mozilla/5.0 (Windows NT 5.1; rv:2.0b7pre) Gecko/20100928 Firefox/4.0b7pre

as of the recent daily builds (ffx 4 beta7pre) there seem to be some severe memory handling bugs (memory allocation?). taskmanager/processexplorer (sysinternals) shows red-curve (system load) for the firefox.exe process.

whole firefox goes nonresponsive for many minutes or even like half an hour or an hour long (but for example still downloading a file via http meanwhile), then returns back to interactive mode and reactivity for even just as little as a few seconds or a minute, then returning to the nonresponsive status.

sometimes whole firefox.exe crashes (winxp, sp3, x86) in mozalloc.dll with drwatson/windowserrorreporting (but no crashhandling via mozilla breackpad/crashreporter)

measuring with amd codeanalyst 
(see mozilla developer pages https://developer.mozilla.org/Profiling_with_AMD_CodeAnalyst )

reveals that during such times withing firefox.exe the following load scenario
Process Name 	64-bit 	Timer samples 	
ntkrnlpa.exe 	       	98.01         	
hal.dll      	       	1.06          	
xul.dll      	       	0.4           	
ntdll.dll    	       	0.16          	
MOZCRT19.dll 	       	0.07          	

and inside the ntkrnlpa.exe the following subroutines:
CS:EIP     	Symbol + Offset               	64-bit 	Timer samples 	
0x805b9408 	MiGetNextNode                 	       	84.69         	
0x805b950a 	MiFindEmptyAddressRangeInTree 	       	10.66         	
0x8052be48 	RtlFindClearBits              	       	3.88          	
0x80523e74 	MiDeleteVirtualAddresses      	       	0.17          	

4 functions, 64 instructions, Total: 9193 samples, 95.11% of samples in the module, 46.08% of total session samples
-----------



in general: i am a heavy user of firefox, great many number of windows, inside most of those many tabs, and most recently as of the introduction of firefox tabcandy/panorama/grouptabs i am also using those ;)

the winxp machine has 4gigs of ram, and during such load scenarios firefox.exe was consuming only like 1,4gigabytes of ram (taskmganager), but the machine can also run other applications without problems of paging/memory swapping or similar, so its not really a windows memory load but rather a firefox memory management problem i think.

firefox4 b7pre builds from last week were still behaving much better and this mozalloc crashes and firefox ntkrnlpa load only occurs as of the last few days.


at the moment i am with:
about:buildconfig
Source

Built from http://hg.mozilla.org/mozilla-central/rev/c257bfb8cad0
Build platform
target
i686-pc-mingw32
Build tools
Compiler 	Version 	Compiler flags
d;D:\mozilla-build\msys\mozilla-build\python25\python2.5.exe -O e;D:\mozilla-build\msys\builds\moz2_slave\mozilla-central-win32-nightly\build\build\cl.py cl 	14.00.50727.762 	-TC -nologo -W3 -Gy -Fdgenerated.pdb -DNDEBUG -DTRIMMED -Zi -Zi -UDEBUG -DNDEBUG -GL -wd4624 -wd4952 -O1
d;D:\mozilla-build\msys\mozilla-build\python25\python2.5.exe -O e;D:\mozilla-build\msys\builds\moz2_slave\mozilla-central-win32-nightly\build\build\cl.py cl 	14.00.50727.762 	-GR- -TP -nologo -Zc:wchar_t- -W3 -Gy -Fdgenerated.pdb -wd4800 -DNDEBUG -DTRIMMED -Zi -Zi -UDEBUG -DNDEBUG -GL -wd4624 -wd4952 -O1
Configure arguments

--enable-application=browser --enable-update-channel=nightly --enable-update-packaging --enable-jemalloc --enable-tests
------------


Reproducible: Sometimes

Comment 1

[abittner]

still experiencing the very same with:
Build identifier: Mozilla/5.0 (Windows NT 5.1; rv:2.0b7pre) Gecko/20100930 Firefox/4.0b7pre

about:buildconfig
Source

Built from http://hg.mozilla.org/mozilla-central/rev/5a2012482a63
Build platform
target
i686-pc-mingw32
Build tools
Compiler 	Version 	Compiler flags
d;D:\mozilla-build\msys\mozilla-build\python25\python2.5.exe -O e;D:\mozilla-build\msys\builds\moz2_slave\mozilla-central-win32-nightly\build\build\cl.py cl 	14.00.50727.762 	-TC -nologo -W3 -Gy -Fdgenerated.pdb -DNDEBUG -DTRIMMED -Zi -Zi -UDEBUG -DNDEBUG -GL -wd4624 -wd4952 -O1
d;D:\mozilla-build\msys\mozilla-build\python25\python2.5.exe -O e;D:\mozilla-build\msys\builds\moz2_slave\mozilla-central-win32-nightly\build\build\cl.py cl 	14.00.50727.762 	-GR- -TP -nologo -Zc:wchar_t- -W3 -Gy -Fdgenerated.pdb -wd4800 -DNDEBUG -DTRIMMED -Zi -Zi -UDEBUG -DNDEBUG -GL -wd4624 -wd4952 -O1
Configure arguments

--enable-application=browser --enable-update-channel=nightly --enable-update-packaging --enable-jemalloc --enable-tests

-------------

the longer the firefox.exe runs, the more severe it gets. more lagging when navigating http (f5, reload pages, or simple klicking on standard html links, sometimes take ages til the browser actually does the network access and refreshes or loads the page, no matter how simple the target webpage is (so not a target system load problem), independent from network load or other external factors). whole app is really lagging.

then these red-curve load situations occur, at the moment (i am writing this update via parallel google chrome browser) the redcurve load events take like 10-20seconds, no crash yet.


one more symptom that is present is, that for example i am at the moment downloading a set of files via http and the progress bar in the separate standard download window of firefox is nowhere moving (after the browser returns to normal reactive mode), so i need to generate mouseover (moving the mouse inside the download window) events so that maybe windows dicates additional cpu cycles to the app or that firefox needs to redraw its download window, and all of a sudden all the download bars that were still progressing, jump up in their completion percentage and again get updated continuously for a while again and the download resumes also in the graphical representation (it somehow continues internally until the mouse-movement, but even filemon.exe (sysinternal tools, etc.) lacks output for the firefox.exe process and doesnt actually capture events of file activity for firefox.exe process for quite a while, only when the mouse moves a whole lot of loglines for firefox.exe appear in the filemon window and the download part-files seem to get updated, appended and are continuing to grow or being filled with data....


not sure though about this file stuff, maybe some internal components of the operating systems or other components of the firefox code actually somehow manage to continue to cache or save the file in the main memory, until the app itself is responsive again to flush the data via the operating system to the disk.


weird behaviour.

it seems that especially sites such as google mail / gmail or google maps with extensive dynamic content javascript and whatnot severly effect and worsen this behaviour and scenrario, as my browser (with all its tabs, windows and tabcandy/panorama) stays pretty much almost the same in terms of tabs/panorama/window numbers over time, only the uptime of the process itself and apparently increasing garbage, memory fragmentation or object and memory leaks and similar mess seem to be the reason of degradation of the firefox app and the situation.


i also have this other bug at
https://bugzilla.mozilla.org/show_bug.cgi?id=579653

which might be related or actually this bug creates a somewhat similar user experience for heavy-duty hardcore firefox users and it seems that although that other bugreport has been fixed by now other areas of the firefox code seem to have went haywire and are new causes for crashes and lagging. :(


btw: it would be good and helpful if mozilla developers would actually be able to catch these mozalloc.dll (and maybe other subcomponents as well) crashes or at least be able to make use of the windows error reporting statistics.

sadly enough to my knowledge, mozilla doesnt seem to care about windows captured crashes via drwatson/wer as i have heard in some of my other bugreports, and some prerequisite for drwatson/wer statistics is a digital certificate for the component(s), so mozalloc.dll as well needing digital signage/certs, which is probably in contrast to the nightlies/trunk builds of the mozilla apps and how all of this gets built and packaged at the moment.

please do get involved with windows error reporting mechanisms, and make use of a countless crashes and reports that are being sent back to microsoft that originate from the mozilla projects and components.

as long as i have been using mozilla apps and components, i came across a great many problems with crashes where breakpad/crashreporter was simply of no use and wasnt able to generate crash logs and reports.

thanks.

http://winqual.microsoft.com/help/default.htm#obtaining_a_verisign_class_3_digital_id.htm

http://www.microsoft.com/whdc/winlogo/maintain/StartWER.mspx
http://winqual.microsoft.com

thanks.

Comment 2

[abittner]

ffx4 just went to beta8pre for the first time:

mozalloc.dll crashed here as well, drwatson/windowserrorreporting external crash, no success in breakpad/crashreporter.


sigh. firefox4 is really barely usable any more as of yesterday/today (last nightly beta7pre build and first nightly beta8pre build)

:(((((

Comment 3

[timeless]

https://developer.mozilla.org/En/How_to_get_a_stacktrace_with_WinDbg

use 32bit firefox w/ 32bit windbg.

Comment 4

[abittner]

Attachment: windebug (windbg) logfile of firefox.exe - ffx4beta8pre

created a windbg output, firefox was consuming completely redcurve/system load, most likely this will have leaded again to a mozalloc.dll crash (cant really tell as this behaves differently in windbg and i dont see the same endresults).

so here is the logoutput.

one thing that i dont understand yet about this windbg stuff is, the int 3 stuff, when the debugger comes along the ntdll!dbgbreakpoint situations.

i observed with processexplorer, and firefox was creating subprocesses for plugin-container.exe, and everytime at least when such a subprocess was being created/started the main firefox.exe process was becoming grey (in process explorer, meaning it was paused/debugged or something) and that was leading to this int 3 pause inside windbg, and so i had to continue/f5 debug/go according to the howto at https://developer.mozilla.org/En/How_to_get_a_stacktrace_with_WinDbg

i had some other windbg run with minefiled ffx4beta8pre but that had like hundreds or thousands of these int 3 pause situations (no access violation or anything else all the time), and it started quite a number of starting/stopping/exiting plugin-container.exe processes, although i use flashblock/flashkill and such stuff preventing from flash/plugins being played automatically, so i am really wondering about this.

also the windbg logfile in that situation became like over 18megs in size, but i had to stop as it most likely lead me nowhere coming near a real crash situation.


so my question is, is there a way to run windb the way, that i dont get annoyed by these int3 pause situations when new subprocesses are being generated but forcing windbg just to run as long as a real crash/problem occurs?

otherwise i will probably never come close to these accessviolations and mozalloc.dll crashes and such situations when firefox needs to run on an avarage like many minutes in this red-curve load situations until it finally crashes....


so when needing to F5 these time-spans all the time this will probably most likely take like several hours or even days until the debugger works its way through the realtime timespan of multi-minute cpucycles but in interactive interrupted stop/continue F5 way :(


about the general behaviour of firefox.exe in this special bug: process explorer shows huge memory consumption, right after starting firefox as a new process, it obviously restores all windows/tabs/tabcandy/panorama and uses mainmemory all right, but that continues for a real long while until this red-curve cpu cycle stuff starts to go off (even without any further interaction at all, not clicking firefox components at all after starting it no matter if normally or from inside the debug-run) and the memory load of firefox.exe process will approach 1.4gigabytes of ram asymptotically and stay there...... and then after a number of minutes of firefox.exe load the crash occurs.

most of the times crash (without debugger) results in entry of windows application log event as mozalloc.dll crash, i also had one plugin-container.dll crash and mozalloc.dll crash right after another, so while the first crash was still ongoing and drwatson was being triggered another second drwatson window appeared and a consecutive probably related crash was being triggered. maybe its just that the firefox.exe crashes in mozalloc and plugin-container also needs the main process or so and crashes due to failing interprocess communication or something.

regards.

Comment 5

[abittner]

Attachment: this time a real mozalloc.dll crash windbg report

this time i did it differently. started firefox normally, and attached windbg later during runtime to this process.

i left firefox (4beta8pre) running a while and went especially to javascript intensive pages such as google-maps, and did a number of dragdrop events such as dragging the little orange-ish dude for streetview around the map, and eventually firefox went into a freeze and again red-curve system-level load and resulting in the mozalloc.dll crash.

at last.

Comment 6

[abittner]

Attachment: screenshot: process and thread situation when aproaching mozalloc.dll crash

screenshot of process explorer (sysinternals/msft) of firefox.exe process and its threads.

when this red-curve system load occurs in the firefox.exe process happens and doesnt calm down any more, firefox.exe is almost totally consuming cpu cycles for !CreateThread+.... only (single core of dualcore machine, nearing 50% cpu) and the stackreport for that thread shows !VirtualAlloc... and !_realloc_crt...

Comment 7

[timeless]

mozalloc_handle_oom
js::gc::MarkObject
js::gc::MarkChildren(struct JSTracer * trc = 0x78187640, struct JSObject * obj = 0x78143280)+0x2e
JS_TraceChildren(struct JSTracer * trc = 0x0181e234, void * thing = 0x00000006, unsigned long kind = 0)+0x81 
nsXPConnect::Traverse(void * p = 0x00000006, class nsCycleCollectionTraversalCallback * cb = 0x00000000)+0x120
nsCycleCollector::MarkRoots(class GCGraphBuilder * builder = 0x00000000)+0xf2
nsCycleCollector::BeginCollection(class nsICycleCollectorListener * aListener = 0x0080b840)+0x8a
nsCycleCollector::Collect(unsigned int aTryCollections = 1, class nsICycleCollectorListener * aListener = 0x00000000)+0x82
nsJSContext::CC(class nsICycleCollectorListener * aListener = 0x100241e8)+0x4c
nsJSContext::IntervalCC(void)+0x29
nsJSContext::CCIfUserInactive(void)+0x11
nsHttpChannel::GetURI(class nsIURI ** aURI = 0x403d2d80)+0x1b
nsDocShell::EndPageLoad(class nsIWebProgress * aProgress = 0x0013da34, class nsIChannel * aChannel = 0x0013d9ec, unsigned int aStatus = 0x811e00)+0xcc

mozalloc is basically saying "we ran out of memory and can't continue safely, we have to quit now, sorry".

Comment 8

[abittner]

well dont know if all my crashes as of recent 4beta7 or 4beta8pre about these mozalloc are the same, but i think its not good to simply crash just because you dont have any memory left.


isnt there swapping via the os, and the whole firefox.exe process was "only" using 1,4gigabyte of ram (taskmgr/processexplorer) sometimes even much less like 1gig only) and still crashing.


btw, i repro-d this mozalloc crash 100% on a clean and completely differnt machine with win7, x64, x86 build of official release level 4beta6.

created a clean firefox4 installation on this machine, opened several tabs with urls from the top 100 alexa international topsites, created only like 2-4 panorama groups and moved various tabs there. quit firefox, and made it start restoring the multiple windows tabs and panorama, and sometimes while still restoring these tabs especially when you go into the panorama thumbnails view everything inside the browser becomes real sluggish, annoyingly slow and lagging and the thumbnails barely get drawn and/or updated/created properly while still starting up firefox4) and eventually you get this crash......


if im not mistaken and my knowledge about windows memory management for win32/32bit apps isnt mistaken, win32 32bit apps should actually be able to use exactly like 2gigabytes of ram, especially also on real x64 (wow64 api) systems, but also on normal win32/winxp operating systems etc.

i have 4igabyte of real physical ram on my test machines, and the operating system can use way beyond 2gigabyte or even 3,7gigs on the win7x64 machine.

so i wonder why mozalloc.dll can give faults and crashes when still working with only 1gigabyte or 1,4gigabytes of ram.


i have never seen recent firefox builds with processexplorer to use more than this magical 1,4gigabyte of ram.

weird.

any more windbg dumps needed? from this win7x64 system?

regards.

Comment 9

[JimFlasch]

I have had problems like these with Firefox beta 5 & 6 (I'm currently on 6) on sites with embedded video. One minute Firefox 4 is fine, the next it's using so much system resources that while it's the active window, response time goes way down - the mouse pointer jumps across the screen rather than moving smoothly, and nothing happens when I click on a button or tab. But when I minimize Firefox 4 or open another maximized window over it, things return to normal. Sometimes this problem goes away at that point, sometimes it stays - even through a restart of Firefox 4. It is getting annoying!

Comment 10

[Nickolay_Ponomarev]

If only Firefox 4 builds are affected, perhaps you can track down the moment when this broke? https://wiki.mozilla.org/QA/Triage#How_to_Help_with_Regressions_--_Finding_Regression_Windows

Comment 11

[RNicoletto]

Is this still reproducible with recent betas?

Comment 12

[JimFlasch]

Yeah - happened when I restarfted after changing skin from Noia 2 to strata deluxe (or whatever it's called - it's an extension of the default skin.) and a second restart didn't help 4b11 was still loading slowly! Could you please fix that problem before the RC comes out?

Comment 13

[Wayne Mery (:wsmwk)]

abittner, please reply ....  (and can you quantify roughly the number of tabs)

(In reply to comment #11)
> Is this still reproducible with recent betas?

Comment 14

[Tim (fmdeveloper)]

abittner -> Any update?

Comment 15

[Wayne Mery (:wsmwk)]

whatever this was, perhaps it has changed because I don't see stacks for the last few days that include both mozalloc_handle_oom and js::gc::MarkObject
=> WFM based on crash stats  (but it could equally be incomplete without abittner input)

Comment 16

[JimFlasch]

The memory problem has returned with Firefox 7.0.1; after it's been running for a while, FF 7.0.1 slows down considerably, with major use of system resources, and at least once became completely unresponsive. What's up with this?

Comment 17

[JimFlasch]

seems to have gotten a bit better - it only slows for a little while then gets better; it would still be better if it didn't slow at all, though