767343 - crash in nsSupportsStringImpl::SetData with abort message: "###!!! ABORT: OOM: file e:\builds\moz2_slave\m-cen-w32-ntly\build\xpcom\string\src\nsTSubstring.cpp, line 348"

Status: VERIFIED FIXED

(Core :: XPCOM, defect)

Description

[Scoobidiver (away)]

It first appeared in 15.0a1/20120511. The regression range might be (low volume):
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=654ac86492e8&tochange=e6a9572b48f7
It's likely a regression from bug 737164.

It might be related to bug 754850.

Signature 	mozalloc_abort(char const* const) | NS_DebugBreak_P | nsSupportsStringImpl::SetData(nsAString_internal const&) More Reports Search
UUID	0931dd83-d46e-4728-99b2-01f402120621
Date Processed	2012-06-21 22:42:51
Uptime	3765
Last Crash	4.5 hours before submission
Install Age	5.3 hours since version was first installed.
Install Time	2012-06-21 17:25:36
Product	Firefox
Version	16.0a1
Build ID	20120621053048
Release Channel	nightly
OS	Windows NT
OS Version	6.2.8400
Build Architecture	x86
Build Architecture Info	GenuineIntel family 6 model 42 stepping 7
Crash Reason	EXCEPTION_BREAKPOINT
Crash Address	0x7437198a
App Notes 	
AdapterVendorID: 0x8086, AdapterDeviceID: 0x0116, AdapterSubsysID: 17121043, AdapterDriverVersion: 9.17.10.2736
Has dual GPUs. GPU #2: AdapterVendorID2: 0x10de, AdapterDeviceID2: 0x0df6, AdapterSubsysID2: 17121043, AdapterDriverVersion2: 9.18.12.9665D2D? D2D+ DWrite? DWrite+ D3D10 Layers? D3D10 Layers+ xpcom_runtime_abort(###!!! ABORT: OOM: file e:\builds\moz2_slave\m-cen-w32-ntly\build\xpcom\string\src\nsTSubstring.cpp, line 348)
EMCheckCompatibility	False
Adapter Vendor ID	0x8086
Adapter Device ID	0x0116
Total Virtual Memory	4294836224
Available Virtual Memory	154865664
System Memory Use Percentage	77
Available Page File	3879383040
Available Physical Memory	1430482944

Frame 	Module 	Signature 	Source
0 	mozalloc.dll 	mozalloc_abort 	memory/mozalloc/mozalloc_abort.cpp:23
1 	xul.dll 	NS_DebugBreak_P 	xpcom/base/nsDebugImpl.cpp:369
2 	xul.dll 	nsSupportsStringImpl::SetData 	xpcom/ds/nsSupportsPrimitives.cpp:143
3 	xul.dll 	NS_InvokeByIndex_P 	xpcom/reflect/xptcall/src/md/win32/xptcinvoke.cpp:70
4 	xul.dll 	XPCWrappedNative::CallMethod 	js/xpconnect/src/XPCWrappedNative.cpp:2390
5 	xul.dll 	XPCWrappedNative::SetAttribute 	js/xpconnect/src/xpcprivate.h:2749
6 	xul.dll 	XPC_WN_GetterSetter 	js/xpconnect/src/XPCWrappedNativeJSOps.cpp:1540
7 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:311
8 	mozjs.dll 	js::Invoke 	js/src/jsinterp.cpp:354
9 	mozjs.dll 	js::Shape::set 	js/src/jsscopeinlines.h:292
10 	mozjs.dll 	js::baseops::SetPropertyHelper 	js/src/jsobj.cpp:5445
11 	mozjs.dll 	js::SetPropertyOperation 	js/src/jsinterpinlines.h:311
12 	mozjs.dll 	js::Interpret 	js/src/jsinterp.cpp:2348
13 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:322
14 	mozjs.dll 	js::Invoke 	js/src/jsinterp.cpp:354
15 	mozjs.dll 	JS_CallFunctionValue 	js/src/jsapi.cpp:5489
16 	xul.dll 	nsXPCWrappedJSClass::CallMethod 	js/xpconnect/src/XPCWrappedJSClass.cpp:1475
17 	xul.dll 	nsXPCWrappedJS::CallMethod 	js/xpconnect/src/XPCWrappedJS.cpp:580
18 	xul.dll 	PrepareAndDispatch 	xpcom/reflect/xptcall/src/md/win32/xptcstubs.cpp:85
19 	xul.dll 	SharedStub 	xpcom/reflect/xptcall/src/md/win32/xptcstubs.cpp:112
20 	xul.dll 	nsTimerImpl::Fire 	xpcom/threads/nsTimerImpl.cpp:481
21 	nspr4.dll 	nspr4.dll@0x8d2f 	
22 	xul.dll 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:624
23 	xul.dll 	mozilla::ipc::MessagePump::Run 	ipc/glue/MessagePump.cpp:113
24 	xul.dll 	MessageLoop::RunHandler 	ipc/chromium/src/base/message_loop.cc:201
25 	xul.dll 	_SEH_epilog4 	
26 	xul.dll 	MessageLoop::Run 	ipc/chromium/src/base/message_loop.cc:175
27 	xul.dll 	nsBaseAppShell::Run 	widget/xpwidgets/nsBaseAppShell.cpp:163
28 	xul.dll 	nsAppShell::Run 	widget/windows/nsAppShell.cpp:232
29 	xul.dll 	nsAppStartup::Run 	toolkit/components/startup/nsAppStartup.cpp:256
30 	xul.dll 	XREMain::XRE_mainRun 	toolkit/xre/nsAppRunner.cpp:3786
31 	xul.dll 	XREMain::XRE_main 	toolkit/xre/nsAppRunner.cpp:3863
32 	xul.dll 	XRE_main 	toolkit/xre/nsAppRunner.cpp:3939
...

More reports at:
https://crash-stats.mozilla.com/report/list?signature=mozalloc_abort%28char+const*+const%29+|+NS_DebugBreak_P+|+nsSupportsStringImpl%3A%3ASetData%28nsAString_internal+const%26%29

Comment 1

[Boris Zbarsky [:bzbarsky]]

Oh, for JS stacks in crash-stats.  :(

Comment 3

[Wayne Mery (:wsmwk)]

mine: bp-17324de9-c4cf-430f-869b-dfb782120811 and bp-f04ca0c8-253d-4705-85c8-653ba2120811 among others

Comment 4

[ellisdee23]

Attachment: current state of firefox

Comment 5

[Wayne Mery (:wsmwk)]

Attachment: windbg stacktrace 2012-09-29

captured this in windbg and have dump

Comment 6

[Wayne Mery (:wsmwk)]

~top 100 crash

I'm crashing frequently the last 2 days. two examples
bp-70a7be84-c6c2-4e51-a92d-e2ac22121101
bp-c9cc5d49-3baa-4957-a7a4-dc2ae2121101

0	mozalloc.dll	mozalloc_abort	memory/mozalloc/mozalloc_abort.cpp:23
1	xul.dll	NS_DebugBreak_P	xpcom/base/nsDebugImpl.cpp:410
2	xul.dll	nsSupportsStringImpl::SetData	xpcom/ds/nsSupportsPrimitives.cpp:143
3	xul.dll	NS_InvokeByIndex_P	xpcom/reflect/xptcall/src/md/win32/xptcinvoke.cpp:70
4	xul.dll	XPCWrappedNative::CallMethod	js/xpconnect/src/XPCWrappedNative.cpp:2418
5	xul.dll	XPCWrappedNative::SetAttribute	js/xpconnect/src/xpcprivate.h:2803
6	xul.dll	XPC_WN_GetterSetter	js/xpconnect/src/XPCWrappedNativeJSOps.cpp:1528
7	mozjs.dll	js::InvokeKernel	js/src/jsinterp.cpp:367
8	mozjs.dll	js::Invoke	js/src/jsinterp.cpp:412
9	mozjs.dll	js::Shape::set	js/src/jsscopeinlines.h:313
10	mozjs.dll	js::baseops::SetPropertyHelper	js/src/jsobj.cpp:4694

Comment 7

[Benjamin Smedberg]

Wayne, if you could catch this in a debugger again, could you figure out what data is actually being set in nsSupportsStringImpl::SetData and also call DumpJSStack to see the JS stack?

Comment 8

[Wayne Mery (:wsmwk)]

I'd love to. Is ti possible for me to do this via windbg?  If so, how?

Comment 9

[Benjamin Smedberg]

I'm sure it's possible, yes. I don't actually know windbg that well, but I'm sure there are people in IRC who do (I used Visual Studio primarily).

Comment 10

[Wayne Mery (:wsmwk)]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #7)
> Wayne, if you could catch this in a debugger again, could you figure out
> what data is actually being set in nsSupportsStringImpl::SetData and also
> call DumpJSStack to see the JS stack?

here it is ... with assistance on IRC on 11-01, we found it was pointing to the sessionstore data

0:000> .frame 2
02 0013b794 60427b30 xul!nsSupportsStringImpl::SetData+0x204fb9 [e:\builds\moz2_slave\m-cen-w32-ntly\build\xpcom\ds\nssupportsprimitives.cpp @ 143]
0:000> .call DumpJSStack()
Thread is set up for call, 'g' will execute.
WARNING: This can have serious side-effects,
including deadlocks and corruption of the debuggee.
0:000> g
(1c28.1838): Access violation - code c0000005 (!!! second chance !!!)
eax=00000000 ebx=6107cf28 ecx=69cb3896 edx=00000003 esi=69c61ec6 edi=69cb379c
eip=73c5198d esp=0013b340 ebp=0013b34c iopl=0         nv up ei pl nz na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010206
mozalloc!mozalloc_abort+0x30:
73c5198d c705000000007b000000 mov dword ptr ds:[0],7Bh ds:0023:00000000=????????
0:000> .frame 2
02 0013b794 60427b30 xul!nsSupportsStringImpl::SetData+0x204fb9 [e:\builds\moz2_slave\m-cen-w32-ntly\build\xpcom\ds\nssupportsprimitives.cpp @ 143]
0:000> dd poi(aData) l3
0260e258  79c00000 00f4729c 00000001
0:000> du 0x79c00000
79c00000  "{"windows":[{"tabs":[{"entries":"
79c00040  "[{"url":"about:home","title":"Ni"
79c00080  "ghtly Start Page","ID":317395111"
79c000c0  "2,"docshellID":11229,"docIdentif"
79c00100  "ier":90},{"url":"https://bugzill"
...

so something about sessionstore data was not liked. Perhaps size being near 16mb is not a good thing?  But most or all of these crashes, sessionstore.js on disk is in the 14-15mb range.    (but note, sessionstore.js on disk can definitely get over 16mb)

I also see bug 581510 and Bug 511135 from time to time

Comment 11

[Scoobidiver (away)]

It spiked in 23.0a1/20130427 (four times the previous volume - same regression as bug 866526) in its form without debug symbols.

Comment 12

[John Hesling [:John99] (NeedInfo me)]

This may just be coincidental, but a user with a string of CrashIDs relating to this bug has reported in SUMO that upgrading FlashPlayer resolved the issue. https://support.mozilla.org/en-US/questions/953285

Comment 13

[Andrew McCreight [:mccr8]]

There are also a lot of EMPTY dumps, which is relevant to bug 837835.  I posted in there.  Also, it looks like the user's crashes were not fixed by upgrading Flash, as they started happening again.

Comment 14

[(not currently active) Ted Mielczarek]

That's not really surprising, since any crash which is due to OOM has a high probability of winding up with an empty dump on Windows (not having enough memory to write the dump properly).

Comment 17

[u279076]

The qawanted request on this bug is pretty old so I'm dropping it. Please re-add if there's something we can currently do to help. Marcia noted to me on IRC that this recently showed up at #23 on Firefox 23.0b10 and is now at #21.

Comment 18

[Scoobidiver (away)]

It's #17 browser crasher in 23.0 and #15 in 24.0b1.

Comment 19

[Tanner M. Young [:tmyoung]]

Mozilla/5.0 (Windows NT 6.0; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0

I am encountering this crash a lot: roughly three or four times each day.

https://crash-stats.mozilla.com/report/index/6e5b650a-5784-4906-b64d-4ea0a2130815
https://crash-stats.mozilla.com/report/index/dff19a89-a613-478e-9807-9ef1e2130815
https://crash-stats.mozilla.com/report/index/d12b0e8b-0bc6-4ab7-98cd-cbb102130815

Those links give some recent examples, also the following is an empty crash that I believe is the same problem.

https://crash-stats.mozilla.com/report/index/6d1f47cd-51cc-4eb5-b40d-a49582130815

It's a little hard to zero in on some STR, but I have noticed a lot of problems with Vimeo and Flickr.  Users in the crash comments are reporting of trouble with NPR streams and FB too.  I can't really say what exactly does it, as I have a number of tabs open at any given time--usually with FB, Vimeo and Flickr together as app tabs.  I know the former two use Flash constantly and the latter probably does a lot as well.  It usually hits when the plug-in has a lot chew on, like selecting dozens of photos in Flickr's organizer or when FB is loading a new list of items in the newsfeed, etc.  Part of the problem though is that half of crashes occur while I'm away from my computer which hints at background code in flash instead of it always being related to an action I take--like facebook automatically checking for updates or something similar.

Also this is a flash hang/crash that the browser recovered from which appeared to behave a lot like the other crashes but the browser was able to handle it: https://crash-stats.mozilla.com/report/index/b5790977-29d8-4b86-94fd-0c8262130815

Comment 20

[Tanner M. Young [:tmyoung]]

One other thing I've discovered.  I haven't had this crash appear since I reduced the number of tabs open in my session.  I have a number of tabs open (or disabled through an extension).  So it could be related to sites that use plugins or a lot of DOM/HTML like FB and a large number of tabs open or other sources of load.  Since my computer has a large amount of horse power, it may take proportionally more load to cause the crash than it otherwise might.

Comment 21

[Wayne Mery (:wsmwk)]

(In reply to Tanner M. Young [:tmyoung] from comment #20)
> One other thing I've discovered.  I haven't had this crash appear since I
> reduced the number of tabs open in my session.  

This mirrors my experience.

> I have a number of tabs open
> (or disabled through an extension).  So it could be related to sites that
> use plugins or a lot of DOM/HTML like FB and a large number of tabs open or
> other sources of load.  

I have no correlation to plugins.  90-99% of tabs are:
- crash-stats (topcrash pages seem to be especially onerous) 
- bugzilla.m.o
- getsatisfaction.com/mozilla_messaging/
all of which are complex and/or heavy js pages.


> Since my computer has a large amount of horse power,
> it may take proportionally more load to cause the crash than it otherwise
> might.

unrelated to horsepower afaict and more to do with cycle collector behavior with high number of tabs. Suggest you install memchaser addon.  see the bugs mentioned in comment 10.

Comment 22

[bhavana bajaj [:bajaj]]

Tracking as its a top-crash;

needsinfo on :bsmedberg here as this is discussed in the stability workweek but I am unsure of the next steps here to make it actionable .

Comment 23

[Benjamin Smedberg]

There are no next steps for this bug. It is a generic out of memory bug that has to be investigated at the source. One likely suspect for a problem is bug 859955, which is currently unowned.

Comment 24

[u279076]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #23)
> There are no next steps for this bug. It is a generic out of memory bug that
> has to be investigated at the source. One likely suspect for a problem is
> bug 859955, which is currently unowned.

Should this be whiteboard tagged as unactionable at this time?

Comment 25

[Benjamin Smedberg]

If that helps somebody track it, sure.

Comment 26

[u279076]

Current standing in crash-stats:
> 24.0   #37 0.22%
> 25.0b  #23 0.30%
> 26.0a2 #44 0.22%
> 27.0a1 #59 0.19%

This signature no longer qualifies as a top-crash.

Comment 27

[kumar.ganesha]

I have found that my Firefox crashes with similar stacktrace. The default crash reporter produces crash report with empty thread information. When I ran Firefox with WinDbg, it produces the following stacktrace:
----------------------------------------------------------------------------------------------------------
FAULTING_IP: 
mozalloc!mozalloc_abort+2d [e:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\memory\mozalloc\mozalloc_abort.cpp @ 30]
715f119f c705000000007b000000 mov dword ptr ds:[0],7Bh

EXCEPTION_RECORD:  ffffffffffffffff -- (.exr 0xffffffffffffffff)
.exr 0xffffffffffffffff
ExceptionAddress: 00000000715f119f (mozalloc!mozalloc_abort+0x000000000000002d)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 0000000000000001
   Parameter[1]: 0000000000000000
Attempt to write to address 0000000000000000

FAULTING_THREAD:  0000000000001194

PROCESS_NAME:  firefox.exe

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

EXCEPTION_PARAMETER1:  0000000000000001

EXCEPTION_PARAMETER2:  0000000000000000

WRITE_ADDRESS:  0000000000000000 

FOLLOWUP_IP: 
mozalloc!mozalloc_abort+2d [e:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\memory\mozalloc\mozalloc_abort.cpp @ 30]
715f119f c705000000007b000000 mov dword ptr ds:[0],7Bh

APPLICATION_VERIFIER_FLAGS:  0

APP:  firefox.exe

BUGCHECK_STR:  APPLICATION_FAULT_NULL_POINTER_WRITE_ZEROED_STACK

PRIMARY_PROBLEM_CLASS:  NULL_POINTER_WRITE

DEFAULT_BUCKET_ID:  NULL_POINTER_WRITE

LAST_CONTROL_TRANSFER:  from 00000000715f1218 to 00000000715f119f

STACK_TEXT:  
0582f884 715f1218 0582f89c 00008000 71254bff mozalloc!mozalloc_abort+0x2d
0582f8d4 715f10a2 00008000 715f1087 8db58a24 mozalloc!mozalloc_handle_oom+0x5f
0582f8e4 507b1cc8 00008000 8db589c0 00000000 mozalloc!moz_xmalloc+0x1b
0582f8f4 506fa733 8db589f8 00000000 506fa644 xul!nsSegmentedBuffer::AppendNewSegment+0x58
0582f900 506fa644 0582f924 0582f920 00008000 xul!nsPipe::GetWriteSegment+0x33
0582f934 50853a6e 8db589f8 506fa230 b6658800 xul!nsPipeOutputStream::WriteSegments+0x34
0582f95c 50853b50 0b26d3d4 00008000 0582f980 xul!nsHttpTransaction::WriteSegments+0x2d
0582f978 506be2b2 00000000 08470970 08470a10 xul!nsHttpConnection::OnSocketReadable+0xb0
0582f988 50850024 0b26d3d8 08470a10 0b26d3d8 xul!nsHttpConnection::OnInputStreamReady+0x26
0582f998 506da5ee 00000000 00000001 00aa3040 xul!nsSocketInputStream::OnSocketReady+0x84
0582f9ac 5083fc43 0afb0320 00000001 00aa30a0 xul!nsSocketTransport::OnSocketReady+0x5e
0582f9cc 507fb69a 00000000 00000001 00aa30a0 xul!nsSocketTransportService::DoPollIteration+0x103
0582f9e8 507a7a51 00aa3000 0582fa84 0582faa0 xul!nsSocketTransportService::Run+0xba
0582fa5c 507fb1b8 00aa3000 00000001 0582fa8c xul!nsThread::ProcessNextEvent+0x221
0582fa84 5ef5e927 00aa3001 00000000 00000000 xul!nsThread::ThreadFunc+0x98
0582faa0 5ef6329d 00a0f530 6a4cc6de 00a0f530 nss3!_PR_NativeRunThread+0x167
0582faa8 6a4cc6de 00a0f530 6ad72338 00000000 nss3!pr_root+0xd
0582fae0 6a4cc788 00000000 0582faf8 76a7336a MSVCR100!_callthreadstartex+0x1b
0582faec 76a7336a 0075b8b8 0582fb38 77669f72 MSVCR100!_threadstartex+0x64
0582faf8 77669f72 0075b8b8 530ff328 00000000 kernel32!BaseThreadInitThunk+0xe
0582fb38 77669f45 6a4cc724 0075b8b8 00000000 ntdll32!__RtlUserThreadStart+0x70
0582fb50 00000000 6a4cc724 0075b8b8 00000000 ntdll32!_RtlUserThreadStart+0x1b


FAULTING_SOURCE_LINE:  e:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\memory\mozalloc\mozalloc_abort.cpp

FAULTING_SOURCE_FILE:  e:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\memory\mozalloc\mozalloc_abort.cpp

FAULTING_SOURCE_LINE_NUMBER:  30

FAULTING_SOURCE_CODE:  
No source found for 'e:\builds\moz2_slave\rel-m-rel-w32_bld-000000000000\build\memory\mozalloc\mozalloc_abort.cpp'


SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  mozalloc!mozalloc_abort+2d

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: mozalloc

IMAGE_NAME:  mozalloc.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  522fa829

STACK_COMMAND:  ~7s ; kb

FAILURE_BUCKET_ID:  NULL_POINTER_WRITE_c0000005_mozalloc.dll!mozalloc_abort

BUCKET_ID:  X64_APPLICATION_FAULT_NULL_POINTER_WRITE_ZEROED_STACK_mozalloc!mozalloc_abort+2d

----------------------------------------------------------------------------------------------------------

Is this the same bug or a new one?

Comment 28

[Kyle Wooten]

Thanks!

Comment 29

[(not currently active) Ted Mielczarek]

(In reply to kumar.ganesha from comment #27)
> Is this the same bug or a new one?

This looks to be a completely different bug. Your Firefox is crashing trying to allocate a buffer for a pipe for an incoming HTTP request. That should not be very large, so I suspect something else is using up all of your memory first. If you can reproduce this then I'd suggest you check about:memory periodically before crashing and see if anything stands out.

Comment 30

[Alex Keybl [:akeybl]]

(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #26)
> Current standing in crash-stats:
> > 24.0   #37 0.22%
> > 25.0b  #23 0.30%
> > 26.0a2 #44 0.22%
> > 27.0a1 #59 0.19%
> 
> This signature no longer qualifies as a top-crash.

Is that true when adding all signatures?

Comment 31

[u279076]

(In reply to Alex Keybl [:akeybl] from comment #30)
> (In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #26)
> > Current standing in crash-stats:
> > > 24.0   #37 0.22%
> > > 25.0b  #23 0.30%
> > > 26.0a2 #44 0.22%
> > > 27.0a1 #59 0.19%
> > 
> > This signature no longer qualifies as a top-crash.
> 
> Is that true when adding all signatures?

I'm not sure how to aggregate multiple signature rankings. KaiRo, can you help out here?

Comment 32

[Benjamin Smedberg]

Use the script from https://bugzilla.mozilla.org/show_bug.cgi?id=915373#c2

Comment 33

[u279076]

Tracy showed me how to do a manual aggregation, here are the numbers:
 * Release: #40 (1624 crashes)
 * Beta:    #39 (437 crashes)
 * Aurora:  #26 (38 crashes)
 * Nightly: #38 (32 crashes)

Using Benjamin's script I get slightly different numbers:
 * Release: #49
 * Beta:    #43
 * Aurora:  #29
 * Nightly: #41

In either case this does not qualify as a top-crash.

Comment 34

[Robert Kaiser]

Well, we would need to aggregate in a way where we make sure all those have this abort message, for which we have no tooling at all right now. But even the umbrella for this seems to not qualify as topcrash any more, so we are good here.

Comment 37

[Robert Kaiser]

This is topcrash #8 on 25 release atm.

Comment 38

[User Dderss]

I have similar crash as Robert Kaiser provided in the the latest message:

https://crash-stats.mozilla.com/report/index/4abf0b19-8e5e-414a-a9fb-7ab732131107

Crash Signature:

mozalloc_abort(char const* const) | xul.dll@0xdd6788 | xul.dll@0x8a686b | xul.dll@0x6aea28 | xul.dll@0xc3138 | xul.dll@0xe70e6 | js::Invoke(JSContext*, JS::Value const&, JS::Value const&, unsigned int, JS::Value*, JS::MutableHandle<JS::Value>)

Comment 39

[Robert Kaiser]

The question came up why the recent signatures for this all contain unsymbolized xul.dll frames. Here's some info I could find out:

First, it turns out that there's no debug ID (or PDB file name) for xul.dll in those reports, which in turn makes us unable to find symbols.

Second, Ted thinks that the Windows function we use for gathering the minidump might need to allocate memory for reading that info and as we are in an OOM situation, the large size of xul.dll might cause that to fail and as a result not gather the debug ID.

Comment 40

[Robert Kaiser]

FWIW, I just used the pretty new Socorro functionality of customizing columns in the Reports tab of https://crash-stats.mozilla.com/report/list?signature=mozalloc_abort%28char%20const*%20const%29%20|%20xul.dll%400xdf4367%20|%20xul.dll%400x8b1965%20|%20xul.dll%400x6a9549%20|%20xul.dll%400x1e502b%20|%20xul.dll%400x1d2481%20|%20js%3A%3AInvoke%28JSContext*%2C%20JS%3A%3AValue%20const%26%2C%20JS%3A%3AValue%20const%26%2C%20unsigned%20int%2C%20JS%3A%3AValue*%2C%20JS%3A%3AMutableHandle%3CJS%3A%3AValue%3E%29 (new signature for Fx 26.0b4) to look at AvailablePageFile, AvailablePhysicalMemory, AvailableVirtualMemory, and OOMAllocationSize of those crashes, and interestingly all those have plenty of available memory in all three categories and nothing in OOMAllocationSize which IIUC means this was not an infallible allocation that ran OOM. Is that expected with the abort message we get here?

Comment 41

[Benjamin Smedberg]

That is expected, because in our string code we're using the fallible allocator and later aborting if it failed, rather that using the infallible allocator directly. I've been considering whether we can modify the annotation in this case also, but haven't done anything about it yet.

FWIW, I don't think "plenty of available memory" is quite correct. In most of these crashes, we're seeing AvailableVirtualMemory < 500MB. And because AvailableVirtualMemory doesn't take VM fragmentation into account, it's quite possible that we're running out of VM space.

Comment 42

[User Dderss]

In the crash I have got (see http://bugzilla.mozilla.org/show_bug.cgi?id=767343#c38 above), there was 9 GB of available memory (out of 16 GB; 7 GBs occupied). In such case, what might cause the crash?

Comment 43

[Robert Kaiser]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #41)
> FWIW, I don't think "plenty of available memory" is quite correct. In most
> of these crashes, we're seeing AvailableVirtualMemory < 500MB. And because
> AvailableVirtualMemory doesn't take VM fragmentation into account, it's
> quite possible that we're running out of VM space.

Ah, OK, makes sense. I was taking "surely multi-MB" as "plenty" (hoping we don't do as large allocations too often) but with VM fragmentation I see how all those numbers can lie. I guess there's no easy way for us to report how much would be available to our allocation and how much it tries to allocate?

(In reply to User Dderss from comment #42)
> In the crash I have got (see
> http://bugzilla.mozilla.org/show_bug.cgi?id=767343#c38 above), there was 9
> GB of available memory (out of 16 GB; 7 GBs occupied). In such case, what
> might cause the crash?

Note that what usually counts here is the amount of non-fragmented virtual memory available to our process (which is a 32bit process so has 4GB VM space available at max). I'm not sure if there's any easy way to diagnose here, though Benjamin might know something (unlike me, he's an engineer and has a lot of knowledge about those things).

Comment 44

[Benjamin Smedberg]

> all those numbers can lie. I guess there's no easy way for us to report how
> much would be available to our allocation and how much it tries to allocate?

In fact there is! Bug 939141 covers annotating the largest free VM block (requires JSON MDSW to be deployed).

And bug 938794 covers annotating the allocation size we're attempting for these string aborts.

Comment 45

[(not currently active) Ted Mielczarek]

(In reply to User Dderss from comment #38)
> https://crash-stats.mozilla.com/report/index/4abf0b19-8e5e-414a-a9fb-
> 7ab732131107

FWIW, the stack from this crash is:
Thread 0 (crashed)
 0  mozalloc.dll!mozalloc_abort(char const * const) [mozalloc_abort.cpp:d86ad7db
1de3 : 30 + 0x0]
    eip = 0x53b6119c   esp = 0x0047db60   ebp = 0x0047df90   ebx = 0x53479b44
    esi = 0x54121ec6   edi = 0x5417379c   eax = 0x00000000   ecx = 0x54173896
    edx = 0x00000003   efl = 0x00200206
    Found by: recovered by external stack walker
 1  xul.dll!NS_DebugBreak [nsDebugImpl.cpp:d86ad7db1de3 : 417 + 0xb]
    eip = 0x52f76788   esp = 0x0047db6c   ebp = 0x0047df90
    Found by: call frame info
 2  xul.dll!nsSupportsStringImpl::SetData(nsAString_internal const &) [nsSupport
sPrimitives.cpp:d86ad7db1de3 : 143 + 0x18]
    eip = 0x52a4686b   esp = 0x0047df98   ebp = 0x0047dfac
    Found by: call frame info
 3  xul.dll!NS_InvokeByIndex [xptcinvoke.cpp:d86ad7db1de3 : 70 + 0x3]
    eip = 0x5284ea28   esp = 0x0047dfb4   ebp = 0x0047dfc0
    Found by: previous frame's frame pointer
 4  xul.dll!XPCWrappedNative::CallMethod(XPCCallContext &,XPCWrappedNative::Call
Mode) [XPCWrappedNative.cpp:d86ad7db1de3 : 2114 + 0x3eb]
    eip = 0x52263138   esp = 0x0047dfc8   ebp = 0x0047e18c
    Found by: call frame info
 5  xul.dll!XPC_WN_GetterSetter(JSContext *,unsigned int,JS::Value *) [XPCWrappe
dNativeJSOps.cpp:d86ad7db1de3 : 1343 + 0xe]
    eip = 0x522870e6   esp = 0x0047e194   ebp = 0x0047e48c
    Found by: call frame info
 6  xul.dll!XPC_WN_JSOp_ThisObject(JSContext *,JS::Handle<JSObject *>) [XPCWrappedNativeJSOps.cpp:d86ad7db1de3 : 1117 + 0x12]
    eip = 0x52467a82   esp = 0x0047e48c   ebp = 0x0047e500
    Found by: call frame info
 7  0x1
    eip = 0x00000001   esp = 0x0047e49c   ebp = 0x0047e500
    Found by: call frame info
 8  0xffffff82
    eip = 0xffffff82   esp = 0x0047e508   ebp = 0x00000000
    Found by: previous frame's frame pointer
 9  mozjs.dll!js::Shape::set(JSContext *,JS::Handle<JSObject *>,JS::Handle<JSObject *>,bool,JS::MutableHandle<JS::Value>) [Shape-inl.h:d86ad7db1de3 : 282 + 0x4b]
    eip = 0x53bd9b97   esp = 0x0047e558   ebp = 0x0047e5e0
    Found by: stack scanning
10  mozjs.dll!js::Invoke(JSContext *,JS::CallArgs,js::MaybeConstruct) [Interpreter.cpp:d86ad7db1de3 : 486 + 0x8]
    eip = 0x53bf6d0a   esp = 0x0047e57c   ebp = 0x0047e5e0
    Found by: call frame info
11  mozjs.dll!GetPropertyOperation(JSContext *,js::StackFrame *,JS::Handle<JSScript *>,unsigned char *,JS::MutableHandle<JS::Value>,JS::MutableHandle<JS::Value>) [Interpreter.cpp:d86ad7db1de3 : 286 + 0x1d]
    eip = 0x53c0d620   esp = 0x0047e72c   ebp = 0x0047e610
    Found by: call frame info with scanning
12  mozjs.dll!Interpret [Interpreter.cpp:d86ad7db1de3 : 2518 + 0x28]
    eip = 0x53c09daf   esp = 0x0047e7b4   ebp = 0x49377840
    Found by: call frame info with scanning
13  mozjs.dll!js::Invoke(JSContext *,JS::Value const &,JS::Value const &,unsigned int,JS::Value *,JS::MutableHandle<JS::Value>) [Interpreter.cpp:d86ad7db1de3 : 536 + 0x12c]
    eip = 0x53bf8e1f   esp = 0x0047ef60   ebp = 0x53ba7730
    Found by: call frame info with scanning
14  mozjs.dll!JS_CallFunctionValue(JSContext *,JSObject *,JS::Value,unsigned int,JS::Value *,JS::Value *) [jsapi.cpp:d86ad7db1de3 : 5394 + 0x4b]
    eip = 0x53c39524   esp = 0x0047f020   ebp = 0x0047f054
    Found by: call frame info
15  xul.dll!nsXPCWrappedJSClass::CallMethod(nsXPCWrappedJS *,unsigned short,XPTMethodDescriptor const *,nsXPTCMiniVariant *) [XPCWrappedJSClass.cpp:d86ad7db1de3 : 1444 + 0x26]
    eip = 0x5227b738   esp = 0x0047f05c   ebp = 0x0047f2b8
    Found by: call frame info
16  xul.dll!nsXPCWrappedJS::CallMethod(unsigned short,XPTMethodDescriptor const 
*,nsXPTCMiniVariant *) [XPCWrappedJS.cpp:d86ad7db1de3 : 589 + 0x2a]
    eip = 0x522454fb   esp = 0x0047f2c0   ebp = 0x0047f33c
    Found by: call frame info
17  xul.dll!PrepareAndDispatch [xptcstubs.cpp:d86ad7db1de3 : 85 + 0x15]
    eip = 0x5284eb26   esp = 0x0047f2ec   ebp = 0x0047f33c
    Found by: call frame info
18  xul.dll!SharedStub [xptcstubs.cpp:d86ad7db1de3 : 112 + 0x5]
    eip = 0x5284eb9a   esp = 0x0047f3ac   ebp = 0x0047f3c0
    Found by: call frame info
19  xul.dll!nsTimerImpl::Fire() [nsTimerImpl.cpp:d86ad7db1de3 : 552 + 0x10]
    eip = 0x5226910e   esp = 0x0047f3c8   ebp = 0x0047f3c0
    Found by: call frame info

Comment 46

[User Dderss]

How to interpret this stack and the crash signature?

Which changes to the source code should be done to avoid this crash at least in the upcoming FF 26 and later versions?

Comment 47

[User Dderss]

Another crash, this time not empty:

https://crash-stats.mozilla.com/report/index/14bdd66e-eac9-44d2-9c8e-dbaaa2131122

Signature:
mozalloc_abort(char const* const) | xul.dll@0xdd7368 | xul.dll@0x8a52c8 | xul.dll@0x6adc4c | xul.dll@0x169988 | xul.dll@0x14a6b6 | js::Invoke(JSContext*, JS::Value const&, JS::Value const&, unsigned int, JS::Value*, JS::MutableHandle<JS::Value>) 

Can please anyone say if there is chance to fix those Malloc crashes in the foreseeable future? I mean I have the crashes almost every day. It is almost impossible to work that way. Really unbearable experience.

Comment 48

[Boris Zbarsky [:bzbarsky]]

User Dderss, mozalloc_abort is the general thing that gets called in two cases:

1)  Out of memory (virtual or physical).
2)  Failed fatal runtime assertion.

For this last stack of yours (and actually the previous one too), it looks like the latter.  This bug is about the former.

I suggest doing the following:

1)  File a separate bug about your crash.
2)  Check whether any of your extensions might be causing the problem (e.g. by disabling
    them for a few days; if you crash as consistently as you do that might be enough to
    determine whether they're causing the problem).

Step 2 is particularly important if you have any binary extensions or extensions that might be doing low-level things with the JS engine or DOM guts, because your aborts are almost certainly compartment mismatch failures given their stacks.

Comment 49

[Boris Zbarsky [:bzbarsky]]

Oh, and please cc me on your new bug, ok?

Comment 50

[User Dderss]

Thanks, Boris. This Malloc crash half of the time creates empty/corrupt-mini dump, so I had to run the browser under debugger to confirm it is the same crash as those which come with correct dump and signature you saw above. 

I already created separate bug report on that; with stack and everything:
https://bugzilla.mozilla.org/show_bug.cgi?id=930797

It does not look that it depends on add-ons, right? Nothing weird runs in my system. (I even switched off the damn Adobe Flash plug-in, which crashed my system by itself every day.)

Comment 51

[Robert Kaiser]

(In reply to Boris Zbarsky [:bz] from comment #48)
> For this last stack of yours (and actually the previous one too), it looks
> like the latter.  This bug is about the former.

Sorry Boris to disagree with you, but from all I see here, going back to comment #0, this bug was always about the case that User Dderss is seeing. The abort message is there and even contains "OOM" in the message, the stack looks pretty much the same (just that we don't resolve the xul.dll frames, see comment #39).

Comment 52

[Boris Zbarsky [:bzbarsky]]

> I already created separate bug report on that; with stack and everything:

Great, thank you!  I'll follow up there.

Comment 53

[Robert Kaiser]

As a note, the mozalloc_abort(char const* const) | NS_DebugBreak | nsSupportsStringImpl::SetData(nsAString_internal const&) signature seems to be rising in topcrash stats as a result of reserving memory for Breakpad and decreasing the xull.dll-containing other signatures here or the EMPTY signature.

Comment 54

[Benjamin Smedberg]

The primary signature now is almost certainly going to be NS_ABORT_OOM | nsStringsomething

Comment 55

[Benjamin Smedberg]

It sucks that we don't have JS stack traces from the point where we died: this is pretty much exclusively being called from nsTimerImpl::Fire -> unknown JS. But we can use the fallible allocator in SetString and see what happens.

Comment 56

[Benjamin Smedberg]

Attachment: bug767343-SupportsPrimitive-fallible

Comment 57

[Nathan Froyd [:froydnj]]

Comment on attachment 8343791 [details] [diff] [review]
bug767343-SupportsPrimitive-fallible

Review of attachment 8343791 [details] [diff] [review]:
-----------------------------------------------------------------

Some men just want to watch what burns when memory cannot be allocated and real error codes returned.  I am one of those men.  r=me

Comment 58

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/fac17c3d3efd
https://hg.mozilla.org/mozilla-central/rev/d441322e0ad0

Comment 59

[Benjamin Smedberg]

Attachment: branch patch

[Approval Request Comment]
Bug caused by (feature/regressing bug #): Unknown code sticking very large strings into wrapper class, crashing due to infallible alloc
User impact if declined: more OOM crashes
Testing completed (on m-c, etc.): landed, no regressions reported
Risk to taking this patch (and alternatives if risky): fairly low risk: the main problem is that this might throw exceptions into session-restore or other code which isn't expecting them, and that code would stop working. But I think that's still better than crashing.
String or IDL/UUID changes made by this patch: None

Comment 60

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/f36745a543a4

Comment 61

[u279076]

Flagging for verification by checking crashstats.

Comment 62

[Bogdan Maris, Desktop Test Engineering]

Verified as fixed on Firefox 27 beta 4 and latest Aurora 28.0a2 (buildID: 20140107004003). No crashes were registered in the last 3 weeks on all signatures. For crash stats links please check this etherpad https://etherpad.mozilla.org/Bug767343-CrashStats