731179 - OOM Crash [@ Length] due to unhandled alloc failure in nsXBLPrototypeBinding::InstantiateInsertionPoints

Status: RESOLVED WONTFIX

(Core :: XBL, defect)

Description

[Christian Holler (:decoder)]

Tested on m-c revision 66e4d53697c2: The method nsXBLPrototypeBinding::InstantiateInsertionPoints calls nsHashtable::Enumerate on one of its members. Although this method is of return type "void", it seems to be fallible and allocation failure will cause the following crash:


Program received signal SIGSEGV, Segmentation fault.
0x00002aaaaca70f3b in Length (this=0x0) at ../../../dist/include/nsTArray.h:224
224         return mHdr->mLength;
#0  0x00002aaaaca70f3b in Length (this=0x0) at ../../../dist/include/nsTArray.h:224
#1  InstantiateInsertionPoint (aKey=<optimized out>, aData=<optimized out>, aClosure=<optimized out>) at /srv/repos/browser/mozilla-central/content/xbl/src/nsXBLPrototypeBinding.cpp:712
#2  0x00002aaaad25b01f in hashEnumerate (table=<optimized out>, hdr=<optimized out>, i=<optimized out>, arg=<optimized out>) at /srv/repos/browser/mozilla-central/xpcom/ds/nsHashtable.cpp:130
#3  0x00002aaaad24979f in PL_DHashTableEnumerate (table=0x2aaabee8d5a0, etor=0x2aaaad25b00a <hashEnumerate(PLDHashTable*, PLDHashEntryHdr*, PRUint32, void*)>, arg=0x7fffffff9c20) at /srv/repos/browser/mozilla-central/objdir-ff-gcc64dbg/xpcom/build/pldhash.cpp:754
#4  0x00002aaaad25b781 in nsHashtable::Enumerate (this=0x2aaabee8d590, aEnumFunc=<optimized out>, aClosure=<optimized out>) at /srv/repos/browser/mozilla-central/xpcom/ds/nsHashtable.cpp:314
#5  0x00002aaaaca6b103 in nsXBLPrototypeBinding::InstantiateInsertionPoints (this=<optimized out>, aBinding=<optimized out>) at /srv/repos/browser/mozilla-central/content/xbl/src/nsXBLPrototypeBinding.cpp:746
#6  0x00002aaaaca6a260 in nsXBLBinding::GenerateAnonymousContent (this=0x2aaabee50660) at /srv/repos/browser/mozilla-central/content/xbl/src/nsXBLBinding.cpp:683
#7  0x00002aaaaca8250e in nsXBLService::LoadBindings (this=0x0, aContent=0x2aaabd3bf100, aURL=0x2aaabedd1190, aOriginPrincipal=0x2aaabda802e0, aAugmentFlag=false, aBinding=0x2aaabe614350, aResolveStyle=0x7fffffffa1af) at /srv/repos/browser/mozilla-central/content/xbl/src/nsXBLService.cpp:611


The backtrace of the failing allocation is as follows:

#0 /srv/repos/browser/mozilla-central/objdir-ff-gcc64dbg/dist/bin/libmozalloc.so(moz_malloc+0x5f) [0x2aaaaab2415c] (aab2415c)
#1 PL_DHashTableInit at objdir-ff-gcc64dbg/xpcom/build/pldhash.cpp:270
#2 nsTHashtable<nsBaseHashtableET<nsISupportsHashKey, nsAutoPtr<nsTArray<nsRefPtr<nsXBLInsertionPoint>, nsTArrayDefaultAllocator> > > >::Init(unsigned int) at objdir-ff-gcc64dbg/dist/include/nsTHashtable.h:425
#3 nsTArray_base<nsTArrayDefaultAllocator>::Length() const at objdir-ff-gcc64dbg/dist/include/nsTArray.h:224
#4 hashEnumerate at xpcom/ds/nsHashtable.cpp:125
#5 PL_DHashTableEnumerate at objdir-ff-gcc64dbg/xpcom/build/pldhash.cpp:755
#6 nsHashtable::Enumerate(bool (*)(nsHashKey*, void*, void*), void*) at xpcom/ds/nsHashtable.cpp:315
#7 nsXBLPrototypeBinding::InstantiateInsertionPoints(nsXBLBinding*) at content/xbl/src/nsXBLPrototypeBinding.cpp:747
#8 ContentListData at content/xbl/src/nsXBLBinding.cpp:431
#9 nsXBLService::LoadBindings(nsIContent*, nsIURI*, nsIPrincipal*, bool, nsXBLBinding**, bool*) at content/xbl/src/nsXBLService.cpp:614
[...]


I'm not sure how this OOM should be handled, as Enumerate does not return anything that the caller could use to check for allocation failure. I suppose it should have either a return type to indicate that it's fallible, or another way/method to check for success.

Comment 1

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

InstantiateInsertionPoints needs to add a boolean to the closure data indicating whether the enumeration succeeded or failed, and that needs to be set in the right places and checked in the right places.

Comment 2

[Sylvestre Ledru [:Sylvestre]]

Closing because no crash reported since 12 weeks.

Comment 3

[Sylvestre Ledru [:Sylvestre]]

Closing because no crash reported since 12 weeks.