OOM Crash [@ Length] due to unhandled alloc failure in nsXBLPrototypeBinding::InstantiateInsertionPoints

NEW
Unassigned

Status

()

Core
XBL
--
critical
6 years ago
6 years ago

People

(Reporter: decoder, Unassigned)

Tracking

(Blocks: 1 bug, {crash})

Trunk
x86_64
Linux
crash
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(crash signature)

(Reporter)

Description

6 years ago
Tested on m-c revision 66e4d53697c2: The method nsXBLPrototypeBinding::InstantiateInsertionPoints calls nsHashtable::Enumerate on one of its members. Although this method is of return type "void", it seems to be fallible and allocation failure will cause the following crash:


Program received signal SIGSEGV, Segmentation fault.
0x00002aaaaca70f3b in Length (this=0x0) at ../../../dist/include/nsTArray.h:224
224         return mHdr->mLength;
#0  0x00002aaaaca70f3b in Length (this=0x0) at ../../../dist/include/nsTArray.h:224
#1  InstantiateInsertionPoint (aKey=<optimized out>, aData=<optimized out>, aClosure=<optimized out>) at /srv/repos/browser/mozilla-central/content/xbl/src/nsXBLPrototypeBinding.cpp:712
#2  0x00002aaaad25b01f in hashEnumerate (table=<optimized out>, hdr=<optimized out>, i=<optimized out>, arg=<optimized out>) at /srv/repos/browser/mozilla-central/xpcom/ds/nsHashtable.cpp:130
#3  0x00002aaaad24979f in PL_DHashTableEnumerate (table=0x2aaabee8d5a0, etor=0x2aaaad25b00a <hashEnumerate(PLDHashTable*, PLDHashEntryHdr*, PRUint32, void*)>, arg=0x7fffffff9c20) at /srv/repos/browser/mozilla-central/objdir-ff-gcc64dbg/xpcom/build/pldhash.cpp:754
#4  0x00002aaaad25b781 in nsHashtable::Enumerate (this=0x2aaabee8d590, aEnumFunc=<optimized out>, aClosure=<optimized out>) at /srv/repos/browser/mozilla-central/xpcom/ds/nsHashtable.cpp:314
#5  0x00002aaaaca6b103 in nsXBLPrototypeBinding::InstantiateInsertionPoints (this=<optimized out>, aBinding=<optimized out>) at /srv/repos/browser/mozilla-central/content/xbl/src/nsXBLPrototypeBinding.cpp:746
#6  0x00002aaaaca6a260 in nsXBLBinding::GenerateAnonymousContent (this=0x2aaabee50660) at /srv/repos/browser/mozilla-central/content/xbl/src/nsXBLBinding.cpp:683
#7  0x00002aaaaca8250e in nsXBLService::LoadBindings (this=0x0, aContent=0x2aaabd3bf100, aURL=0x2aaabedd1190, aOriginPrincipal=0x2aaabda802e0, aAugmentFlag=false, aBinding=0x2aaabe614350, aResolveStyle=0x7fffffffa1af) at /srv/repos/browser/mozilla-central/content/xbl/src/nsXBLService.cpp:611


The backtrace of the failing allocation is as follows:

#0 /srv/repos/browser/mozilla-central/objdir-ff-gcc64dbg/dist/bin/libmozalloc.so(moz_malloc+0x5f) [0x2aaaaab2415c] (aab2415c)
#1 PL_DHashTableInit at objdir-ff-gcc64dbg/xpcom/build/pldhash.cpp:270
#2 nsTHashtable<nsBaseHashtableET<nsISupportsHashKey, nsAutoPtr<nsTArray<nsRefPtr<nsXBLInsertionPoint>, nsTArrayDefaultAllocator> > > >::Init(unsigned int) at objdir-ff-gcc64dbg/dist/include/nsTHashtable.h:425
#3 nsTArray_base<nsTArrayDefaultAllocator>::Length() const at objdir-ff-gcc64dbg/dist/include/nsTArray.h:224
#4 hashEnumerate at xpcom/ds/nsHashtable.cpp:125
#5 PL_DHashTableEnumerate at objdir-ff-gcc64dbg/xpcom/build/pldhash.cpp:755
#6 nsHashtable::Enumerate(bool (*)(nsHashKey*, void*, void*), void*) at xpcom/ds/nsHashtable.cpp:315
#7 nsXBLPrototypeBinding::InstantiateInsertionPoints(nsXBLBinding*) at content/xbl/src/nsXBLPrototypeBinding.cpp:747
#8 ContentListData at content/xbl/src/nsXBLBinding.cpp:431
#9 nsXBLService::LoadBindings(nsIContent*, nsIURI*, nsIPrincipal*, bool, nsXBLBinding**, bool*) at content/xbl/src/nsXBLService.cpp:614
[...]


I'm not sure how this OOM should be handled, as Enumerate does not return anything that the caller could use to check for allocation failure. I suppose it should have either a return type to indicate that it's fallible, or another way/method to check for success.
InstantiateInsertionPoints needs to add a boolean to the closure data indicating whether the enumeration succeeded or failed, and that needs to be set in the right places and checked in the right places.

Updated

6 years ago
Depends on: 653881
You need to log in before you can comment on or make changes to this bug.