Closed Bug 913138 Opened 6 years ago Closed 6 years ago

Intermittent test_service_login.js | test failed (with xpcshell return code: 1) | application crashed [@ nsContentUtils::IsCallerChrome()]

Categories

(Core :: XPConnect, defect)

x86_64
macOS
defect
Not set

Tracking

()

VERIFIED FIXED
mozilla31
Tracking Status
firefox29 --- verified
firefox30 --- verified
firefox31 --- verified
firefox-esr24 --- unaffected

People

(Reporter: RyanVM, Assigned: bholley)

References

Details

(Keywords: crash, intermittent-failure)

Crash Data

Attachments

(4 files)

https://tbpl.mozilla.org/php/getParsedLog.php?id=27432866&tree=Fx-Team

Rev5 MacOSX Mountain Lion 10.8 fx-team debug test xpcshell on 2013-09-05 08:08:49 PDT for push efe592efe709
slave: talos-mtnlion-r5-084

08:20:51     INFO -  TEST-INFO | /builds/slave/talos-slave/test/build/tests/xpcshell/tests/services/sync/tests/unit/test_service_login.js | running test ...
08:21:00  WARNING -  TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/tests/xpcshell/tests/services/sync/tests/unit/test_service_login.js | test failed (with xpcshell return code: 1), see following log:
08:21:01     INFO -  >>>>>>>
08:21:01     INFO -  System JS : WARNING /builds/slave/talos-slave/test/build/tests/xpcshell/tests/services/sync/tests/unit/head_http_server.js:885
08:21:01     INFO -                       function handleStorage does not always return a value
08:21:01     INFO -  System JS : WARNING /builds/slave/talos-slave/test/build/tests/xpcshell/tests/services/sync/tests/unit/head_http_server.js:890
08:21:01     INFO -                       function handleStorage does not always return a value
08:21:01     INFO -  System JS : WARNING /builds/slave/talos-slave/test/build/tests/xpcshell/tests/services/sync/tests/unit/head_http_server.js:892
08:21:01     INFO -                       function handleStorage does not always return a value
08:21:01     INFO -  System JS : WARNING /builds/slave/talos-slave/test/build/tests/xpcshell/tests/services/sync/tests/unit/head_http_server.js:898
08:21:01     INFO -                       function handleStorage does not always return a value
08:21:01     INFO -  System JS : WARNING /builds/slave/talos-slave/test/build/tests/xpcshell/tests/services/sync/tests/unit/head_http_server.js:907
08:21:01     INFO -                       function handleStorage does not always return a value
08:21:01     INFO -  System JS : WARNING /builds/slave/talos-slave/test/build/tests/xpcshell/tests/services/sync/tests/unit/head_http_server.js:938
08:21:01     INFO -                       function handleStorage does not always return a value
08:21:01     INFO -  System JS : WARNING /builds/slave/talos-slave/test/build/tests/xpcshell/tests/services/sync/tests/unit/head_http_server.js:954
08:21:01     INFO -                       function handleStorage does not always return a value
08:21:01     INFO -  System JS : WARNING /builds/slave/talos-slave/test/build/tests/xpcshell/tests/services/sync/tests/unit/head_http_server.js:956
08:21:01     INFO -                       function handleStorage does not always return a value
08:21:01     INFO -  TEST-INFO | (xpcshell/head.js) | test MAIN run_test pending (1)
08:21:01     INFO -  TEST-INFO | (xpcshell/head.js) | test run_next_test 0 pending (2)
08:21:01     INFO -  TEST-INFO | (xpcshell/head.js) | test MAIN run_test finished (2)
08:21:01     INFO -  TEST-INFO | (xpcshell/head.js) | running event loop

blah blah blah...

08:21:01     INFO -  WARNING: NS_ENSURE_TRUE(mThread != PR_GetCurrentThread()) failed: file ../../../xpcom/threads/nsThread.cpp, line 435
08:21:01     INFO -  1378394457404	Sync.AddonsReconciler	DEBUG	Stopping listening and removing AddonManager listeners.
08:21:01     INFO -  WARNING: nsExceptionService ignoring thread destruction after shutdown: file ../../../xpcom/base/nsExceptionService.cpp, line 151
08:21:01     INFO -  1378394457408	Sync.Tracker.Passwords	DEBUG	Saving changed IDs to passwords
08:21:01     INFO -  1378394458403	Sync.Tracker.Prefs	DEBUG	Saving changed IDs to prefs
08:21:01     INFO -  1378394458405	Sync.Tracker.Tabs	DEBUG	Saving changed IDs to tabs
08:21:01     INFO -  1378394458408	Sync.Tracker.Clients	DEBUG	Saving changed IDs to clients
08:21:01     INFO -  <<<<<<<
08:21:15  WARNING -  PROCESS-CRASH | /builds/slave/talos-slave/test/build/tests/xpcshell/tests/services/sync/tests/unit/test_service_login.js | application crashed [@ nsContentUtils::IsCallerChrome()]
08:21:15     INFO -  Crash dump filename: /var/folders/xx/59ykv66n1hj3nt4n04tr9sh400000w/T/tmpKnRtO9/3B021205-3DA9-4C37-9FEC-D1BD08CA08AD.dmp
08:21:15     INFO -  Operating system: Mac OS X
08:21:15     INFO -                    10.8.0 12A269
08:21:15     INFO -  CPU: amd64
08:21:15     INFO -       family 6 model 42 stepping 7
08:21:15     INFO -       8 CPUs
08:21:15     INFO -  Crash reason:  EXC_BAD_ACCESS / KERN_INVALID_ADDRESS
08:21:15     INFO -  Crash address: 0x0
08:21:15     INFO -  Thread 0 (crashed)
08:21:15     INFO -   0  XUL!nsContentUtils::IsCallerChrome() [nsContentUtils.cpp:efe592efe709 : 1751 + 0x7]
08:21:15     INFO -      rbx = 0x0000000106524f50   r12 = 0x00007fff5fbf80c8
08:21:15     INFO -      r13 = 0x0000000000000001   r14 = 0x00007fff5fbf80c8
08:21:15     INFO -      r15 = 0x0000000106528ab0   rip = 0x00000001007e94da
08:21:15     INFO -      rsp = 0x00007fff5fbf7e00   rbp = 0x00007fff5fbf7e10
08:21:15     INFO -      Found by: given as instruction pointer in context
08:21:15     INFO -   1  XUL!mozJSComponentLoader::Import(nsACString_internal const&, JS::Value const&, JSContext*, unsigned char, JS::Value*) [mozJSComponentLoader.cpp:efe592efe709 : 1108 + 0x4]
08:21:15     INFO -      rbx = 0x0000000106524f50   r12 = 0x00007fff5fbf80c8
08:21:15     INFO -      r13 = 0x0000000000000001   r14 = 0x00007fff5fbf80c8
08:21:15     INFO -      r15 = 0x0000000106528ab0   rip = 0x0000000101237215
08:21:15     INFO -      rsp = 0x00007fff5fbf7e20   rbp = 0x00007fff5fbf7f00
08:21:15     INFO -      Found by: call frame info
08:21:15     INFO -   2  XUL!nsXPCComponents_Utils::Import(nsACString_internal const&, JS::Value const&, JSContext*, unsigned char, JS::Value*) [XPCComponents.cpp:efe592efe709 : 2860 + 0x17]
08:21:15     INFO -      rbx = 0x0000000000000000   r12 = 0x00007fff5fbf80c8
08:21:15     INFO -      r13 = 0x0000000000000001   r14 = 0x0000000080004005
08:21:15     INFO -      r15 = 0x0000000106528ab0   rip = 0x00000001011c241c
08:21:15     INFO -      rsp = 0x00007fff5fbf7f10   rbp = 0x00007fff5fbf7f60
08:21:15     INFO -      Found by: call frame info
08:21:15     INFO -   3  XUL!NS_InvokeByIndex [xptcinvoke_x86_64_unix.cpp : 162 + 0x3]
08:21:15     INFO -      rbx = 0x0000000000000004   r12 = 0x00007fff75eb4400
08:21:15     INFO -      r13 = 0x00007fff5fbf8070   r14 = 0x00007fff5fbf7f70
08:21:15     INFO -      r15 = 0x0000000000000003   rip = 0x0000000101d99d1d
08:21:15     INFO -      rsp = 0x00007fff5fbf7f70   rbp = 0x00007fff5fbf8000
08:21:15     INFO -      Found by: call frame info
08:21:15     INFO -   4  XUL!CallMethodHelper::Call() [XPCWrappedNative.cpp:efe592efe709 : 2809 + 0x4]
08:21:15     INFO -      rbx = 0x000000010686fc00   r12 = 0x00007fff75eb4400
08:21:15     INFO -      r13 = 0x00007fff5fbf8070   r14 = 0x0000000106504e03
08:21:15     INFO -      r15 = 0x0000000000000003   rip = 0x0000000101204fa8
08:21:15     INFO -      rsp = 0x00007fff5fbf8010   rbp = 0x00007fff5fbf8040
08:21:15     INFO -      Found by: call frame info
So, we're segfaulting because we're calling into nsContentUtils::IsCallerChrome() at a point where sSecurityManager is null. This stack is below.

My gut reaction was that nsContentUtils must have been shut down, because this is happening during shutdown. But on closer inspection it looks like this happens when we notify xpcom-shutdown-threads, which I would think would happen before shutting down layout. Is this a correct assumption bsmedberg? Also please let me know if there's anyone less busy that I can ask these kinds of xpcom questions to. ;-)


08:21:15     INFO -   0  XUL!nsContentUtils::IsCallerChrome() [nsContentUtils.cpp:efe592efe709 : 1751 + 0x7]
08:21:15     INFO -   1  XUL!mozJSComponentLoader::Import(nsACString_internal const&, JS::Value const&, JSContext*, unsigned char, JS::Value*) [mozJSComponentLoader.cpp:efe592efe709 : 1108 + 0x4]
08:21:15     INFO -   2  XUL!nsXPCComponents_Utils::Import(nsACString_internal const&, JS::Value const&, JSContext*, unsigned char, JS::Value*) [XPCComponents.cpp:efe592efe709 : 2860 + 0x17]
08:21:15     INFO -   3  XUL!NS_InvokeByIndex [xptcinvoke_x86_64_unix.cpp : 162 + 0x3]
08:21:15     INFO -   4  XUL!CallMethodHelper::Call() [XPCWrappedNative.cpp:efe592efe709 : 2809 + 0x4]
08:21:15     INFO -   5  XUL!XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode) [XPCWrappedNative.cpp:efe592efe709 : 2115 + 0x7]
08:21:15     INFO -   6  XUL!XPC_WN_CallMethod(JSContext*, unsigned int, JS::Value*) [XPCWrappedNativeJSOps.cpp:efe592efe709 : 1316 + 0x9]
08:21:15     INFO -   7  XUL!js::CallJSNative(JSContext*, bool (*)(JSContext*, unsigned int, JS::Value*), JS::CallArgs const&) [jscntxtinlines.h:efe592efe709 : 219 + 0x7]
08:21:15     INFO -   8  XUL!js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) [Interpreter.cpp:efe592efe709 : 482 + 0xa]
08:21:15     INFO -   9  XUL!Interpret [Interpreter.cpp:efe592efe709 : 2459 + 0x2c]
08:21:15     INFO -  10  XUL!js::RunScript(JSContext*, js::RunState&) [Interpreter.cpp:efe592efe709 : 446 + 0xa]
08:21:15     INFO -  11  XUL!js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) [Interpreter.cpp:efe592efe709 : 508 + 0x7]
08:21:15     INFO -  12  XUL!js_fun_call(JSContext*, unsigned int, JS::Value*) [jsfun.cpp:efe592efe709 : 873 + 0x2c]
08:21:15     INFO -  13  XUL!js_fun_apply(JSContext*, unsigned int, JS::Value*) [jsfun.cpp:efe592efe709 : 912 + 0x12]
08:21:15     INFO -  14  XUL!js::CallJSNative(JSContext*, bool (*)(JSContext*, unsigned int, JS::Value*), JS::CallArgs const&) [jscntxtinlines.h:efe592efe709 : 219 + 0x7]
08:21:15     INFO -  15  XUL!js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) [Interpreter.cpp:efe592efe709 : 482 + 0xa]
08:21:15     INFO -  16  XUL!Interpret [Interpreter.cpp:efe592efe709 : 2459 + 0x2c]
08:21:15     INFO -  17  XUL!js::RunScript(JSContext*, js::RunState&) [Interpreter.cpp:efe592efe709 : 446 + 0xa]
08:21:15     INFO -  18  XUL!js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) [Interpreter.cpp:efe592efe709 : 508 + 0x7]
08:21:15     INFO -  19  XUL!js::Invoke(JSContext*, JS::Value const&, JS::Value const&, unsigned int, JS::Value*, JS::MutableHandle<JS::Value>) [Interpreter.cpp:efe592efe709 : 539 + 0x2c]
08:21:15     INFO -  20  XUL!js::DirectProxyHandler::call(JSContext*, JS::Handle<JSObject*>, JS::CallArgs const&) [jsproxy.cpp:efe592efe709 : 448 + 0x4]
08:21:15     INFO -  21  XUL!js::CrossCompartmentWrapper::call(JSContext*, JS::Handle<JSObject*>, JS::CallArgs const&) [jswrapper.cpp:efe592efe709 : 454 + 0x12]
08:21:15     INFO -  22  XUL!js::Proxy::call(JSContext*, JS::Handle<JSObject*>, JS::CallArgs const&) [jsproxy.cpp:efe592efe709 : 2611 + 0x15]
08:21:15     INFO -  23  XUL!proxy_Call [jsproxy.cpp:efe592efe709 : 3188 + 0x7]
08:21:15     INFO -  24  XUL!js::CallJSNative(JSContext*, bool (*)(JSContext*, unsigned int, JS::Value*), JS::CallArgs const&) [jscntxtinlines.h:efe592efe709 : 219 + 0x7]
08:21:15     INFO -  25  XUL!js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) [Interpreter.cpp:efe592efe709 : 482 + 0xa]
08:21:15     INFO -  26  XUL!js::Invoke(JSContext*, JS::Value const&, JS::Value const&, unsigned int, JS::Value*, JS::MutableHandle<JS::Value>) [Interpreter.cpp:efe592efe709 : 539 + 0x2c]
08:21:15     INFO -  27  XUL!js::InvokeGetterOrSetter(JSContext*, JSObject*, JS::Value, unsigned int, JS::Value*, JS::MutableHandle<JS::Value>) [Interpreter.cpp:efe592efe709 : 610 + 0x1c]
08:21:15     INFO -  28  XUL!js::Shape::get(JSContext*, JS::Handle<JSObject*>, JSObject*, JSObject*, JS::MutableHandle<JS::Value>) [Shape-inl.h:efe592efe709 : 256 + 0x13]
08:21:15     INFO -  29  XUL!NativeGetInline<1> [jsobj.cpp:efe592efe709 : 4070 + 0xe]
08:21:15     INFO -  30  XUL!js_NativeGet(JSContext*, JS::Handle<JSObject*>, JS::Handle<JSObject*>, JS::Handle<js::Shape*>, unsigned int, JS::MutableHandle<JS::Value>) [jsobj.cpp:efe592efe709 : 4086 + 0xa]
08:21:15     INFO -  31  XUL!js::NativeGet(JSContext*, JSObject*, JSObject*, js::Shape*, unsigned int, JS::MutableHandle<JS::Value>) [Interpreter-inl.h:efe592efe709 : 143 + 0x11]
08:21:15     INFO -  32  XUL!bool js::FetchName<false>(JSContext*, JS::Handle<JSObject*>, JS::Handle<JSObject*>, JS::Handle<js::PropertyName*>, JS::Handle<js::Shape*>, JS::MutableHandle<JS::Value>) [Interpreter-inl.h:efe592efe709 : 207 + 0x18]
08:21:15     INFO -  33  XUL!Interpret [Interpreter.cpp:efe592efe709 : 345 + 0xa]
08:21:15     INFO -  34  XUL!js::RunScript(JSContext*, js::RunState&) [Interpreter.cpp:efe592efe709 : 446 + 0xa]
08:21:15     INFO -  35  XUL!js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) [Interpreter.cpp:efe592efe709 : 508 + 0x7]
08:21:15     INFO -  36  XUL!js::CallOrConstructBoundFunction(JSContext*, unsigned int, JS::Value*) [jsfun.cpp:efe592efe709 : 1254 + 0x2c]
08:21:15     INFO -  37  XUL!js::CallJSNative(JSContext*, bool (*)(JSContext*, unsigned int, JS::Value*), JS::CallArgs const&) [jscntxtinlines.h:efe592efe709 : 219 + 0x7]
08:21:15     INFO -  38  XUL!js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) [Interpreter.cpp:efe592efe709 : 482 + 0xa]
08:21:15     INFO -  39  XUL!js_fun_apply(JSContext*, unsigned int, JS::Value*) [jsfun.cpp:efe592efe709 : 1026 + 0x2c]
08:21:15     INFO -  40  XUL!js::CallJSNative(JSContext*, bool (*)(JSContext*, unsigned int, JS::Value*), JS::CallArgs const&) [jscntxtinlines.h:efe592efe709 : 219 + 0x7]
08:21:15     INFO -  41  XUL!js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) [Interpreter.cpp:efe592efe709 : 482 + 0xa]
08:21:15     INFO -  42  XUL!js::Invoke(JSContext*, JS::Value const&, JS::Value const&, unsigned int, JS::Value*, JS::MutableHandle<JS::Value>) [Interpreter.cpp:efe592efe709 : 539 + 0x2c]
08:21:15     INFO -  43  XUL!js::DirectProxyHandler::call(JSContext*, JS::Handle<JSObject*>, JS::CallArgs const&) [jsproxy.cpp:efe592efe709 : 448 + 0x4]
08:21:15     INFO -  44  XUL!js::CrossCompartmentWrapper::call(JSContext*, JS::Handle<JSObject*>, JS::CallArgs const&) [jswrapper.cpp:efe592efe709 : 454 + 0x12]
08:21:15     INFO -  45  XUL!js::Proxy::call(JSContext*, JS::Handle<JSObject*>, JS::CallArgs const&) [jsproxy.cpp:efe592efe709 : 2611 + 0x15]
08:21:15     INFO -  46  XUL!proxy_Call [jsproxy.cpp:efe592efe709 : 3188 + 0x7]
08:21:15     INFO -  47  XUL!js::CallJSNative(JSContext*, bool (*)(JSContext*, unsigned int, JS::Value*), JS::CallArgs const&) [jscntxtinlines.h:efe592efe709 : 219 + 0x7]
08:21:15     INFO -  48  XUL!js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) [Interpreter.cpp:efe592efe709 : 482 + 0xa]
08:21:15     INFO -  49  XUL!Interpret [Interpreter.cpp:efe592efe709 : 2459 + 0x2c]
08:21:15     INFO -  50  XUL!js::RunScript(JSContext*, js::RunState&) [Interpreter.cpp:efe592efe709 : 446 + 0xa]
08:21:15     INFO -  51  XUL!js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) [Interpreter.cpp:efe592efe709 : 508 + 0x7]
08:21:15     INFO -  52  XUL!js::Invoke(JSContext*, JS::Value const&, JS::Value const&, unsigned int, JS::Value*, JS::MutableHandle<JS::Value>) [Interpreter.cpp:efe592efe709 : 539 + 0x2c]
08:21:15     INFO -  53  XUL!JS_CallFunctionValue(JSContext*, JSObject*, JS::Value, unsigned int, JS::Value*, JS::Value*) [jsapi.cpp:efe592efe709 : 5437 + 0x29]
08:21:15     INFO -  54  XUL!nsXPCWrappedJSClass::CallMethod(nsXPCWrappedJS*, unsigned short, XPTMethodDescriptor const*, nsXPTCMiniVariant*) [XPCWrappedJSClass.cpp:efe592efe709 : 1445 + 0x26]
08:21:15     INFO -  55  XUL!nsXPCWrappedJS::CallMethod(unsigned short, XPTMethodDescriptor const*, nsXPTCMiniVariant*) [XPCWrappedJS.cpp:efe592efe709 : 587 + 0x12]
08:21:15     INFO -  56  XUL!PrepareAndDispatch [xptcstubs_x86_64_darwin.cpp:efe592efe709 : 122 + 0x20]
08:21:15     INFO -  57  XUL!SharedStub + 0x5a
08:21:15     INFO -  58  XUL!nsTimerImpl::Fire() [nsTimerImpl.cpp:efe592efe709 : 549 + 0xb]
08:21:15     INFO -  59  XUL!nsTimerEvent::Run() [nsTimerImpl.cpp:efe592efe709 : 630 + 0x4]
08:21:15     INFO -  60  XUL!nsThread::ProcessNextEvent(bool, bool*) [nsThread.cpp:efe592efe709 : 622 + 0x5]
08:21:15     INFO -  61  XUL!NS_ProcessNextEvent(nsIThread*, bool) [nsThreadUtils.cpp:efe592efe709 : 238 + 0xc]
08:21:15     INFO -  62  XUL!nsThread::Shutdown() [nsThread.cpp:efe592efe709 : 463 + 0xd]
08:21:15     INFO -  63  XUL!nsThreadPool::Shutdown() [nsThreadPool.cpp:efe592efe709 : 316 + 0x5]
08:21:15     INFO -  64  XUL!_ZThn16_N24nsStreamTransportService7ObserveEP11nsISupportsPKcPKt [nsStreamTransportService.cpp:efe592efe709 : 513 + 0x5]
08:21:15     INFO -  65  XUL!nsObserverList::NotifyObservers(nsISupports*, char const*, unsigned short const*) [nsObserverList.cpp:efe592efe709 : 96 + 0xe]
08:21:15     INFO -  66  XUL!nsObserverService::NotifyObservers(nsISupports*, char const*, unsigned short const*) [nsObserverService.cpp:efe592efe709 : 334 + 0x10]
08:21:15     INFO -  67  XUL!mozilla::ShutdownXPCOM(nsIServiceManager*) [nsXPComInit.cpp:efe592efe709 : 668 + 0xf]
Flags: needinfo?(benjamin)
I'm sorry, I totally missed this NEEDINFO earlier. It should be impossible that  LayoutModuleDtor has been called. *However* sSecurityManager is not managed by the layout module; it is instead managed by nsLayoutStatics. That is cleared by Shutdown() in nsLayoutModule.cpp, which is called from an xpcom-shutdown observer. That seems unfortunate; it should be later, perhaps a lot later.

The existing situation dates back to bug 209804, see the comment at http://hg.mozilla.org/mozilla-central/annotate/59beb1868522/layout/build/nsLayoutModule.cpp#l352

I don't think that reasoning is valid any more, now that everything is in libxul and we don't unload component libraries.

But of course this is giant live wire, because shutdown is a pile of hacks, and reordering layout could have unknown cascading effects. So we have to tread carefully.
Flags: needinfo?(benjamin)
Moving nsLayoutStatics shutdown to the layout module dtor seems to pass my local smoketesting. Let's see what linux64 try has to say.

https://tbpl.mozilla.org/?tree=Try&rev=61f63b201ef7
Looks good on linux. Expanding to all platforms.

https://tbpl.mozilla.org/?tree=Try&rev=8f08607860b0
Attachment #807360 - Flags: review?(benjamin) → review+
(In reply to Bobby Holley (:bholley) from comment #4)
> Looks good on linux. Expanding to all platforms.
> 
> https://tbpl.mozilla.org/?tree=Try&rev=8f08607860b0

So this is totally green except for windows debug, which is perma-orange, probably due to a botched assertion. But for whatever reason, TBPL isn't giving crash dumps.

I'm guessing it's something on the simple side, and this patch is probably well worth-it to get landed, and would probably solve a large class of crashes. bsmedberg, can you or someone on your team with a windows build setup take a quick look and see what's going on?
Flags: needinfo?(benjamin)
The crashtest log says:

15:33:22     INFO -  Assertion failed at e:/builds/moz2_slave/try-w32-d-00000000000000000000/build/gfx/cairo/cairo/src/cairo-hash.c:196: hash_table->live_entries == 0
(probably not fatal, perhaps not interesting)

15:38:52     INFO -  Hit MOZ_CRASH() at e:/builds/moz2_slave/try-w32-d-00000000000000000000/build/xpcom/build/mozPoisonWriteBase.cpp:172
(fatal, interesting)

Which is http://hg.mozilla.org/mozilla-central/annotate/8f8a683dfc42/xpcom/build/mozPoisonWriteBase.cpp#l171

Without a stack, not so useful though. I don't think I have time to follow up myself.
Flags: needinfo?(benjamin)
Is there anyone on the stability team who could spend an hour with this?
Flags: needinfo?(benjamin)
I might be able on Tuesday. dmajor is working on other things.
Flags: needinfo?(benjamin)
Blocks: 910435
Flags: needinfo?(benjamin)
I hit a crash here, trying to enter a deleted critical section:

00 ntdll!RtlpWaitOnCriticalSection
01 ntdll!RtlEnterCriticalSection
02 gkmedias!_cairo_atomic_int_dec_and_test
03 gkmedias!_moz_cairo_surface_destroy
04 xul!gfxASurface::Release
05 xul!nsAutoPtr<imgFrame>::~nsAutoPtr<imgFrame>
06 xul!nsTArray_Impl<mozilla::image::FrameDataPair,nsTArrayInfallibleAllocator>::RemoveElementsAt
07 xul!mozilla::image::FrameSequence::ClearFrames
08 xul!mozilla::image::FrameSequence::~FrameSequence
09 xul!mozilla::image::FrameSequence::Release
0a xul!mozilla::image::FrameBlender::~FrameBlender
0b xul!mozilla::image::RasterImage::~RasterImage
0c xul!mozilla::image::RasterImage::`scalar deleting destructor'
0d xul!mozilla::image::RasterImage::Release
0e xul!imgRequest::~imgRequest
0f xul!imgRequest::`scalar deleting destructor'
10 xul!imgRequest::Release
11 xul!imgCacheEntry::~imgCacheEntry
12 xul!imgCacheEntry::Release
13 xul!nsRefPtr<imgCacheEntry>::`scalar deleting destructor'
14 xul!nsTArray_Impl<nsRefPtr<imgCacheEntry>,nsTArrayInfallibleAllocator>::DestructRange
15 xul!nsTArray_Impl<nsRefPtr<imgCacheEntry>,nsTArrayInfallibleAllocator>::RemoveElementsAt
16 xul!imgLoader::EvictEntries
17 xul!imgLoader::~imgLoader
18 xul!imgLoader::`scalar deleting destructor'
19 xul!imgLoader::Release
1a xul!nsContentUtils::Shutdown
1b xul!nsLayoutStatics::Shutdown
1c xul!LayoutModuleDtor
1d xul!nsComponentManagerImpl::KnownModule::~KnownModule
1e xul!nsAutoPtr<nsComponentManagerImpl::KnownModule>::~nsAutoPtr<nsComponentManagerImpl::KnownModule>
1f xul!nsTArray_Impl<nsAutoPtr<nsComponentManagerImpl::KnownModule>,nsTArrayInfallibleAllocator>::RemoveElementsAt
20 xul!nsComponentManagerImpl::Shutdown
21 xul!mozilla::ShutdownXPCOM
22 xul!ScopedXPCOMStartup::~ScopedXPCOMStartup
23 xul!XREMain::XRE_main
24 xul!XRE_main
25 firefox!do_main
26 firefox!NS_internal_main
27 firefox!wmain
28 firefox!__tmainCRTStartup
29 firefox!wmainCRTStartup
2a kernel32!BaseThreadInitThunk
2b ntdll!__RtlUserThreadStart
2c ntdll!_RtlUserThreadStart

_cairo_atomic_mutex was deleted here:

00 ntdll!RtlDeleteCriticalSection
01 gkmedias!_cairo_mutex_finalize
02 xul!gfxPlatform::~gfxPlatform
03 xul!gfxWindowsPlatform::`scalar deleting destructor'
04 xul!gfxPlatform::Shutdown
05 xul!nsComponentManagerImpl::KnownModule::~KnownModule
06 xul!nsAutoPtr<nsComponentManagerImpl::KnownModule>::~nsAutoPtr<nsComponentManagerImpl::KnownModule>
07 xul!nsTArray_Impl<nsAutoPtr<nsComponentManagerImpl::KnownModule>,nsTArrayInfallibleAllocator>::RemoveElementsAt
08 xul!nsComponentManagerImpl::Shutdown
09 xul!mozilla::ShutdownXPCOM
0a xul!ScopedXPCOMStartup::~ScopedXPCOMStartup
0b xul!XREMain::XRE_main
0c xul!XRE_main
0d firefox!do_main
0e firefox!NS_internal_main
0f firefox!wmain
10 firefox!__tmainCRTStartup
11 firefox!wmainCRTStartup
12 kernel32!BaseThreadInitThunk
13 ntdll!__RtlUserThreadStart
14 ntdll!_RtlUserThreadStart
Ok, that seems simple enough. We end up shutting down gfx first, so gfxPlatform::Shutdown goes along and destroys cairo. Then, we release the image cache from the Layout dtor, which ends up calling back into cairo. Boom.

It seems like we should move the management of imgLoader singletons out of nsContentUtils. Taking a crack at that:

https://tbpl.mozilla.org/?tree=Try&rev=adec0c72b792
Flags: needinfo?(benjamin)
Ok, so nsLayoutStylesheetCache::Shutdown() basically runs into the same problem (calling into cairo, which has been shut down).

Is there a way for us to force the gfx module to be destroyed after layout?
Flags: needinfo?(benjamin)
I can't think of a simple way to do that. IIRC modules are destroyed in random (hash enumerator) order. You'd have to add some specific code to those two modules.
Flags: needinfo?(benjamin)
These last two were real bustage on an uplift, not this bug.
For reference, here are the patches I was working on: https://github.com/bholley/mozilla-central/commits/layout_shutdown3
Blocks: 976181
Assignee: nobody → bobbyholley
This looks green. Uploading patches.

I'm dividing these into logical parts, but note that, due to craziness that is shutdown, only the final revision is expected to build without crashing (contrary to my normal incremental patch policy).
Comment on attachment 807360 [details] [diff] [review]
Part 1 - Release nsLayoutStatics when the layout module is unloaded. v1

Also note that I opted to build on top of this patch (which moves layout destruction into the layout module destructor), and just explicitly do the shutdown for the modules Layout depends on in the layout module dtor.
Attachment #807360 - Attachment description: Release nsLayoutStatics when the layout module is unloaded. v1 → Part 1 - Release nsLayoutStatics when the layout module is unloaded. v1
Due to the size of the patch, it would be nice to have this in beta6 (next Monday)  or 7 (next Thursday) to have enough time to test it.
Attachment #8397874 - Flags: review?(benjamin) → review+
Attachment #8397875 - Flags: review?(benjamin) → review+
Comment on attachment 8397876 [details] [diff] [review]
Part 4 - Shut down imagelib at the end of layout shutdown. v1

I'm really sorry it took this long to get to these.
Attachment #8397876 - Flags: review?(benjamin) → review+
Comment on attachment 8397876 [details] [diff] [review]
Part 4 - Shut down imagelib at the end of layout shutdown. v1

I'm flagging this for uplift, even though we might want to wait a day or two to see what the impact here is. See bug 976181.

[Approval Request Comment]
Bug caused by (feature/regressing bug #): Longstanding, but recently got worse (bug 976181).
User impact if declined: Crashes 
Testing completed (on m-c, etc.): Just landed to m-c
Risk to taking this patch (and alternatives if risky): Moderate risk. We're rejiggering the module code, which could have unintended consequences.
String or IDL/UUID changes made by this patch: None
Attachment #8397876 - Flags: approval-mozilla-beta?
Attachment #8397876 - Flags: approval-mozilla-aurora?
Blocks: 993918
Comment on attachment 8397876 [details] [diff] [review]
Part 4 - Shut down imagelib at the end of layout shutdown. v1

Bobby, actually, except if you said moderate to make me happy (instead of super high risk ;) ), I am going to accept it right now:
Aurora will have it tomorrow (Thursday) and beta7 (GTB tomorrow) will have it (instead of waiting for next Tuesday for beta8).
It will give us more information and more time to backout if it does not improve the situation.
If you disagree, don't hesitate to let me know!
Attachment #8397876 - Flags: approval-mozilla-beta?
Attachment #8397876 - Flags: approval-mozilla-beta+
Attachment #8397876 - Flags: approval-mozilla-aurora?
Attachment #8397876 - Flags: approval-mozilla-aurora+
Depends on: 994740
No longer blocks: 993918
Depends on: 993918
Duplicate of this bug: 976181
The build ID facet of https://crash-stats.mozilla.com/search/?signature=~IsCallerChrome&version=30.0a2&date=%3E2014-04-01&_facets=build_id&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform says this ceased after the 2014-04-09 on 30.0a2 - on 31.0a1 is apparently stopped with the 2014-03-24 build.
And the signature summary of https://crash-stats.mozilla.com/report/list?signature=nsContentUtils::IsCallerChrome() says it's not happening on any 29 beta after 29.0b6, so I call it verified on all channels.
Status: RESOLVED → VERIFIED
Crash Signature: [@ nsContentUtils::IsCallerChrome()]
You need to log in before you can comment on or make changes to this bug.