1271657 - Various Windows debug e10s test suites hitting "Assertion failure: IsInitialized(), at gfxFeature.h:80" and crashing [@ mozilla::gfx::FeatureState::AssertInitialized()] on Ash

Status: RESOLVED FIXED

(Core :: Graphics, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

Bug 1269387 seems like the most likely candidate. Affects multiple WinXP e10s runs and also reftest-noaccel-e10s on Win7/Win8.

https://treeherder.mozilla.org/logviewer.html#?job_id=176552&repo=ash#L62429

07:25:05     INFO -  REFTEST TEST-START | file:///C:/slave/test/build/tests/reftest/tests/dom/canvas/test/reftest/webgl-clear-test.html
07:25:05     INFO -  REFTEST INFO | RESTORE PREFERENCE pref(webgl.disabled,false)
07:25:05     INFO -  REFTEST INFO | RESTORE PREFERENCE pref(media.useAudioChannelAPI,false)
07:25:05     INFO -  REFTEST INFO | RESTORE PREFERENCE pref(webgl.force-enabled,false)
07:25:05     INFO -  REFTEST INFO | SET PREFERENCE pref(webgl.force-enabled,true)
07:25:05     INFO -  REFTEST INFO | SET PREFERENCE pref(media.useAudioChannelAPI,true)
07:25:05     INFO -  REFTEST TEST-LOAD | file:///C:/slave/test/build/tests/reftest/tests/dom/canvas/test/reftest/webgl-clear-test.html | 4912 / 13554 (36%)
07:25:05     INFO -  ++DOMWINDOW == 218 (10564C00) [pid = 3380] [serial = 15967] [outer = 0C31C000]
07:25:05     INFO -  Assertion failure: IsInitialized(), at c:\builds\moz2_slave\ash-w32-d-00000000000000000000\build\src\gfx\config\gfxFeature.h:80
07:25:05     INFO -  #01: mozilla::gl::GLLibraryEGL::EnsureInitialized(bool) [gfx/gl/GLLibraryEGL.cpp:416]
07:25:05     INFO -  #02: mozilla::gl::GLContextProviderEGL::CreateOffscreen(mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const &,mozilla::gl::SurfaceCaps const &,mozilla::gl::CreateContextFlags) [gfx/gl/GLContextProviderEGL.cpp:988]
07:25:05     INFO -  #03: mozilla::CreateGLWithANGLE [dom/canvas/WebGLContext.cpp:602]
07:25:05     INFO -  #04: mozilla::WebGLContext::CreateAndInitGLWith(already_AddRefed<mozilla::gl::GLContext> (*)(mozilla::gl::SurfaceCaps const &,mozilla::gl::CreateContextFlags,mozilla::WebGLContext *),mozilla::gl::SurfaceCaps const &,mozilla::gl::CreateContextFlags) [dom/canvas/WebGLContext.cpp:661]
07:25:05     INFO -  #05: mozilla::WebGLContext::CreateAndInitGL(bool) [dom/canvas/WebGLContext.cpp:703]
07:25:05     INFO -  #06: mozilla::WebGLContext::SetDimensions(int,int) [dom/canvas/WebGLContext.cpp:863]
07:25:05     INFO -  #07: mozilla::dom::CanvasRenderingContextHelper::UpdateContext(JSContext *,JS::Handle<JS::Value>,mozilla::ErrorResult &) [dom/canvas/CanvasRenderingContextHelper.cpp:246]
07:25:05     INFO -  #08: mozilla::dom::CanvasRenderingContextHelper::GetContext(JSContext *,nsAString_internal const &,JS::Handle<JS::Value>,mozilla::ErrorResult &) [dom/canvas/CanvasRenderingContextHelper.cpp:197]
07:25:05     INFO -  #09: mozilla::dom::HTMLCanvasElement::GetContext(JSContext *,nsAString_internal const &,JS::Handle<JS::Value>,mozilla::ErrorResult &) [dom/html/HTMLCanvasElement.cpp:898]
07:25:05     INFO -  #10: mozilla::dom::HTMLCanvasElementBinding::getContext [obj-firefox/dom/bindings/HTMLCanvasElementBinding.cpp:228]
07:25:05     INFO -  #11: mozilla::dom::GenericBindingMethod(JSContext *,unsigned int,JS::Value *) [dom/bindings/BindingUtils.cpp:2781]
07:25:05     INFO -  #12: js::CallJSNative(JSContext *,bool (*)(JSContext *,unsigned int,JS::Value *),JS::CallArgs const &) [js/src/jscntxtinlines.h:235]
07:25:05     INFO -  #13: js::InternalCallOrConstruct(JSContext *,JS::CallArgs const &,js::MaybeConstruct) [js/src/vm/Interpreter.cpp:480]
07:25:05     INFO -  #14: InternalCall [js/src/vm/Interpreter.cpp:525]
07:25:05     INFO -  #15: Interpret [js/src/vm/Interpreter.cpp:2831]
07:25:05     INFO -  #16: js::RunScript(JSContext *,js::RunState &) [js/src/vm/Interpreter.cpp:426]
07:25:05     INFO -  #17: js::InternalCallOrConstruct(JSContext *,JS::CallArgs const &,js::MaybeConstruct) [js/src/vm/Interpreter.cpp:498]
07:25:05     INFO -  #18: InternalCall [js/src/vm/Interpreter.cpp:525]
07:25:05     INFO -  #19: js::Call(JSContext *,JS::Handle<JS::Value>,JS::Handle<JS::Value>,js::AnyInvokeArgs const &,JS::MutableHandle<JS::Value>) [js/src/vm/Interpreter.cpp:544]
07:25:05     INFO -  #20: JS::Call(JSContext *,JS::Handle<JS::Value>,JS::Handle<JS::Value>,JS::HandleValueArray const &,JS::MutableHandle<JS::Value>) [js/src/jsapi.cpp:2926]
07:25:05     INFO -  #21: mozilla::dom::EventHandlerNonNull::Call(JSContext *,JS::Handle<JS::Value>,mozilla::dom::Event &,JS::MutableHandle<JS::Value>,mozilla::ErrorResult &) [obj-firefox/dom/bindings/EventHandlerBinding.cpp:259]
07:25:05     INFO -  #22: mozilla::dom::EventHandlerNonNull::Call<nsISupports *>(nsISupports * const &,mozilla::dom::Event &,JS::MutableHandle<JS::Value>,mozilla::ErrorResult &,char const *,mozilla::dom::CallbackObject::ExceptionHandling,JSCompartment *) [obj-firefox/dist/include/mozilla/dom/EventHandlerBinding.h:356]
07:25:05     INFO -  #23: mozilla::JSEventHandler::HandleEvent(nsIDOMEvent *) [dom/events/JSEventHandler.cpp:215]
07:25:05     INFO -  #24: mozilla::EventListenerManager::HandleEventSubType(mozilla::EventListenerManager::Listener *,nsIDOMEvent *,mozilla::dom::EventTarget *) [dom/events/EventListenerManager.cpp:1111]
07:25:05     INFO -  #25: mozilla::EventListenerManager::HandleEventInternal(nsPresContext *,mozilla::WidgetEvent *,nsIDOMEvent * *,mozilla::dom::EventTarget *,nsEventStatus *) [dom/events/EventListenerManager.cpp:1282]
07:25:05     INFO -  #26: mozilla::EventListenerManager::HandleEvent(nsPresContext *,mozilla::WidgetEvent *,nsIDOMEvent * *,mozilla::dom::EventTarget *,nsEventStatus *) [obj-firefox/dist/include/mozilla/EventListenerManager.h:350]
07:25:05     INFO -  #27: mozilla::EventTargetChainItem::HandleEvent(mozilla::EventChainPostVisitor &,mozilla::ELMCreationDetector &) [dom/events/EventDispatcher.cpp:278]
07:25:05     INFO -  #28: mozilla::EventTargetChainItem::HandleEventTargetChain(nsTArray<mozilla::EventTargetChainItem> &,mozilla::EventChainPostVisitor &,mozilla::EventDispatchingCallback *,mozilla::ELMCreationDetector &) [dom/events/EventDispatcher.cpp:381]
07:25:05     INFO -  #29: mozilla::EventDispatcher::Dispatch(nsISupports *,nsPresContext *,mozilla::WidgetEvent *,nsIDOMEvent *,nsEventStatus *,mozilla::EventDispatchingCallback *,nsTArray<mozilla::dom::EventTarget *> *) [dom/events/EventDispatcher.cpp:713]
07:25:05     INFO -  #30: nsDocumentViewer::LoadComplete(nsresult) [layout/base/nsDocumentViewer.cpp:997]
07:25:05     INFO -  #31: nsDocShell::EndPageLoad(nsIWebProgress *,nsIChannel *,nsresult) [docshell/base/nsDocShell.cpp:7525]
07:25:05     INFO -  #32: nsDocShell::OnStateChange(nsIWebProgress *,nsIRequest *,unsigned int,nsresult) [docshell/base/nsDocShell.cpp:7332]
07:25:05     INFO -  #33: nsDocLoader::DoFireOnStateChange(nsIWebProgress * const,nsIRequest * const,int &,nsresult) [uriloader/base/nsDocLoader.cpp:1250]
07:25:05     INFO -  #34: nsDocLoader::doStopDocumentLoad(nsIRequest *,nsresult) [uriloader/base/nsDocLoader.cpp:834]
07:25:05     INFO -  #35: nsDocLoader::DocLoaderIsEmpty(bool) [uriloader/base/nsDocLoader.cpp:726]
07:25:05     INFO -  #36: nsDocLoader::OnStopRequest(nsIRequest *,nsISupports *,nsresult) [uriloader/base/nsDocLoader.cpp:609]
07:25:05     INFO -  #37: nsLoadGroup::RemoveRequest(nsIRequest *,nsISupports *,nsresult) [netwerk/base/nsLoadGroup.cpp:633]
07:25:05     INFO -  #38: nsDocument::DoUnblockOnload() [dom/base/nsDocument.cpp:9202]
07:25:05     INFO -  #39: nsDocument::UnblockOnload(bool) [dom/base/nsDocument.cpp:9129]
07:25:05     INFO -  #40: nsDocument::DispatchContentLoadedEvents() [dom/base/nsDocument.cpp:5271]
07:25:05     INFO -  #41: nsRunnableMethodImpl<void ( nsDocument::*)(void),1,0>::Run() [obj-firefox/dist/include/nsThreadUtils.h:743]
07:25:05     INFO -  #42: nsThread::ProcessNextEvent(bool,bool *) [xpcom/threads/nsThread.cpp:991]
07:25:05     INFO -  #43: NS_ProcessNextEvent(nsIThread *,bool) [xpcom/glue/nsThreadUtils.cpp:290]
07:25:05     INFO -  #44: mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate *) [ipc/glue/MessagePump.cpp:98]
07:25:05     INFO -  #45: mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate *) [ipc/glue/MessagePump.cpp:303]
07:25:05     INFO -  #46: MessageLoop::RunInternal() [ipc/chromium/src/base/message_loop.cc:233]
07:25:05     INFO -  #47: MessageLoop::RunHandler() [ipc/chromium/src/base/message_loop.cc:227]
07:25:05     INFO -  #48: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:207]
07:25:05     INFO -  #49: nsBaseAppShell::Run() [widget/nsBaseAppShell.cpp:158]
07:25:05     INFO -  #50: nsAppShell::Run() [widget/windows/nsAppShell.cpp:264]
07:25:05     INFO -  #51: XRE_RunAppShell [toolkit/xre/nsEmbedFunctions.cpp:802]
07:25:05     INFO -  #52: mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate *) [ipc/glue/MessagePump.cpp:271]
07:25:05     INFO -  #53: MessageLoop::RunInternal() [ipc/chromium/src/base/message_loop.cc:233]
07:25:05     INFO -  #54: MessageLoop::RunHandler() [ipc/chromium/src/base/message_loop.cc:227]
07:25:05     INFO -  #55: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:207]
07:25:05     INFO -  #56: XRE_InitChildProcess [toolkit/xre/nsEmbedFunctions.cpp:641]
07:25:05     INFO -  #57: content_process_main(int,char * * const) [ipc/contentproc/plugin-container.cpp:238]
07:25:05     INFO -  #58: wmain [toolkit/xre/nsWindowsWMain.cpp:138]
07:25:05     INFO -  #59: __scrt_common_main_seh [f:/dd/vctools/crt/vcstartup/src/startup/exe_common.inl:255]
07:25:05     INFO -  #60: kernel32.dll + 0x53c45
07:25:05     INFO -  #61: ntdll.dll + 0x637f5
07:25:05     INFO -  #62: ntdll.dll + 0x637c8

Comment 1

[Jeff Muizelaar [:jrmuizel]]

Attachment: Ensure that the D3D11_ANGLE feature is initialized

I expect the early returns caused this. We don't ever want to be bitten by that so let's just move it out of InitializeDevices completely.

Comment 2

[David Anderson [:dvander] - inactive, e-mail if emergency]

Comment on attachment 8750815 [details] [diff] [review]
Ensure that the D3D11_ANGLE feature is initialized

It looks like this would make it only run after a device reset. Maybe the thing to do is add an InitializeWebGLConfig function and make sure it gets called anytime InitializeDevices is called. It could assert that the D3D11 status has been initialized.

Comment 3

[Jean-Yves Avenard [:jya]]

I hit the issue in 100% of the media try run here:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5362caa2f4b1a8af849a495c458126a46af0eec1

Using RyanVM change to force e10s on tests:
https://hg.mozilla.org/try/rev/74ce462a6070ce62b38011400c86d5cbc21df9ad

Comment 4

[Jeff Muizelaar [:jrmuizel]]

Attachment: Redo gfxConfig support for HW accel d3d11

This changes the D3D11_ANGLE gfxConfig code a bunch. The goal here is to avoid hitting assertions from using uninitiated values. As part of this I modified a bunch of FeatureState functions to be no-ops in the case of uninitialization. This simplifies code that handles run-time prefs because it does not need to check if the feature has been initialized. I'm not certain this is the right approach so would welcome feedback.

Comment 5

[Intermittent Failures Robot]

70 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* ash: 67
* try: 3

Platform breakdown:
* windowsxp: 58
* windows8-64: 6
* windows7-32: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1271657&startday=2016-05-10&endday=2016-05-16&tree=all

Comment 6

[Intermittent Failures Robot]

22 automation job failures were associated with this bug yesterday.

Repository breakdown:
* ash: 22

Platform breakdown:
* windowsxp: 18
* windows8-64: 2
* windows7-32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1271657&startday=2016-05-17&endday=2016-05-17&tree=all

Comment 7

[Intermittent Failures Robot]

22 automation job failures were associated with this bug yesterday.

Repository breakdown:
* ash: 22

Platform breakdown:
* windowsxp: 19
* windows7-32: 2
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1271657&startday=2016-05-18&endday=2016-05-18&tree=all

Comment 8

[Jeff Muizelaar [:jrmuizel]]

Attachment: Redo gfxConfig support for HW accel d3d11

This redoes things again. Changes of note:
 - should always initialize (keeps the assertions in)
 - reenables the blocklist checking that was unintentionally disabled previously

Comment 9

[Jeff Muizelaar [:jrmuizel]]

Attachment: Redo gfxConfig support for HW accel d3d11

Fix a compile issue

Comment 10

[David Anderson [:dvander] - inactive, e-mail if emergency]

Comment on attachment 8755017 [details] [diff] [review]
Redo gfxConfig support for HW accel d3d11

Review of attachment 8755017 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks!

Comment 11

[Intermittent Failures Robot]

78 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* ash: 78

Platform breakdown:
* windowsxp: 65
* windows7-32: 7
* windows8-64: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1271657&startday=2016-05-16&endday=2016-05-22&tree=all

Comment 12

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/727ca0c6f57d

Comment 13

[Intermittent Failures Robot]

23 automation job failures were associated with this bug yesterday.

Repository breakdown:
* ash: 23

Platform breakdown:
* windowsxp: 19
* windows8-64: 2
* windows7-32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1271657&startday=2016-05-24&endday=2016-05-24&tree=all

Comment 14

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/727ca0c6f57d

Comment 15

[Intermittent Failures Robot]

23 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* ash: 23

Platform breakdown:
* windowsxp: 19
* windows8-64: 2
* windows7-32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1271657&startday=2016-05-23&endday=2016-05-29&tree=all