Closed Bug 1386263 Opened 7 years ago Closed 7 years ago

Perma failure when 56 merges to beta application crashed [@ nsLayoutUtils::SurfaceFromElement(mozilla::dom::HTMLVideoElement *,unsigned int,RefPtr<mozilla::gfx::DrawTarget> &)] for gl and mda on Windows 8

Categories

(Core :: Graphics: CanvasWebGL, defect, P1)

defect

Tracking

()

RESOLVED FIXED
mozilla56
Tracking Status
firefox56 blocking fixed
firefox57 --- fixed

People

(Reporter: aryx, Assigned: jgilbert)

References

Details

(Keywords: intermittent-failure, Whiteboard: gfx-noted)

Attachments

(1 file)

Not sure if this bug is a duplicate.

Merge is tomorrow (August 2nd)

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=try&revision=6f5695dcff2eaf89a798caa4ba14b5f330b32aa8&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable

The Windows 8 x64 gl and mda suites fail.

gl1 + mda with dom/canvas/test/webgl-conf/generated/test_2_conformance2__textures__video__tex-2d-r11f_g11f_b10f-rgb-float.html | application crashed [@ nsLayoutUtils::SurfaceFromElement(mozilla::dom::HTMLVideoElement *,unsigned int,RefPtr<mozilla::gfx::DrawTarget> &)]

gl2 + gl3 with dom/canvas/test/webgl-conf/generated/test_2_conformance__textures__misc__texture-upload-size.html | getError expected: NO_ERROR. Was OUT_OF_MEMORY : when calling texImage2D

Please file new bug if these are two different issues.
Flags: needinfo?(jgilbert)
Confirmed that these are very-much hitting Beta now since yesterday's merge and are now one of a few bugs keeping the tree closed.
Severity: normal → blocker
Flags: needinfo?(milan)
Priority: -- → P1
Peter, while we're waiting for Jeff to come back from siggraph, can somebody on your team take a look at this?
Flags: needinfo?(milan) → needinfo?(howareyou322)
Flags: needinfo?(milan)
Depends on: 1387242
Nothing obvious to crash at the indicated line:
https://hg.mozilla.org/try/file/6f5695dcff2eaf89a798caa4ba14b5f330b32aa8/layout/base/nsLayoutUtils.cpp#l7504
>  result.mLayersImage = aElement->GetCurrentImage();
>  if (!result.mLayersImage)
>    return result;
> 
>  if (aTarget) {
>    // They gave us a DrawTarget to optimize for, so even though we have a layers::Image,
>    // we should unconditionally grab a SourceSurface and try to optimize it.
>    result.mSourceSurface = result.mLayersImage->GetAsSourceSurface();
>    if (!result.mSourceSurface)
>      return result;
> 
>    RefPtr<SourceSurface> opt = aTarget->OptimizeSourceSurface(result.mSourceSurface); // <<< L7504 (the claimed crash line)
>    if (opt) {
>      result.mSourceSurface = opt;
>    }
>  }

`aTarget`, `result.mLayersImage`, and `result.mSourceSurface` should all be non-null at this point. This is an opt build, but still.

One weird thing here is that it really shouldn't be assigning to result.mSourceSurface directly, but this doesn't seem acutely dangerous here.
Signs point to `aTarget` being null by then.
It may be being passed by reference, and so may be being mutated as a side-effect. If so, it's in the OptimizeSourceSurface call, which is virtual. It looks like there may be a layer of caching somewhere here, so that may be the issue.
Flags: needinfo?(jgilbert)
Assignee: nobody → jgilbert
Whiteboard: gfx-noted
Bad news: We're re-entering the event loop, in the middle of executing a runnable, causing race-condition style side-effects:
> xul.dll!RefPtr<mozilla::gfx::DrawTarget>::swap(mozilla::gfx::DrawTarget * & aRhs) Line 251	C++
> xul.dll!RefPtr<mozilla::gfx::DrawTarget>::forget() Line 261	C++
> xul.dll!mozilla::dom::CanvasRenderingContext2D::ReturnTarget(bool aForceReset) Line 2001	C++
> xul.dll!mozilla::dom::CanvasRenderingContext2D::OnStableState() Line 1590	C++
> xul.dll!mozilla::detail::RunnableMethodArguments<>::applyImpl<mozilla::dom::CanvasRenderingContext2D,void (__cdecl mozilla::dom::CanvasRenderingContext2D::*)(void) __ptr64>(mozilla::dom::CanvasRenderingContext2D * o, void(mozilla::dom::CanvasRenderingContext2D::*)() m, mozilla::Tuple<> & args, mozilla::IndexSequence<> __formal) Line 1123	C++
> xul.dll!mozilla::detail::RunnableMethodArguments<>::apply<mozilla::dom::CanvasRenderingContext2D,void (__cdecl mozilla::dom::CanvasRenderingContext2D::*)(void) __ptr64>(mozilla::dom::CanvasRenderingContext2D * o, void(mozilla::dom::CanvasRenderingContext2D::*)() m) Line 1130	C++
> xul.dll!mozilla::detail::RunnableMethodImpl<mozilla::dom::CanvasRenderingContext2D * __ptr64 const,void (__cdecl mozilla::dom::CanvasRenderingContext2D::*)(void) __ptr64,1,0>::Run() Line 1175	C++
> xul.dll!mozilla::CycleCollectedJSContext::ProcessStableStateQueue() Line 310	C++
> xul.dll!mozilla::CycleCollectedJSContext::AfterProcessTask(unsigned int aRecursionDepth) Line 370	C++
> xul.dll!XPCJSContext::AfterProcessTask(unsigned int aNewRecursionDepth) Line 1014	C++
> xul.dll!nsThread::ProcessNextEvent(bool aMayWait, bool * aResult) Line 1464	C++
> xul.dll!NS_ProcessNextEvent(nsIThread * aThread, bool aMayWait) Line 480	C++
> xul.dll!mozilla::SpinEventLoopUntil<1,bool <lambda>(void) >(nsThreadPool::Dispatch::__l13::bool <lambda>(void) && aPredicate, nsIThread * aThread) Line 323	C++
> xul.dll!nsThreadPool::Dispatch(already_AddRefed<nsIRunnable> aEvent, unsigned int aFlags) Line 268	C++
> xul.dll!mozilla::net::nsStreamTransportService::Dispatch(already_AddRefed<nsIRunnable> task, unsigned int flags) Line 490	C++
> xul.dll!nsIEventTarget::Dispatch(nsIRunnable * aEvent, unsigned int aFlags) Line 37	C++
> xul.dll!mozilla::Preferences::WritePrefFile(nsIFile * aFile, mozilla::Preferences::SaveMethod aSaveMethod) Line 1246	C++
> xul.dll!mozilla::Preferences::SavePrefFileInternal(nsIFile * aFile, mozilla::Preferences::SaveMethod aSaveMethod) Line 1203	C++
> xul.dll!mozilla::Preferences::SavePrefFileBlocking() Line 916	C++
> xul.dll!mozilla::gfx::DriverCrashGuard::FlushPreferences() Line 394	C++
> xul.dll!mozilla::gfx::DriverCrashGuard::ActivateGuard() Line 236	C++
> xul.dll!mozilla::gfx::DriverCrashGuard::Initialize() Line 138	C++
> xul.dll!mozilla::gfx::DriverCrashGuard::InitializeIfNeeded() Line 65	C++
> xul.dll!mozilla::gfx::DriverCrashGuard::Crashed() Line 184	C++
> xul.dll!mozilla::D3D11DXVA2Manager::Init(mozilla::layers::KnowsCompositor * aKnowsCompositor, nsACString & aFailureReason, ID3D11Device * aDevice) Line 693	C++
> xul.dll!mozilla::DXVA2Manager::CreateD3D11DXVA(mozilla::layers::KnowsCompositor * aKnowsCompositor, nsACString & aFailureReason, ID3D11Device * aDevice) Line 1195	C++
> xul.dll!mozilla::layers::D3D11ShareHandleImage::GetAsSourceSurface() Line 90	C++
> xul.dll!nsLayoutUtils::SurfaceFromElement(mozilla::dom::HTMLVideoElement * aElement, unsigned int aSurfaceFlags, RefPtr<mozilla::gfx::DrawTarget> & aTarget) Line 7507	C++
> xul.dll!nsLayoutUtils::SurfaceFromElement(mozilla::dom::Element * aElement, unsigned int aSurfaceFlags, RefPtr<mozilla::gfx::DrawTarget> & aTarget) Line 7540	C++
> xul.dll!mozilla::dom::CanvasRenderingContext2D::DrawImage(const mozilla::dom::HTMLImageElementOrSVGImageElementOrHTMLCanvasElementOrHTMLVideoElementOrImageBitmap & aImage, double aSx, double aSy, double aSw, double aSh, double aDx, double aDy, double aDw, double aDh, unsigned char aOptional_argc, mozilla::ErrorResult & aError) Line 5299	C++
> xul.dll!mozilla::dom::CanvasRenderingContext2D::DrawImage(const mozilla::dom::HTMLImageElementOrSVGImageElementOrHTMLCanvasElementOrHTMLVideoElementOrImageBitmap & aImage, double aDx, double aDy, double aDw, double aDh, mozilla::ErrorResult & aError) Line 234	C++
> xul.dll!mozilla::dom::CanvasRenderingContext2DBinding::drawImage(JSContext * cx, JS::Handle<JSObject *> obj, mozilla::dom::CanvasRenderingContext2D * self, const JSJitMethodCallArgs & args) Line 2692	C++
> xul.dll!mozilla::dom::GenericBindingMethod(JSContext * cx, unsigned int argc, JS::Value * vp) Line 3053	C++
> xul.dll!js::CallJSNative(JSContext * cx, bool(*)(JSContext *, unsigned int, JS::Value *) native, const JS::CallArgs & args) Line 293	C++
> xul.dll!js::InternalCallOrConstruct(JSContext * cx, const JS::CallArgs & args, js::MaybeConstruct construct) Line 469	C++
> xul.dll!InternalCall(JSContext * cx, const js::AnyInvokeArgs & args) Line 515	C++
> xul.dll!js::CallFromStack(JSContext * cx, const JS::CallArgs & args) Line 521	C++
> xul.dll!Interpret(JSContext * cx, js::RunState & state) Line 3064	C++
> xul.dll!js::RunScript(JSContext * cx, js::RunState & state) Line 409	C++
> xul.dll!js::InternalCallOrConstruct(JSContext * cx, const JS::CallArgs & args, js::MaybeConstruct construct) Line 487	C++
> xul.dll!InternalCall(JSContext * cx, const js::AnyInvokeArgs & args) Line 515	C++
> xul.dll!js::Call(JSContext * cx, JS::Handle<JS::Value> fval, JS::Handle<JS::Value> thisv, const js::AnyInvokeArgs & args, JS::MutableHandle<JS::Value> rval) Line 533	C++
> xul.dll!JS::Call(JSContext * cx, JS::Handle<JS::Value> thisv, JS::Handle<JS::Value> fval, const JS::HandleValueArray & args, JS::MutableHandle<JS::Value> rval) Line 2949	C++
> xul.dll!mozilla::dom::EventListener::HandleEvent(JSContext * cx, JS::Handle<JS::Value> aThisVal, mozilla::dom::Event & event, mozilla::ErrorResult & aRv) Line 47	C++
> xul.dll!mozilla::dom::EventListener::HandleEvent<mozilla::dom::EventTarget * __ptr64>(mozilla::dom::EventTarget * const & thisVal, mozilla::dom::Event & event, mozilla::ErrorResult & aRv, const char * aExecutionReason, mozilla::dom::CallbackObject::ExceptionHandling aExceptionHandling, JSCompartment * aCompartment) Line 65	C++
> xul.dll!mozilla::EventListenerManager::HandleEventSubType(mozilla::EventListenerManager::Listener * aListener, nsIDOMEvent * aDOMEvent, mozilla::dom::EventTarget * aCurrentTarget) Line 1139	C++
> xul.dll!mozilla::EventListenerManager::HandleEventInternal(nsPresContext * aPresContext, mozilla::WidgetEvent * aEvent, nsIDOMEvent * * aDOMEvent, mozilla::dom::EventTarget * aCurrentTarget, nsEventStatus * aEventStatus) Line 1311	C++
> xul.dll!mozilla::EventListenerManager::HandleEvent(nsPresContext * aPresContext, mozilla::WidgetEvent * aEvent, nsIDOMEvent * * aDOMEvent, mozilla::dom::EventTarget * aCurrentTarget, nsEventStatus * aEventStatus) Line 377	C++
> xul.dll!mozilla::EventTargetChainItem::HandleEvent(mozilla::EventChainPostVisitor & aVisitor, mozilla::ELMCreationDetector & aCd) Line 319	C++
> xul.dll!mozilla::EventTargetChainItem::HandleEventTargetChain(nsTArray<mozilla::EventTargetChainItem> & aChain, mozilla::EventChainPostVisitor & aVisitor, mozilla::EventDispatchingCallback * aCallback, mozilla::ELMCreationDetector & aCd) Line 466	C++
> xul.dll!mozilla::EventDispatcher::Dispatch(nsISupports * aTarget, nsPresContext * aPresContext, mozilla::WidgetEvent * aEvent, nsIDOMEvent * aDOMEvent, nsEventStatus * aEventStatus, mozilla::EventDispatchingCallback * aCallback, nsTArray<mozilla::dom::EventTarget *> * aTargets) Line 827	C++
> xul.dll!mozilla::EventDispatcher::DispatchDOMEvent(nsISupports * aTarget, mozilla::WidgetEvent * aEvent, nsIDOMEvent * aDOMEvent, nsPresContext * aPresContext, nsEventStatus * aEventStatus) Line 890	C++
> xul.dll!nsINode::DispatchEvent(nsIDOMEvent * aEvent, bool * aRetVal) Line 1345	C++
> xul.dll!nsContentUtils::DispatchEvent(nsIDocument * aDoc, nsISupports * aTarget, const nsAString & aEventName, bool aCanBubble, bool aCancelable, bool aTrusted, bool * aDefaultAction, bool aOnlyChromeDispatch) Line 4464	C++
> xul.dll!nsContentUtils::DispatchTrustedEvent(nsIDocument * aDoc, nsISupports * aTarget, const nsAString & aEventName, bool aCanBubble, bool aCancelable, bool * aDefaultAction) Line 4434	C++
> xul.dll!mozilla::dom::HTMLMediaElement::DispatchEvent(const nsAString & aName) Line 6136	C++
> xul.dll!mozilla::dom::HTMLMediaElement::nsAsyncEventRunner::Run() Line 272	C++
> xul.dll!nsThread::ProcessNextEvent(bool aMayWait, bool * aResult) Line 1446	C++
> xul.dll!NS_ProcessNextEvent(nsIThread * aThread, bool aMayWait) Line 480	C++
> xul.dll!mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate * aDelegate) Line 97	C++
> xul.dll!MessageLoop::RunInternal() Line 327	C++
> xul.dll!MessageLoop::RunHandler() Line 320	C++
> xul.dll!MessageLoop::Run() Line 300	C++
> xul.dll!nsBaseAppShell::Run() Line 158	C++
> xul.dll!nsAppShell::Run() Line 278	C++
> xul.dll!nsAppStartup::Run() Line 287	C++
> xul.dll!XREMain::XRE_mainRun() Line 4608	C++
> xul.dll!XREMain::XRE_main(int argc, char * * argv, const mozilla::BootstrapConfig & aConfig) Line 4772	C++
> xul.dll!XRE_main(int argc, char * * argv, const mozilla::BootstrapConfig & aConfig) Line 4867	C++
> xul.dll!mozilla::BootstrapImpl::XRE_main(int argc, char * * argv, const mozilla::BootstrapConfig & aConfig) Line 46	C++
> firefox.exe!do_main(int argc, char * * argv, char * * envp) Line 237	C++
> firefox.exe!NS_internal_main(int argc, char * * argv, char * * envp) Line 309	C++
> firefox.exe!wmain(int argc, wchar_t * * argv) Line 115	C++
> [External Code]
It looks like this might be due to bug 789945, which landed in 56.
Blocks: 789945
Flags: needinfo?(howareyou322)
Do we know why this isn't breaking on nightly? And does the graphics window reeeeeaally need to do a sync pref flush-to-disk?
It's new-profile-only, and I believe non-e10s-only. I think the call is not actually supposed to be sync here (pref is always true), but rather I suspect (but have not yet confirmed) that querying the pref is failing, and leaving the outvar as false, causing us to request a sync dispatch, which it terrible.

We may just need to fix the pref default or remove the pref itself. I will look more on Saturday PST.
Comment on attachment 8894288 [details]
Bug 1386263 - Disable preferences.allow.omt-write for causing reentrancy bugs by nested-pumping the event loop. -

Try says this makes Win8 m-gl and m-mda much happier!

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ecf649744383651189277d6b54d39c45dfd23837&group_state=expanded
Attachment #8894288 - Flags: feedback+
Canada is on holiday today. Per IRC discussion with Jeff and Liz, we decided that landing this now with review pending to unblock the beta1 GTB was the most prudent plan of action for now.

https://hg.mozilla.org/releases/mozilla-beta/rev/6c489d5df6d4
We'll still need to find the root cause here, but the immediate issue is taken care of so let's take that investigation to a new bug for easier tracking.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla56
Ryan, what's the magic incantation to get this to reproduce on a try run, given that it doesn't fail in nightly?
Flags: needinfo?(milan) → needinfo?(ryanvm)
Attachment #8894288 - Flags: review?(milan) → review+
Flags: needinfo?(milan)
You can push on top of a regular Beta tree (though with comment 12 backed out at this point).
Flags: needinfo?(ryanvm)
The problem could be the "auto save on dirty" (bug 981818) rather than OMT save (bug 789945).  With OMT write enabled, but "auto save on dirty" disabled, try is green:

]https://treeherder.mozilla.org/#/jobs?repo=try&revision=f2efa0972a2905648cce91cf21d1cfd17f661585
Never mind the above is probably just an accident.  Turns out DISPATCH_SYNC does not in fact just "wait" for the call to return - it spins the event loop which then goes on and processes other stuff.  I'll open another bug to fix this.
Flags: needinfo?(milan)
To repro locally, I built beta normally then invoked with e10s disabled:
> ./mach run --setpref browser.tabs.remote.autostart=false --setpref browser.tabs.remote.autostart.2=false

I found I needed to delete the profile between runs, too:
> rm -rf ../path/to/objdir/tmp
(In reply to Milan Sreckovic [:milan] from comment #17)
> Never mind the above is probably just an accident.  Turns out DISPATCH_SYNC
> does not in fact just "wait" for the call to return - it spins the event
> loop which then goes on and processes other stuff.  I'll open another bug to
> fix this.

I'm going to take a stab at renaming that, since its name makes it sound like it does something completely different.
Given the late discovery of this bug, I'm going to suggest we leave this preference off in 56 and let the change and the fix ride 57 instead.
Looks like bug 1388464 did indeed fix this on 57.
No longer depends on: 1387242
You need to log in before you can comment on or make changes to this bug.