Closed Bug 1447563 Opened 2 years ago Closed Last year

Intermittent dom/media/test/test_video_dimensions.html | application terminated with exit code -11

Categories

(Core :: Audio/Video: MediaStreamGraph, defect, P3)

defect

Tracking

()

RESOLVED FIXED
mozilla61
Tracking Status
firefox-esr52 --- unaffected
firefox59 --- unaffected
firefox60 --- fixed
firefox61 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: pehrsons)

References

Details

(Keywords: crash, intermittent-failure, Whiteboard: [stockwell fixed:product])

Attachments

(2 files)

In the last 7 days we have 37 failures.
They occur on Linux x64 (opt, debug, pgo) and linux64-qr (opt).
Recent log example: https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=169779819&lineNumber=3420
:drno, can you have a look at this?
Flags: needinfo?(drno)
Whiteboard: [stockwell needswork]
From https://treeherder.mozilla.org/logviewer.html#?job_id=169770822&repo=autoland&lineNumber=6081

> [task 2018-03-23T00:04:09.834Z] 00:04:09     INFO - GECKO(1588) | [GFX1]: Texture deallocated too late during shutdown
> [task 2018-03-23T00:04:09.834Z] 00:04:09     INFO - GECKO(1588) | Assertion failure: [GFX1]: Texture deallocated too late during shutdown, at /builds/worker/workspace/build/src/gfx/2d/Logging.h:727
> [task 2018-03-23T00:04:44.153Z] 00:04:44     INFO - GECKO(1588) | #01: mozilla::gfx::Log<1, mozilla::gfx::CriticalLogger>::~Log [gfx/2d/Logging.h:731]
> [task 2018-03-23T00:04:44.155Z] 00:04:44     INFO - GECKO(1588) | #02: mozilla::layers::DeallocateTextureClient [gfx/layers/client/TextureClient.cpp:331]
> [task 2018-03-23T00:04:44.158Z] 00:04:44     INFO - GECKO(1588) | #03: mozilla::layers::TextureClient::Destroy [gfx/layers/client/TextureClient.cpp:433]
> [task 2018-03-23T00:04:44.160Z] 00:04:44     INFO - GECKO(1588) | #04: mozilla::layers::TextureClient::~TextureClient [gfx/layers/client/TextureClient.cpp:622]
> [task 2018-03-23T00:04:44.161Z] 00:04:44     INFO - GECKO(1588) | #05: mozilla::layers::TextureClient::~TextureClient [gfx/layers/client/TextureClient.cpp:629]
> [task 2018-03-23T00:04:44.162Z] 00:04:44     INFO - GECKO(1588) | #06: mozilla::AtomicRefCountedWithFinalize<mozilla::layers::TextureClient>::Release [gfx/layers/AtomicRefCountedWithFinalize.h:145]
> [task 2018-03-23T00:04:44.164Z] 00:04:44     INFO - GECKO(1588) | #07: mozilla::layers::TextureClientHolder::Release [memory/mozalloc/mozalloc.h:180]
> [task 2018-03-23T00:04:44.165Z] 00:04:44     INFO - GECKO(1588) | #08: mozilla::layers::TextureClientRecycleAllocator::RecycleTextureClient [mfbt/RefPtr.h:41]
> [task 2018-03-23T00:04:44.167Z] 00:04:44     INFO - GECKO(1588) | #09: mozilla::AtomicRefCountedWithFinalize<mozilla::layers::TextureClient>::Release [gfx/layers/AtomicRefCountedWithFinalize.h:154]
> [task 2018-03-23T00:04:44.169Z] 00:04:44     INFO - GECKO(1588) | #10: mozilla::layers::SharedPlanarYCbCrImage::~SharedPlanarYCbCrImage [xpcom/base/nsAutoRef.h:527]
> [task 2018-03-23T00:04:44.172Z] 00:04:44     INFO - GECKO(1588) | #11: mozilla::layers::SharedPlanarYCbCrImage::~SharedPlanarYCbCrImage [gfx/layers/ipc/SharedPlanarYCbCrImage.cpp:40]
> [task 2018-03-23T00:04:44.174Z] 00:04:44     INFO - GECKO(1588) | #12: mozilla::layers::Image::Release [gfx/layers/ImageContainer.h:205]
> [task 2018-03-23T00:04:44.176Z] 00:04:44     INFO - GECKO(1588) | #13: nsTArray_Impl<mozilla::layers::ImageContainer::OwningImage, nsTArrayInfallibleAllocator>::~nsTArray_Impl [xpcom/ds/nsTArray.h:2072]
> [task 2018-03-23T00:04:44.178Z] 00:04:44     INFO - GECKO(1588) | #14: mozilla::layers::ImageContainer::~ImageContainer [gfx/layers/ImageContainer.cpp:214]
> [task 2018-03-23T00:04:44.180Z] 00:04:44     INFO - GECKO(1588) | #15: mozilla::layers::ImageContainer::Release [memory/mozalloc/mozalloc.h:180]
> [task 2018-03-23T00:04:44.182Z] 00:04:44     INFO - GECKO(1588) | #16: mozilla::VideoFrameContainer::~VideoFrameContainer [dom/media/MediaStreamVideoSink.h:39]
> [task 2018-03-23T00:04:44.183Z] 00:04:44     INFO - GECKO(1588) | #17: mozilla::VideoFrameContainer::~VideoFrameContainer [dom/media/VideoFrameContainer.cpp:57]
> [task 2018-03-23T00:04:44.185Z] 00:04:44     INFO - GECKO(1588) | #18: mozilla::MediaStreamTrackListener::Release [dom/media/MediaStreamListener.h:156]
> [task 2018-03-23T00:04:44.186Z] 00:04:44     INFO - GECKO(1588) | #19: mozilla::MediaStream::~MediaStream [dom/media/MediaStreamGraph.h:175]
> [task 2018-03-23T00:04:44.189Z] 00:04:44     INFO - GECKO(1588) | #20: mozilla::TrackUnionStream::~TrackUnionStream [dom/media/TrackUnionStream.h:18]
> [task 2018-03-23T00:04:44.192Z] 00:04:44     INFO - GECKO(1588) | #21: mozilla::MediaStream::Release [dom/media/MediaStreamGraph.h:254]
> [task 2018-03-23T00:04:44.194Z] 00:04:44     INFO - GECKO(1588) | #22: mozilla::MediaStream::Destroy [dom/media/MediaStreamGraph.cpp:2090]
> [task 2018-03-23T00:04:44.196Z] 00:04:44     INFO - GECKO(1588) | #23: mozilla::DOMMediaStream::Destroy [dom/media/DOMMediaStream.cpp:490]
> [task 2018-03-23T00:04:44.198Z] 00:04:44     INFO - GECKO(1588) | #24: mozilla::DOMMediaStream::cycleCollection::Unlink [dom/media/DOMMediaStream.cpp:378]
> [task 2018-03-23T00:04:44.200Z] 00:04:44     INFO - GECKO(1588) | #25: nsCycleCollector::CollectWhite [xpcom/base/nsCycleCollector.cpp:3403]
> [task 2018-03-23T00:04:44.202Z] 00:04:44     INFO - GECKO(1588) | #26: nsCycleCollector::Collect [xpcom/base/nsCycleCollector.cpp:3769]
> [task 2018-03-23T00:04:44.204Z] 00:04:44     INFO - GECKO(1588) | #27: nsCycleCollector::ShutdownCollect [xpcom/base/nsCycleCollector.cpp:3687]
> [task 2018-03-23T00:04:44.206Z] 00:04:44     INFO - GECKO(1588) | #28: nsCycleCollector_shutdown [mfbt/RefPtr.h:63]
> [task 2018-03-23T00:04:44.207Z] 00:04:44     INFO - GECKO(1588) | #29: mozilla::ShutdownXPCOM [xpcom/build/XPCOMInit.cpp:983]

I think the easy fix here is to clear mVideoOutputs on xpcom shutdown (rather than cyclecollector shutdown), just like we do with mTrackListeners: https://searchfox.org/mozilla-central/rev/003262ae12ce937950ffb8d3b0fa520d1cc38bff/dom/media/MediaStreamGraph.cpp#2048
Assignee: nobody → apehrson
Status: NEW → ASSIGNED
Rank: 25
Component: Audio/Video → Audio/Video: MediaStreamGraph
Flags: needinfo?(drno)
Priority: P5 → P3
Duplicate of this bug: 1442588
Comment on attachment 8962653 [details]
Bug 1447563 - Clear video outputs during shutdown.

https://reviewboard.mozilla.org/r/231202/#review237266
Attachment #8962653 - Flags: review?(padenot) → review+
Comment on attachment 8962654 [details]
Bug 1447563 - Clear direct track listeners from TrackUnionStream during shutdown.

https://reviewboard.mozilla.org/r/231276/#review237268
Attachment #8962654 - Flags: review?(padenot) → review+
Pushed by pehrsons@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/7c2efd417ec5
Clear video outputs during shutdown. r=padenot
https://hg.mozilla.org/integration/autoland/rev/3f2758cf09a1
Clear direct track listeners from TrackUnionStream during shutdown. r=padenot
Whiteboard: [stockwell needswork] → [stockwell fixed:product]
https://hg.mozilla.org/mozilla-central/rev/7c2efd417ec5
https://hg.mozilla.org/mozilla-central/rev/3f2758cf09a1
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla61
Duplicate of this bug: 1449823
:drno- there are a lot of intermittents failing in audio/video tests- can you look at these or find some engineers to help out?
Flags: needinfo?(drno)
Whiteboard: [stockwell fixed:product] → [stockwell needswork]
If only one of these happened in a debug build it would be a lot easier to diagnose. So far since the landing there has only been opt and pgo crashes.
I hope Andreas can continue to take care of this.

Is this the same as bug 1442732?
Is bug 1442732 comment #3 what we landed here before?
Flags: needinfo?(drno)
See Also: → 1442732
Duplicate of this bug: 1442732
Duplicate of this bug: 1442391
Duplicate of this bug: 1393761
Duplicate of this bug: 1449387
Duplicate of this bug: 1449063
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #19)
> :drno- there are a lot of intermittents failing in audio/video tests- can
> you look at these or find some engineers to help out?

Not a single of the intermittents I closed as duplicates of this bug has been recorded since this first landed. Clearly we have some failure mode left in opt but the debug intermittents seem to be gone.

To act on this I need one crash in debug so I can see the stack, or plenty of time to dig deeper. Currently I have neither of those, but I'll keep my eyes open for the former.
(In reply to Nils Ohlmeier [:drno] from comment #22)
> I hope Andreas can continue to take care of this.
> 
> Is this the same as bug 1442732?
> Is bug 1442732 comment #3 what we landed here before?

Yes and yes.
Interestingly, after this landed on March 29th, there was an outbreak of failures on April 2nd and 3rd for some reason. After that it's been completely quiet again.

I feel inclined to re-close this, but let's give it some more time.
Whiteboard: [stockwell disable-recommended] → [stockwell fixed:product]
Status: REOPENED → RESOLVED
Closed: 2 years ago2 years ago
Resolution: --- → FIXED
Seeing a couple of these on Beta again recently :-\
https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-beta&job_id=173947148
https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-beta&job_id=173949453

Given the timing of it starting today, that narrows it down to one your recent uplifts I guess?
Flags: needinfo?(apehrson)
Hard to say, it might just be a slight timing change triggering this on beta. We could try uplifting this fix to beta too.
Flags: needinfo?(apehrson)
Comment on attachment 8962653 [details]
Bug 1447563 - Clear video outputs during shutdown.

This approval request applies to both patches on this bug.

Approval Request Comment
[Feature/Bug causing the regression]: It's unclear what has triggered this latent issue.
[User impact if declined]: (fairly rare) shutdown crash
[Is this code covered by automated tests?]: Yes, it shows as a fairly rare intermittent.
[Has the fix been verified in Nightly?]: Yes.
[Needs manual test from QE? If yes, steps to reproduce]: No
[List of other uplifts needed for the feature/fix]: None
[Is the change risky?]: It's low risk
[Why is the change risky/not risky?]: It does change timing of some events related to shutdown/cleanup, but there are no thread safety concerns, and it has been well baked on Nightly.
[String changes made/needed]: None
Attachment #8962653 - Flags: approval-mozilla-beta?
Comment on attachment 8962653 [details]
Bug 1447563 - Clear video outputs during shutdown.

avoid some intermittent shutdown issues, approved for 60.0b14
Attachment #8962653 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Duplicate of this bug: 1458166
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Status: REOPENED → RESOLVED
Closed: 2 years ago2 years ago
Resolution: --- → FIXED
It is not solved. See the dup I linked to.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Plus comment 39 shows it is still Occurring
Yes, sorry, I should have put an explanatory comment here too. For some reason bugzilla doesn't show a record of me unduping the dub you linked to.

See bug 1458166 comment 17. That bug and this have similar symptoms but different root causes. Since this has already had patches landed I prefer to do followup fixes in a separate bug. I see no indication that the root cause identified here is not fixed.
Status: REOPENED → RESOLVED
Closed: 2 years ago2 years ago
Resolution: --- → FIXED
Sure, but comment 13 shop that it is still happening and that occurred less than a day ago. As such, this bug can't be considered fixed.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Like I said, same symptoms, different root causes. To me that means different bugs.

These opt crashes should IMHO be linked to an opt version of bug 1458166.
(In reply to OrangeFactor Robot from comment #46)
> 2 failures in 632 pushes (0.003 failures/push) were associated with this bug
> in the last 7 days.    
> 
> Repository breakdown:
> * autoland: 2
> 
> Platform breakdown:
> * linux64-qr: 1
> * linux64: 1
> 
> For more details, see:
> https://treeherder.mozilla.org/intermittent-failures.html#/
> bugdetails?bug=1447563&startday=2018-05-21&endday=2018-05-27&tree=trunk

The one occurrence in that treeherder link is a bad attribution.
Status: REOPENED → RESOLVED
Closed: 2 years agoLast year
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.