1316145 - Intermittent dom/media/tests/mochitest/test_peerConnection_capturedVideo.html | application crashed [@ mozilla::TaskQueue::Dispatch]

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect, P3)

Description

[Treeherder Bug Filer]

Filed by: wkocher [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=38830567&repo=mozilla-inbound

https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux-pgo/1478610504/mozilla-inbound_ubuntu32_vm_test_pgo-mochitest-media-bm05-tests1-linux32-build63.txt.gz

Comment 1

[Olli Pettay [:smaug]]

I'm still reading the code. As far as I see it is rather easy to enter the state where the thread is closing yet we're still dispatching to it. For example Canonical::Set.
And all the StableState usage... if one happens to try to close the AbstractThread in a nested event loop, StableState stuff will just run using closing thread.

But haven't managed to reproduce locally, so don't know which case this is about.

Comment 2

[Intermittent Failures Robot]

29 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 16
* autoland: 12
* try: 1

Platform breakdown:
* linux32: 24
* linux64: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316145&startday=2016-11-07&endday=2016-11-13&tree=all

Comment 3

[Olli Pettay [:smaug]]

jww, jya, do you think you could have time to look at this a bit?

The current guess is that some thread somewhere is shutting down and after that we still try to dispatch tasks to it. (Possibly nested event loop spinning in the main thread is related here?)
As far as I see bug 1306591 shouldn't be able to cause this, but it could certainly reveal wrong assumptions in code.

Comment 4

[JW Wang [:jwwang] [:jw_wang]]

Bug 1225731 will help us figure out which callsite is doing Dispatch() after task queue shutdown. Let's just wait for reports to reveal it.

Comment 5

[Intermittent Failures Robot]

17 failures in 144 pushes (0.118 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* autoland: 10
* mozilla-inbound: 5
* try: 1
* mozilla-aurora: 1

Platform breakdown:
* linux32: 14
* linux64: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316145&startday=2016-11-15&endday=2016-11-15&tree=all

Comment 6

[Intermittent Failures Robot]

16 failures in 119 pushes (0.134 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* autoland: 8
* mozilla-inbound: 6
* mozilla-central: 1
* mozilla-aurora: 1

Platform breakdown:
* linux32: 15
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316145&startday=2016-11-16&endday=2016-11-16&tree=all

Comment 7

[Intermittent Failures Robot]

19 failures in 147 pushes (0.129 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* autoland: 7
* mozilla-inbound: 6
* try: 5
* mozilla-central: 1

Platform breakdown:
* linux32: 16
* linux64: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316145&startday=2016-11-17&endday=2016-11-17&tree=all

Comment 8

[JW Wang [:jwwang] [:jw_wang]]

This is a regression caused by bug 1315581.

Comment 9

[JW Wang [:jwwang] [:jw_wang]]

Attachment: Bug 1316145 - ensure compositor events are dispatched to readers before Shutdown().

To prevent bug 1318226, we use 2-phase dispatch
(main thread -> MDSM task queue -> reader task queue) to ensure events arrive
in correct order.

Review commit: https://reviewboard.mozilla.org/r/93980/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/93980/

Comment 10

[JW Wang [:jwwang] [:jw_wang]]

Attachment: Bug 1316145 - pass DontAssertDispatchSuccess when dispatching events because it is hard to enforce notification happens before AbstractThread shutdown.

Review commit: https://reviewboard.mozilla.org/r/93994/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/93994/

Comment 11

[Jean-Yves Avenard [:jya]]

Comment on attachment 8812115 [details]
Bug 1316145 - pass DontAssertDispatchSuccess when dispatching events because it is hard to enforce notification happens before AbstractThread shutdown.

https://reviewboard.mozilla.org/r/93994/#review94142

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6e593b6f2ee6
pass DontAssertDispatchSuccess when dispatching events because it is hard to enforce notification happens before AbstractThread shutdown. r=jya

Comment 13

[Intermittent Failures Robot]

74 failures in 715 pushes (0.103 failures/push) were associated with this bug in the last 7 days. 

This is the #32 most frequent failure this week. 

** This failure happened more than 50 times this week! Resolving this bug is a high priority. **

Repository breakdown:
* mozilla-inbound: 30
* autoland: 30
* try: 7
* mozilla-central: 4
* mozilla-aurora: 3

Platform breakdown:
* linux32: 63
* linux64: 11

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316145&startday=2016-11-14&endday=2016-11-20&tree=all

Comment 14

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/6e593b6f2ee6

Comment 15

[Ryan VanderMeulen [:RyanVM]]

Please request Aurora on this when you get a chance.

Comment 16

[JW Wang [:jwwang] [:jw_wang]]

Comment on attachment 8812115 [details]
Bug 1316145 - pass DontAssertDispatchSuccess when dispatching events because it is hard to enforce notification happens before AbstractThread shutdown.

Approval Request Comment
[Feature/regressing bug #]:1315581
[User impact if declined]:assertion
[Describe test coverage new/current, TreeHerder]:TreeHerder
[Risks and why]: Low. The change is quite simple.
[String/UUID change made/needed]:none

Comment 17

[Julien Cristau [:jcristau]]

Comment on attachment 8812115 [details]
Bug 1316145 - pass DontAssertDispatchSuccess when dispatching events because it is hard to enforce notification happens before AbstractThread shutdown.

one-line fix for intermittent regression, let's get this in aurora52

Comment 18

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/8bcdcb64549b