Intermittent /webrtc/RTCPeerConnection-addTrack.html | addTrack when pc is closed should throw InvalidStateError - promise_test: Unhandled rejection with value: object "NotFoundError: The object can not be found here."

RESOLVED FIXED

Status

()

P1
normal
Rank:
15
RESOLVED FIXED
a year ago
8 months ago

People

(Reporter: intermittent-bug-filer, Unassigned)

Tracking

({intermittent-failure})

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [stockwell disabled])

32 failures in 179 pushes (0.179 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 29
* try: 3

Platform breakdown:
* macosx64-stylo: 32

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388550&startday=2017-08-10&endday=2017-08-10&tree=all
All failures are osx-stylo. This appears to be perma-fail. It occurs in the same job as bug 1387684 and was starred against that bug for at least several days. Is this failure a consequence of bug 1387684? A dup?
See Also: → bug 1387684
Whiteboard: [stockwell needswork]
31 failures in 166 pushes (0.187 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 25
* mozilla-central: 6

Platform breakdown:
* macosx64-stylo: 31

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388550&startday=2017-08-11&endday=2017-08-11&tree=all
92 failures in 901 pushes (0.102 failures/push) were associated with this bug in the last 7 days. 

This is the #9 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 71
* mozilla-central: 17
* try: 4

Platform breakdown:
* macosx64-stylo: 92

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388550&startday=2017-08-07&endday=2017-08-13&tree=all
(In reply to Geoff Brown [:gbrown] from comment #3)
> First failure seems to be
> https://treeherder.mozilla.org/#/
> jobs?repo=autoland&revision=e5e994cb9d84ea35d4034c0fd21bb7e870d5f586

The set of patches appear to have nothing directly to do with webrtc.  I note that there are a ton of stylo failures, and also in that patchset I see "Enable stylo on static analysis builds".

Stylo *should* have nothing to do with webrtc -- but this may have changed timing and uncovered a race condition.

jib?  pehrsons? (who's back the 21st)
Rank: 15
Flags: needinfo?(jib)
Priority: P5 → P1
The bug here seems to be that these tests are run at all. Logs [1] show all the "RTCPeerConnection-*" tests are failing.

Tests like RTCPeerConnection-addTrack.html [2] use getUserMedia, which fails with NotFoundError because there are no cameras or mics available in the test environment. - But we shouldn't be relying on the native environment at all, because this tends to be unreliable. Test machines often have no hardware devices at all. Do stylo tests run on different machines?

Instead, we should use fake devices like we do in mochitests (with pref media.navigator.streams.fake) on all platforms but linux where we have low-end loopback devices instead. We should also use media.navigator.permission.disabled to prevent opening permission prompts.

I see these tests are currently expected to TIMEOUT, probably because of the permission prompt. I also see there's been recent efforts (bug 1386604) [3] to expect OK for certain versions of OSX (not stylo though) perhaps from empirical testing? We probably shouldn't do that either.

See Bug 1376960 which should improve this situation by using the aforementioned prefs to enable several mediastream wpt tests (though not these particular ones AFAICT). It hasn't landed yet though.

[1] https://public-artifacts.taskcluster.net/WTzEA7EeS2mMz_zTeC_lBQ/0/public/logs/live_backing.log
[2] https://dxr.mozilla.org/mozilla-central/rev/5322c03f4c8587fe526172d3f87160031faa6d75/testing/web-platform/tests/webrtc/RTCPeerConnection-addTrack.html#43-44,54
[3] https://hg.mozilla.org/mozilla-central/rev/d15a98cd1761#l5.4
Flags: needinfo?(jib)
Flags: needinfo?(james)
38 failures in 149 pushes (0.255 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 30
* mozilla-central: 8

Platform breakdown:
* macosx64-stylo: 38

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388550&startday=2017-08-14&endday=2017-08-14&tree=all
So the expectation from these tests is just set based on the result they had when they are imported. I don't know much about the tests in specific, but if there's some way we can set up the configuration so we don't get permission prompts and use fake devices we should do that. Is there something here that needs to be standardised (e.g. a standard for faking media devices for testing)? Chrome have been experimenting with that approach for writing sharable WebUSB and WebBluetooth tests. Another option would be to extend WebDriver with some protocol for faking media input (there is already work underway to allow handling permission prompts via WebDriver).
Flags: needinfo?(james)
(In reply to James Graham [:jgraham] from comment #9)
> So the expectation from these tests is just set based on the result they had
> when they are imported. I don't know much about the tests in specific, but
> if there's some way we can set up the configuration so we don't get
> permission prompts and use fake devices we should do that. Is there
> something here that needs to be standardised (e.g. a standard for faking
> media devices for testing)? 

No, and to be honest, fake:true streams are useful for testing but we have no interest in standardizing them.  And compared to the "real" fake devices we use on Linux, fake:true doesn't test the real capture codepaths (which has bitten us a few times).  We've looked at fake devices for other OSes, but have never agreed on anything (ted did most of this, note).

> Chrome have been experimenting with that
> approach for writing sharable WebUSB and WebBluetooth tests. Another option
> would be to extend WebDriver with some protocol for faking media input
> (there is already work underway to allow handling permission prompts via
> WebDriver).

In theory it could be added to WebDriver; this would not be a simple thing to do.  To really test the real codepaths for capture, you need to act like a real OS device.  If you add hooks for forcing in input as if it was from a camera/mic, that would work for selenium/WebDriver, but as mentioned would be quite a bit of work (especially if you don't want it to just be what fake:true gets you - sine waves and a color-field (with coded frame number/etc data in squares).

Chrome/Webrtc.org does have some automated audio and video tests, though, so it may be worth looking at that at some point
Yeah the fake cam and mic devices are too limiting. Instead, having full-stack loopback devices, like we have on linux [1], on all platforms would be awesome.

The fake devices also predate other options for producing fake tracks: https://blog.mozilla.org/webrtc/warm-up-with-replacetrack/ shows already-web-compatible techniques to create black video and silent audio tracks, if that's all your test needs.

Bug 1088621 hopes to make our fake devices behave more like cams and mics, if it ever lands.

[1] https://dxr.mozilla.org/mozilla-central/rev/b95b1638db48fc3d450b95b98da6bcd2f9326d2f/testing/mochitest/mochitest_options.py#837
21 failures in 180 pushes (0.117 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 18
* mozilla-central: 3

Platform breakdown:
* macosx64-stylo: 21

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388550&startday=2017-08-15&endday=2017-08-15&tree=all
39 failures in 188 pushes (0.207 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 25
* mozilla-central: 9
* try: 5

Platform breakdown:
* macosx64-stylo: 39

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388550&startday=2017-08-16&endday=2017-08-16&tree=all
19 failures in 194 pushes (0.098 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 10
* try: 6
* mozilla-central: 3

Platform breakdown:
* macosx64-stylo: 19

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388550&startday=2017-08-17&endday=2017-08-17&tree=all
32 failures in 160 pushes (0.2 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 30
* try: 2

Platform breakdown:
* macosx64-stylo: 32

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388550&startday=2017-08-18&endday=2017-08-18&tree=all
169 failures in 949 pushes (0.178 failures/push) were associated with this bug in the last 7 days. 

This is the #7 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 127
* mozilla-central: 27
* try: 15

Platform breakdown:
* macosx64-stylo: 169

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388550&startday=2017-08-14&endday=2017-08-20&tree=all
34 failures in 129 pushes (0.264 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 23
* mozilla-central: 8
* try: 3

Platform breakdown:
* macosx64-stylo: 34

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388550&startday=2017-08-21&endday=2017-08-21&tree=all

Comment 18

a year ago
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/fde25580274f
Disable /webrtc/RTCPeerConnection-addTrack.html on osx-stylo. r=me a=testonly
leaving this open, please consider looking into the failure and fixing this in the near future :)
Keywords: leave-open
Whiteboard: [stockwell needswork] → [stockwell disabled]
21 failures in 143 pushes (0.147 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 17
* try: 4

Platform breakdown:
* macosx64-stylo: 21

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388550&startday=2017-08-22&endday=2017-08-22&tree=all
70 failures in 908 pushes (0.077 failures/push) were associated with this bug in the last 7 days. 

This is the #25 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 46
* try: 16
* mozilla-central: 8

Platform breakdown:
* macosx64-stylo: 70

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1388550&startday=2017-08-21&endday=2017-08-27&tree=all
Depends on: 1390521
Fixed by 1390521.
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.