Closed Bug 1388550 Opened 7 years ago Closed 7 years ago

Intermittent /webrtc/RTCPeerConnection-addTrack.html | addTrack when pc is closed should throw InvalidStateError - promise_test: Unhandled rejection with value: object "NotFoundError: The object can not be found here."

Categories

(Core :: WebRTC, defect, P1)

Product:
Core
Component:
WebRTC
Type:
defect

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: intermittent-bug-filer, Unassigned)

References

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell disabled]) 

All failures are osx-stylo. This appears to be perma-fail. It occurs in the same job as bug 1387684 and was starred against that bug for at least several days. Is this failure a consequence of bug 1387684? A dup?
See Also: → 1387684
Whiteboard: [stockwell needswork]
(In reply to Geoff Brown [:gbrown] from comment #3)
> First failure seems to be
> https://treeherder.mozilla.org/#/
> jobs?repo=autoland&revision=e5e994cb9d84ea35d4034c0fd21bb7e870d5f586

The set of patches appear to have nothing directly to do with webrtc.  I note that there are a ton of stylo failures, and also in that patchset I see "Enable stylo on static analysis builds".

Stylo *should* have nothing to do with webrtc -- but this may have changed timing and uncovered a race condition.

jib?  pehrsons? (who's back the 21st)
Rank: 15
Flags: needinfo?(jib)
Priority: P5 → P1
The bug here seems to be that these tests are run at all. Logs [1] show all the "RTCPeerConnection-*" tests are failing.

Tests like RTCPeerConnection-addTrack.html [2] use getUserMedia, which fails with NotFoundError because there are no cameras or mics available in the test environment. - But we shouldn't be relying on the native environment at all, because this tends to be unreliable. Test machines often have no hardware devices at all. Do stylo tests run on different machines?

Instead, we should use fake devices like we do in mochitests (with pref media.navigator.streams.fake) on all platforms but linux where we have low-end loopback devices instead. We should also use media.navigator.permission.disabled to prevent opening permission prompts.

I see these tests are currently expected to TIMEOUT, probably because of the permission prompt. I also see there's been recent efforts (bug 1386604) [3] to expect OK for certain versions of OSX (not stylo though) perhaps from empirical testing? We probably shouldn't do that either.

See Bug 1376960 which should improve this situation by using the aforementioned prefs to enable several mediastream wpt tests (though not these particular ones AFAICT). It hasn't landed yet though.

[1] https://public-artifacts.taskcluster.net/WTzEA7EeS2mMz_zTeC_lBQ/0/public/logs/live_backing.log
[2] https://dxr.mozilla.org/mozilla-central/rev/5322c03f4c8587fe526172d3f87160031faa6d75/testing/web-platform/tests/webrtc/RTCPeerConnection-addTrack.html#43-44,54
[3] https://hg.mozilla.org/mozilla-central/rev/d15a98cd1761#l5.4
Flags: needinfo?(jib)
Flags: needinfo?(james)
So the expectation from these tests is just set based on the result they had when they are imported. I don't know much about the tests in specific, but if there's some way we can set up the configuration so we don't get permission prompts and use fake devices we should do that. Is there something here that needs to be standardised (e.g. a standard for faking media devices for testing)? Chrome have been experimenting with that approach for writing sharable WebUSB and WebBluetooth tests. Another option would be to extend WebDriver with some protocol for faking media input (there is already work underway to allow handling permission prompts via WebDriver).
Flags: needinfo?(james)
(In reply to James Graham [:jgraham] from comment #9)
> So the expectation from these tests is just set based on the result they had
> when they are imported. I don't know much about the tests in specific, but
> if there's some way we can set up the configuration so we don't get
> permission prompts and use fake devices we should do that. Is there
> something here that needs to be standardised (e.g. a standard for faking
> media devices for testing)? 

No, and to be honest, fake:true streams are useful for testing but we have no interest in standardizing them.  And compared to the "real" fake devices we use on Linux, fake:true doesn't test the real capture codepaths (which has bitten us a few times).  We've looked at fake devices for other OSes, but have never agreed on anything (ted did most of this, note).

> Chrome have been experimenting with that
> approach for writing sharable WebUSB and WebBluetooth tests. Another option
> would be to extend WebDriver with some protocol for faking media input
> (there is already work underway to allow handling permission prompts via
> WebDriver).

In theory it could be added to WebDriver; this would not be a simple thing to do.  To really test the real codepaths for capture, you need to act like a real OS device.  If you add hooks for forcing in input as if it was from a camera/mic, that would work for selenium/WebDriver, but as mentioned would be quite a bit of work (especially if you don't want it to just be what fake:true gets you - sine waves and a color-field (with coded frame number/etc data in squares).

Chrome/Webrtc.org does have some automated audio and video tests, though, so it may be worth looking at that at some point
Yeah the fake cam and mic devices are too limiting. Instead, having full-stack loopback devices, like we have on linux [1], on all platforms would be awesome.

The fake devices also predate other options for producing fake tracks: https://blog.mozilla.org/webrtc/warm-up-with-replacetrack/ shows already-web-compatible techniques to create black video and silent audio tracks, if that's all your test needs.

Bug 1088621 hopes to make our fake devices behave more like cams and mics, if it ever lands.

[1] https://dxr.mozilla.org/mozilla-central/rev/b95b1638db48fc3d450b95b98da6bcd2f9326d2f/testing/mochitest/mochitest_options.py#837
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/fde25580274f
Disable /webrtc/RTCPeerConnection-addTrack.html on osx-stylo. r=me a=testonly
leaving this open, please consider looking into the failure and fixing this in the near future :)
Keywords: leave-open
Whiteboard: [stockwell needswork] → [stockwell disabled]
Depends on: 1390521
Fixed by 1390521.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.