1321547 - Improve WebRTC test coverage on Android

Status: RESOLVED INCOMPLETE

(Core :: WebRTC, defect, P3)

Description

[Dan Minor [:dminor]]

Looking at the manifest, we have the following tests disabled on Autophone (excluding screen share, which isn't implemented for Android):

test_dataChannel_basicAudioVideoCombined.html
test_getUserMedia_audioCapture.html
test_peerConnection_basicAudioVideoCombined.html
test_peerConnection_basicH264Video.html
test_peerConnection_bug1042791.html
test_peerConnection_capturedVideo.html
test_peerConnection_certificates.html
test_peerConnection_forwarding_basicAudioVideoCombined.html
test_peerConnection_iceFailure.html
test_peerConnection_offerRequiresReceiveVideoAudio.html
test_peerConnection_relayOnly.html
test_peerConnection_replaceTrack.html
test_peerConnection_scaleResolution.html
test_peerConnection_trackDisabling.html
test_peerConnection_twoAudioVideoStreamsCombined.html
identity/test_fingerprints.html
identity/test_getIdentityAssertion.html
identity/test_idpproxy.html]
identity/test_loginNeeded.html
identity/test_peerConnection_asymmetricIsolation.html
identity/test_peerConnection_peerIdentity.html
identity/test_setIdentityProvider.html
identity/test_setIdentityProviderWithErrors.html

We have bugs on file for all of these, although in some cases, multiple disabled tests are tracked by the same bug.

Comment 1

[Dan Minor [:dminor]]

Randell, please let me know if there is anything here that you would like me to look at first. Thanks!

Making this a P1 as I think getting at least some of these enabled will be a Q1 goal for me.

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

replaceTrack, basicAudioVideoCombined, scaleResolution, basicH264 probably.

Most of these aren't really directly dependent on Android, and the only advantage of testing directly is android can expose timing problems (and perf) we don't see on other platforms.

Comment 3

[Dan Minor [:dminor]]

I ran the tests with each failing test enabled individually using the mochitest harness directly. The following tests fail in that case:

dom/media/tests/mochitest/test_getUserMedia_audioCapture.html (times out)

test_peerConnection_basicH264Video.html:

FAIL H.264 should be present in the SDP offer
FAIL H.264 codec is present in SDP
FAIL H.264 codec is present in SDP
FAIL All expected tracks have been observed
expected: {"{9a9a720f-b618-4800-a2f9-8d461fc201f8}":{"type":"video","streamId":"{e6af154f-6164-459c-9fca-cc94a5695614}"}}
observed: {}
FAIL ICE username is present in SDP
FAIL ICE password is present in SDP
FAIL H.264 codec is present in SDP
FAIL RTCP Mux is offered in SDP
FAIL ICE username is present in SDP
FAIL ICE password is present in SDP
FAIL H.264 codec is present in SDP
FAIL RTCP Mux is offered in SDP
FAIL PeerConnectionWrapper (pcRemote): legal ICE state transition from new to failed
FAIL PeerConnectionWrapper (pcLocal): legal ICE state transition from new to failed
FAIL Error in test execution: TypeError: e is undefined execute/<@http://mochi.test:8888/tests/dom/media/tests/mochitest/head.js:751:19 ...

test_peerConnection_bug1042791.html
-------------------------------------------------------------
FAIL H.264 should be present in the SDP offer
FAIL H264 offer contains profile-level-id=42e0

test_peerConnection_iceFailure.html (times out)

test_peerConnection_scaleResolution.html (times out)

The intersection of this set and Randell's is basicH264, so I think I'll start there.

I haven't yet rooted my phone, so it's possible that some of the failures that don't reproduce locally will reproduce once the phone is rooted and the tests are run through Autophone.

Comment 4

[Dan Minor [:dminor]]

Attachment: Update autophone webrtc manifests

:bc, this might be redundant with your work to get rid of these manifests, but on the other hand, it might be worth turning these on to see if any problems show up in the meantime, I'm not sure how close you are to landing.

try run here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=c49df13eb73f7774f64063dc078c80ef4de0a49e

Comment 5

[Bob Clary [:bc] (inactive)]

I'll look in a bit but I'm *really* close to landing the dom media stuff which would eliminate the autophone webrtc manifests altogether. Come to mobile/testing today and we can chat.

Comment 6

[Bob Clary [:bc] (inactive)]

FYI, I ran these tests against my local devices using the soon to be available
manifests from bug 1358118. Mdm2 are the dom/media/tests/mochitest/mochitest.ini tests and Mdm4 are the dom/media/tests/mochitest/identity/mochitest.ini tests. Note I have the wrong bug number in the try commit messages.

https://treeherder.allizom.org/#/jobs?repo=try&revision=c426ff84ea0587f607c0e4f6b8275603193ed86c

Comment 7

[Bob Clary [:bc] (inactive)]

Attachment: 1321547-errors.txt

I ended up running the tests at least twice and on some devices more times due to wifi / device disconnection issues. This is a rough summary of the failures.

Comment 8

[Bob Clary [:bc] (inactive)]

Comment on attachment 8857521 [details]
Update autophone webrtc manifests

>diff --git a/testing/mochitest/manifests/autophone-webrtc.ini b/testing/mochitest/manifests/autophone-webrtc.ini
>--- a/testing/mochitest/manifests/autophone-webrtc.ini
>+++ b/testing/mochitest/manifests/autophone-webrtc.ini

> [../tests/dom/media/tests/mochitest/test_dataChannel_basicAudioVideoCombined.html]
>-skip-if = true  # Bug 1189784

Ok.

> [../tests/dom/media/tests/mochitest/test_peerConnection_basicAudioVideoCombined.html]
>-skip-if = true  # Bug 1189784

Ok.

> [../tests/dom/media/tests/mochitest/test_peerConnection_basicH264Video.html]
>-skip-if = true # Bug 1149374
>+skip-if = true # Bug 1355786

Ok.

> [../tests/dom/media/tests/mochitest/test_peerConnection_bug1042791.html]
>-skip-if = true # Bug 1149374
>+skip-if = true # Bug 1355786

Ok.

> [../tests/dom/media/tests/mochitest/test_peerConnection_forwarding_basicAudioVideoCombined.html]
>-skip-if = true  # Bug 1189784

Not ok.

> [../tests/dom/media/tests/mochitest/test_peerConnection_replaceTrack.html]
>-skip-if = true  # Bug 1189784

Ok.

> [../tests/dom/media/tests/mochitest/test_peerConnection_twoAudioVideoStreamsCombined.html]
>-skip-if = true  # Bug 1189784

Ok.

All of the identity tests are Ok.

Update dom/media/tests/mochitest/mochitest.ini and continue to skip test_peerConnection_forwarding_basicAudioVideoCombined.html or fix it and I think we will be good to go. r? me when you're ready and I'll hit it asap.

Comment 9

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

This is an assigned P1 bug without activity in two weeks. 

If you intend to continue working on this bug for the current release/iteration/sprint, remove the 'stale-bug' keyword.

Otherwise we'll reset the priority of the bug back to '--' on Monday, August 28th.

Comment 10

[Bulk Bug Changes for mreavy's org]

Mass change P2->P3 to align with new Mozilla triage process.

Comment 11

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:dminor, maybe it's time to close this bug?

Comment 12

[Dan Minor [:dminor]]

We probably do want to improve WebRTC test coverage on Android at some point, so I'll reset the bug to NEW rather than closing it.