1189784 - Enable mochitests on Android 4.3

Status: RESOLVED FIXED

(Core :: WebRTC, defect, P1)

Description

[Maire Reavy [:mreavy]]

We need to get tests enabled on Android 4.3.  Android 4.0 tests are being turned off.

Comment 1

[Randell Jesup [:jesup] (needinfo me)]

See the suppressions at the head of dom/media/tests/mochitest/mochitest.in

IIRC when the 2.3 tests moved to VMs there were problems getting the tests to run in that environment, and it carried over to 4.3, but my memory is a little hazy there.

Comment 2

[Kim Moir [:kmoir] ET]

Talked to gbrown (ateam) regarding this issue.  These tests were disabled because they were failing running on emulators and we didn't have a way to run them.  He will push a patch to try to renable them and see if the failures still exist.

Comment 3

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=40701771833f

Comment 4

[Geoff Brown [:gbrown]]

test_dataChannel_basicAudioVideo.html still fails on Android 2.3 and 4.3 - bug 981881. It crashes, so it may be masking other media mochitest failures.

On 4.3, test_peerConnection_addDataChannel.html fails and test_peerConnection_addDataChannelNoBundle.html crashes, similar to bug 981881.

Comment 5

[Geoff Brown [:gbrown]]

Attachment: update mochitest.ini for Android 4.3 opt

With the change to c3.xlarge, we can run many of these tests reliably on Android 4.3 Opt (Debug is a little different -- see next patch). There are several tests that still fail, often intermittently, often with a timing-related failure:
 - "Test timed out"
 - "Rtcp rtt 64104 < 1 min" (bug 1195549)
 - "timeout after 60s: failed to close data channel pair timerGuard"

This patch re-enables all the dom/media/tests/mochitest mochitests for Android 4.3, then disables about 14 tests that still fail on Android 4.3 Opt.

Comment 6

[Geoff Brown [:gbrown]]

Attachment: update mochitest.ini for Android 4.3 debug

On Android 4.3 *Debug* many more tests hit intermittent timeouts. This patch disables those tests on 4.3 Debug only.

Running Debug tests on Android emulators has been very challenging in general: Debug runs much slower than Opt and Android runs slower than Desktop, so it's a real stretch sometimes. We usually run what we can on Android Debug but don't worry too much about the tests that are too demanding.

Here's a try run with both patches applied: https://treeherder.mozilla.org/#/jobs?repo=try&revision=5a20c6805a62

Comment 7

[Gian-Carlo Pascutto [:gcp]]

Comment on attachment 8651124 [details] [diff] [review]
update mochitest.ini for Android 4.3 debug

Review of attachment 8651124 [details] [diff] [review]:
-----------------------------------------------------------------

I assume there's another part of this that's actually enabling the rests? Because this just adds more skip-ifs and it's supposed to enable tests.

Comment 8

[Geoff Brown [:gbrown]]

Yes, the first patch (comment 5) removes the global skip-if which was disabling all dom/media tests on 4.3 Opt and 4.3 Debug.

Comment 9

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/907068a4b1e1
https://hg.mozilla.org/integration/mozilla-inbound/rev/55322eeb1425

Comment 10

[Geoff Brown [:gbrown]]

:mreavy -- With the patches that are landing now, most tests in dom/media/tests/mochitest will start running on Android 4.3. There are a few tests in there that remain disabled on Android 4.3 Opt (comment 5) and several more that are disabled on Android 4.3 Debug (comment 6). That's the best that I can do. I'm leaving this bug open for follow-up on the remaining disabled tests; for that, I think https://bugzilla.mozilla.org/show_bug.cgi?id=1195549#c4 would be a great place to start.

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/907068a4b1e1
https://hg.mozilla.org/mozilla-central/rev/55322eeb1425

Comment 12

[Maire Reavy [:mreavy]]

Thanks, Geoff, for moving this forward!  Instead of leaving this bug open, can you instead file a follow up bug listing all the tests we want to get re-enabled?  (I'll likely wind up making it a meta.)

Comment 13

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/0c18d9d5bbab

Comment 14

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/0c18d9d5bbab

Comment 15

[Geoff Brown [:gbrown]]

(In reply to Maire Reavy [:mreavy] (Plz needinfo me) from comment #12)
> Thanks, Geoff, for moving this forward!  Instead of leaving this bug open,
> can you instead file a follow up bug listing all the tests we want to get
> re-enabled?  (I'll likely wind up making it a meta.)

Filed bug 1200411.

Comment 16

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/044b9ffab63c

Comment 17

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/a3635b06f95c

Comment 18

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/044b9ffab63c
https://hg.mozilla.org/mozilla-central/rev/a3635b06f95c