Closed Bug 1189784 Opened 4 years ago Closed 4 years ago

Enable mochitests on Android 4.3

Categories

(Core :: WebRTC, defect, P1)

defect

Tracking

()

RESOLVED FIXED
Tracking Status
firefox41 --- affected
firefox42 --- affected
firefox43 --- fixed
Blocking Flags:

People

(Reporter: mreavy, Assigned: gbrown)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

Attachments

(2 files)

We need to get tests enabled on Android 4.3.  Android 4.0 tests are being turned off.
Rank: 15
Priority: -- → P1
backlog: --- → webRTC+
See the suppressions at the head of dom/media/tests/mochitest/mochitest.in

IIRC when the 2.3 tests moved to VMs there were problems getting the tests to run in that environment, and it carried over to 4.3, but my memory is a little hazy there.
See Also: → 981881
Talked to gbrown (ateam) regarding this issue.  These tests were disabled because they were failing running on emulators and we didn't have a way to run them.  He will push a patch to try to renable them and see if the failures still exist.
Depends on: 981881
See Also: 981881
test_dataChannel_basicAudioVideo.html still fails on Android 2.3 and 4.3 - bug 981881. It crashes, so it may be masking other media mochitest failures.

On 4.3, test_peerConnection_addDataChannel.html fails and test_peerConnection_addDataChannelNoBundle.html crashes, similar to bug 981881.
Depends on: 1195549
Depends on: 1195893
With the change to c3.xlarge, we can run many of these tests reliably on Android 4.3 Opt (Debug is a little different -- see next patch). There are several tests that still fail, often intermittently, often with a timing-related failure:
 - "Test timed out"
 - "Rtcp rtt 64104 < 1 min" (bug 1195549)
 - "timeout after 60s: failed to close data channel pair timerGuard"

This patch re-enables all the dom/media/tests/mochitest mochitests for Android 4.3, then disables about 14 tests that still fail on Android 4.3 Opt.
Assignee: nobody → gbrown
Attachment #8651122 - Flags: review?(gpascutto)
On Android 4.3 *Debug* many more tests hit intermittent timeouts. This patch disables those tests on 4.3 Debug only.

Running Debug tests on Android emulators has been very challenging in general: Debug runs much slower than Opt and Android runs slower than Desktop, so it's a real stretch sometimes. We usually run what we can on Android Debug but don't worry too much about the tests that are too demanding.

Here's a try run with both patches applied: https://treeherder.mozilla.org/#/jobs?repo=try&revision=5a20c6805a62
Attachment #8651124 - Flags: review?(gpascutto)
Comment on attachment 8651124 [details] [diff] [review]
update mochitest.ini for Android 4.3 debug

Review of attachment 8651124 [details] [diff] [review]:
-----------------------------------------------------------------

I assume there's another part of this that's actually enabling the rests? Because this just adds more skip-ifs and it's supposed to enable tests.
Attachment #8651124 - Flags: review?(gpascutto) → review+
Yes, the first patch (comment 5) removes the global skip-if which was disabling all dom/media tests on 4.3 Opt and 4.3 Debug.
Attachment #8651122 - Flags: review?(gpascutto) → review+
Keywords: leave-open
:mreavy -- With the patches that are landing now, most tests in dom/media/tests/mochitest will start running on Android 4.3. There are a few tests in there that remain disabled on Android 4.3 Opt (comment 5) and several more that are disabled on Android 4.3 Debug (comment 6). That's the best that I can do. I'm leaving this bug open for follow-up on the remaining disabled tests; for that, I think https://bugzilla.mozilla.org/show_bug.cgi?id=1195549#c4 would be a great place to start.
Assignee: gbrown → nobody
Flags: needinfo?(mreavy)
Whiteboard: [test disabled on android 4.3]
Thanks, Geoff, for moving this forward!  Instead of leaving this bug open, can you instead file a follow up bug listing all the tests we want to get re-enabled?  (I'll likely wind up making it a meta.)
Flags: needinfo?(mreavy) → needinfo?(gbrown)
See Also: → 1200411
(In reply to Maire Reavy [:mreavy] (Plz needinfo me) from comment #12)
> Thanks, Geoff, for moving this forward!  Instead of leaving this bug open,
> can you instead file a follow up bug listing all the tests we want to get
> re-enabled?  (I'll likely wind up making it a meta.)

Filed bug 1200411.
Assignee: nobody → gbrown
Flags: needinfo?(gbrown)
Keywords: leave-open
Whiteboard: [test disabled on android 4.3]
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Blocks: 1321547
You need to log in before you can comment on or make changes to this bug.