Closed Bug 1613690 Opened 5 years ago Closed 5 years ago

Webrtc mochitests (mda mochitest jobs) don't run at all on android

Categories

(Core :: WebRTC: Signaling, defect, P1)

Product:
Core
Component:
WebRTC: Signaling
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla75
Tracking Flags:
Tracking Status
firefox75 --- fixed

People

(Reporter: bwc, Assigned: bwc)

References

Details

Attachments

(3 files)

Bug 1613690: Get mochitest-media running on android-em again. r?jmaher,jib,bryce
47 bytes, text/x-phabricator-request
Details | Review
Bug 1613690: Re-enable some tests on android, and update some bug numbers. r?dminor
47 bytes, text/x-phabricator-request
Details | Review
Bug 1613690: Detect support for h264 in this test instead of making assumptions based on platform. r?dminor
47 bytes, text/x-phabricator-request
Details | Review

I am not sure when this happened, but the mda mochitest jobs aren't scheduled on android-em builds any more. We do appear to be scheduling the webrtc web-platform-tests, so at least we have some test coverage for webrtc on android, but this is still pretty bad.

I ran the mochitests on a local emulator this week and everything looks fine, but we should get this fixed before things start breaking.

I should note that passing --full to mach try fuzzy does not yield any android-em mochitest-media jobs either.

It looks like jobs for the media subsuite were simply never added to the geckoview-based android platform.

Hi Joel, do you know if this was a deliberate decision to conserve testing resources or an oversight? Thanks!

Flags: needinfo?(jmaher)

we run tests on physical phones (not emulators):
https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=android%2Cmda&fromchange=9c90cbf62eeeff797a9ca582cf4051b6c3405c1c

and on mozilla-central opt/debug for the aarch64 platform.

in addition we do run mda tests on win10/aarch64.

is there a strong desire to run these on emulators, or more to just ensure coverage on try prior to landing?

Flags: needinfo?(jmaher) → needinfo?(docfaraday)

To answer your question, the same failure that tripped up the landing of bug 1591199 on android-hw also happens on android-em on the push in comment 5:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=601d69122d8a9f092afe286156bec59802a35d4a&selectedJob=287786135

So, android-em coverage on try looks valuable to me.

Flags: needinfo?(docfaraday)

:gbrown, thoughts on enabling mda jobs for android emulators? we should do something to get try coverage and there appears to be a correlation between em and hw; maybe em for try only will be close enough?

Flags: needinfo?(gbrown)

Our original plan for packet.net + bitbar android testing was to run each suite on either android-em or android-hw, but not both: We would run suites requiring or benefiting from hardware on android-hw, and the rest on android-em. mochitest-media was identified as something that should run on hardware, so it is only run on android-hw.

In case anyone is unaware, since android-hw resources are limited, android-hw tests are normally hidden from view in 'mach try fuzzy', etc, but can be selected with 'mach try fuzzy --full': mochitest-media can be run on android-hw this way, today.

Running mochitest-media on both android-hw and android-em seems slightly wasteful to me, but if anyone feels there is value in that, I am okay with it.

Flags: needinfo?(gbrown)

I would be happy to have this stuff run only on android-hw once we have enough test-devices to run them by default on try. Don't forget, ./mach try fuzzy still cannot be used to run all of the testing we need for webrtc; we're still stuck using try strings when we want to run all of our testing, unless we want to run every test an unconstrained ./mach try fuzzy --full selects. Which we don't.

For now, I'd like to have these running on android-em.

So, what's the best way to write a mochitest.ini skip-if for android-em, but not android-hw? There's some cleanup that needs doing in our mochitest.ini that I'd like to do.

Flags: needinfo?(jmaher)

mochitest supports "is_emulator" in manifest annotations, like:

https://searchfox.org/mozilla-central/rev/3811b11b5773c1dccfe8228bfc7143b10a9a2a99/dom/media/tests/mochitest/mochitest.ini#168

Flags: needinfo?(jmaher)

Thanks!

Have an intermittent failure on that try push. Retriggering to see how frequent it is.

Blocks: 1610975

Seems fairly frequent. Will either need to disable, or fix. Looking into it.

Yeah, that test case is racy, and can pass ICE candidates too early. Should not be hard to fix.

Actually, this looks like an H264 problem.

(In reply to Byron Campen [:bwc] from comment #22)

Actually, this looks like an H264 problem.

We don't built a fake h.264 plugin on Android, so we don't normally run h.264 tests there. I guess when running on hardware that may not be an issue if we are using the hardware codec.

Yeah, that's what I'm seeing.

Assignee: nobody → docfaraday

Try pushes look ok.

Pushed by bcampen@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/17dc195c8126 Get mochitest-media running on android-em again. r=jmaher,jib,bryce https://hg.mozilla.org/integration/autoland/rev/1eee4a9a497c Re-enable some tests on android, and update some bug numbers. r=dminor https://hg.mozilla.org/integration/autoland/rev/70d4ff6dd267 Detect support for h264 in this test instead of making assumptions based on platform. r=dminor
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: