Webrtc mochitests (mda mochitest jobs) don't run at all on android
Categories
(Core :: WebRTC: Signaling, defect, P1)
Tracking
()
Tracking | Status | |
---|---|---|
firefox75 | --- | fixed |
People
(Reporter: bwc, Assigned: bwc)
References
Details
Attachments
(3 files)
I am not sure when this happened, but the mda mochitest jobs aren't scheduled on android-em builds any more. We do appear to be scheduling the webrtc web-platform-tests, so at least we have some test coverage for webrtc on android, but this is still pretty bad.
Comment 1•4 years ago
|
||
I ran the mochitests on a local emulator this week and everything looks fine, but we should get this fixed before things start breaking.
Assignee | ||
Comment 2•4 years ago
|
||
I should note that passing --full to mach try fuzzy does not yield any android-em mochitest-media jobs either.
Assignee | ||
Comment 3•4 years ago
|
||
It looks like jobs for the media subsuite were simply never added to the geckoview-based android platform.
Comment 4•4 years ago
|
||
Hi Joel, do you know if this was a deliberate decision to conserve testing resources or an oversight? Thanks!
Assignee | ||
Comment 5•4 years ago
|
||
Comment 6•4 years ago
|
||
we run tests on physical phones (not emulators):
https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=android%2Cmda&fromchange=9c90cbf62eeeff797a9ca582cf4051b6c3405c1c
and on mozilla-central opt/debug for the aarch64 platform.
in addition we do run mda tests on win10/aarch64.
is there a strong desire to run these on emulators, or more to just ensure coverage on try prior to landing?
Assignee | ||
Comment 7•4 years ago
|
||
To answer your question, the same failure that tripped up the landing of bug 1591199 on android-hw also happens on android-em on the push in comment 5:
So, android-em coverage on try looks valuable to me.
Comment 8•4 years ago
|
||
:gbrown, thoughts on enabling mda jobs for android emulators? we should do something to get try coverage and there appears to be a correlation between em and hw; maybe em for try only will be close enough?
Assignee | ||
Comment 9•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=151f753c0d2353b18cc9a0e7d12ed4d041cb7094
Comment 10•4 years ago
|
||
Our original plan for packet.net + bitbar android testing was to run each suite on either android-em or android-hw, but not both: We would run suites requiring or benefiting from hardware on android-hw, and the rest on android-em. mochitest-media was identified as something that should run on hardware, so it is only run on android-hw.
In case anyone is unaware, since android-hw resources are limited, android-hw tests are normally hidden from view in 'mach try fuzzy', etc, but can be selected with 'mach try fuzzy --full': mochitest-media can be run on android-hw this way, today.
Running mochitest-media on both android-hw and android-em seems slightly wasteful to me, but if anyone feels there is value in that, I am okay with it.
Assignee | ||
Comment 11•4 years ago
•
|
||
I would be happy to have this stuff run only on android-hw once we have enough test-devices to run them by default on try. Don't forget, ./mach try fuzzy still cannot be used to run all of the testing we need for webrtc; we're still stuck using try strings when we want to run all of our testing, unless we want to run every test an unconstrained ./mach try fuzzy --full selects. Which we don't.
For now, I'd like to have these running on android-em.
Assignee | ||
Comment 12•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=6c2531cf97576c7837ddd39a2d7b0c1394c4967e
Assignee | ||
Comment 13•4 years ago
|
||
So, what's the best way to write a mochitest.ini skip-if for android-em, but not android-hw? There's some cleanup that needs doing in our mochitest.ini that I'd like to do.
Assignee | ||
Comment 14•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5b7511ee51cc16b88b2358289ada8cea7586f94a
Comment 15•4 years ago
|
||
mochitest supports "is_emulator" in manifest annotations, like:
Assignee | ||
Comment 16•4 years ago
|
||
Thanks!
Assignee | ||
Comment 17•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=699dfe204e03556943a95aaf68a085440dfa7055
Assignee | ||
Comment 18•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8783415c46b411d356ebfcabb5b434b15f70e42d
Assignee | ||
Comment 19•4 years ago
|
||
Have an intermittent failure on that try push. Retriggering to see how frequent it is.
Assignee | ||
Comment 20•4 years ago
|
||
Seems fairly frequent. Will either need to disable, or fix. Looking into it.
Assignee | ||
Comment 21•4 years ago
|
||
Yeah, that test case is racy, and can pass ICE candidates too early. Should not be hard to fix.
Assignee | ||
Comment 22•4 years ago
|
||
Actually, this looks like an H264 problem.
Comment 23•4 years ago
|
||
(In reply to Byron Campen [:bwc] from comment #22)
Actually, this looks like an H264 problem.
We don't built a fake h.264 plugin on Android, so we don't normally run h.264 tests there. I guess when running on hardware that may not be an issue if we are using the hardware codec.
Assignee | ||
Comment 24•4 years ago
|
||
Yeah, that's what I'm seeing.
Assignee | ||
Comment 25•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d920605f24bc228ceae5a448586e0d2a14c14edf
Assignee | ||
Comment 26•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a4990e44bc45f256a0f0ef96000424025a07fa3b
Assignee | ||
Comment 27•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a622d0bc362172af1f2048dbcffc448a8c6bf3a0
Assignee | ||
Comment 28•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=34337080641f4a5b7b90adee3e63d9521baad979
Assignee | ||
Comment 29•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=35755d775db41593f47c605d813c7735b5879add
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Comment 30•4 years ago
|
||
Try pushes look ok.
Assignee | ||
Comment 31•4 years ago
|
||
Assignee | ||
Comment 32•4 years ago
|
||
Depends on D63195
Assignee | ||
Comment 33•4 years ago
|
||
Depends on D63196
Assignee | ||
Comment 34•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=dd34b9cc1acf8e9ee595337b88a28c877852505d
Comment 35•4 years ago
|
||
Pushed by bcampen@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/17dc195c8126 Get mochitest-media running on android-em again. r=jmaher,jib,bryce https://hg.mozilla.org/integration/autoland/rev/1eee4a9a497c Re-enable some tests on android, and update some bug numbers. r=dminor https://hg.mozilla.org/integration/autoland/rev/70d4ff6dd267 Detect support for h264 in this test instead of making assumptions based on platform. r=dminor
Comment 36•4 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/17dc195c8126
https://hg.mozilla.org/mozilla-central/rev/1eee4a9a497c
https://hg.mozilla.org/mozilla-central/rev/70d4ff6dd267
Description
•