Get webrender reftests and crashtests running on android emulator
Categories
(Core :: Graphics: WebRender, task)
Tracking
()
| Tracking | Status | |
|---|---|---|
| firefox72 | --- | fixed |
People
(Reporter: jnicol, Assigned: jnicol)
References
Details
Attachments
(3 files)
Bug 1525314 enabled reftests and crashtests on Pixel 2 devices, but didn't enable them on the android emulator. This was due to lots of issues at the time. However, the situation has since improved, due to upgrading the emulator version, switching to the gpu backend (rather than swiftshader) and a couple of bugfixes/workarounds. So let's try to enable them now!
Initial try run here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=ef9787747362ac83ee2c1014471cc6ad74ca51cb
It looks like we have:
- a couple of crashes, maybe emulator bugs.
- a few unexpected passes - from tests marked
fail-if(geckoview&&webrender)because they are broken on the pixel 2, but seem to work on the emulator - some fuzziness - either needing increased or decreased compared to the pixel 2
- and a few additional failures
but overall it looks in okay shape.
|
Assignee | |
Comment 1•5 years ago
|
||
One notable thing is that for some tests we have different results between the emulator and pixel 2. Either in terms of amount of fuzz, or simply passing/failing. On the wrench reftests we are able to say fails-if(emulator) or fuzzy-if(device). Is that possible, or even desirable, to be able to do for regular non-wrench reftests?
|
||
Comment 2•5 years ago
|
||
Non-wrench reftests have Android, AndroidVersion, and geckoview in the sandbox today, but nothing for emulator vs device. We have gotten by like that, in part because implementing emulator detection is a little tricky: wrench relies on WRENCH_REFTEST_CONDITION_ env vars, and mochitest uses https://searchfox.org/mozilla-central/rev/8a63fc190b39ed6951abb4aef4a56487a43962bc/testing/mochitest/runtestsremote.py#104. Adding that capability for reftests is possible and would give us an option for writing more accurate android annotations.
|
|
Assignee | |
Comment 3•5 years ago
•
|
||
Thanks for the link. If you don't disagree I think I'll try to add that capability? - For the fuzzy ones it's probably not as important, but for ones which outright work on the emulator and don't on devices, I'm not sure we have any other option. Otherwise we'll get unexpected passes or fails depending on the platform, or we mark the tests as random or disable them, which don't seem like great options either.
Especially if I'm unable to find a fix for the crashing tests - it would be a shame to have to disable the tests on devices because of a potential emulator bug causing a crash.
|
|
||
Comment 4•5 years ago
|
||
(In reply to Jamie Nicol [:jnicol] from comment #3)
Thanks for the link. If you don't disagree I think I'll try to add that capability?
That would be great. I'm happy to review.
|
|
Assignee | |
Comment 5•5 years ago
|
||
Add new test platforms "android-em-7.0-x86_64-qr/opt" and
"android-em-7.0-x86_64-qr/debug", which run a new test-set
"android-x86_64-qr-tests", which includes reftests and crashtests.
Depends on D51294
|
|
Assignee | |
Comment 6•5 years ago
|
||
Depends on D51295
|
|
Assignee | |
Comment 7•5 years ago
|
||
Depends on D51296
|
|
Assignee | |
Comment 8•5 years ago
|
||
Thanks for the reviews Geoff. I've been holding off on landing until bug 1594303 is fixed, as they are intermittently hitting that assertion.
Pushed by jnicol@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/0effdf5676a9 Run reftests and crashtests on android emulator with webrender enabled. r=gbrown https://hg.mozilla.org/integration/autoland/rev/1f760cf44ba3 Add "emulator" and "device" conditions to reftest sandbox. r=gbrown https://hg.mozilla.org/integration/autoland/rev/b7c4186eb089 Update reftest expectations for webrender on android emulator. r=gbrown
|
||
Comment 10•5 years ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/0effdf5676a9
https://hg.mozilla.org/mozilla-central/rev/1f760cf44ba3
https://hg.mozilla.org/mozilla-central/rev/b7c4186eb089
Description
•