Run reftests on GeckoView x86_64
Categories
(Firefox for Android Graveyard :: Testing, enhancement, P2)
Tracking
(firefox68 wontfix, firefox69 fixed)
People
(Reporter: cpeterson, Assigned: kats)
References
(Depends on 3 open bugs)
Details
(Whiteboard: [geckoview:fenix:p2])
Attachments
(4 files)
We currently run reftests on Android 4.3 (Fennec), Bitbar ARM32 (Moto G5), and Packet x86. We'd like to run them on our new GeckoView platforms where possible: Bitbar ARM64 (Pixel 2) Packet x86-64 (which is bug 1460411) There are some reftests failures in bug 1494388 on Packet x86.
Reporter | ||
Comment 1•6 years ago
|
||
bc, do we run reftests on Bitbar's Moto G5 devices now? My test matrix notes [1] say yes, but in today's mobile testing meeting, I think you said that reftests and mochitests are impractical for real devices because they would take hours to run. If so, I can WONTFIX this bug. [1] https://docs.google.com/spreadsheets/d/17hp38bKgyLYs-ydqM0kauOgwr13ytYxh3aii6RrIBAk/edit#gid=0&range=E17
Comment 2•6 years ago
|
||
No. we only run raptor-speedometer-geckoview and raptor-unity-webgl-geckoview on moto-g5.
Updated•6 years ago
|
Reporter | ||
Comment 3•6 years ago
|
||
OK. We can make this bug P5 to keep it open as unfixed but probably not something we will spend time on.
Reporter | ||
Updated•6 years ago
|
Comment 4•5 years ago
|
||
reftest (reftest-plain) takes a long time -- impractical for real devices, but we would like to see them running on packet.net, against the x86_64 build.
Comment 5•5 years ago
|
||
Reporter | ||
Comment 6•5 years ago
|
||
snorp says he wants to postpone fixing GV reftests until we have Android WebRender (bug 1525314).
Reporter | ||
Comment 7•5 years ago
|
||
With Fennec (and its reftest coverage) moving to ESR 68 and GV needing to support devices that are not WebRender-capable, we probably need to green up the reftests on GV soon.
Reporter | ||
Comment 8•5 years ago
|
||
Kats estimates only about 35% of Android devices will be WebRender compatible in phase 1 (Android 7+ with GL ES 3.2) based on https://developer.android.com/about/dashboards.
Kats also says: "the vast majority of the [GV reftest failures] just need fuzzing which should be pretty easy to green up. There are some larger differences that will need investigation though. Given that the GV gfx codepath is a hybrid between existing Fennec and desktop codepaths, I wouldn't expect there to be much in the way actual failures, and if there are, they're likely due to misplaced ifdefs somewhere which would valuable to fix for production GV anyway."
Assignee | ||
Comment 9•5 years ago
|
||
Here's a new try push to see what's failing now: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b41e9f13f604e16743c7f451c0a4700323338669
Assignee | ||
Comment 10•5 years ago
|
||
Taking this since I'm kind of working on it already.
Assignee | ||
Comment 11•5 years ago
|
||
I did one round of annotation based on the opt failures in the above try push and re-pushed: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=182ee572e222a771f3515e9b9dd601043a43cb0b
Either I made a lot of typos or the results are nondeterministic. I'll keep iterating.
Assignee | ||
Comment 12•5 years ago
|
||
I'm expecting this to be all green: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=0ea2a5d71d396a33115b0142203a289b8574ef54 - if it is, patches will be up shortly.
I filed bugs hanging off this one for the various actual failures.
Assignee | ||
Comment 13•5 years ago
|
||
A few more intermittent fuzzes came crawling out, I updated my patches locally to account for those.
Assignee | ||
Comment 14•5 years ago
|
||
Assignee | ||
Comment 15•5 years ago
|
||
There are a number of failures, for which I've filed separate bugs.
And then a lot of fuzziness. I manually inspected the reftest analyzer
results on try pushes to distinguish failures vs fuzziness.
Depends on D34537
Assignee | ||
Comment 16•5 years ago
|
||
The fuzziness in the position-dynamic-changes reftests seems nondeterministic.
The fuzziness annotations in the previous patch were what I got after a few
iterations of do-a-try-push-and-update-annotations, but there are still more
failures showing up in subsequent try pushes. I visually checked all the
failures and they are all just fuzzy in different places, but intermittent.
This patch updates the fuzziness annotations on these tests to the maximum
that I encountered on any test, which is (2, 1382).
I'm keeping this as a separate patch because I think it might be valuable
in version control history to have the actual numbers seen on try which
are in the previous patch.
Depends on D34538
Assignee | ||
Comment 17•5 years ago
|
||
Depends on D34539
Comment 18•5 years ago
|
||
Pushed by kgupta@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/679372a486c0 Add a geckoview condition to the reftest sandbox. r=gbrown https://hg.mozilla.org/integration/autoland/rev/c3b63aae0869 Mark geckoview failures. r=gbrown https://hg.mozilla.org/integration/autoland/rev/1f61c85281d4 Increase position-dynamic-changes to a blanket fuzz. r=gbrown https://hg.mozilla.org/integration/autoland/rev/e954e6e848d1 Run reftests on GeckoView. r=gbrown
Comment 19•5 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/679372a486c0
https://hg.mozilla.org/mozilla-central/rev/c3b63aae0869
https://hg.mozilla.org/mozilla-central/rev/1f61c85281d4
https://hg.mozilla.org/mozilla-central/rev/e954e6e848d1
Reporter | ||
Comment 20•5 years ago
|
||
68=wontfix because we don't need to enable these tests in GV 68 Beta.
Updated•3 years ago
|
Description
•