1501582 - (geckoview_reftests) Run reftests on GeckoView x86_64

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Testing, enhancement, P2)

Description

[Chris Peterson [:cpeterson]]

We currently run reftests on Android 4.3 (Fennec), Bitbar ARM32 (Moto G5), and Packet x86. We'd like to run them on our new GeckoView platforms where possible:

Bitbar ARM64 (Pixel 2)
Packet x86-64 (which is bug 1460411)

There are some reftests failures in bug 1494388 on Packet x86.

Comment 1

[Chris Peterson [:cpeterson]]

bc, do we run reftests on Bitbar's Moto G5 devices now? My test matrix notes [1] say yes, but in today's mobile testing meeting, I think you said that reftests and mochitests are impractical for real devices because they would take hours to run.

If so, I can WONTFIX this bug.

[1] https://docs.google.com/spreadsheets/d/17hp38bKgyLYs-ydqM0kauOgwr13ytYxh3aii6RrIBAk/edit#gid=0&range=E17

Comment 2

[Bob Clary [:bc] (inactive)]

No. we only run raptor-speedometer-geckoview and raptor-unity-webgl-geckoview on moto-g5.

Comment 3

[Chris Peterson [:cpeterson]]

OK. We can make this bug P5 to keep it open as unfixed but probably not something we will spend time on.

Comment 4

[Geoff Brown [:gbrown]]

reftest (reftest-plain) takes a long time -- impractical for real devices, but we would like to see them running on packet.net, against the x86_64 build.

Comment 5

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c0c6c419a4a1ac769217198dc4d08c0007ba99ee

Comment 6

[Chris Peterson [:cpeterson]]

snorp says he wants to postpone fixing GV reftests until we have Android WebRender (bug 1525314).

Comment 7

[Chris Peterson [:cpeterson]]

With Fennec (and its reftest coverage) moving to ESR 68 and GV needing to support devices that are not WebRender-capable, we probably need to green up the reftests on GV soon.

Comment 8

[Chris Peterson [:cpeterson]]

Kats estimates only about 35% of Android devices will be WebRender compatible in phase 1 (Android 7+ with GL ES 3.2) based on https://developer.android.com/about/dashboards.

Kats also says: "the vast majority of the [GV reftest failures] just need fuzzing which should be pretty easy to green up. There are some larger differences that will need investigation though. Given that the GV gfx codepath is a hybrid between existing Fennec and desktop codepaths, I wouldn't expect there to be much in the way actual failures, and if there are, they're likely due to misplaced ifdefs somewhere which would valuable to fix for production GV anyway."

Comment 9

[Kartikaya Gupta (email:[e-mail])]

Here's a new try push to see what's failing now: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b41e9f13f604e16743c7f451c0a4700323338669

Comment 10

[Kartikaya Gupta (email:[e-mail])]

Taking this since I'm kind of working on it already.

Comment 11

[Kartikaya Gupta (email:[e-mail])]

I did one round of annotation based on the opt failures in the above try push and re-pushed: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=182ee572e222a771f3515e9b9dd601043a43cb0b

Either I made a lot of typos or the results are nondeterministic. I'll keep iterating.

Comment 12

[Kartikaya Gupta (email:[e-mail])]

I'm expecting this to be all green: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=0ea2a5d71d396a33115b0142203a289b8574ef54 - if it is, patches will be up shortly.

I filed bugs hanging off this one for the various actual failures.

Comment 13

[Kartikaya Gupta (email:[e-mail])]

A few more intermittent fuzzes came crawling out, I updated my patches locally to account for those.

Comment 14

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1501582 - Add a geckoview condition to the reftest sandbox. r?gbrown

Comment 15

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1501582 - Mark geckoview failures. r?gbrown

There are a number of failures, for which I've filed separate bugs.
And then a lot of fuzziness. I manually inspected the reftest analyzer
results on try pushes to distinguish failures vs fuzziness.

Depends on D34537

Comment 16

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1501582 - Increase position-dynamic-changes to a blanket fuzz. r?gbrown

The fuzziness in the position-dynamic-changes reftests seems nondeterministic.
The fuzziness annotations in the previous patch were what I got after a few
iterations of do-a-try-push-and-update-annotations, but there are still more
failures showing up in subsequent try pushes. I visually checked all the
failures and they are all just fuzzy in different places, but intermittent.
This patch updates the fuzziness annotations on these tests to the maximum
that I encountered on any test, which is (2, 1382).

I'm keeping this as a separate patch because I think it might be valuable
in version control history to have the actual numbers seen on try which
are in the previous patch.

Depends on D34538

Comment 17

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1501582 - Run reftests on GeckoView. r?gbrown

Depends on D34539

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/679372a486c0
Add a geckoview condition to the reftest sandbox. r=gbrown
https://hg.mozilla.org/integration/autoland/rev/c3b63aae0869
Mark geckoview failures. r=gbrown
https://hg.mozilla.org/integration/autoland/rev/1f61c85281d4
Increase position-dynamic-changes to a blanket fuzz. r=gbrown
https://hg.mozilla.org/integration/autoland/rev/e954e6e848d1
Run reftests on GeckoView. r=gbrown

Comment 19

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/679372a486c0
https://hg.mozilla.org/mozilla-central/rev/c3b63aae0869
https://hg.mozilla.org/mozilla-central/rev/1f61c85281d4
https://hg.mozilla.org/mozilla-central/rev/e954e6e848d1

Comment 20

[Chris Peterson [:cpeterson]]

68=wontfix because we don't need to enable these tests in GV 68 Beta.