Closed Bug 1501582 (geckoview_reftests) Opened Last year Closed 4 months ago

Run reftests on GeckoView x86_64

Categories

(Firefox for Android :: Testing, enhancement, P2)

x86_64
Android
enhancement

Tracking

()

RESOLVED FIXED
Firefox 69
Tracking Status
firefox68 --- wontfix
firefox69 --- fixed

People

(Reporter: cpeterson, Assigned: kats)

References

(Depends on 5 open bugs, Regressed 1 open bug)

Details

(Whiteboard: [geckoview:fenix:p2])

Attachments

(4 files)

We currently run reftests on Android 4.3 (Fennec), Bitbar ARM32 (Moto G5), and Packet x86. We'd like to run them on our new GeckoView platforms where possible:

Bitbar ARM64 (Pixel 2)
Packet x86-64 (which is bug 1460411)

There are some reftests failures in bug 1494388 on Packet x86.
bc, do we run reftests on Bitbar's Moto G5 devices now? My test matrix notes [1] say yes, but in today's mobile testing meeting, I think you said that reftests and mochitests are impractical for real devices because they would take hours to run.

If so, I can WONTFIX this bug.

[1] https://docs.google.com/spreadsheets/d/17hp38bKgyLYs-ydqM0kauOgwr13ytYxh3aii6RrIBAk/edit#gid=0&range=E17
Flags: needinfo?(bob)
Hardware: Unspecified → ARM64
No. we only run raptor-speedometer-geckoview and raptor-unity-webgl-geckoview on moto-g5.
Flags: needinfo?(bob)
Priority: -- → P3
OK. We can make this bug P5 to keep it open as unfixed but probably not something we will spend time on.
Priority: P3 → P5
Hardware: ARM64 → ARM
Summary: Run reftests for GeckoView on Bitbar ARM64 (Pixel 2) → Run reftests for GeckoView on Bitbar
Whiteboard: [geckoview] → [geckoview:p5]

reftest (reftest-plain) takes a long time -- impractical for real devices, but we would like to see them running on packet.net, against the x86_64 build.

Alias: geckoview_reftests
Summary: Run reftests for GeckoView on Bitbar → Run reftests for GeckoView on x86_64 packet.net
See Also: → 1525314

snorp says he wants to postpone fixing GV reftests until we have Android WebRender (bug 1525314).

Depends on: 1525314
See Also: 1525314

With Fennec (and its reftest coverage) moving to ESR 68 and GV needing to support devices that are not WebRender-capable, we probably need to green up the reftests on GV soon.

Priority: P5 → P2
Hardware: ARM → x86_64
Summary: Run reftests for GeckoView on x86_64 packet.net → Run reftests on GeckoView x86_64
Whiteboard: [geckoview:p5] → [geckoview:fenix:p2]

Kats estimates only about 35% of Android devices will be WebRender compatible in phase 1 (Android 7+ with GL ES 3.2) based on https://developer.android.com/about/dashboards.

Kats also says: "the vast majority of the [GV reftest failures] just need fuzzing which should be pretty easy to green up. There are some larger differences that will need investigation though. Given that the GV gfx codepath is a hybrid between existing Fennec and desktop codepaths, I wouldn't expect there to be much in the way actual failures, and if there are, they're likely due to misplaced ifdefs somewhere which would valuable to fix for production GV anyway."

Taking this since I'm kind of working on it already.

Assignee: nobody → kats

I did one round of annotation based on the opt failures in the above try push and re-pushed: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=182ee572e222a771f3515e9b9dd601043a43cb0b

Either I made a lot of typos or the results are nondeterministic. I'll keep iterating.

Depends on: 1558285
Depends on: 1558286

I'm expecting this to be all green: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=0ea2a5d71d396a33115b0142203a289b8574ef54 - if it is, patches will be up shortly.

I filed bugs hanging off this one for the various actual failures.

A few more intermittent fuzzes came crawling out, I updated my patches locally to account for those.

There are a number of failures, for which I've filed separate bugs.
And then a lot of fuzziness. I manually inspected the reftest analyzer
results on try pushes to distinguish failures vs fuzziness.

Depends on D34537

The fuzziness in the position-dynamic-changes reftests seems nondeterministic.
The fuzziness annotations in the previous patch were what I got after a few
iterations of do-a-try-push-and-update-annotations, but there are still more
failures showing up in subsequent try pushes. I visually checked all the
failures and they are all just fuzzy in different places, but intermittent.
This patch updates the fuzziness annotations on these tests to the maximum
that I encountered on any test, which is (2, 1382).

I'm keeping this as a separate patch because I think it might be valuable
in version control history to have the actual numbers seen on try which
are in the previous patch.

Depends on D34538

Pushed by kgupta@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/679372a486c0
Add a geckoview condition to the reftest sandbox. r=gbrown
https://hg.mozilla.org/integration/autoland/rev/c3b63aae0869
Mark geckoview failures. r=gbrown
https://hg.mozilla.org/integration/autoland/rev/1f61c85281d4
Increase position-dynamic-changes to a blanket fuzz. r=gbrown
https://hg.mozilla.org/integration/autoland/rev/e954e6e848d1
Run reftests on GeckoView. r=gbrown

68=wontfix because we don't need to enable these tests in GV 68 Beta.

You need to log in before you can comment on or make changes to this bug.