Open Bug 1556042 Opened 6 months ago Updated 25 days ago

Support Fenix as an app in mozregression

Categories

(Testing :: mozregression, defect, P3)

Version 3
Unspecified
Android
defect

Tracking

(Not tracked)

People

(Reporter: botond, Unassigned)

References

(Blocks 1 open bug)

Details

(Whiteboard: [geckoview:fenix:p2])

Now that there are no more Fennec nightlies built from the 69 branch, we can no longer use mozregression -n fennec to find regression ranges for Android-specific platform regressions.

The current alternative seems to be mozregression -n gve which runs the GeckoView Example app, but this app is very primitive and lacks many browser features like history.

It would be good to add Fenix as a supported app so we can find regression ranges using Fenix nightlies.

I did look at this a little while ago since it seemed likely to be useful soon, but struggled to see the needed abilities in the GitHub API to get pushes instead of just commits. Unless Fenix is brought into hg.mozilla.org I don't see this happening any time soon.

It might be easier to enhance GVE to have more features.

(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #2)

It might be easier to enhance GVE to have more features.

Won't Fenix need to have a regression window finding story anyways, for front-end regressions? Or regressions where it's not clear if it's front-end or platform?

Yeah, that's true.

In the case where it turns out to be a platform regression, the fenix regression window will likely just point to a "bump m-c from version X to version Y" commit, and then we'll need to further bisect m-c using GVE to isolate between X and Y. And if GVE doesn't have enough features to do that bisection it would still be bad. So we will need both - features in GVE to allow bisecting usefully, and the ability to bisect the final fenix product.

the fenix regression window will likely just point to a "bump m-c from version X to version Y" commit, and then we'll need to further bisect m-c using GVE to isolate between X and Y.

This is correct. Fenix currently only updates its GV snapshot every few days to a week, so mozregression will need to support GVE to bisect Gecko platform regressions on Android.

Blocks: 1509087
OS: Unspecified → Android
Whiteboard: [geckoview:fenix:p2]

The priority flag is not set for this bug.
:wlach, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(wlachance)
Flags: needinfo?(wlachance)
Priority: -- → P3

Note: it appears that fenix is being built with taskcluster these days, so at least the process of getting builds might not be incredibly difficult. You can see builds appear on treeherder here:

https://treeherder.mozilla.org/#/jobs?repo=fenix

As :Kwan mentioned in comment 1, we would indeed still have to sort out how to get mozregression to refer to github commits instead of mercurial pushes.

:catlee tells me that :jlund is the person to ask about Fenix-related automation questions. CC'ed him on this bug.

One thought I had before was that as an interim measure we could maybe do at least a date-based bisection, since that's doable with just taskcluster routes like so: https://tools.taskcluster.net/index/project.mobile.fenix.v2.nightly.2019.9.16/latest

Looking at the cadence of commits that might even be more than enough.

Of course I'm not sure how woven-in mozregression's assumptions about the existence of a pushlog are, and thus how doable that actually is.

(In reply to Ian Moody [:Kwan] (UTC+1) from comment #8)

One thought I had before was that as an interim measure we could maybe do at least a date-based bisection, since that's doable with just taskcluster routes like so: https://tools.taskcluster.net/index/project.mobile.fenix.v2.nightly.2019.9.16/latest

Looking at the cadence of commits that might even be more than enough.

That would be an excellent start! ++

Of course I'm not sure how woven-in mozregression's assumptions about the existence of a pushlog are, and thus how doable that actually is.

I suspect this part shouldn't be too bad, at worst I would think we could just create a simplified codepath for dealing with git repositories.

You need to log in before you can comment on or make changes to this bug.