define a good command line interface for b2g-device builds

RESOLVED FIXED

Status

Testing
mozregression
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: parkouss, Assigned: parkouss)

Tracking

Trunk
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox44 affected)

Details

Attachments

(1 attachment)

(Assignee)

Description

2 years ago
So right now (current master branch, nothing released so far) the command line for bisecting b2g devices looks like:

# Regression finding by date range with aries-opt builds (defaults to b2g-inbound)
> mozregression --app b2g-device --bad 2015-09-10 --good 2015-09-07 --taskcluster b2g.aries-opt --artifact-name aries.zip

# Regression finding on mozilla-inbound
> mozregression --app b2g-device --bad 2015-09-10 --good 2015-09-07 --taskcluster b2g.aries-opt --artifact-name aries.zip --inbound-branch mozilla-inbound

# Flame eng opt builds with a good and bad revision
> mozregression --app b2g-device --bad-rev c4bf8c0c204434c0c83d10caf3e19eeab60a5fd4 --good-rev b93dd434b3cdc248c114f9289c37b2fb92fb6229 --taskcluster b2g.flame-kk-eng-opt --artifact-name flame-kk.zip

Thinking more about it, I would propose that we use instead:

# Regression finding by date range with aries-opt builds (defaults to b2g-inbound)
> mozregression --app b2g-aries --bad 2015-09-10 --good 2015-09-07

# Regression finding on mozilla-inbound for debug builds (reusing new option from bug 1211138)
> mozregression --app b2g-aries --build-type debug --bad 2015-09-10 --good 2015-09-07 --inbound-branch mozilla-inbound

# Flame builds with a good and bad revision
> mozregression --app b2g-flame --bad-rev c4bf8c0c204434c0c83d10caf3e19eeab60a5fd4 --good-rev b93dd434b3cdc248c114f9289c37b2fb92fb6229

So basically more app-name possible values, but command lines looks better to me.


So, looking at https://tools.taskcluster.net/index/#gecko.v2.b2g-inbound.revision.000eeb380b8a686ee5632a9a20372c6173c3a417.b2g/gecko.v2.b2g-inbound.revision.000eeb380b8a686ee5632a9a20372c6173c3a417.b2g.aries-debug you can see the possibilities. Some notes:

 - the "-debug" or "-opt" suffix is handled by the "--build-type" flag, defaulting to "opt"
 - I'm not sure what to do with all the other options, e.g. like "aries" versus "aries-en". Should we make --app-name=b2g-aries-en works, or add new flag(s) for those options ? I have no strong preference here.


See this branch for a starting implementation: https://github.com/parkouss/mozregression/tree/b2g-device-interface - more precisely, commit https://github.com/parkouss/mozregression/commit/18d0cfdbced05ad3f4104b89ef4c5a76f8d1d4d8. the implementation simple, and does the for for flame-kk, aries and emulator for now.

Please tell me what you think. :)
I think combining the app and taskcluster flags is a good idea. One thing I like about the --taskcluster flag currently is that the 'b2g.aries-opt' matches exactly the Taskcluster index (eg: https://tools.taskcluster.net/index/artifacts/#gecko.v2.mozilla-central.latest.b2g/gecko.v2.mozilla-central.latest.b2g.aries-opt ) - the "b2g.aries-opt" can be copy/pasted directly from the URL. So maybe just replacing the --app flag with what --taskcluster currently is? Does separating out the --build-type flag make things simpler?

Note I'm not tied to the above, but I just thought it would be handy to be able to look through the index in the browser, and then know exactly what the string is for the device you want to test.
(Assignee)

Comment 2

2 years ago
Well separating the --build-type flag only provide abstraction in case we want to implement that on nightlies. For now, nightlies are not using TC, so the logic to get debug builds would be different.
Plus for now we have the --app option that allow {firefox, fennec, ...}. Another solution would be to allow here {firefox-debug, fennec-debug, ...} but I like the --build-type flag that i find more intuitive (note it is possible to create a simple alias, like "-B"). Plus this is possible to add that flag in the conf file in case you are interested mainly in debug builds - for all apps.

In a general way, I would prefer to keep an abstraction over taskcluster. I think lots of people using mozregression do not care about how the builds are downloaded, and they don't want/need to - that may be different for the b2g devices usage though.

Note that if we register all flavors of b2g available builds for --app (without the debug flag), running 'mozregression -h' would list them automatically.
(Assignee)

Comment 3

2 years ago
Ok, so another possibility would be to use the different devices as different --app, and to allow different build flavors using the new --build-type. Something like:

mozregression --app b2g-flame --build-type en,debug

This would allow to use the same interface for firefox builds for example,

mozregression --app firefox --build-type asan,opt


So as you can see, allowing multiple build variation using commas (for example). This looks like TC routes, but we are not tied to it. Maybe we can replace the separator with '-' so copy pasting from TC routes (v2) would work fine (I don't find "--build-type asan-opt" easily readable for those who don't know about TC, but that's a possibility).

Then we could add a --list-build-types option, to list available build types for a given app for example.


What do you think ? (NIing :wlach and :nhirata to be sure everybody can say a word on this)
Flags: needinfo?(wlachance)
Flags: needinfo?(nhirata.bugzilla)
I like the latest proposal! It seems very intuitive to me from a user perspective. Perhaps we could have a debugging only option to see how those combinations map onto taskcluster index, if that's really needed.
Flags: needinfo?(wlachance)
(Assignee)

Comment 5

2 years ago
Created attachment 8673856 [details] [review]
good command line interface for b2g-device builds

I intend to add some unit tests soon, but this is a start.

Tell me what you think about it. Thanks!
Assignee: nobody → j.parkouss
Status: NEW → ASSIGNED
Attachment #8673856 - Flags: review?(wlachance)
Comment on attachment 8673856 [details] [review]
good command line interface for b2g-device builds

The code/design looks pretty good to me!
Attachment #8673856 - Flags: review?(wlachance) → review+
(Assignee)

Comment 7

2 years ago
Thanks Will - I added a few tests and landed the patch in https://github.com/mozilla/mozregression/commit/6c74272aff08e0f67ca680b31299901d4e76963c.

Closing this bug, please reopen it or open a new one if needed.
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
No longer blocks: 1227780
fyi, I like the proposal.  (probably too late for the chime in, though I guess it doesn't matter in the end )
Flags: needinfo?(nhirata.bugzilla)
You need to log in before you can comment on or make changes to this bug.