1211185 - define a good command line interface for b2g-device builds

Status: RESOLVED FIXED

(Testing :: mozregression, defect)

Description

[Julien Pagès (:parkouss)]

So right now (current master branch, nothing released so far) the command line for bisecting b2g devices looks like:

# Regression finding by date range with aries-opt builds (defaults to b2g-inbound)
> mozregression --app b2g-device --bad 2015-09-10 --good 2015-09-07 --taskcluster b2g.aries-opt --artifact-name aries.zip

# Regression finding on mozilla-inbound
> mozregression --app b2g-device --bad 2015-09-10 --good 2015-09-07 --taskcluster b2g.aries-opt --artifact-name aries.zip --inbound-branch mozilla-inbound

# Flame eng opt builds with a good and bad revision
> mozregression --app b2g-device --bad-rev c4bf8c0c204434c0c83d10caf3e19eeab60a5fd4 --good-rev b93dd434b3cdc248c114f9289c37b2fb92fb6229 --taskcluster b2g.flame-kk-eng-opt --artifact-name flame-kk.zip

Thinking more about it, I would propose that we use instead:

# Regression finding by date range with aries-opt builds (defaults to b2g-inbound)
> mozregression --app b2g-aries --bad 2015-09-10 --good 2015-09-07

# Regression finding on mozilla-inbound for debug builds (reusing new option from bug 1211138)
> mozregression --app b2g-aries --build-type debug --bad 2015-09-10 --good 2015-09-07 --inbound-branch mozilla-inbound

# Flame builds with a good and bad revision
> mozregression --app b2g-flame --bad-rev c4bf8c0c204434c0c83d10caf3e19eeab60a5fd4 --good-rev b93dd434b3cdc248c114f9289c37b2fb92fb6229

So basically more app-name possible values, but command lines looks better to me.


So, looking at https://tools.taskcluster.net/index/#gecko.v2.b2g-inbound.revision.000eeb380b8a686ee5632a9a20372c6173c3a417.b2g/gecko.v2.b2g-inbound.revision.000eeb380b8a686ee5632a9a20372c6173c3a417.b2g.aries-debug you can see the possibilities. Some notes:

 - the "-debug" or "-opt" suffix is handled by the "--build-type" flag, defaulting to "opt"
 - I'm not sure what to do with all the other options, e.g. like "aries" versus "aries-en". Should we make --app-name=b2g-aries-en works, or add new flag(s) for those options ? I have no strong preference here.


See this branch for a starting implementation: https://github.com/parkouss/mozregression/tree/b2g-device-interface - more precisely, commit https://github.com/parkouss/mozregression/commit/18d0cfdbced05ad3f4104b89ef4c5a76f8d1d4d8. the implementation simple, and does the for for flame-kk, aries and emulator for now.

Please tell me what you think. :)

Comment 1

[Michael Shal [:mshal]]

I think combining the app and taskcluster flags is a good idea. One thing I like about the --taskcluster flag currently is that the 'b2g.aries-opt' matches exactly the Taskcluster index (eg: https://tools.taskcluster.net/index/artifacts/#gecko.v2.mozilla-central.latest.b2g/gecko.v2.mozilla-central.latest.b2g.aries-opt ) - the "b2g.aries-opt" can be copy/pasted directly from the URL. So maybe just replacing the --app flag with what --taskcluster currently is? Does separating out the --build-type flag make things simpler?

Note I'm not tied to the above, but I just thought it would be handy to be able to look through the index in the browser, and then know exactly what the string is for the device you want to test.

Comment 2

[Julien Pagès (:parkouss)]

Well separating the --build-type flag only provide abstraction in case we want to implement that on nightlies. For now, nightlies are not using TC, so the logic to get debug builds would be different.
Plus for now we have the --app option that allow {firefox, fennec, ...}. Another solution would be to allow here {firefox-debug, fennec-debug, ...} but I like the --build-type flag that i find more intuitive (note it is possible to create a simple alias, like "-B"). Plus this is possible to add that flag in the conf file in case you are interested mainly in debug builds - for all apps.

In a general way, I would prefer to keep an abstraction over taskcluster. I think lots of people using mozregression do not care about how the builds are downloaded, and they don't want/need to - that may be different for the b2g devices usage though.

Note that if we register all flavors of b2g available builds for --app (without the debug flag), running 'mozregression -h' would list them automatically.

Comment 3

[Julien Pagès (:parkouss)]

Ok, so another possibility would be to use the different devices as different --app, and to allow different build flavors using the new --build-type. Something like:

mozregression --app b2g-flame --build-type en,debug

This would allow to use the same interface for firefox builds for example,

mozregression --app firefox --build-type asan,opt


So as you can see, allowing multiple build variation using commas (for example). This looks like TC routes, but we are not tied to it. Maybe we can replace the separator with '-' so copy pasting from TC routes (v2) would work fine (I don't find "--build-type asan-opt" easily readable for those who don't know about TC, but that's a possibility).

Then we could add a --list-build-types option, to list available build types for a given app for example.


What do you think ? (NIing :wlach and :nhirata to be sure everybody can say a word on this)

Comment 4

[William Lachance (:wlach)]

I like the latest proposal! It seems very intuitive to me from a user perspective. Perhaps we could have a debugging only option to see how those combinations map onto taskcluster index, if that's really needed.

Comment 5

[Julien Pagès (:parkouss)]

Attachment: good command line interface for b2g-device builds

I intend to add some unit tests soon, but this is a start.

Tell me what you think about it. Thanks!

Comment 6

[William Lachance (:wlach)]

Comment on attachment 8673856 [details] [review]
good command line interface for b2g-device builds

The code/design looks pretty good to me!

Comment 7

[Julien Pagès (:parkouss)]

Thanks Will - I added a few tests and landed the patch in https://github.com/mozilla/mozregression/commit/6c74272aff08e0f67ca680b31299901d4e76963c.

Closing this bug, please reopen it or open a new one if needed.

Comment 8

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

fyi, I like the proposal.  (probably too late for the chime in, though I guess it doesn't matter in the end )