Closed Bug 1205560 Opened 9 years ago Closed 6 years ago

start to integrate b2g device bisection in mozregression

Categories

(Testing :: mozregression, defect)

Product:
Testing
Component:
mozregression
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED INVALID

People

(Reporter: parkouss, Unassigned)

References

Details

Attachments

(1 file)

Basic support for manual b2g device downloads
49 bytes, text/x-github-pull-request
parkouss
: review+
Details | Review 
Well it should be easy to add into mozregression a basic bisection for b2g devices builds.

For now we can start by only downloading builds (or ask the user to do that, because of bug 1204109) and ask the user to install manually the build on the device. Still it would be an easier process!

So b2g devices is a bit different than our other apps, we have no nightlies for example, we only bisect inbounds.

:mshal already have something working for this, https://github.com/mshal/mozregression/tree/b2g-device.

I would go to integrate that soon, we will iterate to make it better over time.
I'm not sure if this totally fits the existing schemes. I think we want to be able to use TC indexed builds both by date or by revision range, so I made the Taskcluster config try to use both the Inbound (for revision range) and the Nightly (for date range, which gets converted to a good/bad revision) in cli.py, but the actual regression finding uses the inbound regression logic with the good/bad revisions.

The existing tests work, but I haven't added new tests yet.
Attachment #8662493 - Flags: review?(j.parkouss)
Comment on attachment 8662493 [details] [review]
Basic support for manual b2g device downloads

Thanks a lot :mshal! This looks good to me - mozregression was not structured for this exactly, but I'm totally fine to clean things later.

I noted some things on the PR, it would be awesome if you could address that. Also, we use flake8 on travis-ci to check our code style (sorry I forgot to say that!), and there is some long lines in there, https://travis-ci.org/mozilla/mozregression/builds/80872187#L443. Could you address that too ?

Thank you!
Attachment #8662493 - Flags: review?(j.parkouss) → review+
:nhirata, we are providing two options to find b2g device builds to bisect in mozregression, e.g:

--taskcluster:
  the Taskcluster build product/name/type, such as 'b2g.aries-opt'
--artifact-name:
  the Taskcluster artifact name to look for, such as 'aries.zip'

For now they have no defaults (they are required). Maybe you know what could be a good default for them ? What you (or most people if you know that) most often use ?
Flags: needinfo?(nhirata.bugzilla)
Oh!  This is a wonderful bug!

+Askeing; FYI.  We can probably all work together in getting b2g device builds working for mozregression.

For Aries we currently test on the dogfood builds.  For inbound regression testing, it might be worth it to use engineering builds in the even that we need to profile or something.  ie : https://tools.taskcluster.net/index/artifacts/#gecko.v2.mozilla-inbound.revision.0021aceac846d189cbdaf38e845cac52a0d917de.b2g/gecko.v2.mozilla-inbound.revision.0021aceac846d189cbdaf38e845cac52a0d917de.b2g.aries-eng-opt

I think we should be able to specify the platform(device, simulator) ->  device (flame, aries) or simulator (emulator, b2g desktop, mulet, simulator) -> build type (userdebug, eng) -> inbound (b2g-inbound, fxteam, mozilla-inbound)

Also as a requirement, we need to be able to save these builds to directory (whether local or an attached server folder) in some sort of formatted way so if one wanted, they could potentially build a local server that has the builds in the event that they need to recheck the window.
Flags: needinfo?(fyen)
Testing the patch : 
Build_type isn't defined either:

nhirata-19333:mozr nhirata$ mozregression --bad 2015-09-10 --good 2015-09-07 --taskcluster b2g.aries-opt --artifact-name aries.zip
**********
You should use a config file. Please use the --write-config command line flag to help you create one.
**********

Using good rev 5bc2bf65bcf59e01fed8aa3f5de0c0f7df7b2e77 for date 2015-09-07
Using bad rev 0a18e5db95a4505d26260e6690a3e6241f0e16de for date 2015-09-10
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/bin/mozregression", line 9, in <module>
    load_entry_point('mozregression==1.0.1', 'console_scripts', 'mozregression')()
  File "build/bdist.macosx-10.6-x86_64/egg/mozregression/main.py", line 266, in main
  File "build/bdist.macosx-10.6-x86_64/egg/mozregression/cli.py", line 445, in validate
AttributeError: 'FirefoxConfig' object has no attribute 'set_build_type'
(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #4)
> Oh!  This is a wonderful bug!

Eh, glad to know that!

> I think we should be able to specify the platform(device, simulator) -> 
> device (flame, aries) or simulator (emulator, b2g desktop, mulet, simulator)
> -> build type (userdebug, eng) -> inbound (b2g-inbound, fxteam,
> mozilla-inbound)

Well, b2g desktop is already supported, with the --app=b2g flag, and the inbound
branch choice also with --inbound-branch (defaulting to b2g-inbound for these cases).

for the other choices, I wonder if the options we are adding are sufficient ?
(--taskcluster and --artifact-name) I don't know yet what we can get from taskcluster
and if we can get everything required from here also.
 
> Also as a requirement, we need to be able to save these builds to directory
> (whether local or an attached server folder) in some sort of formatted way
> so if one wanted, they could potentially build a local server that has the
> builds in the event that they need to recheck the window.

If I understand well, this is what the --persist flag already do (it store build files locally).
Still we can't download builds yet (bug 1204109) but this should be easy to add after that.
Maybe we can combine Askeing's tool with the code that we have here?  I made a comment in the other bug.
I've updated the PR with the review feedback & made it flake8 compatible. Do you get notified of that automatically? It looks like GitHub updated it, but I don't know if I need to re-request it or something.
Flags: needinfo?(j.parkouss)
(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #5)
> Testing the patch : 
> Build_type isn't defined either:
> 
> nhirata-19333:mozr nhirata$ mozregression --bad 2015-09-10 --good 2015-09-07
> --taskcluster b2g.aries-opt --artifact-name aries.zip

You also need to specify the '--app b2g-device' flag, otherwise it assumes you are doing Firefox desktop builds. Some examples:

# Regression finding by date range with aries-opt builds (defaults to b2g-inbound)
mozregression --app b2g-device --bad 2015-09-10 --good 2015-09-07 --taskcluster b2g.aries-opt --artifact-name aries.zip

# Regression finding on mozilla-inbound
mozregression --app b2g-device --bad 2015-09-10 --good 2015-09-07 --taskcluster b2g.aries-opt --artifact-name aries.zip --inbound-branch mozilla-inbound

# Flame eng opt builds with a good and bad revision
mozregression --app b2g-device --bad-rev c4bf8c0c204434c0c83d10caf3e19eeab60a5fd4 --good-rev b93dd434b3cdc248c114f9289c37b2fb92fb6229 --taskcluster b2g.flame-kk-eng-opt --artifact-name flame-kk.zip
(In reply to Michael Shal [:mshal] from comment #8)
> I've updated the PR with the review feedback & made it flake8 compatible. Do
> you get notified of that automatically? It looks like GitHub updated it, but
> I don't know if I need to re-request it or something.

So no, github do not send mail when a PR changes - so thanks to let me know that :) And thanks for the patch update!

I merged the code as https://github.com/mozilla/mozregression/commit/efdf1ee79591f6a36cab4f2cea004911c5e338fc, then added a few more things related to that:

 - fixed the error seen by :nhirata by looking at the app name used to know if this is b2g-device
 - fixed an error that prevented to print the push log url for b2g devices

and a few other minor things, see the changelog for details. I did the changes since I know
well the code of mozreg, (sorry :mshal I should have noticed that before!).

So now we now have a basic integration of b2 devices bisection in mozregression. Though I still have
a few questions for the the arguments we added (see comment 3):

 - I would really like to provide default values for those, so basically for most users it will become possible to do "mozregression --app b2g-device --bad 2015-09-10 --good 2015-09-07". From what I can understand from comment 4, we could use "b2g.aries-eng-opt" and "aries.zip". (Also we should add as a follow up the ability to store user customized values in the config file).

 - Do you guys think that the option names are intuitive ? (they are for b2g device only, so at least I would add that information in the help message associated, but maybe we can find more appropriate names)

NI :wlach for the option names here: since you know well mozregression command line usage, maybe you have ideas :)

Also, I feel like bisecting b2g devices should be a bit explained on the mozregression gh-pages documentation, maybe
some examples like :mshal gave in comment 9. That would be a good thing to do once we are sure about the options.
Flags: needinfo?(j.parkouss) → needinfo?(wlachance)
(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #7)
> Maybe we can combine Askeing's tool with the code that we have here?  I made
> a comment in the other bug.

That looks really useful, thanks! That need investigation for sure, we can talk about that in the other bug
and/or in a follow up mozregression bug - I would like o keep things as simple as possible for this bug, i.e. only provide a basic integration (just pointing taskcluster urls, and let the user download/test manually for now).
Sorry for the delay in replying to this.

It would IMO be nice if we could avoid having to specify anything to do with taskcluster in the commandline arguments. That's an implementation detail which ideally people using this wouldn't have to care about. At least maybe have some kind of simplified option covering the common case (like, e.g. --device-type aries).

Documenting some common workflows on mozilla.github.io/mozregression would be great. :)
Flags: needinfo?(wlachance)
I agree William.  The interface should be transparent to the end user.

Having said that, I will shoot for working first, before working out the finer details.  :)
It's getting there.

The mozregression tool should allow for the Task Cluster credentials like the python util that Askeing created.  Could you incorporate that too please?  Otherwise it just lists a bunch of paths.
Flags: needinfo?(j.parkouss)
(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #14)
> It's getting there.
> 
> The mozregression tool should allow for the Task Cluster credentials like
> the python util that Askeing created.  Could you incorporate that too
> please?  Otherwise it just lists a bunch of paths.

Yep, sounds really doable, I think a good place for credentials could be in our ~/.mozilla/mozregression/mozregression.cfg. I will work on this (unless someone wants to), however I won't be able to test it as I am a contributor, and I don't have a @mozilla.com mail address (required by https://auth.taskcluster.net/).
Flags: needinfo?(j.parkouss)
Depends on: 1207631
Depends on: 1207662
1) Julien, I can test that portion for you.  :) Selene explained the dependencies.  It will change so that LDAP authentication takes place eventually.


2) We also need to be able to determine the appropriate inbound : inbound (b2g-inbound, fxteam, mozilla-inbound).  We tried to find a regression range, the problem is that the issue comes from a mozilla-inbound not b2g-inbound so we have this huge range still to sort through without means to find the appropriate build.

Just ping me with a Needinfo when it's ready and I will test again.
Flags: needinfo?(nhirata.bugzilla) → needinfo?(j.parkouss)
(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #16)
> 1) Julien, I can test that portion for you.  :) Selene explained the
> dependencies.  It will change so that LDAP authentication takes place
> eventually.

Cool thanks ! There is a PR in bug 1207662 comment 1 that you can try. I asked
:wlach first, but please go ahead!


> 2) We also need to be able to determine the appropriate inbound : inbound
> (b2g-inbound, fxteam, mozilla-inbound).  We tried to find a regression
> range, the problem is that the issue comes from a mozilla-inbound not
> b2g-inbound so we have this huge range still to sort through without means
> to find the appropriate build.

So yeah, just use the --inbound-branch=mozilla-inbound flag then (see comment 9).
Flags: needinfo?(j.parkouss) → needinfo?(nhirata.bugzilla)
Ok!  Thanks Julien!

We also need to be able to access Mozilla Central (to check to see if we need to go to which inbound for a narrower view):
mozregression --bad 2015-09-10 --good 2015-09-07 --taskcluster b2g.aries-opt --artifact-name aries.zip --repo mozilla-central --app b2g-device

This didn't quite work...
Flags: needinfo?(nhirata.bugzilla)
(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #18)
> Ok!  Thanks Julien!

You're welcome!

> We also need to be able to access Mozilla Central (to check to see if we
> need to go to which inbound for a narrower view):
> mozregression --bad 2015-09-10 --good 2015-09-07 --taskcluster b2g.aries-opt
> --artifact-name aries.zip --repo mozilla-central --app b2g-device
> 
> This didn't quite work...

Yes, this is not taken into account, because for now we use taskcluster only to get "inbound" builds, and we crawl https://archive.mozilla.org/ for "nightlies".
But for b2g device, only the taskcluster way works, so this needs a trick. I will look into that.

Ultimately, we should probably think into how mozregression should be used - at least once we use taskcluster for eveything, we should consider using mozregression to bisect builds on a given branch - in that case, using dates or changesets won't change anything but the way to choose the range, and the distinction --repo versus --inbound-branch should go away. This seems really interesting and will need to be discussed! But this is out of that bug;

Thanks Naoki for the report!
Depends on: 1208745
Naoki, see bug 1208745 for making at least mozilla-central work.
FYI : https://docs.google.com/document/d/1UmtV6IQzddV_1JZhGI5CboKJy2OcyrKqSOrX7-k52GI/edit

The Regression range hunting is a bit different on B2G because we have to deal with the merging of fx-team, b2g-inbound, and mozilla-inbound.

This doc talks about starting at mozilla-central, having said that down towards the bottom, you'll see that we have to check where the patch landed first.  I'm not sure if there's a systematic way of figuring it out; picking the wrong inbound will get you the wrong regression window is what I found out.
BTW, thanks for the quick patches!  I'll take another look soon.
On b2g, when developers land a patch they land it on inbound first, and then later they're merged into other inbounds and central. That's why we backtrack this process by starting in central.

An example on a Gaia landing - bug 1198976 comment 8
An example on a Gecko (b2g-inbound) landing - bug 1121463 comment 43
An example on a Gecko (mozilla-inbound) landing - bug 1097464 comment 89

For Gaia issues they always land on b2g-inbound. For Gecko issues we read the pushlog from mozilla-central and determine which inbound to go further. 

Another important step that this tool is missing is that we manually swap Gecko and Gaia to find out whether it's a Gecko or Gaia issue.

Eventually you need to find out the which one of the three examples I listed above is your bug: start the regression range in central, do a swap to determine whether it's a Gecko or Gaia issue, progress further in whichever inbound that leads you to, and get the final inbound pushlog.

If you don't do this and just start from any inbound you could end up with a long pushlog that contains back-merges from another inbound or from central, or you could be looking at a Gecko pushlog full of merges from Gaia when you should be looking at a Gaia pushlog.
Thanks Pi Wei, this is plenty of great information.

(In reply to Pi Wei Cheng [:piwei] from comment #23)
> Another important step that this tool is missing is that we manually swap
> Gecko and Gaia to find out whether it's a Gecko or Gaia issue.
> 
> Eventually you need to find out the which one of the three examples I listed
> above is your bug: start the regression range in central, do a swap to
> determine whether it's a Gecko or Gaia issue, progress further in whichever
> inbound that leads you to, and get the final inbound pushlog.
> 
> If you don't do this and just start from any inbound you could end up with a
> long pushlog that contains back-merges from another inbound or from central,
> or you could be looking at a Gecko pushlog full of merges from Gaia when you
> should be looking at a Gaia pushlog.

So, if I understand well, we need to first test if the bug comes from gaia or Gecko. I suppose this is two distinct artifacts.

What we could do about that is first try a single changeset or date, using the --launch mozregression flag. E.G, something like:

> mozregression --launch 2015-09-10 --taskcluster b2g.aries-opt --artifact-name aries.zip --repo mozilla-central --app b2g-device

then hit the same command again to test the gaia or gecko build, just changing the artifact name (I must confess I don't know which one "aries.zip" is).

Once you know that, this time bisect on m-c, using the artifact name you want. Then bisect on the right inbound (for now user defined, but maybe we could do better later).

So basically mozregression would help to:

1. download gecko and gaia artifacts for one given build (using --launch)
2. search on m-c given a bisection range and a given artifact
3. reduce the regression range by searching for a bisection on the right inbound

So still manual steps, but we could improve that later.

Does that make sense ?


Note that --launch probably does not work for b2g-device yet, but that would be easy to fix. Also we should try to give more meaningful (simplified?) options and option names than --artifact-name and --taskcluster, but we can address that later.
> 1. download gecko and gaia artifacts for one given build

This tool would help us QA in finding regression windows via Taskcluster the most if it could auto-download builds for us and label them by their build IDs.

> 2. search on m-c given a bisection range and a given artifact

I've only used this tool once so unfamiliar with it, but does the tool correct itself if user inputs a wrong initial guess of a range? Say one's "fairly sure" the bug doesn't happen last week but it actually does, will this tool correct this initial range?

> 3. reduce the regression range by searching for a bisection on the right inbound

The auto-generated pushlog is a great feature as well, but I think the rest needs user to manually test/confirm.

It would be great if the tool does this:

1) Ask user which date of build to download (instead of giving a range, because that range is usually wrong) and from where (central, mozilla-inbound, b2g-inbound, or fx-team). Once user gives a date, the tool downloads gaia and gecko artifacts from mozilla central and label them in a folder named by their build ID

2) User gives 'repro' or 'no repro' results until they're on the smallest window possible on the given branch. And the tool asks user to swap their Gecko/Gaia within their last working/first broken build. User will enter 'Gaia' or 'Gecko' based on their swap results, and the tool generates a central pushlog based on it.

3) Repeat the same process from step 1, but this is up to the user as they could ask the tool to download from which branch.
Correction on this part on comment 25:

> User will enter 'Gaia' or 'Gecko' based on their swap results, and the tool generates a central pushlog based on it.

Remove the word central in there. It should be whichever branch pushlog the user asked at step 1 and has given at step 2. I'm confusing myself when writing this but this is all very self explanatory when we do it.
Depends on: 1211185
Flags: needinfo?(fyen) → needinfo?(sho)
I think we can just simply leverage the taskcluster-util-python written by Askeing, you can ref the repo here : https://github.com/askeing/taskcluster-util-python. Call the taskcluster_download command and providing the right credential for it, then the build should be downloaded.

If that still bother you, we could have send patch for this ..
Let us know if you need any help.
Flags: needinfo?(sho)
Well, downloading private artifacts is done in bug 1207662. We should be there already, but I can't test it since I am a contributor - So I'm waiting for someone to give it a try. :) I think :wlach will help me for that when he will be back if nobody can try before (should be next week).
Hm, not sure we need indexing by date - :mshal already implemented something to find changesets for dates (this is based on json pushes api, http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmo/pushlog.html#pushloghtml-command.

Though that won't hurt, I believe we could save a few requests (and maybe be more accurate ?)
mozregression should continue to use the pushlog for getting a revision list based on the date range as it does now. The indexing-by-date is more for human browsing convenience, not for automation purposes.
No longer blocks: 1227780
I think we can close this bug as soon as bug 1232873 is fixed. We will then have a basic b2g device support, with downloads and gecko/gaia distinction at the end.

OFC we will have other things to do to to make it nicer (automatic flashing, looking more at the emulator support) - but still I believe we will have then something usable for aries and flame devices.
Depends on: 1232873
No longer depends on: 1201771
B2G is gone.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → INVALID
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: