1194941 - mozregression feature request: Offer a way to launch a specific build

Status: RESOLVED FIXED

(Testing :: mozregression, defect)

Description

[Ray Satiro]

I think it would be good to have a way to run a specific build that mozregression has identified as the bad build. Or any build really. I think such a feature would be useful if someone wanted to launch a specific commit, for example:
mozregresssion --launch c458465578e0

I recently used mozregression trying to identify when HSTS was enabled by default. It ended like this:

10:04.20 LOG: MainThread Bisector INFO Narrowed inbound regression window from [eafffe75, 57273aac] (3 revisions) to [104b0bbd, 57273aac] (2 revisions) (~1 steps left)
10:04.20 LOG: MainThread Bisector INFO Oh noes, no (more) inbound revisions :(
10:04.20 LOG: MainThread Bisector INFO Last good revision: 104b0bbd714f
10:04.20 LOG: MainThread Bisector INFO First bad revision: 57273aac7996
10:04.20 LOG: MainThread Bisector INFO Pushlog: https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=104b0bbd714f&tochange=57273aac7996

Now I don't understand how 57273aac7996 caused HSTS by default and I thought the first time maybe I just made a mistake on whether a build was good or bad. An easy way to check this would have been if I could have done something to confirm like 
mozregression --launch 104b0bbd714f
confirm it's good
mozregression --launch 57273aac7996
confirm it's bad
But I couldn't do that so I ran mozregression again to make sure I didn't make a mistake and that took more time.

There may very well be an easy way to do this, I'm just not experienced with these tools. As far as the more generic way goes, say someone says they're running Aurora 20150811004008 and they see some problem. Well I'd like to be able to quickly launch that to see if I can reproduce the problem, like
mozregression --launch 20150811004008

Comment 1

[Julien Pagès (:parkouss)]

Hmm, this was possible before but we dropped that because there is mozdownload:

https://github.com/mozilla/mozdownload

I think this may help - though you may have to do a manual install.

Looking at your use case, I actually like the --launch argument. And I am thinking now that this could be useful.

Comment 2

[Julien Pagès (:parkouss)]

@Will

I'm thinking now that this can be useful, and I am working on making this easy to integrate.

I actually like the proposal to trigger that using a simple flag. --launch is good to me, but that can be something else if you have another idea.

I also like the idea that this flag should be able to take a date or a changeset. This won't be hard to parse and this is powerful for the user (not having to use another flag).

Of course, this flag should use every other option defined on the command line (--app, --repo, and so on). I looked at the code, it will be easy.

So Will, thoughts on this ?

Comment 3

[William Lachance (:wlach)]

This makes a lot of sense to me! I think we should do it.

Comment 4

[Julien Pagès (:parkouss)]

So , Mikeling is interested for that bug, and I think it will be good for him to learn more of mozregression's internals.

This can only be addressed once Bug 1197744 and Bug 1197568 are resolved. I will give a lot of details to fix that in order to help Mikeling.

Required changes:

1. In cli.py file: add the --launch argument. if this option is defined by the user, we should try to convert it to a date (using parse_date). If it is a success, update options.launch to be that date instance, else, leave the raw string and we would assume it is an inbound changeset. In any case if that option is defined we should set the action attribute of the configuration to "launch".

2. in main.py file: We should add a method 'launch()' in the Application class. In there:
 - create a NightlyFetchInfo or InboundFetchInfo class dependending on the value of the options.launch argument (nightly if it is a date, Inbound in the other case).
 - use that fetch_info object to get the BuildInfo (using the appropriate method)
 - give that build_info object to the build_downloader (using focus_download), so it will download the build file if necessary
 - then give that build_info to the test_runner.run_once method (Bug 1197744) and return that error code.

Comment 5

[Julien Pagès (:parkouss)]

MikeLing, as discussed I assign that to you. :) Feel free to ask if needed!

Comment 6

[MikeLing]

(In reply to Julien Pagès from comment #4)

> 2. in main.py file: We should add a method 'launch()' in the Application
> class. In there:
>  - create a NightlyFetchInfo or InboundFetchInfo class dependending on the
> value of the options.launch argument (nightly if it is a date, Inbound in
> the other case).

Could you tell me more detail about this? What kind of info I need to fetch in order to get the right build data? Something like https.get('build_url')?

Comment 7

[MikeLing]

Don't we have method to get changeset url already(https://github.com/mozilla/mozregression/blob/master/mozregression/bisector.py#L75)?

Comment 8

[Julien Pagès (:parkouss)]

(In reply to MikeLing from comment #6)
> (In reply to Julien Pagès from comment #4)
> 
> > 2. in main.py file: We should add a method 'launch()' in the Application
> > class. In there:
> >  - create a NightlyFetchInfo or InboundFetchInfo class dependending on the
> > value of the options.launch argument (nightly if it is a date, Inbound in
> > the other case).
> 
> Could you tell me more detail about this? What kind of info I need to fetch
> in order to get the right build data? Something like https.get('build_url')?

So the info you need to fetch are available with the FetchInfo classes (in the fetch_build_info module).
You can use them this way (for a nightly):

from datetime import date
from mozregression.fetch_build_info import NightlyFetchInfo

# the fetch_build_info will help us to find all the information
# we need to know about a build. This information is a build_info object (see the build_info.py file)
fetch_build_info = NightlyFetchInfo(fetch_config)
build_info = fetch_build_info.find_build_info(date(2015, 10, 08))

# here you have a build_info. it contains the url for example.
# now you give that to the downloader, then to the launcher to respectively download if required and run the build.

Comment 9

[Julien Pagès (:parkouss)]

Some precisions:

to run the build, don't use the launcher directly, use the test_runner (in the Application class).

It has a new method for that, run_once:

https://github.com/mozilla/mozregression/blob/master/mozregression/test_runner.py#L120

Also the NightlyFetchInfo has a new param in find_build_info that you should set to True for your call:

https://github.com/mozilla/mozregression/blob/master/mozregression/fetch_build_info.py#L90

Comment 10

[MikeLing]

Attachment: PR for bug 1184941

Please tell me if anything else need to be addressed. Thank you for your help :)

Comment 11

[MikeLing]

Furthermore, I would like to write a test for it if it's ok :)

Comment 12

[Julien Pagès (:parkouss)]

Comment on attachment 8656968 [details] [review]
PR for bug 1184941

Looks good, thanks!

Just fixed myself some minor issues with the --help output.

Comment 13

[Julien Pagès (:parkouss)]

Merged in https://github.com/mozilla/mozregression/commit/215a875e3c7305c026c0cf25d5fb863d016c15bb.

MikeLing, we will have to implement this on the GUI side. :) Are you interested to work on it ?

Comment 14

[MikeLing]

(In reply to Julien Pagès from comment #13)
> Merged in
> https://github.com/mozilla/mozregression/commit/
> 215a875e3c7305c026c0cf25d5fb863d016c15bb.
> 
> MikeLing, we will have to implement this on the GUI side. :) Are you
> interested to work on it ?

hmm, Due to I still get the same error when I use launch option locally, I decide to give up the further work related with this feature. maybe there are other bugs I can work with?

Comment 15

[MikeLing]

Ok, seems like I can use this launch feature. please cc me if you ready to start that feature on GUI.

BTW, could you tell me where should I address if would like to wrote a test for it? I just would like to see all the check is green on git :) Thank you!

Comment 16

[Julien Pagès (:parkouss)]

(In reply to MikeLing from comment #15)
> Ok, seems like I can use this launch feature. please cc me if you ready to
> start that feature on GUI.
> 
> BTW, could you tell me where should I address if would like to wrote a test
> for it? I just would like to see all the check is green on git :) Thank you!

Yeah, I love that motivation to write tests! So basically you should look in the tests folder, and in the file named test_[FILE_NAME_OF_SOURCE_CODE]. so here, basically you need to edit tests/test_main.py, and maybe tests/test_cli.py.

After that, you need to understand what is done in those files, and try to add tests in there for your particular case. You will have to look at py.test, mock, pytest-mock and maybe other things that we use to write tests.

Best way to learn then is to lokk at the test examples and try to understand them.

From a bugzilla point of view, we can create a bug, or just use a github PR (as you like) since it not something that will be useful to end users (or more bug reporters). github PR is fine for me here.

Comment 17

[Ray Satiro]

Thanks for your work on this guys, and with the other bugs I filed. I just updated today and it's running a lot better now than it was last month. I am able to launch by revision if I have the changeset for example, 159d5d2946d3.

I cannot launch by date build code though. Is that possible? For example I see someone has a bug and in the build report is by default their Build ID, and I want to see what they're seeing but I cannot launch with that ID code. For example I tried to launch a code today and this is what happened:

mozregression --launch 20150918104554
 0:00.60 LOG: MainThread INFO bits option not specified, using 64-bit builds.
 0:01.36 LOG: MainThread ERROR The url 'https://hg.mozilla.org/integration/mozilla-inbound/json-pushes?changeset=20150918104554' returned a 404 error. Please check the validity of the given changeset '20150918104554'.