1111340 - possible bug for nightly and inbound regression hunting in mozregression

Status: RESOLVED FIXED

(Testing :: mozregression, defect)

Description

[Julien Pagès (:parkouss)]

I was working on bug 1111314 and discover a strange behavior in regression hunting. I think that could be simple to explain with an example:

Let's say we want to bissect with the following command:

mozregression --good 2014-11-10 --bad 2014-11-20

That means that we know that 2014-11-10 was good, and that 2014-11-20 was bad. We have a range of 10 builds, 8 being testables.

We will hit a build on 2014-11-15, and test it. If we say good, the current implementation will split the range in such a way that we will only have a range of four builds remaining instead of five. (we do data[(mid+1):]). That is wrong is my opinion, because we can no more say that build 2014-11-16 was good! we only know for 2014-11-15, so we should have a range of five builds, from 2014-11-15 to 2014-11-20.

That is not hapenning when we say bad. It happens on both nightly and inbound. The bug is in nightly since we use build_data I suppose, but I think it was in inbound from a long time ago because I remember that I looked at this +1 thing, but thinking that was normal I let it the way it was.

I think it is pretty serious, so I'll make a patch now. Waiting for your review so you can confirm or infirm the issue.

Comment 1

[Julien Pagès (:parkouss)]

Attachment: fix a bug that miss builds when the last verdict is good

Also I fixed a bug in BuildData.mid_point: we should return 0 when there is less than 3 elements (we can not return a mid_point of only two). My git comment does not reflect this, and it's in only one patch - tell me if you want me to rebase it to be more accurate.

Comment 2

[Elbart]

Attachment: 1111340_log.txt

First step: 11-15
11-15 is good
11-10 to 11-15 has four builds in between, just like 11-15 to 11-20.

Second step: 11-17 
Let's say 11-17 is bad.
Range is 11-15 to 11-17

Third step: 11-16.

Where's the problem?

Comment 3

[Julien Pagès (:parkouss)]

What is the version of mozregression used ? I believe you run the latest version from pypi, which is not impacted (on nightly...). With the current master, I got this:

mozregression --good 2014-11-10 --bad 2014-11-20
 0:23.26 LOG: MainThread Bisector INFO Running nightly for 2014-11-15
 0:23.26 LOG: MainThread Regression Runner INFO Downloading build from: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/11/2014-11-15-03-02-05-mozilla-central/firefox-36.0a1.en-US.linux-x86_64.tar.bz2
===== Downloaded 100% =====

Was this nightly build good, bad, or broken? (type 'good', 'bad', 'skip', 'retry', or 'exit' and press Enter): g
 1:34.47 LOG: MainThread Bisector INFO Narrowed nightly regression window from [2014-11-10, 2014-11-20] (10 days) to [2014-11-15, 2014-11-20] (5 days) (~2 steps left)
 1:34.48 LOG: MainThread Bisector INFO Running nightly for 2014-11-18
 1:34.48 LOG: MainThread Regression Runner INFO Downloading build from: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/11/2014-11-18-14-40-12-mozilla-central/firefox-36.0a1.en-US.linux-x86_64.tar.bz2
===== Downloaded 100% =====

Was this nightly build good, bad, or broken? (type 'good', 'bad', 'skip', 'retry', or 'exit' and press Enter): g
 2:42.73 LOG: MainThread Bisector INFO Narrowed nightly regression window from [2014-11-16, 2014-11-20] (4 days) to [2014-11-18, 2014-11-20] (2 days) (~1 steps left)
 2:42.73 LOG: MainThread Bisector INFO Got as far as we can go bisecting nightlies...



Note that for me, the second download is 2014-11-18 instead of yours 2014-11-17 (2014-11-17 is good). Also the remaining steps printed are wrong for me. I suppose this has been introduced with bug 1106258, because we reused some logic to bisect from inbound. That's why I am thinking that inbound hunting may be innacurate for quite some time.

Comment 4

[Elbart]

(In reply to Julien Pagès from comment #3)
> What is the version of mozregression used ? I believe you run the latest
> version from pypi, which is not impacted (on nightly...).

Yes, 0.30.

> Note that for me, the second download is 2014-11-18 instead of yours
> 2014-11-17 (2014-11-17 is good). Also the remaining steps printed are wrong
> for me. I suppose this has been introduced with bug 1106258, because we
> reused some logic to bisect from inbound. That's why I am thinking that
> inbound hunting may be innacurate for quite some time.

Sounds like what I described in bug 1109731.
I guess the patch here will fix that bug too. Thanks!

Comment 5

[Julien Pagès (:parkouss)]

(In reply to Elbart from comment #4)
> Sounds like what I described in bug 1109731.
> I guess the patch here will fix that bug too. Thanks!

Good to know about this one. :) I will check to see if that resolves it.

We are currently doing some nice work to test mozregression more with unit tests (you can see a new coverage badge here https://github.com/mozilla/mozregression) - currently the regression.py file (in which I found the bug) is not tested, this is intended to be done soon in bug 1111314. I will add some testing around this for sure. :)

Comment 6

[William Lachance (:wlach)]

Comment on attachment 8536158 [details] [review]
fix a bug that miss builds when the last verdict is good

Good catch. The only thing I'd change would be the comment. Feel free to merge this in yourself once you've updated that.

Comment 7

[Julien Pagès (:parkouss)]

Done, and merged in.