1235127 - [gui] Mozregression sometimes fails to continue bisection in mozilla-inbound builds

Status: UNCONFIRMED

(Testing :: mozregression, defect)

Description

[arni2033]

Attachment: 2015.12.25 10-57-28.png

>>> My Info:      Version 0.7.0   Using mozregression version 2.1.0
1. Start new bisection, choose "firefox,32bit,opt"
2. Press Next twice
3. Set last good date -> 2015-10-27, first bad date -> 2015-10-28
4. For the first 2 suggested builds, answer "Bad"

Result:
 Bisection stops at the third build, and last record in Log View is:
> DEBUG : Found commit message: Backed out changeset 90edb8c62dee (bug 1214293) for Valgrind test
>         failures on a CLOSED TREE

 but if I click at the last tested build, I see the following pushlog_url with lots of builds:
> https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=9a8f2342fb3116d23989087e026448d38a3768c5&tochange=5430b2dba98b2a39ccdfd3700131f780a27be17c
 and I can test some of them using "Run a single build" feature.

Expectations:
 It should start testing inbound builds. Please also explain me whether this is valid bug or not.

Comment 1

[Julien Pagès (:parkouss)]

Since mozregression 2.0, mozregression will bisect on integration branches by looking at the commit message - the commit should look like "Merged inbound to central" for that to work. So in the commit message you have, there is no information to know which branch originated the merge (and even if it was a merge) - so the bisection is done.

See [1] for explanations of the mozregression changes in 2.0.

I guess in cases like this, if you know that it comes from mozilla-inbound, I think you can try another bisection with the parameters:

- use the branch "inbound"
- bisection range should be in changesets, say ef10857254a0 to 1988a43888ae (reading your push log url)

Does that explain it better, and does that help ?

[1] https://parkouss.wordpress.com/2015/11/14/mozregression-new-way-for-handling-merges/

Comment 2

[arni2033]

> in cases like this, if you know that it comes from mozilla-inbound,
This is it, I don't know. What I know is that using Mozregression Gui 5 I got more precise result
> pushlog_7:   https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=9a8f2342fb3116d23989087e026448d38a3768c5&tochange=5430b2dba98b2a39ccdfd3700131f780a27be17c
> pushlog_5:   https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=34564c10054d5864e98152b238c9c16b0bad1b80&tochange=90edb8c62dee69fe55faf84507d70570f9d8eaad

I don't have enough time to understand the whole article, but as I understand, there is an improvement: moving from testing all builds in the previous 4 days to a smaller number using smart algorithm. I was OK with "4-days" approach, because it's reliable.
So, the decision is already made, but IMO it kind of ruins "Gui" part.

I haven't seen any clear manuals on what to do with all those unclear pushlogs version 7 sometimes returns, so I'll rather use version 5 or leave keyword "regressionwindow-wanted" if I need to get more
definite results. Please close this bug in appropriate way - idk how to do that; it doesn't "wfm" (

Comment 3

[Julien Pagès (:parkouss)]

(In reply to arni2033 from comment #2)
> > in cases like this, if you know that it comes from mozilla-inbound,
> This is it, I don't know. What I know is that using Mozregression Gui 5 I
> got more precise result
> > pushlog_7:   https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=9a8f2342fb3116d23989087e026448d38a3768c5&tochange=5430b2dba98b2a39ccdfd3700131f780a27be17c
> > pushlog_5:   https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=34564c10054d5864e98152b238c9c16b0bad1b80&tochange=90edb8c62dee69fe55faf84507d70570f9d8eaad
> 
> I don't have enough time to understand the whole article, but as I
> understand, there is an improvement: moving from testing all builds in the
> previous 4 days to a smaller number using smart algorithm. I was OK with
> "4-days" approach, because it's reliable.

Well - it is only when the regression comes from mozilla inbound. If it comes from fx-team or b2g-inbound, the old way does not work.

> So, the decision is already made, but IMO it kind of ruins "Gui" part.
>
> I haven't seen any clear manuals on what to do with all those unclear
> pushlogs version 7 sometimes returns, so I'll rather use version 5 or leave
> keyword "regressionwindow-wanted" if I need to get more
> definite results. Please close this bug in appropriate way - idk how to do
> that; it doesn't "wfm" (

So the best way to do this with the new version in situations like this (when mozregression does not understand that it should continue to another branch, here mozilla-inbound) is to manually start again a new bisection using the changesets you have from m-c. Once you reduced the range to:

> https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=9a8f2342fb3116d23989087e026448d38a3768c5&tochange=5430b2dba98b2a39ccdfd3700131f780a27be17c

on m-c and that you know that you should continue on mozilla-inbound, just note the changesets and use them in a new bisection, using the branch mozilla-inbound and specifying the good and bad changesets you see:

1988a43888ae and ef10857254a0 (look in https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=9a8f2342fb3116d23989087e026448d38a3768c5&tochange=5430b2dba98b2a39ccdfd3700131f780a27be17c, these are the firsts and lasts commits in there, except for the merge commit itself).

You should then have the same result. Though, you have some more manual work to do (a new bisection), but it should be rare cases - and maybe we can improve it. As a last option we could offer the "old bisection way" as an option, but I would prefer to see some better alternatives.

But definitely I want to avoid you from using an old version (or being blocked by our decision), so let's try to find a solution if the manual way of doing two bisections in such cases is not enough.

Comment 4

[arni2033]

(In reply to Julien Pagès (:parkouss) from comment #3)
> 1988a43888ae and ef10857254a0 (look in https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=9a8f2342fb3116d23989087e026448d38a3768c5&tochange=5430b2dba98b2a39ccdfd3700131f780a27be17c,
> these are the firsts and lasts commits in there, except for the merge commit itself).
There's a bug even with that. I mean, it's a bug, right?

STR_1:
1.  Start new bisection, set last good -> ef10857254a0, first bad -> 1988a43888ae, click "Finish"
2.  Look at the 1st item in the panel "Bisection progress"
AR: it says "[cab34e0b - 1988a438]", i.e.:
> https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=cab34e0b0a7b02fa3b4e86e89c24d48cd96c1cbe&tochange=1988a43888ae243c2866b8ad7ae61137ff0de2c0
    Pushlog doesn't include 2 changesets:   cab34e0b0a7b and ef10857254a0
ER: it should be possible to get any of those 2 changesets as result of bisection process.


STR_2 (I also tried this one):
1.  Start new bisection, set last good -> 9a8f2342fb3116d23989087e026448d38a3768c5,
    first bad -> 1988a43888ae, click "Finish"
2.  Look at the 1st item in the panel "Bisection progress"
AR: it says "[66d59599 - 1988a438]", i.e.:
> https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=66d59599465c708d4dc985c774c99ccc26aca69d&tochange=1988a43888ae243c2866b8ad7ae61137ff0de2c0
ER: if STR_1 isn't supposed to let me bisect those 2 changesets, then STR_2 should allow that.

Comment 5

[Julien Pagès (:parkouss)]

Well, this is not a bug - this is the normal behavior of mozregression.

When you ask to test a bisection, you are giving a good and a bad build. Mozregression assume that they are good and bad, without testing them - and the pushlog will always contains the bad, but not the good one (so at the end you may end up with only one push in the log, which obviously is the bad one).

And this is the default behavior of the pushlog view anyway,

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=ef10857254a0&tochange=1988a43888ae
`Changes pushed after changeset ef10857254a0, up to and including changeset 1988a43888ae` is at the top of the page


That being said, I did the test with your input. I have the following log (in the log view):

> 2016-01-20T09:41:29: WARNING : Skipping build ef10857254a01368861d9c1cc0105de89be6d169: Unable to find build info using the taskcluster route 'buildbot.revisions.ef10857254a01368861d9c1cc0105de89be6d169.mozilla-inbound.linux64'

mozregression do not find builds for ef10857254a0 and it skips it. So what happen is that

1. you gave the range ef10857254a0 - 1988a43888ae
2. mozregression can't find ef10857254a0, so range is reduced to cab34e0b0a7b - 1988a43888ae
3. you now have an initial range of [cab34e0b - 1988a438], as you saw. Pushlog do not contain cab34e0b0a7b because it is the good one (thus it is in the url) and it do not contain ef10857254a0 because we couldn't find it.

For the STR2, you can't do that. You are using the changeset of a merge (which appear only in mozilla-central) to find something in mozilla-inbound.

Comment 6

[Julien Pagès (:parkouss)]

So, maybe that may help, I just added the default behavior in mozregression (command line only for now, but I will work on it for the GUI) to test the two first/bad builds of a range at first. So you are sure that you are in a good/bad range before going really into bisection.

I think it may help for case like this, because:

- you are looking for a regression between ef10857254a0 - 1988a43888ae
- unfortunately, ef10857254a0 can't be used to test, so it is reduced to cab34e0b0a7b - 1988a43888ae
- now mozregression make you test cab34e0b0a7b - 1988a43888ae
- if initially you say good for the two, then mozregression will stop. And you can be pretty sure that cab34e0b0a7b (which has been untested) is the bad one, given that ef10857254a0 was really good.