1195294 - mozregression should be more aggressive in its 'skip' feature.

Status: RESOLVED FIXED

(Testing :: mozregression, defect)

Description

[Jukka Jylänki]

I'm attempting to bisect a WebGL rendering failure between a large time span (2014-01-01 -> 2015-08-17). That bisection led me to a datetime 2014-12-12 where WebGL got broken by something else altogether, so I am using the 'skip' feature to avoid those commits. However, when I skip 2014-12-12, mozregression then tries 2014-12-11, which is broken as well. I then skip that to get to 2014-12-13, which is broken, so I skip, and get 2014-12-10. Skip -> 2014-12-15. Skip -> 2014-12-10, and so on, and my bisecting has come to a slow crawl.

It would be better if mozregression instead used any one of these three scenarios:

 a) If A and B are the endpoints, and C = (A+B)/2, and C was skipped, then try (A+C)/2 or (B+C)/2 in the hope that the broken/skipped revision is lost to the "dropped" uninteresting side of the fence.

 b) Instead of doing a "try the closest untested date from the target skipped date" which ends up in that kind of linear one day differences which have a high probability of being broken too, do geometric incremental hops away from the original skipped date, e.g. 2014-12-12 ->(-1 day)-> 2014-12-11 ->(+2 days)-> 2014-12-14 ->(-4 days)-> 2014-12-08 and so on.

 c) Try any random date in the range.

Any one of these should have a higher probability of escaping the uninteresting breakage quicker and resuming the fast bisection process.

Comment 2

[Julien Pagès (:parkouss)]

Thanks for the use case and details Jukka. I will think more about how to implement it mozregression.

Comment 3

[Julien Pagès (:parkouss)]

Oh, I can see another possibility - if the user chooses skip, we could let him choose the next build to try. So for example:

[G, X1, X2, X3, X4, X5, B]

We are trying X3 - unfortunately broken so we have to skip it (we hit 's'). Build range is now:

[G, X1, X2, X4, X5, B]

Now, we could ask to choose a build to test, based on the index. I like the idea of a middle-based index:

[G, X1, X2, X4, X5, B]
    -1  0   1   2

'0' is what we are doing every time currently (ie, choose the next build in the middle of the range).

Now, if user type '1', we will test X4. If it is good, the range becomes:

[G, X5, B]

If it is bad:

[G, X1, X2, B]

and if build is broken again, remove X4 from the range and ask again.


I started to look at the code to implement it, it seems doable without much pain.

What do you think ?

Comment 4

[Jukka Jylänki]

Hmm, I suppose asking the user could be interesting, since it lets the user have some extra control. I think I like the idea. However, I think the question needs to be presented with enough information that I can easily see what the distance from the skipped version is to good and bad, so that I can estimate what e.g. -163 means in terms of the search ranges. Perhaps show the valid number range, like "Please enter a revision to test in the range [-163, 162]?" or something similar to that?

Comment 5

[Julien Pagès (:parkouss)]

(In reply to Jukka Jylänki from comment #4)
> Hmm, I suppose asking the user could be interesting, since it lets the user
> have some extra control. I think I like the idea. However, I think the
> question needs to be presented with enough information that I can easily see
> what the distance from the skipped version is to good and bad, so that I can
> estimate what e.g. -163 means in terms of the search ranges. Perhaps show
> the valid number range, like "Please enter a revision to test in the range
> [-163, 162]?" or something similar to that?

Sure, this makes a lot of sense.

Comment 6

[William Lachance (:wlach)]

Sure, this sounds like it could work well, with the proviso mentioned.

Comment 7

[Julien Pagès (:parkouss)]

Attachment: choose build after skip

I tested it, looks good.

Though another person that can test and confirm would be great!

Comment 8

[Julien Pagès (:parkouss)]

Comment on attachment 8655258 [details] [review]
choose build after skip

So, as discussed on irc wlach is quite busy with other things. I think I tested this enough, it works well and the change looks good to me. This is an interesting feature that we should take, so I just r+ myself here.

Comment 9

[Julien Pagès (:parkouss)]

Oups I should have removed the r? flags instead it seems.

Oh well. Anyway, landed in https://github.com/mozilla/mozregression/commit/cc33004adad16c88a2ed40fa8155e720f943989f.

Comment 10

[Jukka Jylänki]

Hey, thanks. Tried out the new skip feature now in action, looks like it works great! This saved my bacon today! A minor thought is that perhaps the instruction text could be clear on what is a sane default to try (0), or perhaps simply pressing Enter could choose the nearest yet untested revision? I was slightly confused for a moment on whether 0 means "try the same revision again you just tried", or if 0 means "the nearest yet untested revision", but after refreshing my mind from the above discussion, I believe it means the latter.

Comment 11

[Julien Pagès (:parkouss)]

Hey, first, I'm glad it helped!

Right, skipping will "remove" the build (though you can still go back, but with limitations because of bug 1203855). Then the "0" is indeed the choice of the nearest build from the one you just tested. Going on negative numbers will typically go in the "good" range, and positive in the "bad" range (unless you use --find-fix, in that case reverse "good" and "bad" in the sentence).

I am totally open to make it more understandable/easy to use. Let's look at it in bug 1217443.