1233896 - mozregression should have a mode to first double check --good builds and --bad builds before proceeding to bisect

Status: RESOLVED FIXED

(Testing :: mozregression, defect)

Description

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

mozregression should have a mode to first double check --good builds and --bad builds before proceeding to bisect, because when fuzzing, we want to be sure that the --good and --bad builds are indeed --good and --bad respectively.

For example, --good and --bad builds should run against --command if the --command is set.

Comment 1

[Julien Pagès (:parkouss)]

That sounds like a great idea. And I know something similar is requested for b2g support.

Comment 2

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

I'd think that this should even be the default. mozregression should only proceed to bisect further if it can (almost) definitively conclude from the starting earliest-good and latest-bad builds that they are indeed --good and --bad respectively.

Comment 3

[Julien Pagès (:parkouss)]

(In reply to Gary Kwong [:gkw] [:nth10sd] from comment #2)
> I'd think that this should even be the default. mozregression should only
> proceed to bisect further if it can (almost) definitively conclude from the
> starting earliest-good and latest-bad builds that they are indeed --good and
> --bad respectively.

Yes, I agree this is a good default. Mozregression purpose is to find valid pushlog url at the end (well a small number of changes that broke something). It is a waste to try a ton of build, and not be able to find at the end a real regression because we started with a wrong range... Moreover, mozregression won't help you in that case (to at least detect that you are doing something wrong)...;

Another option would be to test for the last build to be bad in case we found everything good before - and if the last build is not bad, that would mean that the given --bad was not correct. But still the user can test a lot of builds for nothing.

I think I prefer Gary proposal - we will always have two more builds to test before the bisection, but we can be sure from the beginning that we are currently looking at a good build range.

If you have thoughts, please shout. :)

Comment 4

[Julien Pagès (:parkouss)]

Attachment: check good and bad at first

This patch introduce bisection 'modes'. 'classic' check first for good and bad builds, and 'no-first-check' does not. The default is now the 'classic' mode, which change mozreg default behavior.

I believe we can add at least another mode later, for the b2g specific bisection.

Comment 5

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Nice, thanks!

Fly-by nit: "firsts good and bad builds to really be good" -> s/firsts/first

Comment 6

[Julien Pagès (:parkouss)]

Thanks Gary! Fixed :)

Comment 7

[William Lachance (:wlach)]

Comment on attachment 8703433 [details] [review]
check good and bad at first

lgtm

Comment 8

[Julien Pagès (:parkouss)]

Great, landed in https://github.com/mozilla/mozregression/commit/e3e881572260d57c4df32f64c54394c169ba800f.