Closed Bug 1233896 Opened 4 years ago Closed 4 years ago

mozregression should have a mode to first double check --good builds and --bad builds before proceeding to bisect

Categories

(Testing :: mozregression, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: gkw, Assigned: parkouss)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

mozregression should have a mode to first double check --good builds and --bad builds before proceeding to bisect, because when fuzzing, we want to be sure that the --good and --bad builds are indeed --good and --bad respectively.

For example, --good and --bad builds should run against --command if the --command is set.
That sounds like a great idea. And I know something similar is requested for b2g support.
I'd think that this should even be the default. mozregression should only proceed to bisect further if it can (almost) definitively conclude from the starting earliest-good and latest-bad builds that they are indeed --good and --bad respectively.
(In reply to Gary Kwong [:gkw] [:nth10sd] from comment #2)
> I'd think that this should even be the default. mozregression should only
> proceed to bisect further if it can (almost) definitively conclude from the
> starting earliest-good and latest-bad builds that they are indeed --good and
> --bad respectively.

Yes, I agree this is a good default. Mozregression purpose is to find valid pushlog url at the end (well a small number of changes that broke something). It is a waste to try a ton of build, and not be able to find at the end a real regression because we started with a wrong range... Moreover, mozregression won't help you in that case (to at least detect that you are doing something wrong)...;

Another option would be to test for the last build to be bad in case we found everything good before - and if the last build is not bad, that would mean that the given --bad was not correct. But still the user can test a lot of builds for nothing.

I think I prefer Gary proposal - we will always have two more builds to test before the bisection, but we can be sure from the beginning that we are currently looking at a good build range.

If you have thoughts, please shout. :)
Blocks: 1236101
This patch introduce bisection 'modes'. 'classic' check first for good and bad builds, and 'no-first-check' does not. The default is now the 'classic' mode, which change mozreg default behavior.

I believe we can add at least another mode later, for the b2g specific bisection.
Assignee: nobody → j.parkouss
Status: NEW → ASSIGNED
Attachment #8703433 - Flags: review?(wlachance)
Nice, thanks!

Fly-by nit: "firsts good and bad builds to really be good" -> s/firsts/first
Thanks Gary! Fixed :)
Comment on attachment 8703433 [details] [review]
check good and bad at first

lgtm
Attachment #8703433 - Flags: review?(wlachance) → review+
Great, landed in https://github.com/mozilla/mozregression/commit/e3e881572260d57c4df32f64c54394c169ba800f.
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.