1240768 - mozregression makes me test the same build multiple times

Status: RESOLVED FIXED

(Testing :: mozregression, defect)

Description

[Marco Castelluccio [:marco]]

Prioritizing already downloaded builds sometimes makes me test the same build multiple times.
If I have already said that it's good, I shouldn't test it again.

This happened to me in a session where I skipped a few tests, I don't know if it's related.

Comment 1

[Julien Pagès (:parkouss)]

Hm, this is not good; I will look into it, and try to reproduce and fix. I you can give me a pattern to reproduce it, that would be awesome. :)

Comment 2

[Julien Pagès (:parkouss)]

Or please attach the log, that would help.

Comment 3

[Marco Castelluccio [:marco]]

I was trying to find a regression range for bug 1240749.
Basically, I have a good and a bad date and a lot of broken builds in the middle (bug 1239855).

I suspect bug 890156 caused both bug 1240749 and bug 1239855. Bug 1239855 was filed on 2016-01-14 and fixed on 2016-01-19. I have to skip all builds during this time, because bug 1239855 makes bug 1240749 impossible to see.

Comment 4

[Marco Castelluccio [:marco]]

Attachment: log.txt

Comment 5

[Marco Castelluccio [:marco]]

As you can see from the log, I've had to run 'mozilla-central build built on 2016-01-14 02:33:52.734000, revision c970bca8' several times.

Comment 6

[Julien Pagès (:parkouss)]

Oh, I understand the problem.

Basically using approx builds is implemented by allowing to use a certain number of files around the one we want, around the one we want.

Basically, for a 10 day range like this:

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

we should test 5, but if we have 4 or 6 instead we choose them. So, this works great, because we are doing that for the mid point (5).

But this is not the case when one choose to skip. Here is a schema:

[0,  1,  2,  3,  4, 5, 6, 7, 8, 9]  # initial range
   [-4, -3, -2, -1, 0, 1, 2, 3]     # choices for the skip.

Now if the user choose to test -4, we will try to find an approx build around his position... So we will take the build 0 (in the initial range) because it is just the ideal candidate (it is next to the build we want to test, and obviously already present...). :(

So, I see two possible ways to fix this:

1. do not allow approx builds when we are choosing a build after a skip.
2. do not allow to use first/last build in the range.

I am not sure which one to choose. Maybe disallowing it in the after skip would be more intuitive (since we ask for a specific build anyway), but it may be more effective (to avoid unnecessary downloads) to just use the approx feature, only avoiding to use first/last in the range.

Thoughts ?

Comment 7

[Julien Pagès (:parkouss)]

So, I can easily reproduce now:

> mozregression -g 2016-01-01 -b 2016-01-19 --persist=

(--persist= is to avoid using my persist dir, and use a temporary one)

Then hit 'good', then 'bad' (to validate the range). A this point we have the two builds in the persist dir, for 2016-01-01 and 2016-01-19.

Now hit 'skip', then choose the build -8 (or 7). mozregression makes you test again one of the already downloaded builds.

Thinking more about it, I'm inclined to fix that with the solution 2: avoiding trying first or last builds in the range. We can still change our mind later, if needed - for now I think it is good.

Comment 8

[Julien Pagès (:parkouss)]

Attachment: fix testing the same build multiple times

This patch fix the issue.

:marco, thanks a lot for filing this bug! I all is fine, I will do a patch release soon with the fix.

Comment 9

[William Lachance (:wlach)]

Comment on attachment 8709590 [details] [review]
fix testing the same build multiple times

This makes sense! Thanks for the sleuthing. :)

Comment 10

[Julien Pagès (:parkouss)]

Great, landed in https://github.com/mozilla/mozregression/commit/e4859cce679dece06c719913f478ca7c61d22c05.

Going to release right now.