962742 - Mozregression bisection of inbound doesn't seem to work.

Status: RESOLVED FIXED

(Testing :: mozregression, defect)

Description

[Jeff Muizelaar [:jrmuizel]]

I'm trying to bisect http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=cf2d1bd796ea&tochange=711f08b0aa1d

mozregression doesn't seem to know about any of the builds after:
9022f0cacaed and therefore concludes the wrong regression range.


$ mozregression --good=2014-01-08 --bad=2014-01-09
Got as far as we can go bisecting nightlies...
Ensuring we have enough metadata to get a pushlog...
Downloading build from: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/01/2014-01-08-03-02-03-mozilla-central/firefox-29.0a1.en-US.mac.dmg
===== Downloaded 100% =====
Installing nightly
Downloading build from: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/01/2014-01-09-03-02-03-mozilla-central/firefox-29.0a1.en-US.mac.dmg
===== Downloaded 100% =====
Installing nightly
Last good revision: cf2d1bd796ea (2014-01-08)
First bad revision: 711f08b0aa1d (2014-01-09)
Pushlog:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=cf2d1bd796ea&tochange=711f08b0aa1d

... attempting to bisect inbound builds
Getting inbound builds between cf2d1bd796ea and 711f08b0aa1d
Testing inbound build with timestamp 1389209742, revision 46f4a621a23d8c9f9e47d94e0cb772fd4bc8f43a
Using local file: firefox-29.0a1.en-US.mac.dmg
Installing nightly
Starting nightly
bad
Testing inbound build with timestamp 1389197746, revision 82cd92e107360c18cf8cddbafec82988c00e0469
Downloading build from: http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-macosx64/1389197746/firefox-29.0a1.en-US.mac.dmg
===== Downloaded 100% =====
Installing nightly
Starting nightly
bad
Testing inbound build with timestamp 1389192524, revision 9e34b5a089e6d5aa3677e072625f5fb2465f4e3c
Downloading build from: http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-macosx64/1389192524/firefox-29.0a1.en-US.mac.dmg
===== Downloaded 100% =====
Installing nightly
Starting nightly
bad
Testing inbound build with timestamp 1389190960, revision 3b06d1997d8cb3989d172482f8fe80e8afe54eed
Downloading build from: http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-macosx64/1389190960/firefox-29.0a1.en-US.mac.dmg
===== Downloaded 100% =====
Installing nightly
Starting nightly
bad
Testing inbound build with timestamp 1389188442, revision 6356fee7e8b8159f25e254faf042fc579411d06b
Downloading build from: http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-macosx64/1389188442/firefox-29.0a1.en-US.mac.dmg
===== Downloaded 100% =====
Installing nightly
Starting nightly
bad
Testing inbound build with timestamp 1389187121, revision 9022f0cacaedff6e42d5a0fa409194508d543324
Downloading build from: http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-macosx64/1389187121/firefox-29.0a1.en-US.mac.dmg
===== Downloaded 100% =====
Installing nightly
Starting nightly
bad
No more inbounds to bisect
Last good revision: cf2d1bd796ea
First bad revision: 9022f0cacaed
Pushlog:
http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=cf2d1bd796ea&tochange=9022f0cacaed

do you want to bisect further by fetching the repository and building? (y or n) n
Exception OSError: (2, 'No such file or directory', 'firefox-29.0a1.en-US.mac.dmg') in <bound method FirefoxNightly.cleanup of <mozregression.runnightly.FirefoxNightly object at 0x104c50810>> ignored

Comment 1

[William Lachance (:wlach)]

I will look into this.

Comment 2

[William Lachance (:wlach)]

Ok, I think I see the problem here. We're getting inbound pushes between two m-c revisions, but what we actually want to do is get inbound pushes between:

1. The last inbound merge before the last good (early) revision
2. The last inbound merge before the first bad (late) revision

I think we're ok for (2) right now (since the nightly revision will either be equal to or greater than that value... and even if it's greater than it, it doesn't matter because there is still nothing in inbound that can help us past the point of the last merge)

(1) is a different story, since there could have been inbound pushes that we care about that might have been incorporated into the build since the last good revision, but will fall off the radar because we're only bisecting after the last good from m-c was merged into m-i.

This might be somewhat clumsy, but I wonder if the best solution isn't to just take the previous day's nightly revision as the starting point of inbound bisection. This means that the user needs to redo a bit of work but guarantees that we're looking in the right range so long as there was an inbound merge the previous day. The "right" alternative is probably to somehow programatically find the last inbound merge into m-i before the last nightly was build and take the pushlog from there, but talking to gps that sounds hard.

(hopefully that made sense, I'm kinda tired right now)

Comment 3

[William Lachance (:wlach)]

Attachment: Inbound range should start from previous day's nightly revision range

Hi Jeff, could you try applying this patch and telling me if it works any better for you? This makes it so we take the previous day's nightly revision as a starting point for the regression range... kind of a hacky solution but I think it should work (at least as long as there was a m-c -> inbound merge the previous day).

Comment 4

[William Lachance (:wlach)]

I'm just going to release this as-is, as I'm 99% certain it works and the old behaviour was definitely broken. If there are problems please reopen the bug.

https://github.com/mozilla/mozregression/commit/4b3ceb229e9a85adba7b8839b4088740357a3d5d

Download mozregression 0.14 to pick up the fix.