393338 - talos tinderbox needs to munge build start times to match code tested

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

The talos tinderbox on http://tinderbox.mozilla.org/Firefox/ is reporting bogus build start times.  The build start times reported need to correspond to the code being tested; otherwise the way the information fits into the tinderbox page doesn't make sense -- the wrong checkins get blamed, etc.  (The performance testing tinderboxes, which also download hourly builds from ftp, already do this correctly.)

Comment 1

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

This is a major problem.  It makes it confusing to figure out who broke the build, and more likely that they've left before the necessary builds have cycled.

Comment 3

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Here's the code in tinderbox that does this:

http://mxr.mozilla.org/seamonkey/source/tools/tinderbox/build-seamonkey-util.pl#981

specifically, http://mxr.mozilla.org/seamonkey/source/tools/tinderbox/build-seamonkey-util.pl#1829

it looks like it basically grabs that last start time of a build from tinderbox, which should correspond to the latest build that got downloaded.

Comment 4

[Mike Schroepfer]

Ben/Rob H/Rob C any chance we can get a quick fix for this.   There are some thoughts in bug 400107 on quick hacks.

Comment 5

[[e-mail] (:bhearsum)]

I took a look at the links from Vlad's comment...using that method I should be able to get something working today/tomorrow.

Comment 6

[[e-mail] (:bhearsum)]

Attachment: patch to let buildbot talos slaves munge start times

This patch adds an option to the TinderboxMailNotifier that lets it use the date of the last change as the 'builddate'. When enabled, the start times will be munged the same way the old perf machines are.

Comment 7

[[e-mail] (:bhearsum)]

Attachment: updated buildbot configs to support start time munging

This patch enables the 'useChangeTime' option on TinderboxMailNotifier. It also removes the MozillaChangePusher step from every builder. If each Build is forced to run with only one Change, the start times will be really wacky.

Comment 8

[Rob Campbell [:rc] (:robcee)]

Comment on attachment 285310 [details] [diff] [review]
patch to let buildbot talos slaves munge start times

the builddate change looks reasonable, I think. Nice fixes on the columnName construction down below. Who let that in? ;)

Comment 9

[[e-mail] (:bhearsum)]

Checking in buildbot/status/tinderbox.py;
/cvsroot/mozilla/tools/buildbot/buildbot/status/tinderbox.py,v  <--  tinderbox.py
new revision: 1.6; previous revision: 1.5
done

Comment 10

[[e-mail] (:bhearsum)]

Checking in master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/master.cfg,v  <--  master.cfg
new revision: 1.13; previous revision: 1.12
done
Checking in perfrunner.py;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/perfrunner.py,v  <--  perfrunner.py
new revision: 1.6; previous revision: 1.5
done

Comment 11

[[e-mail] (:bhearsum)]

The Talos Buildbot master was just restarted to enable the date munging code. It looks to be working perfect.

Comment 12

[(not currently active) Ted Mielczarek]

Mass move of Core:Testing bugs to mozilla.org:Release Engineering:Talos. Filter on RelEngTalosMassMove to ignore.