479261 - try server lies about what it's building

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Attachment: my notes about what I did and what happened

So I pushed a bunch of changesets to the try server (push-to-try) to figure out which of the 10 changesets that I landed caused the startup test to fail on Mac.  I got 11 builds, as described in bug 479260, and 8 talos runs happened that were labeled with the changesets that supposedly generated the builds they were testing.

However, the results don't actually make sense, so I think it's pretty clear that the label that shows up in the tinderbox waterfall showing which try server build was tested isn't actually the build that was tested.

Attached is the text file describing what I did.  I pushed 10 changesets, one-at-a-time, and then later pushed 3 changesets (all at once) that were equivalent (but with different dates) to the first three of the 10.  The pattern of orange and green didn't make sense, although it could have indicated that the problem causing the startup failure was random but happened most of the time.  (However, I have at least 9 runs on the mozilla-central tinderbox saying it happens every time or pretty close to that.)

So, on the assumption that it was random but happened most of the time, I relanded the changes that were on later than the first orange, plus two uninitialized variable fixes that wouldn't cause orange, and the mozilla-central tinderbox turned orange again.

(Sorry if that's not clear; I can explain in more detail if you really need it.)

So I conclude that the try server was lying to me about which builds it was testing.

Comment 1

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Oh, and some more evidence that it's lying:  it actually gets the "rev:" part right, so there's currently a talos cell on tinderbox saying:

[e-mail]
try-1019946addc
id:20090219184000
rev:366ca94d1ccf
FAIL: Busted: ts
FAIL: failed to initialize browser

The rev: indicates that this is actually the revision that I pushed to the try server (and it's correct, since the revision I pushed to the try server fails the startup test).  But the "[e-mail]" and "try-1019946addc" are wrong.

Comment 2

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

And actually, if I look at the log of the *build*, that shows this is a problem with the builder.  The log:

http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1235097179.1235103109.23549.gz starts with:

=== Output ===
TinderboxPrint: [e-mail]
TinderboxPrint: try-1019946addc
Comments: http://hg.mozilla.org/try/rev/1019946addc83301092e2589cfe6faffa743f28b


but then if you go further down, the hg command is:

/tools/python/bin/hg clone --rev 366ca94d1ccfb88ee4a7eeb4738426d0ebdd7761 http://hg.mozilla.org/try /builds/slave/sendchange-mac-hg/mozilla/

Comment 3

[Chris AtLee [:catlee]]

Looks like maybe try server is combining patches together?  The buildbot log for this (at http://sm-try-master.mozilla.org:8010/builders/Try%20server%20mac%20hg%20builder/builds/1574), gives _2_ changes:

   1.

      Changed by: [e-mail]
      Changed at: Thu 19 Feb 2009 18:31:38
      Branch: try
      Revision: 1019946addc83301092e2589cfe6faffa743f28b

      Changed files:
      Comments:

      http://hg.mozilla.org/try/rev/1019946addc83301092e2589cfe6faffa743f28b

   2.

      Changed by: [e-mail]
      Changed at: Thu 19 Feb 2009 18:32:13
      Branch: try
      Revision: 366ca94d1ccfb88ee4a7eeb4738426d0ebdd7761

      Changed files:
      Comments:

      http://hg.mozilla.org/try/rev/366ca94d1ccfb88ee4a7eeb4738426d0ebdd7761

Comment 4

[[e-mail] (:bhearsum)]

I'll look at this later today.

Comment 5

[[e-mail] (:bhearsum)]

Okay, so what's happening here is the multiple changes are getting merged together before MozillaTryProcessing resubmits the extra ones. I'm not sure exactly what's going on under the hood, but I'm pretty sure we can fix this by preventing merging in the Scheduler. I'll give this a try.

Comment 6

[[e-mail] (:bhearsum)]

And to expand on that a bit, the "build" is being labeled as one thing, while it's building another. So, application.ini gets the wrong changeset, which is why talos try labels builds as it does.

Techincally, Talos try is labeling them right but it doesn't match what the build side is saying.

I've got a patch just about ready that should fix this.

Comment 7

[[e-mail] (:bhearsum)]

Attachment: prevent build merging at the Scheduler level

These are the same classes try talos uses, renamed slightly to be consistent with the rest of buildbotcustom.

I tested this locally and it worked great.

Comment 8

[[e-mail] (:bhearsum)]

Comment on attachment 363395 [details] [diff] [review]
prevent build merging at the Scheduler level

buildbotcustom: changeset:   200:625b8ed5603b
buildbot-configs: changeset:   200:625b8ed5603b

Comment 9

[[e-mail] (:bhearsum)]

I've reconfig'ed the master for this. This should be fixed now.