774799 - Double builds scheduled for each platform on try

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Ed Morley [:emorley]]

For every Try push after and inclusive of:
https://tbpl.mozilla.org/?tree=Try&rev=f0096d1ad4b0
2012-07-17 11:06 PDT

There have been two builds scheduled for each platform.

This is going to start to cause issues with Try load very soon :-(

Comment 1

[Steve Fink [:sfink] [:s:]]

I notice that none of the secondary builds have completed yet, though that may just be a matter of time.

Comment 2

[Steve Fink [:sfink] [:s:]]

Then again, that includes the B2G builds, which should have been done for a while.

Comment 3

[Kim Moir [:kmoir] ET]

This looks similar to bug 719260.

Comment 4

[Kim Moir [:kmoir] ET]

Talked to catlee, he looked at the logs and found that at 11:37:20 we got a 500 error from hg.m.o/try

/builds/buildbot/build_scheduler/master/twistd.log.5:2012-07-17 11:37:20-0700 [HTTPPageGetter,client] http://hg.mozilla.org/try has been reset

He suspects this is the root cause and subsequent builds should be fine.

Comment 5

[Ed Morley [:emorley]]

Ok, thank you :-)

Do we need to do anything to mitigate this in the future?

Comment 6

[Kim Moir [:kmoir] ET]

I opened a bug to address this - see bug 774862.

Comment 7

[Kartikaya Gupta (email:[e-mail])]

This is happening again; I pushed changesets f307976a7ae6 and 744b65845b23 both of which got double-builds queued. I manually cancelled the ones I don't care about.

Comment 8

[Chris AtLee [:catlee]]

the try poller reset itself again at 2012-07-18 07:20:54-0700. for some reason the server reports that the previous revision doesn't exist temporarily.

the revision it failed on happened to be yours actually...it was polling http://hg.mozilla.org/try/json-pushes?full=1&fromchange=f307976a7ae68117ad3cf651ad07ceb92cc49cd0&tipsonly=1

Comment 9

[Ed Morley [:emorley]]

We had another batch of double builds on try last night, eg:
https://tbpl.mozilla.org/?tree=Try&rev=eaf06d352463

Comment 10

[Ed Morley [:emorley]]

Ah, due to bug 770811, most of you won't be able to load that page easily :-(

The push in question:
amarchesini@moco – Fri Jul 20 03:23:21 2012 (UTC+1)
eaf06d352463 Andrea Marchesini – try: -b do -p all -u all -t none 

(There were many more like it, this was just the most recent push on which it occurred)

Comment 11

[Ed Morley [:emorley]]

I've cancelled all doubled up running and pending tests (approx 100, all Windows).

There were over 30 pushes last night (UTC) that had doubled up builds; one plus side is that once this is resolved at least bug 772458 gets an easy win :-)

Comment 12

[Chris AtLee [:catlee]]

this shouldn't be a problem now