591074 - Tryserver skipping pushes, building them all at once, then canceling builds.

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[Roy Frostig :froystig]

There are currently about six push entries (see http://tests.themasta.com/tinderboxpushlog/?tree=MozillaTry) that haven't triggered any builds.  Then, Patrick Walton's push of changeset 8381fb9a63e9 triggered a surprisingly large number of builds under his name only, which have now gradually been disappearing.

He tells me that he's presently receiving many emails of all sorts of failures.

Comment 1

[Aki Sasaki (not active)]

Try is currently under massive load (I think Aravind said 102) due to 8? simultaneous pushes. 8 * 14 = 112 simultaneous full clones of the try repository, which isn't the smallest repository ever.

We're currently looking at this.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

I bet all of that is fallout from the master dying, then starting many many jobs when coming back up and overwhelming hg.m.o. 

Try is closed right now.

Comment 3

[Andreas Gal :gal]

Thanks guys for looking into this right away. We are anxiously awaiting the return of our beloved try server :)

Comment 4

[Chris AtLee [:catlee]]

Have the builds actually been getting cancelled?  Or just associated with the wrong user on tbpl?

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Clamp number of clones to 42 (try only)

42 clones is three pushes doing all 14 jobs, and is also a cool number. This affects try clones only because they're directed to a particular host, not things like m-c or builds/tools.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Staging result

This is what happened in staging with maxCount=1. Both builds started at the same time, the leak test builds got to checkout first and the opt compile waited for it's turn.

Comment 7

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 469676 [details] [diff] [review]
Clamp number of clones to 42 (try only)

saw it in staging, love it, 42 is the best number.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 469676 [details] [diff] [review]
Clamp number of clones to 42 (try only)

http://hg.mozilla.org/build/buildbotcustom/rev/d1af3e5a78fb

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

buildbot master reconfig'd. Working on rerunning builds that were affected earlier.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Over to Lukas for the sendchanges to redo earlier failed builds.

Comment 11

[Aki Sasaki (not active)]

Attachment: this fixed it

This fixes try sendchanges after https://bugzilla.mozilla.org/attachment.cgi?id=469065 landed for bug 588756.  r=lsblakk

http://hg.mozilla.org/build/buildbotcustom/rev/6ec8e9031ac9

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Sendchange's are working, advised those people by email no need to repush. Try reopened.