1153012 - A Mac OS X try build from a recent m-c revision times out

Status: RESOLVED DUPLICATE of bug 1154377

(Firefox Build System :: General, defect)

Description

[Chris Manchester (limited bugmail, email directly)]

I pushed something to try with a recent m-c revision to retrigger some tests and assess their failure rate. The build failed due to a time out:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e1bbe0c49472

Comment 1

[Phil Ringnalda (:philor)]

Indeed, did that on three of its last ten jobs, thanks for pointing it out.

Comment 2

[Michael Shal [:mshal]]

I also hit this on bld-lion-r5-035: https://treeherder.mozilla.org/#/jobs?repo=try&revision=ba6f8590eec7
and bld-lion-r5-013: https://treeherder.mozilla.org/#/jobs?repo=try&revision=6e9236d69cdc (the B2G Desktop OSX build. The OS X 10.8 opt was also bld-lion-r5-027.)

Just looking through try on treeherder seems to show a bunch of others too, though I'm not sure if it's all the same problem.

Comment 3

[Phil Ringnalda (:philor)]

Might well be something different, but we did have a previous episode not long ago where the warranty on a bunch apparently expired all at once, and diagnostics found bad disks on all of them.

Comment 4

[Jordan Lund (:jlund)]

what's odd is that this seems to happen only on try unless my starring skills are lacking in https://bugzil.la/1154377

Comment 5

[Phil Ringnalda (:philor)]

Right, it's only on try.

That cannot be because all try builds are clobbers: thanks to a build system bug, whenever someone removes a jarred file, which they do once or twice a week, we have to clobber the OS X opt build (and we generally forget that fact and clobber every OS X build), so we do plenty of clobbers on non-try.

It could be from ccache, though the spots where we seem to be when we time out (one fairly popular one being "while we're doing nsinstall js/src/shell") doesn't much smell like it to my uneducated nose.

It could be that we need to reimage the try build slaves for some unknown reason, but the reimage of bld-lion-r5-016 to test that possibility just died without it coming back and it's just sitting idle.

It could be that we have a whole lot of try build slaves with bad disks, since we did have an episode of this same failure on OS X build slaves just a few months ago, and broken ones would build up on try where nobody stars failures in existing bugs much less files new bugs, whereas if a non-try slave started timing out 3-of-10 it would be disabled, diagnosticized, and fixed. bld-lion-r5-027 and bld-lion-r5-028 got diagnostics started running yesterday to test that theory.

Comment 6

[Phil Ringnalda (:philor)]

But apparently my sense of smell is inaccurate, since it was sccache according to bug 1154377.