386594 - bl-bldxp01 often goes missing after clobber builds

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P3)

Description

[Nick Thomas [:nthomas] (UTC+12)]

For the last few mornings, bl-bldxp01 has stopped doing the perf tests right after the nightlies are made (4am). It's expected that it should miss the nightly (because of where that build is published) but it should pick up the builds after that.

Bugs 386590, 386515, 386353, 385775 (06/25).

The problem seems to be related to cvs up hanging.

Comment 1

[:Gavin Sharp [email: [e-mail]]]

Phil also noticed that it seems to stop building when one of it's builders goes red, so could it be that it doesn't resume testing after failing to retrieve a build, for whatever reason (nightly or bustage)?

Comment 2

[Benjamin Smedberg]

Note that it stopped again after I clobbered around 9:30PT this morning. So it's definitely related to the clobber.

Comment 3

[Phil Ringnalda (:philor)]

Yeah, that was my new theory, too, until cf confirmed that it was still looking for a new a6pre build when there wasn't anything but new a7pre builds to be found.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Turns out that 4 times in a row isn't "every time" - it was fine for this morning's nightlies.

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

lets watch this some more...

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

The latest info on this problem is that cvs hangs updating the tinderbox config; the code does try to kill it after a timeout, but that doesn't work reliably on windows. 

We end up doing that update frequently during the period of the nightly and first hourly builds, as we loop around 
  * update config
  * check for new build
  * if found d/l & start tinderbox

Comment 7

[Robert Helmer [:rhelmer]]

Happened again in bug 398423, same problem. 

Comment 8

[Chris Cooper [:coop] (he/him)]

Still an issue?

Comment 9

[Phil Ringnalda (:philor)]

Tough (for me) to say: bug 399679 certainly was, then bug 400348 was builder-burning, bug 407830 was builder restarted, and bug 410035 and bug 411748 were who-knows?

Comment 10

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

As we are mothballing this machine (see bug#413695) anyway, should we declare this bug done for now? If its a problem on Talos, on Talos hardware, we would need to file a new bug there anyway?