1304673 - All Trees closed for new windows build instances failing to clone

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Carsten Book [:Tomcat]]

10:34 < nthomas|away> aobreja|lunch: aselagea|lunch  - pretty big deal - new windows instances failing to clone, eg 
                      http://buildbot-master70.bb.releng.use1.mozilla.com:8001/builders/fuzzer-win64-rev2/builds/8451/steps/clone_scripts/logs/stdio
10:34 < nthomas|away> Abort: could not find web.cacerts: /etc/mercurial/cacert/cacert.pem
10:34 < nthomas|away> builds retry and fail, queue fills up
10:35 < nthomas|away> Tomcat|lunch: ^^
10:38 < nthomas|away> fallout from https://bugzilla.mozilla.org/show_bug.cgi?id=1303766 I think

all trees closed now 
10:39 < nthomas|away> https://hg.mozilla.org/build/puppet/file/production/modules/mercurial/templates/hgrc.erb#l45 not so good on windows
10:40 < nthomas|away> given the OS limits in https://bug1303766.bmoattachments.org/attachment.cgi?id=8793369

Comment 1

[Carsten Book [:Tomcat]]

affects running build on integration, central etc and also windows nightlys on trunk are busted so far :(

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

<nthomas> ok, so spot-b-2008-2016-09-22-09-44 is going to get whacked, that’s ami-0918c469 in usw2, and ami-f6e991e1 in use1
<nthomas> done. new instances should be OK. going to terminate existing bad instances now
<nthomas> 29 in use1 only

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

We haven't seen any failures on try yet, but it seems likely that the y-2008 AMIs would be affected too.

Comment 4

[Amy Rich [:arr] [:arich]]

markco, aselagea: the path to the cert is going to need to depend on the platform. What works for OS X and Linux won't work for windows. The cert bundle location is currently specified in the ini file for windows. We need to make sure we address this before any new windows AMIs are generated (which will happen automatically tonight).

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Because the axe is already sharpened, I also removed
* spot-y-2008-2016-09-22-09-45, usw2: ami-de1fc3be, use1: ami-2aea923d; and terminated 12 instances running in use1
* spot-t-w732-2016-09-22-08-57, usw2: ami-851dc1e5, use1: ami-91f48c86

Was going to terminate w732 instances but couldn't find any test failures, so may have jumped the gun there.

Comment 6

[Amy Rich [:arr] [:arich]]

Attachment: hgrc.diff

This should set the cert path correct for all platforms. :aselagea: feel free to land this since once you review I'm on PTO.

Mark: we should regenerate the AMIs to make sure they're functional.

Comment 7

[Alin Selagea [:aselagea]]

Comment on attachment 8793717 [details] [diff] [review]
hgrc.diff

http://hg.mozilla.org/build/puppet/rev/74d3ac036c21
http://hg.mozilla.org/build/puppet/rev/1c175c0ebadf

Comment 8

[Carsten Book [:Tomcat]]

trees reopen since nick cleaned up everything we think

Comment 9

[Alin Selagea [:aselagea]]

Reduced the importance of this bug to "major" to stop the alerts.

Comment 10

[Alin Selagea [:aselagea]]

The CA bundle that we added in bug 1303766 (the most recent one, 2016) is different from the one currently stored on the Windows machines (which is 2013). I suppose the latter is integrated in the AMI, so I wonder if we need to update it.

Comment 11

[Intermittent Failures Robot]

7 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 7

Platform breakdown:
* windowsxp: 4
* windows8-64: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1304673&startday=2016-09-19&endday=2016-09-25&tree=all