977032 - Integration Trees closed, high number of pending linux compile jobs about ±2 hour backlog

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Carsten Book [:Tomcat]]

seems the integration trees see a high number of pending linux build jobs and backlog of jobs is about 2 hours on m-i and fx-team. Closing integration trees for this

Comment 1

[Carsten Book [:Tomcat]]

04:17 < mgerva> Tomcat|sheriffduty: there's a bug about  scl3 <-> usw2 link starting to degrade (Bug 975438)
04:18 < Tomcat|sheriffduty> mgerva: ok will comment there , thankx

Comment 2

[Chris AtLee [:catlee]]

Looks like maybe the spot instances are having problems puppetizing:

Feb 26 04:43:25 bld-linux64-ec2-005 puppet-agent[1268]: Could not request certificate: Error 400 on SERVER: this master is not a CA

the hostname mentioned in the log is not what FQDN is in spot_setup

Comment 3

[Carsten Book [:Tomcat]]

trees reopened at 5:13 PST

Comment 4

[Chris AtLee [:catlee]]

Spot instances started since then seem to be working fine...maybe the error above is a red herring

Comment 5

[Armen [:armenzg]]

Bug 975438 does not seem to have been related.

Comment 6

[Carsten Book [:Tomcat]]

seems backlog is starting to build up again :/

Comment 7

[Chris AtLee [:catlee]]

The problem was due to DNS entries not being created for new spot instances. We've created those now, and are seeing good response times from the build farm now.