1109752 - All trees closed due to high AWS pending test backlog

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Ryan VanderMeulen [:RyanVM]]

I'm seeing high numbers of pending AWS tests across all trees and nagios is alerting in #buildduty about high numbers of pending jobs.

All trees closed, including Gaia.

Comment 1

[Chris Cooper [:coop] (he/him)]

New instances are failing to start buildbot because runner can't find hg via hgtool.py. We're bleeding capacity as old instances terminate themselves to pick up the new image.

Comment 2

[Hal Wine [:hwine] use NI!]

Clarifying this is a configuration issue on the AWS machines (can't find the hgtool.py script), not an issue interacting with hg.m.o

Comment 3

[Justin Wood (:Callek)]

Attachment: fix

This patch should fix the root of the problem

In parallel :rail is reverting the golden AMI's to yesterday's ones which will avoid this bustage alltogether

Comment 4

[Ryan VanderMeulen [:RyanVM]]

As an update, we're just waiting for the current pending backlog to come down before reopening. Rail's revert is working for getting the line moving in the right direction :)

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Backlog is looking better and new linux test jobs appear to be starting reasonably fast now. I'm reopening everything.

Comment 6

[Justin Wood (:Callek)]

Pushed http://hg.mozilla.org/build/puppet/rev/a2e182a28c4e

Comment 7

[Justin Wood (:Callek)]

Cautiously optimistic here, marking as fixed.

We'll know for sure after tomorrow's AMI's get generated.

A link that showed the problem today: https://www.hostedgraphite.com/da5c920d/grafana/#/dashboard/temp/e5db589335c850ef95f52b85c2585442aa61c401?panelId=5&fullscreen

Comment 8

[Justin Wood (:Callek)]

and I landed an unsaved version, and tested said version -- thus I caused bustage.

The fix:
https://hg.mozilla.org/build/puppet/rev/01a37f44eafe
https://hg.mozilla.org/build/puppet/rev/521aa8dd8a02

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

I don't like the conditional here, as depending on install order /usr/bin/hg may end up pointing to the releng hg or the system hg.

Maybe the two packages should explicitly conflict, so that only one can be installed?