1675032 - frequent backlog for gecko-t-win10-64-1803-hw Windows worker pool

Status: NEW

(Infrastructure & Operations :: RelOps: Windows OS, task)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Attachment: screenshot win10 backlog

Starting Saturday (!), sometimes a backlog of tasks for the gecko-t-win10-64-1803-hw pool is observed. It grows continuously without big jumps in pending counts from Try pushes requesting many tasks depending on this worker pool.

Mark, any idea what has changed?

Comment 1

[Mark Cornmesser [:markco]]

It looks like a significant number of nodes fell offline. I will do an audit and and work on getting the missing nodes back into the pool.

Comment 2

[Mark Cornmesser [:markco]]

90 nodes have dropped off. I have gone through and forced the chassis cartridges to reboot and reimaged the ones that did not come back. Moving forward, until we have better monitoring and automated recover, i will audit this pool the first week of each month.