1274406 - Trees closed for massive windows test backlog

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Comment 1

[Chris AtLee [:catlee]]

We can reopen when pending counts drop below 1000 per platform.

https://www.hostedgraphite.com/da5c920d/86a8384e-d9cf-4208-989b-9538a1a53e4b/grafana2/dashboard/db/pending

Comment 2

[Selena Deckelmann :selenamarie :selena]

We'd seen an issue with taskcluster-queue/s3/cloudfront and http 503 codes earlier that we re-investigated. Amazon let us know that they'd fixed an underlying issue (probably didn't have enough partitions in the storage backing our buckets). 

The 503s are gone, and responsiveness seems more normal now, so we've ruled out that issue as the cause of the backlog at this point.

Comment 3

[Gregory Szorc [:gps]]

Backlog has cleared on everything except Try. And the backlog amount is about equivalent to what it was at this time yesterday.

Comment 4

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Windows test jobs still seem to have a 24 hour backlog on try, I'd like to bring that down a bit before reopening fully. 

I switched try to approval-required and added a note saying people can land with 'a=notonwindows' if they pinky-swear they're not going to test on Windows.

I'll keep an eye on things a bit longer tonight and probably reopen everything before heading out.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Trees were reopened at 2016-05-20T01:31:06+00:00.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

 ... by Wes.