1414511 - gecko-t-win10-64-gpu workers broken, backlog of windows 10 webgl + gpu jobs which are pending but don't run

Status: RESOLVED FIXED

(Taskcluster :: General, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

gecko-t-win10-64-gpu workers are broken, there is a backlog of windows 10 webgl + gpu jobs which are pending but don't run. See https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=4f270f7da185840a4775d0480c883eb17c4197c3&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=pending&filter-resultStatus=running&group_state=expanded

At the moment >1400 pending jobs.

First affected push? https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=ab983c341420850665acbad872d398cb3b8c9066&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=pending&filter-resultStatus=running

Comment 1

[Greg Arndt [:garndt]]

pending is down to 700 now.  It looks like grenade's cron script to kill off old workers ran and new ones came up about 40 minuts ago.