1560801 - Workers in terraform-packet worker type getting stuck with exception

Status: RESOLVED WORKSFORME

(Taskcluster :: Workers, defect)

Description

[Zsolt Fay [:zfay]]

After monitoring the https://tools.taskcluster.net/provisioners/terraform-packet/worker-types/gecko-t-linux pool today I noticed that a large number of workers stopped picking up tasks. All the affected workers finished their last job as exception.
Treeherder link from earlier:
https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=252607557&searchStr=android%2C7.0%2Cx86-64%2Copt%2Cweb%2Cplatform%2Ctests%2Ctest-android-em-7.0-x86_64%2Fopt-web-platform-tests-reftests-e10s-1%2Cw%28wr1%29&tochange=71d648e912ef3e45b9faf3c63894152e1874c037&fromchange=c497c45f090c25395b1b07680034e7653fb94d7f

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Any idea what's going on with those packet.net instances?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

lets see what :wcosta would know.

Comment 3

[Chris Cooper [:coop] (he/him)]

wcosta is on PTO this week, but I can take a look.

Do we have a list of the affected workerIDs and/or instances?

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Hi Adrian, are there any stuck machines left? The link in comment 0 shows them running as normal.

Comment 5

[Adrian Pop]

I've checked the machines from gecko-t-linux, currently, all the active workers are running. I'll keep on monitoring them