1602637 - packet.net: idle workers

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Andrew Erickson [:aerickson]]

The following packet.net workers aren't reporting to TC (per https://firefox-ci-tc.services.mozilla.co/provisioners/terraform-packet/worker-types/gecko-t-linux):

['machine-0', 'machine-11', 'machine-20', 'machine-42', 'machine-44', 'machine-46']

I have access to the packet.net console and can reboot the hosts, but not sure if there's any debugging to do.

• machine-23 is also not working due to issues discussed in https://bugzilla.mozilla.org/show_bug.cgi?id=1596892

The queue is pretty heavily loaded currently (I have an alert when there are 600+ jobs for 4+ hour that's firing, https://earthangel-b40313e5.influxcloud.net/d/wIJoZ4HWk/android-queues?orgId=1&fullscreen&panelId=10&refresh=5m).

Thanks,
Andy

Comment 1

[Wander Lairson Costa]

I rebooted all faulty machines, but I believe the real problem is that 60 machines is no longer enough.

Comment 2

[Andrew Erickson [:aerickson]]

['machine-0', 'machine-11', 'machine-20', 'machine-42', 'machine-44', 'machine-46'] are working again.

machine-23 is still quarantined. I'll follow up in that ticket.

I think we are very close to needing more instances. I'll keep an eye on the graphs.