1585603 - Switch to new GCP addons workers

Status: VERIFIED FIXED

(Release Engineering :: Release Automation: Other, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

This looks to be a release only worker, can't see anything on nightly.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Bug 1585603 - enable GCP addonworkers. r=catlee

Switch addonworkers from AWS to GCP.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

TODO - add autoscaling. We have 4 prod machines in AWS, need to know average job time, if we can save time by scaling up.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: GitHub Pull Request - k8s-autoscale

Both gecko-1-addon-dev and gecko-1-addon work on try (eg this push), but I am unsure of how to test gecko-3-addon prior to landing on beta. CoT seems to preclude that.

The jwtSecret values needed to be updated for gecko-1-addon, and there was a deploy for that. There is the possibility that gecko-3-addon was also updated but not deployed, so we should look out for 401 errors from amo prod.

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

We had some issues with Jenkins yesterday and couldn't land this in time for Monday betas. Will attempt to land this for Thursday betas.

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

Merged https://github.com/mozilla-releng/k8s-autoscale/pull/45 to master and production.

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: fix missing configs in k8s-autoscale

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Status update:

• Fixed breakage in k8s auto-scaling with https://bugzilla.mozilla.org/attachment.cgi?id=9099591
• set the min-replicas to 0 in gecko-1-addon so we have 0 wasted resources; turns out it works smoothly, we have no pods allocated
• will push in-tree patch once things are stable, hopefully in time for Thursday's beta

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d8ce65453c48
enable GCP addonworkers. r=catlee

Comment 9

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/d8ce65453c48

Comment 10

[Mihai Tabara [:mtabara]⌚️GMT]

Addon workers worker like a charm in beta1 - see https://tools.taskcluster.net/groups/cwGZAl94Ru6DUmMGGspRgQ/tasks/LQ-zaUv8TYeGZIU5Mj3pxg/details