1838648 - Reduce scratch disks for GCP pools

Status: RESOLVED FIXED

(Release Engineering :: Firefox-CI Administration, task)

Description

[Andrew Halberstadt [:ahal]]

We realized that most of our GCP pools have way too many scratch disks. We likely only need a single disk (or the minimum for the given instance type) per pool.

More information on valid # of SSDs by instance type here:
https://cloud.google.com/compute/docs/disks/local-ssd

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1838648 - Add a check to avoid unnecessary GCP SSDs, r?#releng-reviewers!

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1838648 - Reduce number of scratch disks for GCP pools, r?#releng-reviewers!,masterwayz!

Depends on D181112

Comment 3

[Julien Cristau [:jcristau]]

The extra scratch disks make it more likely we run out of quota and can't spin up as many workers as we'd like, so they might be hurting our end-to-end times. We should double check whether they really help performance.

Comment 4

[Julien Cristau [:jcristau]]

Stealing this.

Comment 5

[Julien Cristau [:jcristau]]

Attachment: Bug 1838648 - remove obsolete comment about misc-gcp pool's disk space.

This was copied over from the corresponding aws pool, but is not true in
GCP, since each scratch disk is 375GB.

Comment 6

[Julien Cristau [:jcristau]]

Attachment: Bug 1838648 - reduce scratch disks for misc-gcp pools.

These only run short/simple tasks, they don't need 4x375GB scratch
space, even with capacityPerInstance: 8.

Comment 7

[Julien Cristau [:jcristau]]

Attachment: Bug 1838648 - reduce scratch disks for */linux-gcp pools.

The linux-gcp workers for the ci-t, code-analysis-{1,3}, nss-{1,3},
releng-{t,1,3} and taskgraph-t groups are likely not IO bound and can do
with a single scratch disk.

Comment 8

[Julien Cristau [:jcristau]]

Attachment: Bug 1838648 - reduce scratch disks in gecko-1/b-linux-gcp-bug1797804-n2 pool.

Comment 9

[Julien Cristau [:jcristau]]

Attachment: Bug 1838648 - reduce scratch disks in mobile-{1,3}/b-linux-large-gcp pools.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/78b4facf0ee2
remove obsolete comment about misc-gcp pool's disk space. r=releng-reviewers,ahal
https://hg.mozilla.org/ci/ci-configuration/rev/a478824bf8da
reduce scratch disks for misc-gcp pools. r=releng-reviewers,ahal
https://hg.mozilla.org/ci/ci-configuration/rev/93607eccd3ae
reduce scratch disks for */linux-gcp pools. r=releng-reviewers,ahal
https://hg.mozilla.org/ci/ci-configuration/rev/8d5943ec810e
reduce scratch disks in gecko-1/b-linux-gcp-bug1797804-n2 pool. r=releng-reviewers,ahal
https://hg.mozilla.org/ci/ci-configuration/rev/76a8ee6ac1c7
reduce scratch disks in mobile-{1,3}/b-linux-large-gcp pools. r=ahal
https://hg.mozilla.org/ci/ci-configuration/rev/dfd80dcda744
Add a check to avoid unnecessary GCP SSDs, r=releng-reviewers,gabriel

Comment 11

[Julien Cristau [:jcristau]]

Attachment: Bug 1838648 - check against number of scratch disks smaller than the minimum. r?ahal

Error out if we define a pool with too few scratch disks for the machine type.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/1dee85457c9c
check against number of scratch disks smaller than the minimum. r=ahal

Comment 13

[Julien Cristau [:jcristau]]

Let's call this one done and we can have further improvements in separate bugs.