1797804 - Investigate GCP performance improvements

Status: RESOLVED WORKSFORME

(Release Engineering :: Firefox-CI Administration, task)

Description

[Andrew Halberstadt [:ahal]]

While migrating tasks from AWS to GCP, we've noticed that in general they appear to run a bit slower than they did in AWS. While CPU and memory should be equivalent, there may be another factor like disk speed or network throughput at play here.

We should see if we can tweak the pools to make tasks run a little faster like they do in AWS.

Comment 1

[Julien Cristau [:jcristau]]

A couple of ideas: we're currently using n2 and setting minCpuPlatform to CascadeLake. We could:

• remove the minCpuPlatform setting which would allow us to get newer IceLake CPUs
• look at c2 for build pools, which is also CascadeLake generation but higher frequency and billed as "compute-optimized" where n2 is "general-purpose"; c2 also happens to currently appear twice as cheap as n2, according to this month's spot pricing.

Refs:
https://cloud.google.com/compute/docs/cpu-platforms#intel_processors
https://cloud.google.com/compute/all-pricing

Comment 2

[Julien Cristau [:jcristau]]

Attachment: Bug 1797804 - add gecko-1 linux build pool on gcp using c2 machine series. r?jlorenzo,masterwayz

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/01c7f0b97cd4
add gecko-1 linux build pool on gcp using c2 machine series. r=MasterWayZ

Comment 4

[Julien Cristau [:jcristau]]

Attachment: Bug 1797804 - fix b-linux-gcp-bug1797804-c2 machine type

Custom machine types aren't available here.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/0156a0a00c39
fix b-linux-gcp-bug1797804-c2 machine type r=MasterWayZ

Comment 6

[Julien Cristau [:jcristau]]

Attachment: Bug 1797804 - add b-linux-xlarge-gcp-bug1797804-c2 pool to test shippable builds. r?masterwayz

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/10cd3c367587
add b-linux-xlarge-gcp-bug1797804-c2 pool to test shippable builds. r=MasterWayZ

Comment 8

[Julien Cristau [:jcristau]]

Attachment: Bug 1797804 - fix b-linux-xlarge-gcp-bug1797804-c2 machine type. r?masterwayz

I should have double checked the vCPU count, turns out over 16 they're
not powers of two.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/c7423cfdb54c
fix b-linux-xlarge-gcp-bug1797804-c2 machine type. r=MasterWayZ

Comment 10

[Julien Cristau [:jcristau]]

Pausing my work on this for now.

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

:jcristau- there are worker pools leftover in: https://hg.mozilla.org/ci/ci-configuration/file/tip/worker-pools.yml, is there more work to do on them? should we clean up pools? more experiments to run?

Comment 12

[Julien Cristau [:jcristau]]

Yes, not yet, and yes.

Comment 13

[Julien Cristau [:jcristau]]

Attachment: Bug 1797804 - remove no longer needed test pool. r?#releng

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/450e60ddc287
remove no longer needed test pool. r=releng-reviewers,bhearsum

Comment 15

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:jcristau, maybe it's time to close this bug?
For more information, please visit BugBot documentation.