1640046 - Experiment with larger instances running more tasks for builds

Status: RESOLVED WONTFIX

(Release Engineering :: Firefox-CI Administration, task)

Description

[Mike Hommey [:glandium]]

We're currently building on 4xlarge instances, which have 16 cores. A large part of the build time is actually spent not using those 16 cores at all. It is very likely that getting the next level of instance size (9xlarge), but running more tasks in parallel, would yield better build times, or worst case, similar build times, but with less money spending, since we'd be doing more tasks with less (but larger) instances.

Comment 1

[Mike Hommey [:glandium]]

Attachment: Bug 1640046 - Create a pool to experiment with larger workers for builds.

Start with 9xlarge instead of 4xlarge (36 vs 16 cores), with 3 tasks per
worker. I suspect we could go with more, but let's start small.

Comment 2

[Mike Hommey [:glandium]]

The change landed, and a push to try using the new pool led to only one worker (despite maxCapacity: 2), and no more than one task running at the same time (despite capacityPerInstance: 3)? Any idea what's up?

Looking around, though, it seems weird that gecko-1/b-linux-aws only has maxCapacity: 50, that seems ridiculously low... and impossible to be true.

Also, looking at a bunch of recent decision workers, which are supposed to be running multiple tasks (capacityPerInstance: 2), I actually can't find a worker that has run two tasks at the same time...

Comment 3

[Tom Prince [:tomprince]]

See Bug 1640088 for why the capacity is not effective.

Comment 4

[Mike Hommey [:glandium]]

Attachment: Bug 1640046 - Bump maxCapacity to 6 for the b-linux-experiment pool.

This should run 2 workers as initially intended.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/d0162ed8de64
Bump maxCapacity to 6 for the b-linux-experiment pool. r=tomprince

Comment 6

[Mike Hommey [:glandium]]

IIRC this didn't yield interesting results, and the pool was removed in bug 1818761