Open Bug 1640046 Opened 2 months ago Updated 20 days ago

Experiment with larger instances running more tasks for builds

Categories

(Release Engineering :: Firefox-CI Administration, task)

Tracking

(Not tracked)

People

(Reporter: glandium, Assigned: glandium)

References

Details

Attachments

(2 files)

We're currently building on 4xlarge instances, which have 16 cores. A large part of the build time is actually spent not using those 16 cores at all. It is very likely that getting the next level of instance size (9xlarge), but running more tasks in parallel, would yield better build times, or worst case, similar build times, but with less money spending, since we'd be doing more tasks with less (but larger) instances.

Start with 9xlarge instead of 4xlarge (36 vs 16 cores), with 3 tasks per
worker. I suspect we could go with more, but let's start small.

The change landed, and a push to try using the new pool led to only one worker (despite maxCapacity: 2), and no more than one task running at the same time (despite capacityPerInstance: 3)? Any idea what's up?

Looking around, though, it seems weird that gecko-1/b-linux-aws only has maxCapacity: 50, that seems ridiculously low... and impossible to be true.

Also, looking at a bunch of recent decision workers, which are supposed to be running multiple tasks (capacityPerInstance: 2), I actually can't find a worker that has run two tasks at the same time...

Flags: needinfo?(mozilla)
Flags: needinfo?(dustin)
Flags: needinfo?(aki)
Depends on: 1640088

See Bug 1640088 for why the capacity is not effective.

Flags: needinfo?(mozilla)
Flags: needinfo?(dustin)
Flags: needinfo?(aki)

This should run 2 workers as initially intended.

Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/ci/ci-configuration/rev/d0162ed8de64
Bump maxCapacity to 6 for the b-linux-experiment pool. r=tomprince
You need to log in before you can comment on or make changes to this bug.