1637942 - cost reduction for application-services and glean

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

+++ This bug was initially created as a clone of Bug #1637216 +++

Similarly to android-components, https://github.com/mozilla/application-services/ tasks are quite expensive. Worth investigating whether there's downgrades that we could do that could preserve us roughly same results.

This includes:

• instance type downgrade - see bug 1637216.
• reduce network usage in gradle tasks - see bug 1625126
• SDK baking - see bug 1622339
• stop reinstalling gradle for each tasks - see bug 1637891
• potentially use smaller instances for linter jobs? - see bug 1637205

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

Note: should we file a separate bug for glean or track that here?

Comment 2

[Edouard Oger [:eoger]]

I wrote an email about this about a month ago, reproducing:

Jon Buckley and I spent some time today trying to figure out how come we spend so much money:
First, our Taskcluster configuration seems sub-optimal, here's why:

• We provision c5d.4xlarge machines for our runs that come with 32GiB of memory and 16 vCPUs. We could switch to c5d.2xlarge machines which come up with 16GiB of memory (more than enough to avoid Out Of Memory errors) and 8 vCPUs. They cost half as much, $0.38/hour versus $0.76/hour, and we believe just doing this small change would divide our cost by two. We would need to check if that CPU change affects the build times significantly.
• Our minimal capacity is set to 2 machines, which means at any time of the day we have at least 2 machines potentially idling waiting for runs. Our CI runs take between 3 (pull-requests) to 15 (master merge) machines to run. So this means these 2 machines idling are not even enough to cover the basic case and new machines are started in response (scale-up) anyway. We recommend to set the minimal capacity to 0 as this type of machine can start up pretty fast.
• After scaling-up, machines take 15 minutes (shutdown.afterIdleSeconds = 900) to be shut-down. We believe in with a 40min average run you'd be very lucky to end up re-using a machine that was previously used to run another task. We recommend shutting down the machines right after they are done running their tasks.

We believe these kind of low-hanging fruit changes could reduce the cost of our runs from 50% to 80%.

On the software side, as mentioned on #1502 we build the Rust megazord 13 times (!), which takes about 40 minutes each. We could spend some time (less than a week I estimate) to build the megazord once on a beefy machine (what we use right now) then run the subsequent steps on small machines for a few minutes. This would reduce our costs also quite a bit.

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

FWIW, I gave this a try in bug 1648046 where I baked the resource monitoring tool to see what's our usage on the existing resources.
tl;dr - memory is abour 10% of usage while CPU stands around 50% in the most consuming components (such as full-megazord and lockbox). I've tried reducing the instance types from c5d.4xlarge to c5xlarge (4x cheaper) but the additional time needed is 3x more. So the current instances are having a high factor of utility which makes them good for existing workload.

Moreover, I've been running the costs for the entire project on a monthly basis and it's quite low so not worth investing time right now in improving.

(In reply to Edouard Oger [:eoger] from comment #2)

Jon Buckley and I spent some time today trying to figure out how come we spend so much money

We might want to cross our notes, as the numbers I'm seeing are pretty low.

First, our Taskcluster configuration seems sub-optimal, here's why:

• We provision c5d.4xlarge machines for our runs that come with 32GiB of memory and 16 vCPUs. We could switch to c5d.2xlarge machines which come up with 16GiB of memory (more than enough to avoid Out Of Memory errors) and 8 vCPUs. They cost half as much, $0.38/hour versus $0.76/hour, and we believe just doing this small change would divide our cost by two. We would need to check if that CPU change affects the build times significantly.

I'll give this a try.

• Our minimal capacity is set to 2 machines, which means at any time of the day we have at least 2 machines potentially idling waiting for runs. Our CI runs take between 3 (pull-requests) to 15 (master merge) machines to run. So this means these 2 machines idling are not even enough to cover the basic case and new machines are started in response (scale-up) anyway. We recommend to set the minimal capacity to 0 as this type of machine can start up pretty fast.

minCapacity is now set to 0.

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Note to self: looking at the numbers again, the actual work is relatively cheap, seems like we're spending a bunch of time in the overhead of the workers. I'll look into that at some point, to better manage the resources and caching.

Comment 5

[[e-mail] (:bhearsum)]

(In reply to Edouard Oger [:eoger] from comment #2)

• After scaling-up, machines take 15 minutes (shutdown.afterIdleSeconds = 900) to be shut-down. We believe in with a 40min average run you'd be very lucky to end up re-using a machine that was previously used to run another task. We recommend shutting down the machines right after they are done running their tasks.

We did a similar thing for Firefox CI workers recently - idle time is near zero for most worker types (https://bugzilla.mozilla.org/show_bug.cgi?id=1641624). We should probably do the same here.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

:jlorenzo just did this a week ago for android-components; it is easy to do and relatively safe;

I took a look at the workers for app-services and glean, 90% of the total time we pay for goes to idle- this is broken down into overprovisioned workers (50-65% of total time paid for), post task idletimeout (a bit more than actual time running tasks), and various bootstrapping/hung workers for the small remainder.

If you could create smaller pools (accept some delays in startup or small backlogs) and reduce idleTimeout to 15, I think this could be cleaned up significantly. There is not a lot of spending in this project compared to Firefox CI, but these are easy changes that could save $500-$800/month.

Comment 7

[[e-mail] (:bhearsum)]

All blocking bugs have been fixed. I'm calling this done.