Use larger workers for macOS compilation tasks
Categories
(Release Engineering :: General, enhancement)
Tracking
(firefox162 fixed)
| Tracking | Status | |
|---|---|---|
| firefox162 | --- | fixed |
People
(Reporter: hneiva, Assigned: hneiva)
References
Details
Attachments
(1 external file)
Move macOS compilation tasks from b-linux-docker-amd (16 vCPUs) to b-linux-docker-large-amd (30 vCPUs) to reduce CI build times. The configuration changes are in taskcluster/kinds/build/macosx.yml and taskcluster/kinds/instrumented-build/kind.yml.
Expected improvement
These results represent potential gains with sccache disabled. In normal CI, where sccache is enabled, I expect build-time reductions of roughly 15–40%, depending on the build type and cache hit rate.
Worker-size experiment
The comparison used the same source base (db547b61903f9dee2ddb1a86de3beeab88b2c923) with identical plugin settings. Sccache was disabled (SCCACHE_DISABLE=1, with USE_SCCACHE unset), and both pushes used --no-artifact. Each task had one attempt, with no retries. Durations measure task execution, excluding queue and dependency waits.
All 22 successfully completed tasks ran faster on the larger workers. Summed execution time fell by 34.0%. Allocated vCPU-minutes increased by approximately 23.8% (16 × baseline duration versus 30 × comparison duration); this is an allocation estimate, not a billing measurement. These measurements are one run per configuration.
Execution times with sccache disabled
Durations are minutes:seconds, rounded to the nearest second. Time reductions use the unrounded durations. Each duration links to its Taskcluster task.
| Task | 16 vCPUs | 30 vCPUs | Time reduction |
|---|---|---|---|
build-macosx64-aarch64-add-on-devel/opt |
45:25 | 37:17 | 17.92% |
build-macosx64-aarch64-asan-fuzzing/opt |
46:41 | 27:57 | 40.11% |
build-macosx64-aarch64-devedition/opt |
55:37 | 32:52 | 40.91% |
build-macosx64-aarch64-fuzzing/debug |
51:44 | 32:01 | 38.12% |
build-macosx64-aarch64-nightlyasrelease/opt |
53:51 | 32:20 | 39.96% |
build-macosx64-aarch64-noopt/debug |
38:00 | 20:20 | 46.48% |
build-macosx64-aarch64-reproduced/opt |
54:16 | 31:13 | 42.48% |
build-macosx64-aarch64-shippable/opt |
55:54 | 31:21 | 43.92% |
build-macosx64-aarch64/debug |
47:10 | 29:28 | 37.53% |
build-macosx64-aarch64/opt |
39:41 | 32:48 | 17.37% |
build-macosx64-asan-fuzzing/opt |
46:58 | 27:40 | 41.09% |
build-macosx64-fuzzing/debug |
55:54 | 32:09 | 42.50% |
build-macosx64-noopt/debug |
36:49 | 22:16 | 39.50% |
build-macosx64-reproduced/opt |
58:02 | 33:59 | 41.43% |
build-macosx64-x64-add-on-devel/opt |
44:20 | 37:33 | 15.31% |
build-macosx64-x64-devedition/opt |
58:18 | 36:20 | 37.68% |
build-macosx64-x64-shippable/opt |
56:46 | 32:00 | 43.63% |
build-macosx64/debug |
45:51 | 27:45 | 39.47% |
build-macosx64/opt |
39:20 | 32:18 | 17.89% |
instrumented-build-macosx64-aarch64-nightlyasrelease/opt |
43:55 | 36:36 | 16.65% |
instrumented-build-macosx64-aarch64-shippable/opt |
45:48 | 39:20 | 14.12% |
instrumented-build-macosx64-shippable/opt |
40:34 | 34:44 | 14.37% |
build-macosx64-appservices/opt failed in both the baseline and larger-worker run because sync15 treats deprecated usize::max_value() calls as errors. It is excluded from the timing aggregates.
Production runtime averages
Window: October 12 through November 8, 2026 (28 complete UTC days). Successful runs only, for the exact 22 compilation tasks that completed successfully in the worker-size experiment. Every returned run used gecko-3/b-linux-docker-amd (16 vCPUs). These averages reflect normal production cache settings.
Execution duration is resolved minus started; queue and dependency wait times are excluded. Duplicate task/run records are removed. Successful retry attempts are included, with their count reported separately. Interrupted and failed attempts are excluded. The query groups by repository and worker pool, so different configurations are kept separate.
Total: 28,866 successful runs across 22 task labels and 59 task/repository groups.
Mean execution time by task and repository
Values are minutes, followed by successful-run counts. An em dash means there were no successful runs for that task/repository in this window.
| Task | mozilla-central | autoland | mozilla-beta | mozilla-release |
|---|---|---|---|---|
build-macosx64-aarch64-add-on-devel/opt |
— | — | 18.17 (n=274) | 16.57 (n=24) |
build-macosx64-aarch64-asan-fuzzing/opt |
14.28 (n=85) | 24.20 (n=578) | 12.87 (n=328) | 10.66 (n=37) |
build-macosx64-aarch64-devedition/opt |
— | — | 58.97 (n=334) | — |
build-macosx64-aarch64-fuzzing/debug |
20.79 (n=85) | 31.22 (n=576) | 20.00 (n=328) | — |
build-macosx64-aarch64-nightlyasrelease/opt |
57.31 (n=85) | 56.97 (n=1,102) | — | — |
build-macosx64-aarch64-noopt/debug |
29.67 (n=84) | — | — | — |
build-macosx64-aarch64-reproduced/opt |
56.69 (n=27) | 52.77 (n=2) | — | — |
build-macosx64-aarch64-shippable/opt |
58.26 (n=86) | 58.21 (n=2,018) | 58.15 (n=334) | 58.13 (n=39) |
build-macosx64-aarch64/debug |
14.42 (n=85) | 19.66 (n=3,342) | 18.84 (n=312) | — |
build-macosx64-aarch64/opt |
14.49 (n=84) | 21.57 (n=3,417) | 18.82 (n=121) | — |
build-macosx64-asan-fuzzing/opt |
12.81 (n=85) | 25.43 (n=578) | 13.15 (n=328) | 11.36 (n=37) |
build-macosx64-fuzzing/debug |
20.20 (n=85) | 30.92 (n=578) | 19.99 (n=328) | — |
build-macosx64-noopt/debug |
28.91 (n=84) | — | — | — |
build-macosx64-reproduced/opt |
57.21 (n=26) | 53.59 (n=1) | — | — |
build-macosx64-x64-add-on-devel/opt |
— | — | 18.56 (n=274) | 18.38 (n=24) |
build-macosx64-x64-devedition/opt |
— | — | 59.19 (n=326) | — |
build-macosx64-x64-shippable/opt |
59.13 (n=86) | 58.75 (n=1,553) | 58.53 (n=348) | 58.14 (n=39) |
build-macosx64/debug |
15.51 (n=85) | 21.57 (n=2,094) | 19.27 (n=312) | — |
build-macosx64/opt |
15.76 (n=85) | 22.93 (n=1,935) | 38.78 (n=2) | 26.32 (n=2) |
instrumented-build-macosx64-aarch64-nightlyasrelease/opt |
45.01 (n=85) | 24.77 (n=1,154) | — | — |
instrumented-build-macosx64-aarch64-shippable/opt |
45.71 (n=86) | 22.30 (n=2,094) | 45.46 (n=336) | 46.04 (n=39) |
instrumented-build-macosx64-shippable/opt |
43.78 (n=86) | 21.28 (n=1,529) | 43.45 (n=336) | 43.86 (n=39) |
Distribution and sample details
| Task | Repository | Successful runs | Successful retry runs | Mean (min) | Median (min) | p90 (min) |
|---|---|---|---|---|---|---|
build-macosx64-aarch64-add-on-devel/opt |
mozilla-beta | 274 | 17 | 18.17 | 12.61 | 32.47 |
build-macosx64-aarch64-add-on-devel/opt |
mozilla-release | 24 | 1 | 16.57 | 11.63 | 29.41 |
build-macosx64-aarch64-asan-fuzzing/opt |
autoland | 578 | 28 | 24.20 | 25.33 | 38.91 |
build-macosx64-aarch64-asan-fuzzing/opt |
mozilla-beta | 328 | 7 | 12.87 | 7.96 | 27.21 |
build-macosx64-aarch64-asan-fuzzing/opt |
mozilla-central | 85 | 0 | 14.28 | 7.73 | 34.34 |
build-macosx64-aarch64-asan-fuzzing/opt |
mozilla-release | 37 | 0 | 10.66 | 8.06 | 20.47 |
build-macosx64-aarch64-devedition/opt |
mozilla-beta | 334 | 33 | 58.97 | 58.76 | 61.50 |
build-macosx64-aarch64-fuzzing/debug |
autoland | 576 | 23 | 31.22 | 32.65 | 45.81 |
build-macosx64-aarch64-fuzzing/debug |
mozilla-beta | 328 | 5 | 20.00 | 14.82 | 35.13 |
build-macosx64-aarch64-fuzzing/debug |
mozilla-central | 85 | 2 | 20.79 | 14.73 | 38.47 |
build-macosx64-aarch64-nightlyasrelease/opt |
autoland | 1,102 | 59 | 56.97 | 56.79 | 59.12 |
build-macosx64-aarch64-nightlyasrelease/opt |
mozilla-central | 85 | 3 | 57.31 | 56.95 | 59.27 |
build-macosx64-aarch64-noopt/debug |
mozilla-central | 84 | 4 | 29.67 | 31.42 | 37.18 |
build-macosx64-aarch64-reproduced/opt |
autoland | 2 | 1 | 52.77 | 52.69 | 52.85 |
build-macosx64-aarch64-reproduced/opt |
mozilla-central | 27 | 0 | 56.69 | 56.21 | 58.43 |
build-macosx64-aarch64-shippable/opt |
autoland | 2,018 | 98 | 58.21 | 57.96 | 60.49 |
build-macosx64-aarch64-shippable/opt |
mozilla-beta | 334 | 23 | 58.15 | 57.82 | 60.72 |
build-macosx64-aarch64-shippable/opt |
mozilla-central | 86 | 2 | 58.26 | 58.09 | 60.84 |
build-macosx64-aarch64-shippable/opt |
mozilla-release | 39 | 3 | 58.13 | 57.84 | 61.07 |
build-macosx64-aarch64/debug |
autoland | 3,342 | 97 | 19.66 | 14.70 | 34.22 |
build-macosx64-aarch64/debug |
mozilla-beta | 312 | 10 | 18.84 | 14.04 | 34.14 |
build-macosx64-aarch64/debug |
mozilla-central | 85 | 3 | 14.42 | 13.23 | 17.55 |
build-macosx64-aarch64/opt |
autoland | 3,417 | 143 | 21.57 | 23.19 | 34.65 |
build-macosx64-aarch64/opt |
mozilla-beta | 121 | 2 | 18.82 | 13.14 | 35.15 |
build-macosx64-aarch64/opt |
mozilla-central | 84 | 2 | 14.49 | 12.39 | 28.52 |
build-macosx64-asan-fuzzing/opt |
autoland | 578 | 18 | 25.43 | 26.92 | 40.09 |
build-macosx64-asan-fuzzing/opt |
mozilla-beta | 328 | 12 | 13.15 | 8.35 | 26.89 |
build-macosx64-asan-fuzzing/opt |
mozilla-central | 85 | 0 | 12.81 | 8.08 | 28.07 |
build-macosx64-asan-fuzzing/opt |
mozilla-release | 37 | 0 | 11.36 | 8.51 | 20.07 |
build-macosx64-fuzzing/debug |
autoland | 578 | 26 | 30.92 | 32.74 | 45.78 |
build-macosx64-fuzzing/debug |
mozilla-beta | 328 | 9 | 19.99 | 14.99 | 34.86 |
build-macosx64-fuzzing/debug |
mozilla-central | 85 | 3 | 20.20 | 14.82 | 36.22 |
build-macosx64-noopt/debug |
mozilla-central | 84 | 3 | 28.91 | 30.15 | 36.60 |
build-macosx64-reproduced/opt |
autoland | 1 | 0 | 53.59 | 53.59 | 53.59 |
build-macosx64-reproduced/opt |
mozilla-central | 26 | 0 | 57.21 | 56.55 | 59.53 |
build-macosx64-x64-add-on-devel/opt |
mozilla-beta | 274 | 5 | 18.56 | 12.93 | 33.05 |
build-macosx64-x64-add-on-devel/opt |
mozilla-release | 24 | 1 | 18.38 | 13.37 | 31.16 |
build-macosx64-x64-devedition/opt |
mozilla-beta | 326 | 28 | 59.19 | 58.92 | 61.97 |
build-macosx64-x64-shippable/opt |
autoland | 1,553 | 115 | 58.75 | 58.56 | 61.06 |
build-macosx64-x64-shippable/opt |
mozilla-beta | 348 | 31 | 58.53 | 58.13 | 61.15 |
build-macosx64-x64-shippable/opt |
mozilla-central | 86 | 3 | 59.13 | 58.72 | 61.73 |
build-macosx64-x64-shippable/opt |
mozilla-release | 39 | 1 | 58.14 | 58.25 | 60.03 |
build-macosx64/debug |
autoland | 2,094 | 80 | 21.57 | 17.45 | 35.73 |
build-macosx64/debug |
mozilla-beta | 312 | 11 | 19.27 | 14.58 | 33.09 |
build-macosx64/debug |
mozilla-central | 85 | 2 | 15.51 | 13.51 | 27.29 |
build-macosx64/opt |
autoland | 1,935 | 90 | 22.93 | 24.87 | 34.78 |
build-macosx64/opt |
mozilla-beta | 2 | 0 | 38.78 | 37.53 | 40.03 |
build-macosx64/opt |
mozilla-central | 85 | 2 | 15.76 | 12.62 | 28.14 |
build-macosx64/opt |
mozilla-release | 2 | 1 | 26.32 | 12.93 | 39.70 |
instrumented-build-macosx64-aarch64-nightlyasrelease/opt |
autoland | 1,154 | 43 | 24.77 | 28.88 | 41.66 |
instrumented-build-macosx64-aarch64-nightlyasrelease/opt |
mozilla-central | 85 | 6 | 45.01 | 44.63 | 47.57 |
instrumented-build-macosx64-aarch64-shippable/opt |
autoland | 2,094 | 50 | 22.30 | 15.31 | 41.45 |
instrumented-build-macosx64-aarch64-shippable/opt |
mozilla-beta | 336 | 25 | 45.46 | 45.38 | 47.56 |
instrumented-build-macosx64-aarch64-shippable/opt |
mozilla-central | 86 | 6 | 45.71 | 45.51 | 47.55 |
instrumented-build-macosx64-aarch64-shippable/opt |
mozilla-release | 39 | 0 | 46.04 | 45.85 | 47.94 |
instrumented-build-macosx64-shippable/opt |
autoland | 1,529 | 43 | 21.28 | 11.67 | 39.94 |
instrumented-build-macosx64-shippable/opt |
mozilla-beta | 336 | 22 | 43.45 | 43.24 | 45.24 |
instrumented-build-macosx64-shippable/opt |
mozilla-central | 86 | 1 | 43.78 | 43.61 | 45.90 |
instrumented-build-macosx64-shippable/opt |
mozilla-release | 39 | 0 | 43.86 | 43.71 | 45.20 |
| Assignee | ||
Comment 1•10 days ago
|
||
Comment 3•10 days ago
|
||
| bugherder | ||
Description
•