Closed Bug 2091006 Opened 10 days ago Closed 10 days ago

Use larger workers for macOS compilation tasks

Categories

(Release Engineering :: General, enhancement)

enhancement

Tracking

(firefox162 fixed)

RESOLVED FIXED
Tracking Status
firefox162 --- fixed

People

(Reporter: hneiva, Assigned: hneiva)

References

Details

Attachments

(1 external file)

Move macOS compilation tasks from b-linux-docker-amd (16 vCPUs) to b-linux-docker-large-amd (30 vCPUs) to reduce CI build times. The configuration changes are in taskcluster/kinds/build/macosx.yml and taskcluster/kinds/instrumented-build/kind.yml.

Expected improvement

These results represent potential gains with sccache disabled. In normal CI, where sccache is enabled, I expect build-time reductions of roughly 15–40%, depending on the build type and cache hit rate.

Worker-size experiment

The comparison used the same source base (db547b61903f9dee2ddb1a86de3beeab88b2c923) with identical plugin settings. Sccache was disabled (SCCACHE_DISABLE=1, with USE_SCCACHE unset), and both pushes used --no-artifact. Each task had one attempt, with no retries. Durations measure task execution, excluding queue and dependency waits.

All 22 successfully completed tasks ran faster on the larger workers. Summed execution time fell by 34.0%. Allocated vCPU-minutes increased by approximately 23.8% (16 × baseline duration versus 30 × comparison duration); this is an allocation estimate, not a billing measurement. These measurements are one run per configuration.

Execution times with sccache disabled

Durations are minutes:seconds, rounded to the nearest second. Time reductions use the unrounded durations. Each duration links to its Taskcluster task.

Task 16 vCPUs 30 vCPUs Time reduction
build-macosx64-aarch64-add-on-devel/opt 45:25 37:17 17.92%
build-macosx64-aarch64-asan-fuzzing/opt 46:41 27:57 40.11%
build-macosx64-aarch64-devedition/opt 55:37 32:52 40.91%
build-macosx64-aarch64-fuzzing/debug 51:44 32:01 38.12%
build-macosx64-aarch64-nightlyasrelease/opt 53:51 32:20 39.96%
build-macosx64-aarch64-noopt/debug 38:00 20:20 46.48%
build-macosx64-aarch64-reproduced/opt 54:16 31:13 42.48%
build-macosx64-aarch64-shippable/opt 55:54 31:21 43.92%
build-macosx64-aarch64/debug 47:10 29:28 37.53%
build-macosx64-aarch64/opt 39:41 32:48 17.37%
build-macosx64-asan-fuzzing/opt 46:58 27:40 41.09%
build-macosx64-fuzzing/debug 55:54 32:09 42.50%
build-macosx64-noopt/debug 36:49 22:16 39.50%
build-macosx64-reproduced/opt 58:02 33:59 41.43%
build-macosx64-x64-add-on-devel/opt 44:20 37:33 15.31%
build-macosx64-x64-devedition/opt 58:18 36:20 37.68%
build-macosx64-x64-shippable/opt 56:46 32:00 43.63%
build-macosx64/debug 45:51 27:45 39.47%
build-macosx64/opt 39:20 32:18 17.89%
instrumented-build-macosx64-aarch64-nightlyasrelease/opt 43:55 36:36 16.65%
instrumented-build-macosx64-aarch64-shippable/opt 45:48 39:20 14.12%
instrumented-build-macosx64-shippable/opt 40:34 34:44 14.37%

build-macosx64-appservices/opt failed in both the baseline and larger-worker run because sync15 treats deprecated usize::max_value() calls as errors. It is excluded from the timing aggregates.

Production runtime averages

STMO query 131219

Window: October 12 through November 8, 2026 (28 complete UTC days). Successful runs only, for the exact 22 compilation tasks that completed successfully in the worker-size experiment. Every returned run used gecko-3/b-linux-docker-amd (16 vCPUs). These averages reflect normal production cache settings.

Execution duration is resolved minus started; queue and dependency wait times are excluded. Duplicate task/run records are removed. Successful retry attempts are included, with their count reported separately. Interrupted and failed attempts are excluded. The query groups by repository and worker pool, so different configurations are kept separate.

Total: 28,866 successful runs across 22 task labels and 59 task/repository groups.

Mean execution time by task and repository

Values are minutes, followed by successful-run counts. An em dash means there were no successful runs for that task/repository in this window.

Task mozilla-central autoland mozilla-beta mozilla-release
build-macosx64-aarch64-add-on-devel/opt — — 18.17 (n=274) 16.57 (n=24)
build-macosx64-aarch64-asan-fuzzing/opt 14.28 (n=85) 24.20 (n=578) 12.87 (n=328) 10.66 (n=37)
build-macosx64-aarch64-devedition/opt — — 58.97 (n=334) —
build-macosx64-aarch64-fuzzing/debug 20.79 (n=85) 31.22 (n=576) 20.00 (n=328) —
build-macosx64-aarch64-nightlyasrelease/opt 57.31 (n=85) 56.97 (n=1,102) — —
build-macosx64-aarch64-noopt/debug 29.67 (n=84) — — —
build-macosx64-aarch64-reproduced/opt 56.69 (n=27) 52.77 (n=2) — —
build-macosx64-aarch64-shippable/opt 58.26 (n=86) 58.21 (n=2,018) 58.15 (n=334) 58.13 (n=39)
build-macosx64-aarch64/debug 14.42 (n=85) 19.66 (n=3,342) 18.84 (n=312) —
build-macosx64-aarch64/opt 14.49 (n=84) 21.57 (n=3,417) 18.82 (n=121) —
build-macosx64-asan-fuzzing/opt 12.81 (n=85) 25.43 (n=578) 13.15 (n=328) 11.36 (n=37)
build-macosx64-fuzzing/debug 20.20 (n=85) 30.92 (n=578) 19.99 (n=328) —
build-macosx64-noopt/debug 28.91 (n=84) — — —
build-macosx64-reproduced/opt 57.21 (n=26) 53.59 (n=1) — —
build-macosx64-x64-add-on-devel/opt — — 18.56 (n=274) 18.38 (n=24)
build-macosx64-x64-devedition/opt — — 59.19 (n=326) —
build-macosx64-x64-shippable/opt 59.13 (n=86) 58.75 (n=1,553) 58.53 (n=348) 58.14 (n=39)
build-macosx64/debug 15.51 (n=85) 21.57 (n=2,094) 19.27 (n=312) —
build-macosx64/opt 15.76 (n=85) 22.93 (n=1,935) 38.78 (n=2) 26.32 (n=2)
instrumented-build-macosx64-aarch64-nightlyasrelease/opt 45.01 (n=85) 24.77 (n=1,154) — —
instrumented-build-macosx64-aarch64-shippable/opt 45.71 (n=86) 22.30 (n=2,094) 45.46 (n=336) 46.04 (n=39)
instrumented-build-macosx64-shippable/opt 43.78 (n=86) 21.28 (n=1,529) 43.45 (n=336) 43.86 (n=39)

Distribution and sample details

Task Repository Successful runs Successful retry runs Mean (min) Median (min) p90 (min)
build-macosx64-aarch64-add-on-devel/opt mozilla-beta 274 17 18.17 12.61 32.47
build-macosx64-aarch64-add-on-devel/opt mozilla-release 24 1 16.57 11.63 29.41
build-macosx64-aarch64-asan-fuzzing/opt autoland 578 28 24.20 25.33 38.91
build-macosx64-aarch64-asan-fuzzing/opt mozilla-beta 328 7 12.87 7.96 27.21
build-macosx64-aarch64-asan-fuzzing/opt mozilla-central 85 0 14.28 7.73 34.34
build-macosx64-aarch64-asan-fuzzing/opt mozilla-release 37 0 10.66 8.06 20.47
build-macosx64-aarch64-devedition/opt mozilla-beta 334 33 58.97 58.76 61.50
build-macosx64-aarch64-fuzzing/debug autoland 576 23 31.22 32.65 45.81
build-macosx64-aarch64-fuzzing/debug mozilla-beta 328 5 20.00 14.82 35.13
build-macosx64-aarch64-fuzzing/debug mozilla-central 85 2 20.79 14.73 38.47
build-macosx64-aarch64-nightlyasrelease/opt autoland 1,102 59 56.97 56.79 59.12
build-macosx64-aarch64-nightlyasrelease/opt mozilla-central 85 3 57.31 56.95 59.27
build-macosx64-aarch64-noopt/debug mozilla-central 84 4 29.67 31.42 37.18
build-macosx64-aarch64-reproduced/opt autoland 2 1 52.77 52.69 52.85
build-macosx64-aarch64-reproduced/opt mozilla-central 27 0 56.69 56.21 58.43
build-macosx64-aarch64-shippable/opt autoland 2,018 98 58.21 57.96 60.49
build-macosx64-aarch64-shippable/opt mozilla-beta 334 23 58.15 57.82 60.72
build-macosx64-aarch64-shippable/opt mozilla-central 86 2 58.26 58.09 60.84
build-macosx64-aarch64-shippable/opt mozilla-release 39 3 58.13 57.84 61.07
build-macosx64-aarch64/debug autoland 3,342 97 19.66 14.70 34.22
build-macosx64-aarch64/debug mozilla-beta 312 10 18.84 14.04 34.14
build-macosx64-aarch64/debug mozilla-central 85 3 14.42 13.23 17.55
build-macosx64-aarch64/opt autoland 3,417 143 21.57 23.19 34.65
build-macosx64-aarch64/opt mozilla-beta 121 2 18.82 13.14 35.15
build-macosx64-aarch64/opt mozilla-central 84 2 14.49 12.39 28.52
build-macosx64-asan-fuzzing/opt autoland 578 18 25.43 26.92 40.09
build-macosx64-asan-fuzzing/opt mozilla-beta 328 12 13.15 8.35 26.89
build-macosx64-asan-fuzzing/opt mozilla-central 85 0 12.81 8.08 28.07
build-macosx64-asan-fuzzing/opt mozilla-release 37 0 11.36 8.51 20.07
build-macosx64-fuzzing/debug autoland 578 26 30.92 32.74 45.78
build-macosx64-fuzzing/debug mozilla-beta 328 9 19.99 14.99 34.86
build-macosx64-fuzzing/debug mozilla-central 85 3 20.20 14.82 36.22
build-macosx64-noopt/debug mozilla-central 84 3 28.91 30.15 36.60
build-macosx64-reproduced/opt autoland 1 0 53.59 53.59 53.59
build-macosx64-reproduced/opt mozilla-central 26 0 57.21 56.55 59.53
build-macosx64-x64-add-on-devel/opt mozilla-beta 274 5 18.56 12.93 33.05
build-macosx64-x64-add-on-devel/opt mozilla-release 24 1 18.38 13.37 31.16
build-macosx64-x64-devedition/opt mozilla-beta 326 28 59.19 58.92 61.97
build-macosx64-x64-shippable/opt autoland 1,553 115 58.75 58.56 61.06
build-macosx64-x64-shippable/opt mozilla-beta 348 31 58.53 58.13 61.15
build-macosx64-x64-shippable/opt mozilla-central 86 3 59.13 58.72 61.73
build-macosx64-x64-shippable/opt mozilla-release 39 1 58.14 58.25 60.03
build-macosx64/debug autoland 2,094 80 21.57 17.45 35.73
build-macosx64/debug mozilla-beta 312 11 19.27 14.58 33.09
build-macosx64/debug mozilla-central 85 2 15.51 13.51 27.29
build-macosx64/opt autoland 1,935 90 22.93 24.87 34.78
build-macosx64/opt mozilla-beta 2 0 38.78 37.53 40.03
build-macosx64/opt mozilla-central 85 2 15.76 12.62 28.14
build-macosx64/opt mozilla-release 2 1 26.32 12.93 39.70
instrumented-build-macosx64-aarch64-nightlyasrelease/opt autoland 1,154 43 24.77 28.88 41.66
instrumented-build-macosx64-aarch64-nightlyasrelease/opt mozilla-central 85 6 45.01 44.63 47.57
instrumented-build-macosx64-aarch64-shippable/opt autoland 2,094 50 22.30 15.31 41.45
instrumented-build-macosx64-aarch64-shippable/opt mozilla-beta 336 25 45.46 45.38 47.56
instrumented-build-macosx64-aarch64-shippable/opt mozilla-central 86 6 45.71 45.51 47.55
instrumented-build-macosx64-aarch64-shippable/opt mozilla-release 39 0 46.04 45.85 47.94
instrumented-build-macosx64-shippable/opt autoland 1,529 43 21.28 11.67 39.94
instrumented-build-macosx64-shippable/opt mozilla-beta 336 22 43.45 43.24 45.24
instrumented-build-macosx64-shippable/opt mozilla-central 86 1 43.78 43.61 45.90
instrumented-build-macosx64-shippable/opt mozilla-release 39 0 43.86 43.71 45.20
Regressions: 2091152
Status: NEW → RESOLVED
Closed: 10 days ago
Resolution: --- → FIXED
No longer regressions: 2091152
Blocks: 2090038
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: