Parallelize linters in a more balanced way
Categories
(Developer Infrastructure :: Lint and Formatting, enhancement)
Tracking
(firefox162 fixed)
| Tracking | Status | |
|---|---|---|
| firefox162 | --- | fixed |
People
(Reporter: marco, Assigned: ahochheiden)
References
(Blocks 2 open bugs)
Details
Attachments
(5 external files, 2 obsolete files)
We are kind of running them in parallel already thanks to the ProcessPoolExecutor in roller.py, but this is not enough.
First of all, we are not parallelizing enough, so some take a very long time.
E.g. in a recent task I see:
[task 2025-10-16T09:01:46.349+00:00] 09:01:46.349 avoid-blacklist-and-whitelist (92) | Passing the following paths:
[task 2025-10-16T09:01:46.349+00:00] /builds/worker/checkouts/gecko/intl
[task 2025-10-16T09:01:46.349+00:00] /builds/worker/checkouts/gecko/remote
[task 2025-10-16T09:01:46.350+00:00] /builds/worker/checkouts/gecko/ipc
[task 2025-10-16T09:01:46.350+00:00] /builds/worker/checkouts/gecko/media
[task 2025-10-16T09:01:46.350+00:00] /builds/worker/checkouts/gecko/docshell
[task 2025-10-16T09:01:46.350+00:00] /builds/worker/checkouts/gecko/toolkit
[task 2025-10-16T09:01:46.350+00:00] /builds/worker/checkouts/gecko/testing
[task 2025-10-16T09:01:46.350+00:00] /builds/worker/checkouts/gecko/view
[task 2025-10-16T09:01:46.350+00:00] /builds/worker/checkouts/gecko/mobile
and then later:
avoid-blacklist-and-whitelist (92) | Finished in 352.67 seconds.
Second, we could parallize using threads or asyncio, since the LineType linters (StringType and RegexType) are I/O heavy.
| Reporter | ||
Comment 1•3 months ago
|
||
This doesn't only apply to LineType, but also ExternalType. For example, eslint is often slow and I see things like:
[task 2026-07-29T03:03:29.477+00:00] 03:03:29.477 eslint (241) | ESLint command: /builds/worker/fetches/node/bin/node /builds/worker/checkouts/gecko/node_modules/eslint/bin/eslint.js --format json --no-error-on-unmatched-pattern --ignore-pattern testing/performance/mobile-startup/newssite-nuxt --ignore-pattern testing/xpcshell/node_ip --ignore-pattern testing/web-platform/tests/resources/webidl2 --ignore-pattern testing/talos/talos/tests/kraken --ignore-pattern testing/talos/talos/tests/offscreencanvas/benchmarks/video/demuxer_mp4.js --ignore-pattern testing/xpcshell/node-http2 --ignore-pattern testing/xpcshell/node-ws --ignore-pattern testing/talos/talos/tests/v8_7 --ignore-pattern testing/mochitest/pywebsocket3 --ignore-pattern testing/talos/talos/tests/devtools/addon/content/pages --ignore-pattern testing/web-platform/mozilla/tests/webgpu --ignore-pattern testing/web-platform/tests/tools/third_party --ignore-pattern testing/mozbase/mozproxy/mozproxy/backends/mitm/scripts/catapult --ignore-pattern testing/xpcshell/dns-packet --ignore-pattern testing/modules/sinon-7.2.7.js --ignore-pattern testing/xpcshell/moz-http2/package-lock.json --ignore-pattern testing/talos/talos/tests/offscreencanvas/benchmarks/video/mp4box.all.min.js --ignore-pattern testing/talos/talos/tests/dromaeo --ignore-pattern testing/web-platform/tests/tools/wave/package-lock.json /builds/worker/checkouts/gecko/eslint.config.mjs /builds/worker/checkouts/gecko/.codex /builds/worker/checkouts/gecko/eslint-ignores.config.mjs /builds/worker/checkouts/gecko/hal /builds/worker/checkouts/gecko/testing /builds/worker/checkouts/gecko/startupcache /builds/worker/checkouts/gecko/accessible /builds/worker/checkouts/gecko/.babel-eslint.rc.js
[task 2026-07-29T03:03:30.924+00:00] 03:03:30.924 eslint (240) | Finished in 13.30 seconds
[task 2026-07-29T03:03:56.280+00:00] 03:03:56.278 eslint (244) | Prettier command: /builds/worker/fetches/node/bin/node /builds/worker/checkouts/gecko/node_modules/prettier/bin/prettier.cjs --list-different --no-error-on-unmatched-pattern --ignore-path=.prettierignore --ignore-path=.prettierignore-css /builds/worker/checkouts/gecko/image /builds/worker/checkouts/gecko/editor /builds/worker/checkouts/gecko/gradle /builds/worker/checkouts/gecko/.cargo /builds/worker/checkouts/gecko/stylelint-rollouts.config.js /builds/worker/checkouts/gecko/extensions /builds/worker/checkouts/gecko/js /builds/worker/checkouts/gecko/srcdir-resolver.js
[task 2026-07-29T03:03:57.976+00:00] 03:03:57.976 eslint (245) | Prettier command: /builds/worker/fetches/node/bin/node /builds/worker/checkouts/gecko/node_modules/prettier/bin/prettier.cjs --list-different --no-error-on-unmatched-pattern --ignore-path=.prettierignore --ignore-path=.prettierignore-css /builds/worker/checkouts/gecko/security /builds/worker/checkouts/gecko/modules /builds/worker/checkouts/gecko/servo /builds/worker/checkouts/gecko/.github /builds/worker/checkouts/gecko/docs /builds/worker/checkouts/gecko/eslint-file-globals.config.mjs /builds/worker/checkouts/gecko/taskcluster /builds/worker/checkouts/gecko/docshell
[task 2026-07-29T03:04:01.057+00:00] 03:04:01.057 eslint (244) | Finished in 43.44 seconds
[task 2026-07-29T03:04:05.451+00:00] 03:04:05.450 eslint (245) | Finished in 47.83 seconds
[task 2026-07-29T03:04:07.974+00:00] 03:04:07.973 eslint (243) | Prettier command: /builds/worker/fetches/node/bin/node /builds/worker/checkouts/gecko/node_modules/prettier/bin/prettier.cjs --list-different --no-error-on-unmatched-pattern --ignore-path=.prettierignore --ignore-path=.prettierignore-css /builds/worker/checkouts/gecko/netwerk /builds/worker/checkouts/gecko/memory /builds/worker/checkouts/gecko/intl /builds/worker/checkouts/gecko/chrome /builds/worker/checkouts/gecko/tools /builds/worker/checkouts/gecko/media /builds/worker/checkouts/gecko/storage /builds/worker/checkouts/gecko/eslint-test-paths.config.mjs
[task 2026-07-29T03:04:11.670+00:00] 03:04:11.670 eslint (238) | Prettier command: /builds/worker/fetches/node/bin/node /builds/worker/checkouts/gecko/node_modules/prettier/bin/prettier.cjs --list-different --no-error-on-unmatched-pattern --ignore-path=.prettierignore --ignore-path=.prettierignore-css /builds/worker/checkouts/gecko/config /builds/worker/checkouts/gecko/.prettierrc.js /builds/worker/checkouts/gecko/.stylelintrc.js /builds/worker/checkouts/gecko/remote /builds/worker/checkouts/gecko/layout /builds/worker/checkouts/gecko/package.json /builds/worker/checkouts/gecko/parser /builds/worker/checkouts/gecko/widget
[task 2026-07-29T03:04:13.704+00:00] 03:04:13.704 eslint (241) | Prettier command: /builds/worker/fetches/node/bin/node /builds/worker/checkouts/gecko/node_modules/prettier/bin/prettier.cjs --list-different --no-error-on-unmatched-pattern --ignore-path=.prettierignore --ignore-path=.prettierignore-css /builds/worker/checkouts/gecko/eslint.config.mjs /builds/worker/checkouts/gecko/.codex /builds/worker/checkouts/gecko/eslint-ignores.config.mjs /builds/worker/checkouts/gecko/hal /builds/worker/checkouts/gecko/testing /builds/worker/checkouts/gecko/startupcache /builds/worker/checkouts/gecko/accessible /builds/worker/checkouts/gecko/.babel-eslint.rc.js
[task 2026-07-29T03:04:18.965+00:00] 03:04:18.964 eslint (238) | Finished in 61.35 seconds
[task 2026-07-29T03:04:21.467+00:00] 03:04:21.467 eslint (243) | Finished in 63.85 seconds
[task 2026-07-29T03:04:23.856+00:00] 03:04:23.856 eslint (241) | Finished in 66.24 seconds
[task 2026-07-29T03:05:55.002+00:00] 03:05:55.001 eslint (242) | Prettier command: /builds/worker/fetches/node/bin/node /builds/worker/checkouts/gecko/node_modules/prettier/bin/prettier.cjs --list-different --no-error-on-unmatched-pattern --ignore-path=.prettierignore --ignore-path=.prettierignore-css /builds/worker/checkouts/gecko/gfx /builds/worker/checkouts/gecko/xpcom /builds/worker/checkouts/gecko/.agents /builds/worker/checkouts/gecko/dom /builds/worker/checkouts/gecko/uriloader /builds/worker/checkouts/gecko/python /builds/worker/checkouts/gecko/ipc /builds/worker/checkouts/gecko/.claude
[task 2026-07-29T03:06:09.503+00:00] 03:06:09.502 eslint (242) | Finished in 171.88 seconds
[task 2026-07-29T03:10:41.261+00:00] 03:10:41.261 eslint (239) | Prettier command: /builds/worker/fetches/node/bin/node /builds/worker/checkouts/gecko/node_modules/prettier/bin/prettier.cjs --list-different --no-error-on-unmatched-pattern --ignore-path=.prettierignore --ignore-path=.prettierignore-css /builds/worker/checkouts/gecko/browser /builds/worker/checkouts/gecko/toolkit /builds/worker/checkouts/gecko/devtools /builds/worker/checkouts/gecko/services /builds/worker/checkouts/gecko/eslint-subdirs.config.mjs /builds/worker/checkouts/gecko/.zed /builds/worker/checkouts/gecko/mobile /builds/worker/checkouts/gecko/xpfe
[task 2026-07-29T03:12:42.927+00:00] 03:12:42.927 eslint (239) | Finished in 565.31 seconds
Comment 2•3 months ago
|
||
Proposed fix: https://phabricator.services.mozilla.com/D314888
Root cause is in roller.py::_generate_jobs: chunk_size = ceil(len(paths) / num_procs) produces ~one chunk per worker, which is then processed serially. Since top-level directories vary hugely in size, a worker that draws a few big ones (e.g. toolkit) runs for minutes while others idle, and with only ~num_procs jobs the pool can't rebalance. This hits both the LineType (avoid-blacklist-and-whitelist) and ExternalType (eslint) cases you cited.
The patch oversubscribes the pool (JOBS_PER_PROC=4 smaller jobs per process) so idle workers steal remaining work. It's general across linter types and still capped by MAX_PATHS_PER_JOB. I did not add the thread/asyncio path for LineType — process-level oversubscription already parallelizes their I/O, so that would be redundant. New/updated unit tests cover the chunking; full mozlint suite passes. Note: the actual CI wall-clock improvement wasn't measured here.
This is an automated analysis result. If this result is incorrect please add a needinfo and feel free to correct the error.
Comment 3•3 months ago
|
||
mozlint chunked paths into roughly one chunk per worker (chunk_size = ceil(len(paths) / num_procs)), so each worker processed its whole chunk serially. Because top-level directories vary enormously in size, a worker that happened to draw a few large directories ran for many minutes (e.g. avoid-blacklist-and-whitelist ~352s, eslint ~565s) while other workers sat idle after finishing their single chunk. With only ~num_procs jobs for num_procs workers, the ProcessPoolExecutor had no smaller units to hand a freed worker, so there was no dynamic load-balancing.
This generates JOBS_PER_PROC (=4) smaller jobs per process, so the pool can load-balance dynamically: idle workers steal the remaining work instead of blocking behind one worker's large chunk. The change is applied generally (LineType and ExternalType alike), stays capped by MAX_PATHS_PER_JOB to keep command lines short, and always yields at least one path per job. Global linters continue to run as a single job.
The chunk-size computation is factored into LintRoller._job_chunk_size and covered by new unit tests (oversubscription, the MAX_PATHS_PER_JOB cap, and edge cases); existing job-count tests were updated to the new granularity.
Updated•3 months ago
|
| Reporter | ||
Comment 4•3 months ago
|
||
Updated•3 months ago
|
| Reporter | ||
Comment 5•3 months ago
|
||
| Reporter | ||
Comment 6•3 months ago
|
||
Use os.scandir during recursive traversal so FileFinder can reuse entry
type and existence information instead of calling isdir and lexists for every path.
Updated•3 months ago
|
Updated•3 months ago
|
Updated•3 months ago
|
Updated•3 months ago
|
| Reporter | ||
Updated•2 months ago
|
Comment 8•2 months ago
|
||
| bugherder | ||
Comment 9•2 months ago
|
||
| bugherder | ||
Comment 10•2 months ago
|
||
Comment 11•2 months ago
|
||
| bugherder | ||
Comment 12•2 months ago
|
||
| bugherder | ||
Updated•25 days ago
|
Updated•25 days ago
|
| Reporter | ||
Updated•23 days ago
|
| Reporter | ||
Comment 13•23 days ago
|
||
Given Alex comment in bug 2085705 comment 62, I'm going to reassign this to him.
| Assignee | ||
Comment 15•22 days ago
|
||
A whole tree run gave each job a handful of top level directories, so
the job that got dom or js set the wall time of slow per file
linters. A linter can now set split-dirs to split each directory that
many levels deep. includes uses 2, which took a whole tree run on
Windows from 45 to 17 seconds. Every linter's paths are now handed to
its jobs round robin instead of in consecutive runs. A dot directory
that is passed in is kept whole. The dot entries inside a directory are
dropped unless find-dotfiles is set, matching how the mozlint walkers
treat a directory.
| Assignee | ||
Comment 16•22 days ago
|
||
A fixed split depth leaves one job holding a directory like
browser/components, which then sets the wall time. With
split-dirs: auto, the roller weighs every path in one walk, splits
any directory heavier than an equal share, and hands the pieces out
heaviest first, one job per core. A file the linter checks weighs 1,
every entry walked weighs 0.2, and paths matched by
split-ignore-file or the third party list weigh nothing.
| Assignee | ||
Comment 17•22 days ago
|
||
A whole tree ESLint run gave each job a few top level directories, so
the job with devtools and toolkit set the wall time. On a 12 core
machine one job wrote more JSON than fits in a V8 string, so ESLint
crashed and the results for docshell, layout and netwerk were
lost.
eslint now uses split-dirs: auto with its own ignore list as
split-ignore-file. On try the lint step on 8 cores went from about
300 to 262 seconds, and on 30 cores it takes 88 seconds.
ESLint and Prettier lint the dot files inside a directory, so the split
keeps dot entries, and --no-warn-ignored and --ignore-unknown stop
an explicitly passed ignored entry from producing a warning or an
error. Prettier also starts alongside ESLint instead of after it when
nothing is being fixed.
Comment 18•15 days ago
|
||
Comment 19•15 days ago
|
||
| bugherder | ||
Comment 20•15 days ago
|
||
Alex, I think this can be closed now (leave-open is still on there)?
| Assignee | ||
Comment 21•14 days ago
|
||
Agreed, good to close now.
Updated•14 days ago
|
Description
•