Closed Bug 1994717 Opened 1 year ago Closed 14 days ago

Parallelize linters in a more balanced way

Categories

(Developer Infrastructure :: Lint and Formatting, enhancement)

enhancement

Tracking

(firefox162 fixed)

RESOLVED FIXED
162 Branch
Tracking Status
firefox162 --- fixed

People

(Reporter: marco, Assigned: ahochheiden)

References

(Blocks 2 open bugs)

Details

Attachments

(5 external files, 2 obsolete files)

We are kind of running them in parallel already thanks to the ProcessPoolExecutor in roller.py, but this is not enough.

First of all, we are not parallelizing enough, so some take a very long time.
E.g. in a recent task I see:

[task 2025-10-16T09:01:46.349+00:00] 09:01:46.349 avoid-blacklist-and-whitelist (92) | Passing the following paths:
[task 2025-10-16T09:01:46.349+00:00] /builds/worker/checkouts/gecko/intl  
[task 2025-10-16T09:01:46.349+00:00] /builds/worker/checkouts/gecko/remote  
[task 2025-10-16T09:01:46.350+00:00] /builds/worker/checkouts/gecko/ipc  
[task 2025-10-16T09:01:46.350+00:00] /builds/worker/checkouts/gecko/media  
[task 2025-10-16T09:01:46.350+00:00] /builds/worker/checkouts/gecko/docshell  
[task 2025-10-16T09:01:46.350+00:00] /builds/worker/checkouts/gecko/toolkit  
[task 2025-10-16T09:01:46.350+00:00] /builds/worker/checkouts/gecko/testing  
[task 2025-10-16T09:01:46.350+00:00] /builds/worker/checkouts/gecko/view  
[task 2025-10-16T09:01:46.350+00:00] /builds/worker/checkouts/gecko/mobile

and then later:
avoid-blacklist-and-whitelist (92) | Finished in 352.67 seconds.

Second, we could parallize using threads or asyncio, since the LineType linters (StringType and RegexType) are I/O heavy.

See Also: → 2035010

This doesn't only apply to LineType, but also ExternalType. For example, eslint is often slow and I see things like:

[task 2026-07-29T03:03:29.477+00:00] 03:03:29.477 eslint (241) | ESLint command: /builds/worker/fetches/node/bin/node /builds/worker/checkouts/gecko/node_modules/eslint/bin/eslint.js --format json --no-error-on-unmatched-pattern --ignore-pattern testing/performance/mobile-startup/newssite-nuxt --ignore-pattern testing/xpcshell/node_ip --ignore-pattern testing/web-platform/tests/resources/webidl2 --ignore-pattern testing/talos/talos/tests/kraken --ignore-pattern testing/talos/talos/tests/offscreencanvas/benchmarks/video/demuxer_mp4.js --ignore-pattern testing/xpcshell/node-http2 --ignore-pattern testing/xpcshell/node-ws --ignore-pattern testing/talos/talos/tests/v8_7 --ignore-pattern testing/mochitest/pywebsocket3 --ignore-pattern testing/talos/talos/tests/devtools/addon/content/pages --ignore-pattern testing/web-platform/mozilla/tests/webgpu --ignore-pattern testing/web-platform/tests/tools/third_party --ignore-pattern testing/mozbase/mozproxy/mozproxy/backends/mitm/scripts/catapult --ignore-pattern testing/xpcshell/dns-packet --ignore-pattern testing/modules/sinon-7.2.7.js --ignore-pattern testing/xpcshell/moz-http2/package-lock.json --ignore-pattern testing/talos/talos/tests/offscreencanvas/benchmarks/video/mp4box.all.min.js --ignore-pattern testing/talos/talos/tests/dromaeo --ignore-pattern testing/web-platform/tests/tools/wave/package-lock.json /builds/worker/checkouts/gecko/eslint.config.mjs /builds/worker/checkouts/gecko/.codex /builds/worker/checkouts/gecko/eslint-ignores.config.mjs /builds/worker/checkouts/gecko/hal /builds/worker/checkouts/gecko/testing /builds/worker/checkouts/gecko/startupcache /builds/worker/checkouts/gecko/accessible /builds/worker/checkouts/gecko/.babel-eslint.rc.js
[task 2026-07-29T03:03:30.924+00:00] 03:03:30.924 eslint (240) | Finished in 13.30 seconds
[task 2026-07-29T03:03:56.280+00:00] 03:03:56.278 eslint (244) | Prettier command: /builds/worker/fetches/node/bin/node /builds/worker/checkouts/gecko/node_modules/prettier/bin/prettier.cjs --list-different --no-error-on-unmatched-pattern --ignore-path=.prettierignore --ignore-path=.prettierignore-css /builds/worker/checkouts/gecko/image /builds/worker/checkouts/gecko/editor /builds/worker/checkouts/gecko/gradle /builds/worker/checkouts/gecko/.cargo /builds/worker/checkouts/gecko/stylelint-rollouts.config.js /builds/worker/checkouts/gecko/extensions /builds/worker/checkouts/gecko/js /builds/worker/checkouts/gecko/srcdir-resolver.js
[task 2026-07-29T03:03:57.976+00:00] 03:03:57.976 eslint (245) | Prettier command: /builds/worker/fetches/node/bin/node /builds/worker/checkouts/gecko/node_modules/prettier/bin/prettier.cjs --list-different --no-error-on-unmatched-pattern --ignore-path=.prettierignore --ignore-path=.prettierignore-css /builds/worker/checkouts/gecko/security /builds/worker/checkouts/gecko/modules /builds/worker/checkouts/gecko/servo /builds/worker/checkouts/gecko/.github /builds/worker/checkouts/gecko/docs /builds/worker/checkouts/gecko/eslint-file-globals.config.mjs /builds/worker/checkouts/gecko/taskcluster /builds/worker/checkouts/gecko/docshell
[task 2026-07-29T03:04:01.057+00:00] 03:04:01.057 eslint (244) | Finished in 43.44 seconds
[task 2026-07-29T03:04:05.451+00:00] 03:04:05.450 eslint (245) | Finished in 47.83 seconds
[task 2026-07-29T03:04:07.974+00:00] 03:04:07.973 eslint (243) | Prettier command: /builds/worker/fetches/node/bin/node /builds/worker/checkouts/gecko/node_modules/prettier/bin/prettier.cjs --list-different --no-error-on-unmatched-pattern --ignore-path=.prettierignore --ignore-path=.prettierignore-css /builds/worker/checkouts/gecko/netwerk /builds/worker/checkouts/gecko/memory /builds/worker/checkouts/gecko/intl /builds/worker/checkouts/gecko/chrome /builds/worker/checkouts/gecko/tools /builds/worker/checkouts/gecko/media /builds/worker/checkouts/gecko/storage /builds/worker/checkouts/gecko/eslint-test-paths.config.mjs
[task 2026-07-29T03:04:11.670+00:00] 03:04:11.670 eslint (238) | Prettier command: /builds/worker/fetches/node/bin/node /builds/worker/checkouts/gecko/node_modules/prettier/bin/prettier.cjs --list-different --no-error-on-unmatched-pattern --ignore-path=.prettierignore --ignore-path=.prettierignore-css /builds/worker/checkouts/gecko/config /builds/worker/checkouts/gecko/.prettierrc.js /builds/worker/checkouts/gecko/.stylelintrc.js /builds/worker/checkouts/gecko/remote /builds/worker/checkouts/gecko/layout /builds/worker/checkouts/gecko/package.json /builds/worker/checkouts/gecko/parser /builds/worker/checkouts/gecko/widget
[task 2026-07-29T03:04:13.704+00:00] 03:04:13.704 eslint (241) | Prettier command: /builds/worker/fetches/node/bin/node /builds/worker/checkouts/gecko/node_modules/prettier/bin/prettier.cjs --list-different --no-error-on-unmatched-pattern --ignore-path=.prettierignore --ignore-path=.prettierignore-css /builds/worker/checkouts/gecko/eslint.config.mjs /builds/worker/checkouts/gecko/.codex /builds/worker/checkouts/gecko/eslint-ignores.config.mjs /builds/worker/checkouts/gecko/hal /builds/worker/checkouts/gecko/testing /builds/worker/checkouts/gecko/startupcache /builds/worker/checkouts/gecko/accessible /builds/worker/checkouts/gecko/.babel-eslint.rc.js
[task 2026-07-29T03:04:18.965+00:00] 03:04:18.964 eslint (238) | Finished in 61.35 seconds
[task 2026-07-29T03:04:21.467+00:00] 03:04:21.467 eslint (243) | Finished in 63.85 seconds
[task 2026-07-29T03:04:23.856+00:00] 03:04:23.856 eslint (241) | Finished in 66.24 seconds
[task 2026-07-29T03:05:55.002+00:00] 03:05:55.001 eslint (242) | Prettier command: /builds/worker/fetches/node/bin/node /builds/worker/checkouts/gecko/node_modules/prettier/bin/prettier.cjs --list-different --no-error-on-unmatched-pattern --ignore-path=.prettierignore --ignore-path=.prettierignore-css /builds/worker/checkouts/gecko/gfx /builds/worker/checkouts/gecko/xpcom /builds/worker/checkouts/gecko/.agents /builds/worker/checkouts/gecko/dom /builds/worker/checkouts/gecko/uriloader /builds/worker/checkouts/gecko/python /builds/worker/checkouts/gecko/ipc /builds/worker/checkouts/gecko/.claude
[task 2026-07-29T03:06:09.503+00:00] 03:06:09.502 eslint (242) | Finished in 171.88 seconds
[task 2026-07-29T03:10:41.261+00:00] 03:10:41.261 eslint (239) | Prettier command: /builds/worker/fetches/node/bin/node /builds/worker/checkouts/gecko/node_modules/prettier/bin/prettier.cjs --list-different --no-error-on-unmatched-pattern --ignore-path=.prettierignore --ignore-path=.prettierignore-css /builds/worker/checkouts/gecko/browser /builds/worker/checkouts/gecko/toolkit /builds/worker/checkouts/gecko/devtools /builds/worker/checkouts/gecko/services /builds/worker/checkouts/gecko/eslint-subdirs.config.mjs /builds/worker/checkouts/gecko/.zed /builds/worker/checkouts/gecko/mobile /builds/worker/checkouts/gecko/xpfe
[task 2026-07-29T03:12:42.927+00:00] 03:12:42.927 eslint (239) | Finished in 565.31 seconds
Summary: Parallelize LineType linters → Parallelize linters in a more balanced way

Proposed fix: https://phabricator.services.mozilla.com/D314888

Root cause is in roller.py::_generate_jobs: chunk_size = ceil(len(paths) / num_procs) produces ~one chunk per worker, which is then processed serially. Since top-level directories vary hugely in size, a worker that draws a few big ones (e.g. toolkit) runs for minutes while others idle, and with only ~num_procs jobs the pool can't rebalance. This hits both the LineType (avoid-blacklist-and-whitelist) and ExternalType (eslint) cases you cited.

The patch oversubscribes the pool (JOBS_PER_PROC=4 smaller jobs per process) so idle workers steal remaining work. It's general across linter types and still capped by MAX_PATHS_PER_JOB. I did not add the thread/asyncio path for LineType — process-level oversubscription already parallelizes their I/O, so that would be redundant. New/updated unit tests cover the chunking; full mozlint suite passes. Note: the actual CI wall-clock improvement wasn't measured here.

This is an automated analysis result. If this result is incorrect please add a needinfo and feel free to correct the error.

mozlint chunked paths into roughly one chunk per worker (chunk_size = ceil(len(paths) / num_procs)), so each worker processed its whole chunk serially. Because top-level directories vary enormously in size, a worker that happened to draw a few large directories ran for many minutes (e.g. avoid-blacklist-and-whitelist ~352s, eslint ~565s) while other workers sat idle after finishing their single chunk. With only ~num_procs jobs for num_procs workers, the ProcessPoolExecutor had no smaller units to hand a freed worker, so there was no dynamic load-balancing.

This generates JOBS_PER_PROC (=4) smaller jobs per process, so the pool can load-balance dynamically: idle workers steal the remaining work instead of blocking behind one worker's large chunk. The change is applied generally (LineType and ExternalType alike), stays capped by MAX_PATHS_PER_JOB to keep command lines short, and always yields at least one path per job. Global linters continue to run as a single job.

The chunk-size computation is factored into LintRoller._job_chunk_size and covered by new unit tests (oversubscription, the MAX_PATHS_PER_JOB cap, and edge cases); existing job-count tests were updated to the new granularity.

Assignee: nobody → mcastelluccio
Status: NEW → ASSIGNED
Attachment #9618232 - Attachment description: WIP: Bug 1994717 - Balance mozlint parallelization by oversubscribing the process pool → WIP: Bug 1994717 - Balance mozlint parallelization by oversubscribing the process pool. r=#linter-reviewers

Use os.scandir during recursive traversal so FileFinder can reuse entry
type and existence information instead of calling isdir and lexists for every path.

Attachment #9618232 - Attachment description: WIP: Bug 1994717 - Balance mozlint parallelization by oversubscribing the process pool. r=#linter-reviewers → Bug 1994717 - Balance mozlint parallelization by oversubscribing the process pool. r=#linter-reviewers
Attachment #9619553 - Attachment description: WIP: Bug 1994717 - Balance mozlint parallelization by counting the number of files that should be linted in subdirectories. r=#linter-reviewers → Bug 1994717 - Balance mozlint parallelization by counting the number of files that should be linted in subdirectories. r=#linter-reviewers
Attachment #9621602 - Attachment description: WIP: Bug 1994717 - Don't re-walk the tree multiple times (once per extension), but only once. r=#linter-reviewers → Bug 1994717 - Don't re-walk the tree multiple times (once per extension), but only once. r=#linter-reviewers
Attachment #9621664 - Attachment description: WIP: Bug 1994717 - Avoid redundant filesystem metadata checks in FileFinder. r?#firefox-build-system-reviewers → Bug 1994717 - Avoid redundant filesystem metadata checks in FileFinder. r?#firefox-build-system-reviewers
Keywords: leave-open
Pushed by mcastelluccio@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/1f4361fa3f5a https://hg.mozilla.org/integration/autoland/rev/e4b4dc5667ac Avoid redundant filesystem metadata checks in FileFinder. r=firefox-build-system-reviewers,sergesanspaille
Pushed by mcastelluccio@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/f6f09a425648 https://hg.mozilla.org/integration/autoland/rev/ac0a5a398c81 Don't re-walk the tree multiple times (once per extension), but only once. r=linter-reviewers,sergesanspaille
Depends on: 2085705
Attachment #9619553 - Attachment is obsolete: true
Attachment #9618232 - Attachment is obsolete: true
Blocks: 2085705
No longer depends on: 2085705

Given Alex comment in bug 2085705 comment 62, I'm going to reassign this to him.

Assignee: mcastelluccio → ahochheiden
Duplicate of this bug: 2085859

A whole tree run gave each job a handful of top level directories, so
the job that got dom or js set the wall time of slow per file
linters. A linter can now set split-dirs to split each directory that
many levels deep. includes uses 2, which took a whole tree run on
Windows from 45 to 17 seconds. Every linter's paths are now handed to
its jobs round robin instead of in consecutive runs. A dot directory
that is passed in is kept whole. The dot entries inside a directory are
dropped unless find-dotfiles is set, matching how the mozlint walkers
treat a directory.

A fixed split depth leaves one job holding a directory like
browser/components, which then sets the wall time. With
split-dirs: auto, the roller weighs every path in one walk, splits
any directory heavier than an equal share, and hands the pieces out
heaviest first, one job per core. A file the linter checks weighs 1,
every entry walked weighs 0.2, and paths matched by
split-ignore-file or the third party list weigh nothing.

A whole tree ESLint run gave each job a few top level directories, so
the job with devtools and toolkit set the wall time. On a 12 core
machine one job wrote more JSON than fits in a V8 string, so ESLint
crashed and the results for docshell, layout and netwerk were
lost.

eslint now uses split-dirs: auto with its own ignore list as
split-ignore-file. On try the lint step on 8 cores went from about
300 to 262 seconds, and on 30 cores it takes 88 seconds.
ESLint and Prettier lint the dot files inside a directory, so the split
keeps dot entries, and --no-warn-ignored and --ignore-unknown stop
an explicitly passed ignored entry from producing a warning or an
error. Prettier also starts alongside ESLint instead of after it when
nothing is being fixed.

Pushed by ahochheiden@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/4c9097df41f4 https://hg.mozilla.org/integration/autoland/rev/e0bbe7146a7f Part 1 - Split directories into smaller jobs for slow per file linters r=linter-reviewers,marco https://github.com/mozilla-firefox/firefox/commit/acedc93c3c4e https://hg.mozilla.org/integration/autoland/rev/45375aaba9da Part 2 - Split directories by weight so each job gets an equal share r=linter-reviewers,sergesanspaille https://github.com/mozilla-firefox/firefox/commit/a950498c54ac https://hg.mozilla.org/integration/autoland/rev/3eac5b6ff7fc Part 3 - Split the eslint paths by weight r=linter-reviewers,marco

Alex, I think this can be closed now (leave-open is still on there)?

Flags: needinfo?(ahochheiden)

Agreed, good to close now.

Status: ASSIGNED → RESOLVED
Closed: 14 days ago
Flags: needinfo?(ahochheiden)
Keywords: leave-open
Resolution: --- → FIXED
Target Milestone: --- → 162 Branch
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: