Open Bug 2085705 Opened 23 days ago Updated 20 days ago

[meta] Improve `./mach lint`

Categories

(Developer Infrastructure :: Lint and Formatting, enhancement)

enhancement

Tracking

(Not tracked)

REOPENED

People

(Reporter: ahochheiden, Assigned: ahochheiden)

References

(Blocks 1 open bug)

Details

(Keywords: meta)

Attachments

(24 obsolete files)

./mach lint spends much of its time outside the linters, in repeated tree walks, glob excludes expanded once per pattern, excludes matched one pattern at a time, setup for linters with nothing to lint and repeated toolchain lookups. On Windows, ruff, eslint and stylelint fail on the whole tree with WinError 206 because they pass every exclude on the command line (ruff, eslint, stylelint), and every whole tree run uses a single job.

Fixing these should make most linters several times faster on CI and locally.

A whole tree run passed 477 excludes in one --extend-exclude
argument of 43226 characters, and Windows refuses a command line over
32767 characters with WinError 206. ruff reads its arguments from an
@file instead, which has no such limit.

Each excluded path became an --ignore-pattern, about 477 on a whole
tree run, which went over the Windows command line limit. The eslint
and stylelint configs already read ThirdPartyPaths.txt and
Generated.txt, so both now get only the global excludes. stylelint
also gets those two files through --ignore-path, because its
ignoreFiles setting does not match a directory line. The patterns now
use forward slashes, so Windows applies them as well. The node
processes no longer start through a shell, which limited the command
line to 8191 characters.

_generate_jobs lists the top level entries only when the linted path
equals self.root. roll passes the path through os.path.abspath,
which gives backslashes on Windows, while mach gives the root with
forward slashes, so on Windows mach lint . ran each linter on the
root as a single job.

jj git root snapshots the working copy, and a single mach lint ran it
three times, once from the Sentry thread.

The toolchain digest covered all of tools/lint/mozcheck, including the
Python wrapper __init__.py, which the build does not read. An edit to
the wrapper alone made the prebuilt binary unavailable locally and
rebuilt every mozcheck toolchain on CI.

On macOS and Windows, each mozlint worker starts a fresh Python process.
The workers did not see the binary setup cached, so every mozcheck
linter ran bootstrap_toolchain again. bootstrap_toolchain takes about
3 seconds per worker, and longer when its cache is cold, since it then
runs mach taskgraph.

Checking whether the cached toolchain tasks are still valid took about
3 seconds on every bootstrap_toolchain call, which mach lint makes
for mozcheck. It read and hashed some 6000 files every time, and walked
mobile/android once for each of the three Gradle globs. The hash of
each file is now kept with its size, modification time and inode, so
unchanged files are not read again, and the <dir>/**/<name> globs that
share a directory are expanded in one walk. The cache key changes shape,
so existing entries miss once.

The lint task ran cargo install cargo-audit on every run, which took
about 74 seconds before an audit of about 3 seconds. A toolchain task now
builds the pinned version once, and the lint task puts it on PATH, so
the setup finds the version it wants and skips the install.

should_lint_entire_tree matched each changed file's absolute path
against the support-files patterns. Most patterns are relative to the
root, so only those starting with **/ matched, along with the absolute
path of the linter's own definition. Changing a file like
tools/lint/eslint/__init__.py or tools/lint/python/ruff.py under
--outgoing or --workdir linted only the changed files instead of the
whole tree.

The rules live in eslint.config.mjs and the eslint-*.config.mjs files
it imports, and stylelint reads stylelint-rollouts.config.js, but the
support-files only listed the older .eslintrc.js and .stylelintrc.js.
Changing a rule under --outgoing or --workdir linted only the changed
files instead of the whole tree.

The --fix grouping compared the job paths as strings, so a job on a
directory and a job on a file inside it could run at the same time and
write the same file.

mach lint . passes each top level directory to the linters, including
dot directories like .ruff_cache. A walker skips dot entries only below
the directory it starts from, so every whole tree run walked and linted
the tool caches in a local checkout. .ruff_cache alone holds over
250000 files.

The global excludes were merged into each linter's excludes, which
BaseType then replaced with only the excludes inside the linted paths.
A walk outside those paths, like the whole tree walk in
license-declarations, went into every objdir and node_modules.
expand_exclusions now always applies the global excludes unless
filtering is turned off.

A literal exclude now matches an explicitly passed file only by whole
path components, so subdir1/b no longer excludes subdir1/b.js.

Setup ran for every linter before the paths were filtered, so a run on
Python files still set up typescript, pnpm-audit and the other node
linters, which can take several seconds each. roll now sets up the
linters it generated jobs for. Setup failures also stay in the result
now, since roll used to reset them after setup recorded them.

filterpaths walked each path once per glob exclude through
FileFinder, including the excludes that neither exist nor contain a
glob and so cannot match. Over the whole tree that took about 30 seconds
of CPU in each worker. One scandir walk now tests every name against a
single regex, and does not descend into excluded directories.
mozpath.matcher looks plain path excludes up by ancestor in a set, so
the regex holds only the globs.

FileFinder tested every file and directory against each ignore pattern
in turn. mozlint passes several hundred excludes, so the whole tree walk in
license-declarations spent most of its 104 seconds there. The patterns
are now compiled into one regex when the finder is created.

Every yamllint job started a second mach through mach python to run the
vendored yamllint, which took about a second before yamllint ran. The
linter now calls the yamllint API the way its command line does. The
results also no longer collect in a module global, which reported the
issues of an earlier job again when a worker ran more than one.

FileFinder.find("**") built a File for every file in the tree and
tested it against the excludes before expand_exclusions dropped the
ones with other extensions. The new FileFinder.walk yields the same
paths in the same order without building a File, so the extension is
checked first. A whole tree mach lint -l license on Windows went from
about 22 to 17 seconds.

BaseType._lint_dir walked a directory once for each extension a
linter lists. It now walks it once and skips the other extensions. A
whole tree mach lint -l mscom-init on Windows went from about 39 to
16 seconds.

license-declarations walked the whole tree to gather the declared ids
and then walked the linted paths again, parsing every moz.build twice. The
linted files are now the ones from the first walk under the given paths.
A moz.build in a dot directory is no longer linted when that directory is
passed explicitly, since the tree walk skips dot directories.

filterpaths tested every path against every exclude with
FilterPath.contains, which made both paths absolute, normalized and
split them on each call, and it gathered the excludes under each include
again for every path. The components are now kept on the FilterPath
and the excludes under each include are gathered once. A whole tree
mach lint -l includes on Windows went from about 9.0 to 8.0 seconds.

A whole tree run gave each job a handful of top level directories, so
the job that got dom or js set the wall time of slow per file
linters. A linter can now set split-dirs to split each directory that
many levels deep. includes uses 2, which took a whole tree run on
Windows from 45 to 17 seconds. Every linter's paths are now handed to
its jobs round robin instead of in consecutive runs. A dot directory
that is passed in is kept whole. The dot entries inside a directory are
dropped unless find-dotfiles is set, matching how the mozlint walkers
treat a directory.

A whole tree ESLint run gave each job a few top level directories, so
the job with devtools and toolkit set the wall time. On a 12 core
machine one job wrote more JSON than fits in a V8 string, so ESLint
crashed and the results for docshell, layout and netwerk were
lost.

eslint now uses split-dirs: 1, which took a whole tree run on
Windows from 275 to 172 seconds and its slowest job from 266 to 164,
with no change on CI.
ESLint and Prettier lint the dot files inside a directory, so the split
keeps dot entries, and --no-warn-ignored and --ignore-unknown stop
an explicitly passed ignored entry from producing a warning or an
error. Prettier also starts alongside ESLint instead of after it when
nothing is being fixed.

includes searched every file once per included header, with a regex
joining all of that header's symbols. The lint_std_headers and
lint_cstd_headers checks from diskarzhan and the mfbt check all
searched this way. Every symbol is a plain identifier, so the linter now
collects the words of a file once and looks the symbols up in them. The
linter has its own copies of lint_std_headers and lint_cstd_headers,
which read the symbol tables from diskarzhan, and no longer calls the
diskarzhan versions. The results are the same as the regex search. A
whole tree mach lint -l includes on Windows went from 17 to 8 seconds.

Bug 1994717 was trying to address the eslint imbalance in another, more complicated, way. The only advantage was that it would be dynamically choosing the split instead of hardcoding it (which means it would evolve automatically as the tree evolves, while we might need to tune "split-dirs" at some point).

Blocks: 1907116

Alex, since these are all mostly unrelated changes (except that they all improve mach lint), why not tracking them in separate bugs?

(In reply to Marco Castelluccio [:marco] from comment #26)

Alex, since these are all mostly unrelated changes (except that they all improve mach lint), why not tracking them in separate bugs?

I can make this a [meta] and split them all out into separate bugs if you want. This spawned from an unrelated discussion during the Engineering Summit regarding bug 1975505, where briefly the topic of 'improving ./mach lint' came up. The main talking point there was re-writing it in Rust (which this isn't), but when doing this work I've been thinking about 'improving ./mach lint' as one idea, not the individual improvements themselves.

(In reply to Alex Hochheiden [:ahochheiden] [he/him] from comment #27)

(In reply to Marco Castelluccio [:marco] from comment #26)

Alex, since these are all mostly unrelated changes (except that they all improve mach lint), why not tracking them in separate bugs?

I can make this a [meta] and split them all out into separate bugs if you want. This spawned from an unrelated discussion during the Engineering Summit regarding bug 1975505, where briefly the topic of 'improving ./mach lint' came up. The main talking point there was re-writing it in Rust (which this isn't), but when doing this work I've been thinking about 'improving ./mach lint' as one idea, not the individual improvements themselves.

If it was a manual process, I would say: don't bother too much with the splitting, but given it's partially automated now I would vote for splitting so it's easier to keep track of improvements, regressions, and so on.

Comment on attachment 9667747 [details]
Bug 2085705 - Part 1 - Pass the ruff arguments through a response file r?#linter-reviewers

Revision D336622 was moved to bug 2085844. Setting attachment 9667747 [details] to obsolete.

Attachment #9667747 - Attachment is obsolete: true

Comment on attachment 9667748 [details]
Bug 2085705 - Part 2 - Pass eslint and stylelint only the global excludes r?#linter-reviewers

Revision D336623 was moved to bug 2085844. Setting attachment 9667748 [details] to obsolete.

Attachment #9667748 - Attachment is obsolete: true

Comment on attachment 9667749 [details]
Bug 2085705 - Part 3 - Split a whole tree run into jobs on Windows too r?#linter-reviewers

Revision D336624 was moved to bug 2085845. Setting attachment 9667749 [details] to obsolete.

Attachment #9667749 - Attachment is obsolete: true

(In reply to Marco Castelluccio [:marco] from comment #28)

(In reply to Alex Hochheiden [:ahochheiden] [he/him] from comment #27)

(In reply to Marco Castelluccio [:marco] from comment #26)

Alex, since these are all mostly unrelated changes (except that they all improve mach lint), why not tracking them in separate bugs?

I can make this a [meta] and split them all out into separate bugs if you want. This spawned from an unrelated discussion during the Engineering Summit regarding bug 1975505, where briefly the topic of 'improving ./mach lint' came up. The main talking point there was re-writing it in Rust (which this isn't), but when doing this work I've been thinking about 'improving ./mach lint' as one idea, not the individual improvements themselves.

If it was a manual process, I would say: don't bother too much with the splitting, but given it's partially automated now I would vote for splitting so it's easier to keep track of improvements, regressions, and so on.

Done!

Comment on attachment 9667750 [details]
Bug 2085705 - Part 4 - Cache jj git root with functools.cache r?ahal

Revision D336625 was moved to bug 2085846. Setting attachment 9667750 [details] to obsolete.

Attachment #9667750 - Attachment is obsolete: true

Comment on attachment 9667751 [details]
Bug 2085705 - Part 5 - Hash only the mozcheck crate files for its toolchain r?#linter-reviewers

Revision D336626 was moved to bug 2085847. Setting attachment 9667751 [details] to obsolete.

Attachment #9667751 - Attachment is obsolete: true

Comment on attachment 9667752 [details]
Bug 2085705 - Part 6 - Pass the resolved mozcheck binary to mozlint workers r?#linter-reviewers

Revision D336627 was moved to bug 2085848. Setting attachment 9667752 [details] to obsolete.

Attachment #9667752 - Attachment is obsolete: true

Comment on attachment 9667753 [details]
Bug 2085705 - Part 7 - Hash only the changed inputs in mach_func_cache r?#linter-reviewers

Revision D336628 was moved to bug 2085849. Setting attachment 9667753 [details] to obsolete.

Attachment #9667753 - Attachment is obsolete: true

Comment on attachment 9667754 [details]
Bug 2085705 - Part 8 - Fetch a prebuilt cargo-audit in the lint task r?#linter-reviewers

Revision D336629 was moved to bug 2085851. Setting attachment 9667754 [details] to obsolete.

Attachment #9667754 - Attachment is obsolete: true

Comment on attachment 9667755 [details]
Bug 2085705 - Part 9 - Match support-files against paths relative to the root r?#linter-reviewers

Revision D336630 was moved to bug 2085852. Setting attachment 9667755 [details] to obsolete.

Attachment #9667755 - Attachment is obsolete: true

Comment on attachment 9667756 [details]
Bug 2085705 - Part 10 - List the eslint and stylelint configs in their support-files r?#linter-reviewers

Revision D336631 was moved to bug 2085852. Setting attachment 9667756 [details] to obsolete.

Attachment #9667756 - Attachment is obsolete: true

Comment on attachment 9667757 [details]
Bug 2085705 - Part 11 - Keep --fix jobs on a directory and a file inside it apart r?#linter-reviewers

Revision D336632 was moved to bug 2085853. Setting attachment 9667757 [details] to obsolete.

Attachment #9667757 - Attachment is obsolete: true

Comment on attachment 9667758 [details]
Bug 2085705 - Part 12 - Exclude the local cache and scratch directories from every linter r?#linter-reviewers

Revision D336633 was moved to bug 2085854. Setting attachment 9667758 [details] to obsolete.

Attachment #9667758 - Attachment is obsolete: true

Comment on attachment 9667759 [details]
Bug 2085705 - Part 13 - Apply the global excludes to every walk in expand_exclusions r?#linter-reviewers

Revision D336634 was moved to bug 2085854. Setting attachment 9667759 [details] to obsolete.

Attachment #9667759 - Attachment is obsolete: true

Comment on attachment 9667760 [details]
Bug 2085705 - Part 14 - Only set up the linters that have files to lint r?#linter-reviewers

Revision D336635 was moved to bug 2085855. Setting attachment 9667760 [details] to obsolete.

Attachment #9667760 - Attachment is obsolete: true

Comment on attachment 9667761 [details]
Bug 2085705 - Part 15 - Expand the glob excludes in one walk r?#linter-reviewers

Revision D336636 was moved to bug 2085856. Setting attachment 9667761 [details] to obsolete.

Attachment #9667761 - Attachment is obsolete: true

Comment on attachment 9667762 [details]
Bug 2085705 - Part 16 - Match FileFinder ignores with one regex r?#linter-reviewers

Revision D336637 was moved to bug 2085856. Setting attachment 9667762 [details] to obsolete.

Attachment #9667762 - Attachment is obsolete: true

Comment on attachment 9667764 [details]
Bug 2085705 - Part 18 - Check the extension before building a File in expand_exclusions r?#linter-reviewers

Revision D336639 was moved to bug 2085856. Setting attachment 9667764 [details] to obsolete.

Attachment #9667764 - Attachment is obsolete: true

Comment on attachment 9667765 [details]
Bug 2085705 - Part 19 - Walk each directory once in the string and regex types r?#linter-reviewers

Revision D336640 was moved to bug 2085856. Setting attachment 9667765 [details] to obsolete.

Attachment #9667765 - Attachment is obsolete: true

Comment on attachment 9667767 [details]
Bug 2085705 - Part 21 - Compute the path components of a FilterPath once r?#linter-reviewers

Revision D336642 was moved to bug 2085856. Setting attachment 9667767 [details] to obsolete.

Attachment #9667767 - Attachment is obsolete: true

Comment on attachment 9667763 [details]
Bug 2085705 - Part 17 - Run yamllint in process r?#linter-reviewers

Revision D336638 was moved to bug 2085857. Setting attachment 9667763 [details] to obsolete.

Attachment #9667763 - Attachment is obsolete: true

Comment on attachment 9667766 [details]
Bug 2085705 - Part 20 - Lint the moz.build files license-declarations already walked r?#linter-reviewers

Revision D336641 was moved to bug 2085858. Setting attachment 9667766 [details] to obsolete.

Attachment #9667766 - Attachment is obsolete: true

Comment on attachment 9667768 [details]
Bug 2085705 - Part 22 - Split directories into smaller jobs for slow per file linters r?#linter-reviewers

Revision D336643 was moved to bug 2085859. Setting attachment 9667768 [details] to obsolete.

Attachment #9667768 - Attachment is obsolete: true

Comment on attachment 9667769 [details]
Bug 2085705 - Part 23 - Split the eslint paths one level deep r?#linter-reviewers

Revision D336644 was moved to bug 2085859. Setting attachment 9667769 [details] to obsolete.

Attachment #9667769 - Attachment is obsolete: true

Comment on attachment 9667770 [details]
Bug 2085705 - Part 24 - Look up the includes header symbols in a set of each file's words r?#linter-reviewers

Revision D336645 was moved to bug 2085860. Setting attachment 9667770 [details] to obsolete.

Attachment #9667770 - Attachment is obsolete: true
Keywords: meta
Summary: Improve `./mach lint` → [meta] Improve `./mach lint`

Try push: https://treeherder.mozilla.org/jobs?repo=try&revision=700099e66140af3fef890f1fa4ea3c5e2807cc08

Measured perf wins from the 16 dependent bugs:

Linter Local before (s) Local after (s) Local change CI before (s) CI after (s) CI change
Single small directory
all linters on python/mozlint 15.2 9.2 -39% not measured not measured n/a
Full tree
cargo-audit not measured not measured not measured 80.5 5 -94%
mscom-init 371.9 6.8 -98% 27 7 -74%
test-manifest-toml 308.1 7.7 -98% 46.4 13 -72%
includes 213.2 7.6 -96% 69.2 14 -80%
header-guards 87 5.4 -94% 10.6 4 -62%
license 175.3 16 -91% 54.8 7 -87%
file-whitespace 51.1 5.8 -89% 8.2 4 -51%
ruff 23.2 4.5 -81% 6.6 3 -55%
ruff-format 22.9 4.5 -80% 7.4 3 -59%
eslint 1148 172 -85% 307.6 301 -2%
stylelint 93 35 -62% 35.8 30 -16%
yaml 14.6 10.2 -30% 12.6 9 -29%

Notes:

  • The CI numbers were from an earlier try push, but I don't expect much of a difference.
  • The local columns were measured only on Windows.
  • Some of the Local before (s) values (ruff, ruff-format, eslint, and stylelint) were measured with the base commit including the Windows specific fixes.

Interesting (and sad) to see the huge impact on eslint locally, but very small in CI.

(In reply to Marco Castelluccio [:marco] from comment #55)

Interesting (and sad) to see the huge impact on eslint locally, but very small in CI.

I think that's mostly CPU bound (my local machine has 64 cores, which is why it's so much faster than CI now). Let me experiment with different worker types with different core counts, we might be able to cut the wall time in half.

Worker pool vCPUs Lint phase (s) Lint phase change Task (s) Task change Checkout cache
t-linux-docker-amd (current) 8 300 baseline 464 baseline cold
t-linux-docker-16c32gb-amd 16 194 -35% 381 -18% cold
b-linux-docker-amd 16 185 -38% 227 -51% warm
b-linux-docker-large-amd 30 144 -52% 178 -62% warm (4 of 5)
b-linux-docker-xlarge-amd 60 127 -58% 247 -47% cold

Nice, b-linux-docker-large-amd (30 cores) looks like the best choice. It's more expensive, but a 2x speedup is probably worth it. The 60 core is slightly faster, but I don't think we have many of those workers running, so it'll probably miss the checkout cache more often and not come out ahead.

I'll do the same test with some other linters, too.

Come to think of it, I had noticed that too when working on the balance patches that I mentioned earlier. It makes total sense to bump, as eslint is often the slowest task, thus affecting end to end times for code review bot.

As we're talking about the ESLint perf numbers, there's a couple of things, I though I should note:

  • ESLint does have the facility for a cache, though I've never got around to testing it. I'm not sure if we'd be able to use it on CI either.
  • ESLint also has the concurrency option, however I think it is mainly a slightly different form of what we're doing here. iirc when I tested it before there wasn't a significant improvement on what we do, because they do some single-process set-up things which on our repository still takes a long time.
  • Prettier (run within the ESLint & StyleLint processes as it needs to be run after those) has been experimenting with a new CLI interface. Unfortunately that's still experimental, so probably not quite ready for use yet. A different option would be Oxfmt, though I notice that it doesn't handle our custom css and it has a couple of other minor issues, but it is a lot faster, so might be an option to investigate.

(In reply to Mark Banner (:standard8) from comment #59)

  • ESLint also has the concurrency option, however I think it is mainly a slightly different form of what we're doing here. iirc when I tested it before there wasn't a significant improvement on what we do, because they do some single-process set-up things which on our repository still takes a long time.

I wonder if this might be better anyway. One of the problems I had with my approach when increasing concurrency too much was the overhead of starting Node processes. If eslint is doing things concurrently within a single Node execution, it might have way less overhead.

Authored by https://github.com/jwmossmoz
https://github.com/mozilla-releng/fxci-config/commit/ec63b2297b58b8a65e69cab62f9db8b97d2bc64c
[main] feat(worker-pools): add 30-core gecko-t Linux docker pools for CPU-bound lint (#1338)

(via GitHub webhook, authenticated as shared-secret)

Status: NEW → RESOLVED
Closed: 21 days ago
Flags: qe-verify+
Resolution: --- → FIXED
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

(In reply to Marco Castelluccio [:marco] from comment #60)

(In reply to Mark Banner (:standard8) from comment #59)

  • ESLint also has the concurrency option, however I think it is mainly a slightly different form of what we're doing here. iirc when I tested it before there wasn't a significant improvement on what we do, because they do some single-process set-up things which on our repository still takes a long time.

I wonder if this might be better anyway. One of the problems I had with my approach when increasing concurrency too much was the overhead of starting Node processes. If eslint is doing things concurrently within a single Node execution, it might have way less overhead.

From my limited testing, there's a serialized bottleneck of it finding the files before evenly splitting them between workers, and it ends up being slower than what we have now.(In reply to Marco Castelluccio [:marco] from comment #25)

(In reply to Marco Castelluccio [:marco] from comment #25)

Bug 1994717 was trying to address the eslint imbalance in another, more complicated, way. The only advantage was that it would be dynamically choosing the split instead of hardcoding it (which means it would evolve automatically as the tree evolves, while we might need to tune "split-dirs" at some point).

I'm taking another look at this, with the consideration of improving the core count in CI I think there's more gains we can unlock with your idea.

Approach Worker pool vCPUs Lint phase (s) Lint phase change Task (s) Task change Checkout cache
Baseline t-linux-docker-amd 8 300 baseline 464 baseline cold
split-dirs: 1 b-linux-docker-large-amd 30 144 -52% 178 -62% warm
split-dirs: auto (WIP) t-linux-docker-large-amd 30 89 -70% 211 -55% cold

:jmoss created a new worker type for me (I don't think we were going to be able to use the build pool ones). It'll hit get a cold checkout cache more frequently, and that'll eat away at the wins until we get more checkout time improvements from switching to git. In the meantime, from a quick chat with Sylvestre, it sounds like he's in favor of marginally higher spend for significant end to end time improvements, so I'm going to see what tasks I can improve with more parallelization in addition to this, which would also help improve getting hot checkout caches more frequently if there's more tasks using the pool.

No longer blocks: 1994717
Depends on: 1994717

Authored by https://github.com/jwmossmoz
https://github.com/mozilla-releng/fxci-config/commit/dc258d4b81cc6d6751ca0140c61af3ccbefe3f05
[main] feat(worker-pools): add gecko-t c3d-highcpu-30 pool with SMT off (#1339)

(via GitHub webhook, authenticated as shared-secret)

Status: REOPENED → RESOLVED
Closed: 21 days ago → 21 days ago
Resolution: --- → FIXED
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Depends on: 2086579
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: