Speed up mozlint path filtering and tree walks
Categories
(Developer Infrastructure :: Lint and Formatting, enhancement)
Tracking
(firefox162 fixed)
| Tracking | Status | |
|---|---|---|
| firefox162 | --- | fixed |
People
(Reporter: ahochheiden, Assigned: ahochheiden)
References
(Blocks 1 open bug)
Details
Attachments
(4 external files, 1 obsolete file)
Excludes are matched one pattern at a time, and the tree is walked once per glob or extension.
| Assignee | ||
Comment 1•25 days ago
|
||
filterpaths walked each path once per glob exclude through
FileFinder, including the excludes that neither exist nor contain a
glob and so cannot match. Over the whole tree that took about 30 seconds
of CPU in each worker. One scandir walk now tests every name against a
single regex, and does not descend into excluded directories.
mozpath.matcher looks plain path excludes up by ancestor in a set, so
the regex holds only the globs.
| Assignee | ||
Comment 2•25 days ago
|
||
FileFinder tested every file and directory against each ignore pattern
in turn. mozlint passes several hundred excludes, so the whole tree walk in
license-declarations spent most of its 104 seconds there. The patterns
are now compiled into one regex when the finder is created.
| Assignee | ||
Comment 3•25 days ago
|
||
FileFinder.find("**") built a File for every file in the tree and
tested it against the excludes before expand_exclusions dropped the
ones with other extensions. The new FileFinder.walk yields the same
paths in the same order without building a File, so the extension is
checked first. A whole tree mach lint -l license on Windows went from
about 22 to 17 seconds.
| Assignee | ||
Comment 4•25 days ago
|
||
BaseType._lint_dir walked a directory once for each extension a
linter lists. It now walks it once and skips the other extensions. A
whole tree mach lint -l mscom-init on Windows went from about 39 to
16 seconds.
| Assignee | ||
Comment 5•25 days ago
|
||
filterpaths tested every path against every exclude with
FilterPath.contains, which made both paths absolute, normalized and
split them on each call, and it gathered the excludes under each include
again for every path. The components are now kept on the FilterPath
and the excludes under each include are gathered once. A whole tree
mach lint -l includes on Windows went from about 9.0 to 8.0 seconds.
Updated•23 days ago
|
Updated•23 days ago
|
Updated•23 days ago
|
Updated•23 days ago
|
Comment 7•15 days ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/84591286e09d
https://hg.mozilla.org/mozilla-central/rev/c92fb827d0a6
https://hg.mozilla.org/mozilla-central/rev/f50986122a9e
https://hg.mozilla.org/mozilla-central/rev/7b5beb5b3e36
Description
•