Closed Bug 2085856 Opened 25 days ago Closed 15 days ago

Speed up mozlint path filtering and tree walks

Categories

(Developer Infrastructure :: Lint and Formatting, enhancement)

enhancement

Tracking

(firefox162 fixed)

RESOLVED FIXED
162 Branch
Tracking Status
firefox162 --- fixed

People

(Reporter: ahochheiden, Assigned: ahochheiden)

References

(Blocks 1 open bug)

Details

Attachments

(4 external files, 1 obsolete file)

Excludes are matched one pattern at a time, and the tree is walked once per glob or extension.

filterpaths walked each path once per glob exclude through
FileFinder, including the excludes that neither exist nor contain a
glob and so cannot match. Over the whole tree that took about 30 seconds
of CPU in each worker. One scandir walk now tests every name against a
single regex, and does not descend into excluded directories.
mozpath.matcher looks plain path excludes up by ancestor in a set, so
the regex holds only the globs.

FileFinder tested every file and directory against each ignore pattern
in turn. mozlint passes several hundred excludes, so the whole tree walk in
license-declarations spent most of its 104 seconds there. The patterns
are now compiled into one regex when the finder is created.

FileFinder.find("**") built a File for every file in the tree and
tested it against the excludes before expand_exclusions dropped the
ones with other extensions. The new FileFinder.walk yields the same
paths in the same order without building a File, so the extension is
checked first. A whole tree mach lint -l license on Windows went from
about 22 to 17 seconds.

BaseType._lint_dir walked a directory once for each extension a
linter lists. It now walks it once and skips the other extensions. A
whole tree mach lint -l mscom-init on Windows went from about 39 to
16 seconds.

filterpaths tested every path against every exclude with
FilterPath.contains, which made both paths absolute, normalized and
split them on each call, and it gathered the excludes under each include
again for every path. The components are now kept on the FilterPath
and the excludes under each include are gathered once. A whole tree
mach lint -l includes on Windows went from about 9.0 to 8.0 seconds.

Attachment #9668104 - Attachment description: Bug 2085856 - Part 3 - Check the extension before building a `File` in `expand_exclusions` r?#linter-reviewers → Bug 2085856 - Part 2 - Check the extension before building a `File` in `expand_exclusions` r?#linter-reviewers
Attachment #9668105 - Attachment description: Bug 2085856 - Part 4 - Walk each directory once in the string and regex types r?#linter-reviewers → Bug 2085856 - Part 3 - Walk each directory once in the string and regex types r?#linter-reviewers
Attachment #9668106 - Attachment description: Bug 2085856 - Part 5 - Compute the path components of a `FilterPath` once r?#linter-reviewers → Bug 2085856 - Part 4 - Compute the path components of a `FilterPath` once r?#linter-reviewers
Attachment #9668103 - Attachment is obsolete: true
Pushed by ahochheiden@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/1962c2214aba https://hg.mozilla.org/integration/autoland/rev/84591286e09d Part 1 - Expand the glob excludes in one walk r=linter-reviewers,sylvestre https://github.com/mozilla-firefox/firefox/commit/72b603652452 https://hg.mozilla.org/integration/autoland/rev/c92fb827d0a6 Part 2 - Check the extension before building a `File` in `expand_exclusions` r=linter-reviewers,sergesanspaille https://github.com/mozilla-firefox/firefox/commit/0a303a9d9afb https://hg.mozilla.org/integration/autoland/rev/f50986122a9e Part 3 - Walk each directory once in the string and regex types r=linter-reviewers,sylvestre https://github.com/mozilla-firefox/firefox/commit/9f59778520e5 https://hg.mozilla.org/integration/autoland/rev/7b5beb5b3e36 Part 4 - Compute the path components of a `FilterPath` once r=linter-reviewers,sergesanspaille
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: