2023411 - mozlinter: port some linters to Rust

Status: REOPENED

(Developer Infrastructure :: Lint and Formatting, task)

Comment 1

[Sylvestre Ledru [:Sylvestre]]

Attachment: Bug 2023411 - Rewrite rejected-words linter in Rust with Python wrapper r?#linter-reviewers

Replaces the built-in regex linter type with an external linter backed by
the moz-linter Rust binary. Files are scanned in parallel using rayon.
The Python wrapper builds moz-linter from source if no pre-built binary
is available.

Comment 2

[Sylvestre Ledru [:Sylvestre]]

Attachment: Bug 2023411 - Rewrite file-perm linter in Rust with Python wrapper r?#linter-reviewers

Replaces the Python permission-checking logic with the moz-linter Rust
binary. Checks executable bits in parallel using rayon, supports
--allow-shebang and --fix modes. Falls back to building from source
locally.

Comment 3

[Sylvestre Ledru [:Sylvestre]]

Attachment: Bug 2023411 - Rewrite file-whitespace linter in Rust with Python wrapper r?#linter-reviewers

Replaces the Python whitespace-checking logic with the moz-linter Rust
binary. Detects and fixes trailing whitespace, missing final newlines,
empty trailing lines, and Windows CRLF line endings. Files are processed
in parallel using rayon.

Comment 4

[Sylvestre Ledru [:Sylvestre]]

Attachment: Bug 2023411 - Rewrite trojan-source linter in Rust with Python wrapper r?#linter-reviewers

Replaces the Python Unicode format-character detection with the
moz-linter Rust binary. Scans for Unicode Cf category characters
(CVE-2021-42572) in parallel using rayon.

Comment 5

[Sylvestre Ledru [:Sylvestre]]

Attachment: Bug 2023411 - Fix trailing whitespace in 4 files found by file-whitespace linter r?#linter-reviewers

Comment 6

[Sylvestre Ledru [:Sylvestre]]

Attachment: Bug 2023411 - Rewrite license linter in Rust with Python wrapper r?#linter-reviewers

Implement license check in Rust: validates files contain a recognized
license pattern, adds appropriate headers in fix mode, and checks
toolkit/content/license.html references.

Comment 7

[Marco Castelluccio [:marco]]

If we port the regex ones, we could fix bug 1959706.

Comment 8

[Marco Castelluccio [:marco]]

Ah https://phabricator.services.mozilla.com/D288083 is doing that ;)

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/c8be7046fc81
https://hg.mozilla.org/integration/autoland/rev/fab786399cbf
Fix trailing whitespace in 4 files found by file-whitespace linter r=devtools-reviewers,nchevobbe DONTBUILD

Comment 10

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/fab786399cbf

Comment 11

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/fab786399cbf

Comment 12

[Takanori MATSUURA]

Reopened for unpublished changesets.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/4450b820ea6e
https://hg.mozilla.org/integration/autoland/rev/8833dcfe8085
Rewrite rejected-words linter in Rust with Python wrapper r=linter-reviewers,sergesanspaille,ahal
https://github.com/mozilla-firefox/firefox/commit/13d700d21ede
https://hg.mozilla.org/integration/autoland/rev/bc64f0011c3d
Rewrite file-perm linter in Rust with Python wrapper r=linter-reviewers,ahal,sergesanspaille
https://github.com/mozilla-firefox/firefox/commit/45e041167e4d
https://hg.mozilla.org/integration/autoland/rev/1da6752acedb
Rewrite file-whitespace linter in Rust with Python wrapper r=linter-reviewers,sergesanspaille
https://github.com/mozilla-firefox/firefox/commit/a97b716a4b3a
https://hg.mozilla.org/integration/autoland/rev/89806d24952b
Rewrite trojan-source linter in Rust with Python wrapper r=linter-reviewers,ahal,sergesanspaille
https://github.com/mozilla-firefox/firefox/commit/c29a886b2043
https://hg.mozilla.org/integration/autoland/rev/d0136c69b85b
Rewrite license linter in Rust with Python wrapper r=linter-reviewers,sergesanspaille

Comment 14

[Sylvestre Ledru [:Sylvestre]]

Attachment: Bug 2023411 - Follow-up: rename rejected_words helper to check_reject_words r?#linter-reviewers

Addresses review comment from sergesanspaille on D288083. The inner
function was named after its execution strategy (run_lint_in_parallel)
rather than what it checks; rename it to check_reject_words and update
the batch.rs caller to match.

Comment 15

[Sylvestre Ledru [:Sylvestre]]

Attachment: Bug 2023411 - Follow-up: comment the non-executable early-return in file_perm r?#linter-reviewers

Addresses nit from sergesanspaille on D288084: make the intent of the
mode & 0o111 == 0 guard explicit with a "not executable" comment.

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/95c876dd5b1d
https://hg.mozilla.org/integration/autoland/rev/f5e8406f5136
Rewrite rejected-words linter in Rust with Python wrapper r=linter-reviewers,sergesanspaille,ahal
https://github.com/mozilla-firefox/firefox/commit/f863bc0bf8ab
https://hg.mozilla.org/integration/autoland/rev/1d3cf79cf3a4
Rewrite file-perm linter in Rust with Python wrapper r=linter-reviewers,ahal,sergesanspaille
https://github.com/mozilla-firefox/firefox/commit/98fbb17bc183
https://hg.mozilla.org/integration/autoland/rev/445eb6fd45c0
Rewrite file-whitespace linter in Rust with Python wrapper r=linter-reviewers,sergesanspaille
https://github.com/mozilla-firefox/firefox/commit/0315328b0932
https://hg.mozilla.org/integration/autoland/rev/a8e05981c4b2
Rewrite trojan-source linter in Rust with Python wrapper r=linter-reviewers,ahal,sergesanspaille
https://github.com/mozilla-firefox/firefox/commit/05e7ee23d696
https://hg.mozilla.org/integration/autoland/rev/c8ba14a19ea5
Rewrite license linter in Rust with Python wrapper r=linter-reviewers,sergesanspaille

Comment 17

[Silaghi Andreea]

https://hg.mozilla.org/mozilla-central/rev/f5e8406f5136
https://hg.mozilla.org/mozilla-central/rev/1d3cf79cf3a4
https://hg.mozilla.org/mozilla-central/rev/445eb6fd45c0
https://hg.mozilla.org/mozilla-central/rev/a8e05981c4b2
https://hg.mozilla.org/mozilla-central/rev/c8ba14a19ea5

Comment 18

[Sylvestre Ledru [:Sylvestre]]

Attachment: Bug 2023411 - Follow-up: fix clippy warnings in mozcheck license and file_whitespace r?#linter-reviewers

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/5a2dd4406199
https://hg.mozilla.org/integration/autoland/rev/e256f9ca619f
Follow-up: rename rejected_words helper to check_reject_words r=linter-reviewers,sergesanspaille DONTBUILD
https://github.com/mozilla-firefox/firefox/commit/331fecf6a726
https://hg.mozilla.org/integration/autoland/rev/064c1eb7f594
Follow-up: comment the non-executable early-return in file_perm r=linter-reviewers,sergesanspaille DONTBUILD

Comment 20

[rperta]

https://hg.mozilla.org/mozilla-central/rev/e256f9ca619f
https://hg.mozilla.org/mozilla-central/rev/064c1eb7f594

Comment 21

[Sylvestre Ledru [:Sylvestre]]

Attachment: WIP: Bug 2023411 - Rewrite mingw-capitalization linter in Rust with Python wrapper r?#linter-reviewers

Replaces the Python MinGW header capitalization check with the mozcheck
Rust binary. Validates that Windows #include directives use lowercase
header names matching the MinGW headers list.

Comment 22

[Sylvestre Ledru [:Sylvestre]]

Attachment: Bug 2023411 - Extract Android OS-to-API map into a shared YAML file r?#linter-reviewers

Move android_os_to_api_map out of mozinfo/platforminfo.py and into
mozinfo/android_os_to_api_map.yaml so it can be consumed both by Python
(mozinfo) and by Rust (tools/lint/mozcheck via include_str!).

Comment 23

[Sylvestre Ledru [:Sylvestre]]

Attachment: Bug 2023411 - Port test-manifest-toml linter to Rust r?#linter-reviewers

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/b630fb04c81e
https://hg.mozilla.org/integration/autoland/rev/3e9f26230af2
Rewrite mingw-capitalization linter in Rust with Python wrapper r=linter-reviewers,sergesanspaille
https://github.com/mozilla-firefox/firefox/commit/285374975ce0
https://hg.mozilla.org/integration/autoland/rev/421353098297
Extract Android OS-to-API map into a shared YAML file r=linter-reviewers,sergesanspaille

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/e905604d1410
https://hg.mozilla.org/integration/autoland/rev/6728480377a8
Follow-up: fix clippy warnings in mozcheck license and file_whitespace r=linter-reviewers,sergesanspaille DONTBUILD

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/7190897e6864
https://hg.mozilla.org/integration/autoland/rev/47c76ae98ba6
Revert "Bug 2023411 - Extract Android OS-to-API map into a shared YAML file r=linter-reviewers,sergesanspaille" for causing mass failures.

Comment 27

[Serban Stanca [:SerbanS]]

Reverted this because it was causing mass failures.

• Revert link
• Push with failures
• Failure Log

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/80ea10298fd6
https://hg.mozilla.org/integration/autoland/rev/43533d7870e4
Revert "Bug 2023411 - Follow-up: fix clippy warnings in mozcheck license and file_whitespace r=linter-reviewers,sergesanspaille DONTBUILD" because it is a leftover from the initial backout.

Comment 29

[amarc]

https://hg.mozilla.org/mozilla-central/rev/43533d7870e4

Comment 30

[Sylvestre Ledru [:Sylvestre]]

Attachment: Bug 2023411 - Add manifestparser Rust crate r?#linter-reviewers

Rust port of testing/mozbase/manifestparser/manifestparser/, exposed as a
new crate at testing/mozbase/rust/manifestparser/.

Modules:

• util / ini : path normalization, INI manifest reader, combine_fields
• expression : Pratt-style boolean expression evaluator (nom-lexed)
• filters : skip-if/run-if/fail-if + chunkers + tags + pathprefix
• token : quote-aware whitespace normalization (strip_unquoted)
• parser : nom-based primitives shared with downstream consumers
  (Op, CmpOp, parse_op, split_ops)
• toml : read_toml + sort_paths
• manifest : ManifestParser + TestManifest with active_tests pipeline

The tomlkit-backed formatter helpers (alphabetize_toml_str, add_skip_if,
remove_skip_if, replace_tbd_skip_if) are intentionally not ported because
they depend on tomlkit's stylistic round-trip with no Rust equivalent.

The token vocabulary (legal os/os_version/etc. values) lives with the
test-manifest-toml linter, which loads it from YAML at runtime, rather
than being hardcoded here a second time.

Comment 31

[Sylvestre Ledru [:Sylvestre]]

Attachment: Bug 2023411 - Lowercase MinGW Windows header includes flagged by mozcheck r?tjr

The MinGW capitalization linter requires Windows SDK headers to use
lowercase filenames so the MinGW build can resolve them. The previous
Python implementation was a silent no-op due to a re.search(line, re.I)
bug (the re.I flag was being passed as the pos argument), so these
five files slipped in with <Mferror.h> / <Mfidl.h>. The new mozcheck
Rust port catches them; lowercase the includes to match the headers list.

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/4caddf6d2318
https://hg.mozilla.org/integration/autoland/rev/620086ee0637
Lowercase MinGW Windows header includes flagged by mozcheck r=tjr,media-playback-reviewers,padenot

Comment 33

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/620086ee0637

Comment 34

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/f4abf0169364
https://hg.mozilla.org/integration/autoland/rev/00856bc4cea0
Extract Android OS-to-API map into a shared YAML file r=linter-reviewers,sergesanspaille

Comment 35

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/00856bc4cea0