1793625 - Mozlint based jobs should optimized based on mozlint config files

Status: RESOLVED FIXED

(Thunderbird :: Build Config, task)

Description

[Rob Lemley [:rjl]]

Taskgraph optimization for mozlint jobs duplicates the when-to-run configuration based on affected files with the mozlint yaml files. This leads to cases where CI does not check files that should be checked because the configs are out of sync.

https://phabricator.services.mozilla.com/D158564#5203901

Let's read the mozlint yaml in an optimization strategy and use it to decide when to run.

Comment 1

[Rob Lemley [:rjl]]

Attachment: Bug 1793625 - Taskgraph optimization strategy for mozlint jobs. r=dandarnell

To avoid duplication of file lists between mozlint configs and taskgraph job
configs, for mozlint jobs use the include, exclude, extensions, and support-files
from the mozlint config to determine when to run.

Most of the work is handled by mozlint.pathutils.filterpaths.

Comment 2

[Rob Lemley [:rjl]]

Attachment: Bug 1793625 - Adjust mozlint configs to work with taskgraph skip-unless-mozlint optimization. r=#thunderbird-reviewers

"extensions" cannot include a leading "."

Thunderbird jobs usually do not want to run on upstream code, so set "include" to
"comm/" when a more specific list of includes is not provided.

Depends on D158590

Comment 3

[Rob Lemley [:rjl]]

Attachment: Bug 1793625 - Use "skip-unless-mozlint" optimization for mozlint jobs. r=dandarnell

The optimization only works for Thunderbird when a mozlint file included in
comm/tools/lint. There's a couple of linters that run that use the upstream
configs.

Depends on D158591

Comment 4

[Rob Lemley [:rjl]]

Attachment: Bug 1793625 - Misc mozlint related fixes: codespell and flake8. r=#thunderbird-reviewers

Depends on D158591

Comment 5

[Rob Lemley [:rjl]]

I think the OptimizationStrategy is generic enough now that gecko_taskgraph could use it without having to make changes.

Instead of using "when -> files-changed" in source-test/mozlint.yml, use something like:

optimization:
    skip-unless-mozlint:
        root-path: comm
        mozlint-config: codespell.yml

"root-path" is optional, and would not need to be set for Firefox. It's used to find the YAML files in tools/lint/.
"mozlint-config" is the YAML file to use. Most of these line up with a job name, but I think one or two did not, so this is needed.

The strategy uses mozlint.pathutils.filterpaths and the list of changed files from hg.m.o to decide if a particular job needs to run. I've noticed a couple of instances when it runs different jobs than the current optimizations, but when I looked closer, they were different because the changed files lists were more inclusive than mozlint.

jcristau -- Is this useful for gecko_taskgraph? I can put it there and still use for Thunderbird. (Have not tested on Firefox taskgraphs yet.) D158590 and D158592 are most relevant.

Comment 6

[Julien Cristau [:jcristau]]

(In reply to Rob Lemley [:rjl] from comment #5)

jcristau -- Is this useful for gecko_taskgraph? I can put it there and still use for Thunderbird. (Have not tested on Firefox taskgraphs yet.) D158590 and D158592 are most relevant.

That looks very nice, yes, thanks!

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/00b2693c8fb5
Taskgraph optimization strategy for mozlint jobs. r=dandarnell
https://hg.mozilla.org/comm-central/rev/fe247a4f1566
Adjust mozlint configs to work with taskgraph skip-unless-mozlint optimization. r=mkmelin
https://hg.mozilla.org/comm-central/rev/c53eff8dd08d
Misc mozlint related fixes: codespell and flake8. r=mkmelin
https://hg.mozilla.org/comm-central/rev/3d27c0cc1990
Use "skip-unless-mozlint" optimization for mozlint jobs. r=dandarnell