1899683 - `./mach lint` failure when multiple files in `python/mozperftest` are modified

Status: RESOLVED FIXED

(Developer Infrastructure :: Lint and Formatting, defect)

Description

[Alex Hochheiden [:ahochheiden] [he/him]]

I have D212084 as my head revision. The stack has modifications in these python/mozperftest files:

'D:/mozilla-source/mozilla-unified\\tools/lint/perfdocs/framework_gatherers.py', 
'D:/mozilla-source/mozilla-unified\\python/mozperftest/mozperftest/tests/test_visualtools.py', 
'D:/mozilla-source/mozilla-unified\\python/mozperftest/mozperftest/test/xpcshell.py',
'D:/mozilla-source/mozilla-unified\\python/mozperftest/mozperftest/test/browsertime/visualtools.py'

When I run ./mach lint --fix I get this error:

ahochheiden@Alex-Mozilla-P620 /d/mozilla-source/mozilla-unified
$ ./mach lint --fix
warning: linting the entire repo takes a long time, using --outgoing and --workdir instead. If you want to lint the entire repo, run `./mach lint .`
error: problem with lint setup, skipping android-api-lint, android-checkstyle, android-format, android-javadoc, android-lint, android-test
 0:37.55 INFO Found 10 perfdocs directories in ['D:\\mozilla-source\\mozilla-unified\\devtools\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\dom\\indexedDB\\test\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\python\\mozperftest\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\awsy\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\talos\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\performance\\fxrecord\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\performance\\mach-try-perf\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\performance\\perf-sheriffing\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\performance\\performance-infrastructure\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\raptor\\raptor\\perfdocs']
 0:37.55 INFO Found 10 perfdocs directories in ['D:\\mozilla-source\\mozilla-unified\\devtools\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\dom\\indexedDB\\test\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\python\\mozperftest\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\awsy\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\talos\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\performance\\fxrecord\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\performance\\mach-try-perf\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\performance\\perf-sheriffing\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\performance\\performance-infrastructure\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\raptor\\raptor\\perfdocs']
 0:37.55 INFO Found 10 perfdocs directories in ['D:\\mozilla-source\\mozilla-unified\\devtools\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\dom\\indexedDB\\test\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\python\\mozperftest\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\awsy\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\talos\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\performance\\fxrecord\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\performance\\mach-try-perf\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\performance\\perf-sheriffing\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\performance\\performance-infrastructure\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\raptor\\raptor\\perfdocs']
 0:37.55 INFO Found 10 perfdocs directories in ['D:\\mozilla-source\\mozilla-unified\\devtools\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\dom\\indexedDB\\test\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\python\\mozperftest\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\awsy\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\talos\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\performance\\fxrecord\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\performance\\mach-try-perf\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\performance\\perf-sheriffing\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\performance\\performance-infrastructure\\perfdocs', 'D:\\mozilla-source\\mozilla-unified\\testing\\raptor\\raptor\\perfdocs']
 1:03.88 INFO Regenerating perfdocs...
 1:03.88 INFO Regenerating perfdocs...
Traceback (most recent call last):
  File "d:\mozilla-source\mozilla-unified\python\mozlint\mozlint\roller.py", line 72, in _run_worker
    res = func(paths, config, **lintargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "d:\mozilla-source\mozilla-unified\python\mozlint\mozlint\types.py", line 58, in __call__
    return self._lint(paths, config, **lintargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "d:\mozilla-source\mozilla-unified\python\mozlint\mozlint\types.py", line 201, in _lint
    func(files, config, logger, **lintargs)
  File "D:\mozilla-source\mozilla-unified\tools\lint\perfdocs\__init__.py", line 13, in lint
    return perfdocs.run_perfdocs(config, logger=logger, paths=paths, generate=fix)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\mozilla-source\mozilla-unified\tools\lint\perfdocs\perfdocs.py", line 95, in run_perfdocs
    generator.generate_perfdocs()
  File "D:\mozilla-source\mozilla-unified\tools\lint\perfdocs\generator.py", line 305, in generate_perfdocs
    self._save_perfdocs(perfdocs_tmpdir)
  File "D:\mozilla-source\mozilla-unified\tools\lint\perfdocs\generator.py", line 246, in _save_perfdocs
    shutil.rmtree(str(self.perfdocs_path))
  File "C:\mozilla-build-4.2.pre\python3\Lib\shutil.py", line 820, in rmtree
    return _rmtree_unsafe(path, onexc)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\mozilla-build-4.2.pre\python3\Lib\shutil.py", line 652, in _rmtree_unsafe
    onexc(os.rmdir, path, err)
  File "C:\mozilla-build-4.2.pre\python3\Lib\shutil.py", line 650, in _rmtree_unsafe
    os.rmdir(path)
OSError: [WinError 145] The directory is not empty: 'D:\\mozilla-source\\mozilla-unified\\testing\\perfdocs\\generated'
 1:03.90 INFO Documentation saved to testing\perfdocs\generated/
 1:03.91 INFO Regenerating perfdocs...
 1:03.93 INFO Documentation saved to testing\perfdocs\generated/
 1:03.94 INFO Regenerating perfdocs...
 1:03.96 INFO Documentation saved to testing\perfdocs\generated/
A failure occurred in the perfdocs linter.
✖ 1 problem (0 errors, 0 warnings, 1 failure, 0 fixed)

From what I've gathered, we create one 'lint worker' process per file changed, and since there's 4 perfdocs files changed, we create 4 of those workers.

The --fix causes the perfdocs to be generated during linting (not sure if this is correct?), and when the generated perfdocs are saved we first clear out the directory (rmtree) then copy the generated files over (copytree). The problem is we do this on all 4 workers and they encounter race conditions (the reported failure) when doing that simultaneously.

I do not encounter this issue if I do not provide the --fix, and it's unclear to me why --fix would require the perfdocs to be generated. If generating the perfdocs is not necessary for linting, then maybe preventing that from being run is the way to go, otherwise the perfdocs generation should probably be triggered after the lint workers complete and only run once.

Comment 1

[Alex Hochheiden [:ahochheiden] [he/him]]

https://searchfox.org/mozilla-central/source/tools/lint/perfdocs/perfdocs.py#28-29
https://searchfox.org/mozilla-central/source/tools/lint/perfdocs/__init__.py#13

It looks like it is intentional, but it seems a bit overloaded to me.

Comment 2

[Greg Mierzwinski [:sparky]]

We're planning on moving perfdocs out of the lint tooling eventually to resolve this issue - see bug 1739406.

Comment 3

[BugBot [:suhaib / :marco]]

The severity field is not set for this bug.
:andi, could you have a look please?

For more information, please visit BugBot documentation.

Comment 4

[Greg Mierzwinski [:sparky]]

This should be resolved now with the move to ./mach perfdocs