1710718 - [perfdocs] Failling to delete generated files on subsequent perfdocs runs locally

Status: NEW

(Testing :: Performance, defect, P3)

Description

[Greg Mierzwinski [:sparky]]

This bug is to fix an issue in perfdocs when running it locally with --fix.

The error for this issues varies from one person to another but generally it's that the generated files can't be deleted. If we remove them manually, then run perfdocs, there are no issues. It only happens if there already exist some generated files.

I've noticed that it looks like perfdocs is running multiple times, maybe this is why the deletion is having issues (race condition). Perfdocs should only run once rather than multiple times.

Comment 1

[Nazım Can Altınova [:canova][:canaltinova on phabricator]]

I was hitting the same problem while generating the perfdocs and I was pretty annoyed with it so decided to spend some time to understand.

It looks like you are correct, it runs multiple times and it creates a race condition while multiple of the instances try to delete the generated files directory at the same time.

It looks like it comes down to linter "type". It looks like our linting system is first looking at the total amount of files to be linted, then it splits the files depending on the amount of cpu core that you have in your machine.

For example, spawning all the linting jobs happens in this function.

Looking more deeply into that function revealed that this is the things we might want to change in the perfdocs linter: https://searchfox.org/mozilla-central/rev/fa219b9c1d6c2c64aa139a848573caa03bc81f25/python/mozlint/mozlint/roller.py#296-298

As you can see, it limits the job count to 1, if the linter type is "global". It also has a comment that says "Global linters lint the entire tree in one job.". Looking into the perfdocs, it seems like it's structured_log though. So we might want to change the linter type of perfdocs to global, but then we will lose the structured log capability (I don't know how global linters handle the log?). Or we might want to add the structured_log type in the if check as well to limit it, as the only linter that uses the structured_log is perfdocs.

I don't think I'm the best person to look into it further as I'm not sure why we chose the structured_log type for this linter or how it works. But happy to help in any case.