Open Bug 1486863 Opened 6 years ago Updated 2 years ago

[mozlint] Refactor job creation and setup logic

Categories

(Developer Infrastructure :: Lint and Formatting, task)

Tracking

(Not tracked)

People

(Reporter: ahal, Unassigned)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

A lint job is a tuple of (<linter>, <paths>). Each job is run in it's own multiprocessing work via the Pool pattern. The logic for calculating the jobs and splitting them up across workers all lives in roller.py.

I want to refactor this for two reasons:

1) It is done kind of hackishly, and adds a lot of complexity to roller.py. I'd like to pull all "job" related logic out into a separate 'jobs' module. One major reason I'd like to do this now is because I have plans to add *more* complexity to the job creation logic in the future. For example, I want to introduce "config files" to mozlint, and I want to make sure that all paths in a single "job" have the same configuration apply to them. This will add a fair bit of complexity to the job creation logic.

2) The order things are done in is wrong. Right now the following happens:
  a. Create jobs
  b. Filter paths (within the job)
  c. Run remaining linters (within the same job)

This is bad because we don't know which linters are going to run until *after* we've already created the jobs. Ideally we'd have this information before hand which allows to use it as an input to the job creation logic (resulting in more balanced chunks and more efficient use of resources). So the order should be:

  a. Filter paths
  b. Create jobs with remaining linters
  c. Run linters

Note step a) can still be split up across processes for parallelization. It just wouldn't run within the "main" worker that ultimately runs the linter.
Depends on: 1486866
Assignee: nobody → ahal
Status: NEW → ASSIGNED
I was originally blocking on bug 1486866 to fix some path filtering issues, but I ended up getting into a catch-22 with this one (see bug 1486866 comment 8 for more details).

The result is that I'm going to keep pushing here and fix the path filtering issues at the same time. Things are looking pretty good so far. With some unscientific testing, I've seen perf improvements averaging 50%! Add to that a lot of reduced complexity and easier to maintain code. There are still a few wrinkles left to iron out though.
Blocks: 1556986
Assignee: ahal → nobody
Status: ASSIGNED → NEW
Type: enhancement → task
Blocks: 1597277
Product: Firefox Build System → Developer Infrastructure
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: