Open
Bug 1486863
Opened 6 years ago
Updated 2 years ago
[mozlint] Refactor job creation and setup logic
Categories
(Developer Infrastructure :: Lint and Formatting, task)
Developer Infrastructure
Lint and Formatting
Tracking
(Not tracked)
NEW
People
(Reporter: ahal, Unassigned)
References
(Blocks 1 open bug)
Details
Attachments
(1 file)
A lint job is a tuple of (<linter>, <paths>). Each job is run in it's own multiprocessing work via the Pool pattern. The logic for calculating the jobs and splitting them up across workers all lives in roller.py. I want to refactor this for two reasons: 1) It is done kind of hackishly, and adds a lot of complexity to roller.py. I'd like to pull all "job" related logic out into a separate 'jobs' module. One major reason I'd like to do this now is because I have plans to add *more* complexity to the job creation logic in the future. For example, I want to introduce "config files" to mozlint, and I want to make sure that all paths in a single "job" have the same configuration apply to them. This will add a fair bit of complexity to the job creation logic. 2) The order things are done in is wrong. Right now the following happens: a. Create jobs b. Filter paths (within the job) c. Run remaining linters (within the same job) This is bad because we don't know which linters are going to run until *after* we've already created the jobs. Ideally we'd have this information before hand which allows to use it as an input to the job creation logic (resulting in more balanced chunks and more efficient use of resources). So the order should be: a. Filter paths b. Create jobs with remaining linters c. Run linters Note step a) can still be split up across processes for parallelization. It just wouldn't run within the "main" worker that ultimately runs the linter.
Reporter | ||
Updated•6 years ago
|
Assignee: nobody → ahal
Status: NEW → ASSIGNED
Reporter | ||
Comment 1•6 years ago
|
||
I was originally blocking on bug 1486866 to fix some path filtering issues, but I ended up getting into a catch-22 with this one (see bug 1486866 comment 8 for more details). The result is that I'm going to keep pushing here and fix the path filtering issues at the same time. Things are looking pretty good so far. With some unscientific testing, I've seen perf improvements averaging 50%! Add to that a lot of reduced complexity and easier to maintain code. There are still a few wrinkles left to iron out though.
Reporter | ||
Updated•4 years ago
|
Assignee: ahal → nobody
Status: ASSIGNED → NEW
Type: enhancement → task
Reporter | ||
Comment 2•4 years ago
|
||
Updated•2 years ago
|
Product: Firefox Build System → Developer Infrastructure
Updated•2 years ago
|
Severity: normal → S3
You need to log in
before you can comment on or make changes to this bug.
Description
•