Evaluate not removing rust incremental compilation cache during |mach clobber|
Categories
(Firefox Build System :: General, defect)
Tracking
(firefox69 fixed)
Tracking | Status | |
---|---|---|
firefox69 | --- | fixed |
People
(Reporter: erahm, Assigned: erahm)
References
Details
Attachments
(3 files, 1 obsolete file)
We'd like to see if it's reasonable to leave the rust incremental compilation cache around after a mach clobber
invocation. The idea is that will allow for faster rebuilds when a mach clobber
is necessary for C++ code. It's possible there are reasons this won't work, so for now we just want to evaluate the effect on build times and if it's possible at all.
The main drawbacks that I know of are:
- When we update rust versions we might leave a cache dir around. I don't think this is an actual issue w/in the Firefox build environment.
- If you're locally hacking a rust checkout and messing with the format of the cache files you could get into a bad state. I don't think we need to worry about this.
Assignee | ||
Comment 1•6 years ago
|
||
Updated•6 years ago
|
Assignee | ||
Comment 2•6 years ago
|
||
Pull out the logic for filtering subdirectories and deleting them into
reusable functions.
Assignee | ||
Comment 3•6 years ago
|
||
Pass in the substs dictionary to Clobberer
so that we can use it to query cargo
paths.
Assignee | ||
Comment 4•6 years ago
|
||
Skips over the incremental cache when performing a clobber. The incremental compilation cache is located at:
$(objdir)/$(rust_target)/$(rust_build_kind)/incremental
When cross compiling there can be two caches, one for the host and one for the target so we handle both.
Assignee | ||
Comment 5•6 years ago
|
||
Initial numbers on this for a pretty contrived use case, essentially:
./mach build && ./mach clobber # fill up ccache and IC cache
multitime -q -n 6 -r './mach clobber' ./mach build # Run 6 times and average results
Shows a 494s -> 386s speedup (~20%).
Comment 7•6 years ago
|
||
Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=250235451&repo=autoland&lineNumber=278
Backout link: https://hg.mozilla.org/integration/autoland/rev/a47c1ea455753afd6dfb1e685c213be931f3bb2b
[task 2019-06-05T21:19:24.821Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil
[task 2019-06-05T21:19:24.821Z]
[task 2019-06-05T21:19:24.821Z] Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
[task 2019-06-05T21:22:37.952Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/python/mozbuild/mozbuild/controller/clobber.py:148:9 | too many blank lines (2) (E303)
[task 2019-06-05T21:22:37.952Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/python/mozbuild/mozbuild/controller/clobber.py:149:5 | expected 1 blank line, found 0 (E301)
[taskcluster 2019-06-05 21:22:38.253Z] === Task Finished ===
[taskcluster 2019-06-05 21:22:40.433Z] Unsuccessful task run with exit code: 1 completed in 501.647 seconds
Assignee | ||
Comment 8•6 years ago
|
||
I guess lando did a bad rebase, I'll fix locally.
Assignee | ||
Comment 9•6 years ago
|
||
I tried to lando this but it doesn't seem to have actually landed. I'll leave it to the experts to get this landed.
Comment 10•6 years ago
|
||
Please ask for a review before we can land this.
Removing checkin-needed for now.
Assignee | ||
Comment 11•6 years ago
|
||
(In reply to Dorel Luca [:dluca] from comment #10)
Please ask for a review before we can land this.
Removing checkin-needed for now.
It has been reviewed, it just need to be rebased.
Comment 12•6 years ago
|
||
Pushed by nbeleuzu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/5a8df036968e
Part 1: Factor out subdirectory logic. r=chmanchester
https://hg.mozilla.org/integration/autoland/rev/f35cdcfa0a28
Part 2: Make the Clobberer aware of substs. r=chmanchester
https://hg.mozilla.org/integration/autoland/rev/33fbb6ff15cd
Part 3: Don't remove the rust incremental cache when clobbering. r=chmanchester
Comment 13•6 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/5a8df036968e
https://hg.mozilla.org/mozilla-central/rev/f35cdcfa0a28
https://hg.mozilla.org/mozilla-central/rev/33fbb6ff15cd
Updated•6 years ago
|
Description
•