Speed up the source docs build
Categories
(Developer Infrastructure :: Source Documentation, task)
Tracking
(firefox161 affected)
| Tracking | Status | |
|---|---|---|
| firefox161 | --- | affected |
People
(Reporter: Sylvestre, Assigned: Sylvestre)
References
Details
(Keywords: leave-open)
Attachments
(4 external files)
Expected performance wins (clean ./mach doc -j 4, 3 runs each):
- Overall: 115 s → 47 s, about 2.5x faster.
- Pickling fix: 124 s → 67 s. Doctrees no longer each carry a copy of the build environment
(490 MB → 96 MB). - Garbage collector off during the build: 67 s → 47 s. Peak memory rises from 0.84 to 1.41 GiB,
still below the 1.12 GiB baseline before the performance work. - Parallel read: on its own it adds 7 s, but it's what the other two fixes build on. With them
in place, switching back to a serial read costs 85 s instead of 47 s and uses more memory
(2.48 GiB).
| Assignee | ||
Comment 1•1 month ago
|
||
MyST parses each eval-rst block into a temporary document that shares the
original settings object, and the nodes it moves into the page keep pointing at
it. Sphinx only clears settings.env on a copy of the settings before pickling,
so every such doctree carried a full copy of the BuildEnvironment.
Reattaching those nodes to their doctree shrinks the doctrees from 490 MB to
96 MB. Loading them back in the main process during the write phase drops from
61 s to 6.5 s, and a clean -j 4 build goes from 116 s to 102 s with the same
HTML output.
Updated•1 month ago
|
| Assignee | ||
Comment 2•1 month ago
|
||
mozbuild.sphinx, bzlink and sphinx-js did not declare whether they are safe for parallel reading, so Sphinx fell back to a serial read. None of them keep per-document state.
A clean -j 4 build goes from 102 s to 71 s.
| Assignee | ||
Comment 3•1 month ago
|
||
A CodSpeed walltime profile of a full build showed the main process spending
most of its own time in libpython, which gc.callbacks timing confirmed to be
garbage collection: about 18 s at -j 4, nearly all in full collections over
the growing environment and doctrees. Worker processes forked from it pay the
same cost.
Turn the collector off once reading starts, collect once after the parallel
reads are merged, and turn it back on when the build finishes. A clean -j 4
build goes from 70 s to 51 s, with peak resident memory going from 0.84 GB to
1.41 GB.
| Assignee | ||
Comment 4•1 month ago
|
||
Collect and gc.freeze() when reading and writing start, so the objects that
already exist stay out of later collections and their memory stays shared with
the forked workers. On top of disabling automatic collection this is a small
gain: a clean -j 4 build goes from 54 s to 52 s with the same peak memory.
Comment 6•23 days ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/e2dbda1f3b25
https://hg.mozilla.org/mozilla-central/rev/301ed54c913a
| Assignee | ||
Updated•18 days ago
|
Updated•18 days ago
|
Comment 8•17 days ago
|
||
| bugherder | ||
Description
•