Run doc-upload on 30 core workers
Categories
(Firefox Build System :: Task Configuration, enhancement)
Tracking
(firefox162 fixed)
| Tracking | Status | |
|---|---|---|
| firefox162 | --- | fixed |
People
(Reporter: ahochheiden, Assigned: ahochheiden)
References
(Blocks 1 open bug)
Details
Attachments
(2 external files)
doc-upload builds the docs with Sphinx, which reads and writes them in parallel since bug 2084074, so it scales with cores. On try the docs build takes 154 seconds on the 8 core workers and 102 seconds on b-linux-docker-large-amd, and the whole task 235 seconds instead of 324.
| Assignee | ||
Comment 1•17 days ago
|
||
With the sources read in parallel, doc-upload scales with cores. On try
the docs build takes 102 seconds on 30 core AMD workers instead of 154 on
the 8 core Intel ones, and the whole task 235 seconds instead of 324.
Comment 3•15 days ago
|
||
| bugherder | ||
| Assignee | ||
Comment 4•14 days ago
|
||
doc-upload was the only task on its worker pool with a sparse
profile, so it had a checkout cache to itself that was almost never
warm. A full checkout shares the cache of the other tasks on that
pool, which should raise its cache hit rate from about 5% to about
70%. The remaining misses get slightly slower because a full checkout
writes more files, but the hits more than make up for it in aggregate.
When these tasks switch from hg to Git, nothing changes for
doc-upload. Git tasks already ignore the sphinx-docs sparse
profile, because its # git-checkout: full marker asks for a full
checkout, which is faster than a sparse one on Git.
| Assignee | ||
Updated•14 days ago
|
Updated•14 days ago
|
Updated•14 days ago
|
Comment 6•13 days ago
|
||
| bugherder | ||
Description
•