Closed Bug 2088327 Opened 17 days ago Closed 13 days ago

Run doc-upload on 30 core workers

Categories

(Firefox Build System :: Task Configuration, enhancement)

enhancement

Tracking

(firefox162 fixed)

RESOLVED FIXED
162 Branch
Tracking Status
firefox162 --- fixed

People

(Reporter: ahochheiden, Assigned: ahochheiden)

References

(Blocks 1 open bug)

Details

Attachments

(2 external files)

doc-upload builds the docs with Sphinx, which reads and writes them in parallel since bug 2084074, so it scales with cores. On try the docs build takes 154 seconds on the 8 core workers and 102 seconds on b-linux-docker-large-amd, and the whole task 235 seconds instead of 324.

With the sources read in parallel, doc-upload scales with cores. On try
the docs build takes 102 seconds on 30 core AMD workers instead of 154 on
the 8 core Intel ones, and the whole task 235 seconds instead of 324.

Status: NEW → RESOLVED
Closed: 15 days ago
Resolution: --- → FIXED
Target Milestone: --- → 162 Branch

doc-upload was the only task on its worker pool with a sparse
profile, so it had a checkout cache to itself that was almost never
warm. A full checkout shares the cache of the other tasks on that
pool, which should raise its cache hit rate from about 5% to about
70%. The remaining misses get slightly slower because a full checkout
writes more files, but the hits more than make up for it in aggregate.

When these tasks switch from hg to Git, nothing changes for
doc-upload. Git tasks already ignore the sphinx-docs sparse
profile, because its # git-checkout: full marker asks for a full
checkout, which is faster than a sparse one on Git.

Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Attachment #9675478 - Attachment description: Bug 2088327 - Check out the full tree in doc-upload r?#linter-reviewers → Bug 2088327 - Check out the full tree in doc-generate and doc-upload r?#linter-reviewers
Pushed by ahochheiden@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/93ae40971891 https://hg.mozilla.org/integration/autoland/rev/95bfbec0d9a9 Check out the full tree in doc-generate and doc-upload r=linter-reviewers,sergesanspaille
Status: REOPENED → RESOLVED
Closed: 15 days ago → 13 days ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: