1867845 - Parallelize the compilation of the output from wasm2c

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[Jonathan Watt [:jwatt]]

I'm spinning this off from bug 1867594.

I have an experimental patch that cuts 1 minute 20 seconds off my builds, which on my 16 core M3 Max MBP is a 15% reduction in the time it takes to do a clobber build. The percentage of time saved during incremental builds that require wasm2c to be rerun is of course a lot more.

Comment 1

[Jonathan Watt [:jwatt]]

For background, while digging into the slowness of wasm2c, I spotted some discussion about splitting the output from wasm2c into multiple output files in the wasm2c roadmap ideas issue. See the "Speeding up wasm2c itself on large programs" checklist item, and search for "split" and "w2c2" in the comments.

In fact, that checklist item is checked, despite no comment advertising support as having been added, or how to use it. Digging through the commit history I found this merge that added a new --num-outputs argument to wasm2c, and it turns out we've already updated our vendored copy of wasm2c to a new version that includes this new argument.

Comment 2

[Jonathan Watt [:jwatt]]

Attachment: Bug 1867845 - Split rlbox.wasm.c in multiple parts in local builds.

Comment 3

[Jonathan Watt [:jwatt]]

A profile of a clobber build without this patch (9 minutes and 29 seconds):
https://share.firefox.dev/414j6sV

A profile of a clobber build with this patch (8 minutes and 8 seconds):
https://share.firefox.dev/3uHh8CV

This only parallelizes the rlbox.wasm.o step, not the rlbox.wasm.c step.

Comment 4

[Mike Hommey [:glandium]]

Attachment: Bug 1867845 - Add `len` and `range` to the moz.build sandbox.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ce85df2375bc
Add `len` and `range` to the moz.build sandbox. r=firefox-build-system-reviewers,sergesanspaille
https://hg.mozilla.org/integration/autoland/rev/921a126c2e62
Split rlbox.wasm.c in multiple parts in local builds. r=firefox-build-system-reviewers,sergesanspaille

Comment 6

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/ce85df2375bc
https://hg.mozilla.org/mozilla-central/rev/921a126c2e62

Comment 7

[Florian Quèze [:florian]]

Clobber build profiles before and after. The total time went down from 4min4s to 3min44s (I tried 2 or 3 times both before and after and the numbers were stable +/- 1s). I can't say for sure if the 20s saved are all from this bug or if other improvements were made in the meantime, but rlbox.wasm.o is definitely no longer what we are waiting for.

"file_generate — rlbox.wasm.c" still seems to cause a drop in CPU use until it is finished, as no new Object job seems to start until it is done: https://share.firefox.dev/40gikJj