1886167 - Parallelize the generation part of webidl

Status: RESOLVED FIXED

(Core :: DOM: Bindings (WebIDL), task)

Description

[[:sergesanspaille]]

Once the sequential parsing is done, dumping individual files can be done in parallel (at least conceptually). It would be great if the code could take advantage of that property

Comment 1

[[:sergesanspaille]]

Attachment: Bug 1886167 - Parallelize webidl generation step r=peterv!

Parallelization is process based, which implies a pickle-compatible IR,
thus the few code changes.

It is mandatory to serialize the IR only once per process for the
parallelization to be beneficial.

The parsing is still single-threaded.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bc73456fe2ef
Parallelize webidl generation step r=peterv

Comment 3

[pstanciu]

Backed out for causing WebIDL py3 failures

• Backout link
• Push with failures
• Failure Log
  Failure line: dom/bindings/mozwebidlcodegen/test/test_mozwebidlcodegen.py::TestWebIDLCodegenManager::test_only_rebuild_self TEST-UNEXPECTED-FAIL

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1973c82d95a4
Parallelize webidl generation step r=peterv

Comment 5

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/1973c82d95a4

Comment 6

[Kagami Rosylight [:saschanaz] (they/them) (OOO until Dec 11)]

Attachment: Bug 1886167 - Limit to 4 processes on Windows r=peterv,sergesanspaille

Other than the faulty Python API issue, forking 60 processes is quite expensive on Windows and takes 40sec on my Threadripper machine. This limits on 4 processes because any more than that actually makes things slower.

(Giving 4 processes causes no difference from not spawning extra process at all, but for now I'm not disabling parallel pool here.)

Comment 7

[Phabricator Automation]

Comment on attachment 9395201 [details]
Bug 1886167 - Limit to 4 processes on Windows r=peterv,sergesanspaille

Revision D206740 was moved to bug 1889842. Setting attachment 9395201 [details] to obsolete.