1624670 - Bootstrap fails with 32-core / 64-thread processor

Status: RESOLVED FIXED

(Firefox Build System :: Bootstrap Configuration, defect)

Description

[:kip (Kearwood Gilbert)]

Attachment: TTY output and callstack

Ran mach bootstrap on a fresh mozilla-unified clone and fresh mozillabuild install.
Windows 10 running on a Threadripper 2990wx (32-core 64-thread).

This failed with an error:

ValueError: max_workers must be <= 61

Attached is longer tty output and the callstack.

Comment 1

[:kip (Kearwood Gilbert)]

This appears to be Windows-specific, as mach bootstrap succeeds on a Xeon 7210 (64-core / 256-thread) system running Linux.

Comment 2

[:kip (Kearwood Gilbert)]

Looked a bit deeper...

It appears that the problem is rooted in the 61 worker limitation of the Python3 concurrent.futures library:

https://docs.python.org/3.8/library/concurrent.futures.html

This is implemented using Win32 WaitForMultipleObject calls, which are not able to support more than 63 objects in a single call.

Now that there are consumer oriented, single-socket processors with up to 64-cores / 128-threads, this function call is obsolete for the purpose of synchronizing per-cpu instantiated workers. This is better done with IOCP's (https://docs.microsoft.com/en-us/windows/win32/fileio/i-o-completion-ports). It seems that even the latest version of concurrent.futures has this 61-worker limitation.

Comment 3

[:kip (Kearwood Gilbert)]

I was able to complete a bootstrap by hard-coding max_workers = 61 in python/mozbuild/mozbuild/frontend/reader.py:

        # max_workers = cpu_count()
        max_workers = 61

Comment 4

[Nathan Froyd [:froydnj]]

Easiest fix is probably just clamping the max_workers on Windows only.

Comment 5

[Nathan Froyd [:froydnj]]

Attachment: Bug 1624670 - remove max_workers argument for ProcessPoolExecutor; r=#build

ProcessPoolExecutor will naturally default to the number of CPUs on
the machine and will also handle edge cases on Windows.

Comment 6

[(Away)]

I'd love to hear about what kind of build times you get with that machine. :)

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f6e96e32b5f0
remove max_workers argument for ProcessPoolExecutor; r=dmajor

Comment 8

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/f6e96e32b5f0

Comment 9

[Mike Hommey [:glandium]]

So, fun fact, this actually didn't fix it entirely... because somehow the capping doesn't work properly in python.
See https://bugs.python.org/issue26903#msg365886

Comment 10

[Mike Hommey [:glandium]]

Attachment: Bug 1624670 - Cap ProcessPoolExecutor's max_workers to 60 on Windows.

See https://bugs.python.org/issue26903#msg365886.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/be8356bf09d7
Cap ProcessPoolExecutor's max_workers to 60 on Windows. r=firefox-build-system-reviewers,rstewart

Comment 12

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/be8356bf09d7