923651 - Increase default -j to more than # of cores

Status: RESOLVED INACTIVE

(Firefox Build System :: General, defect, P3)

Description

[Gregory Szorc [:gps]]

With the |binaries| target, I've noticed that even during hardcore C++ compile mode that my machine isn't maxing out at 100% like I would want. It gets close - high 90's - but there's still a few percentage points on the table.

I think we should increase the default -j to #cores + 1 or #cores + 2.

I need to measure this, but I suspect +1 will be sufficient to get us closer to 100% while not incurring too much context switching.

Comment 1

[Mike Hommey [:glandium]]

I've always used #cores * 1.5

Comment 2

[Gregory Szorc [:gps]]

Attachment: Default to building with #cores + 2

I'd rather not use a core # multiplier because I don't believe it will
result in much gains (we're only a few percent away from 100% during
compile) and will likely just increase context switching and system
load.

If we had anonymous reporting of build system resource usage back to
Mozilla, would could experiment with values and choose a scientifically
proven value :)

Comment 3

[Mike Hommey [:glandium]]

Comment on attachment 817382 [details] [diff] [review]
Default to building with #cores + 2

Review of attachment 817382 [details] [diff] [review]:
-----------------------------------------------------------------

::: python/mozbuild/mozbuild/base.py
@@ +407,5 @@
>          if allow_parallel:
>              if num_jobs > 0:
>                  args.append('-j%d' % num_jobs)
>              else:
> +                args.append('-j%d' % (multiprocessing.cpu_count() + 2))

#cores + 2 is huge when #cores < 3.