1641526 - run the libs tier in parallel

Status: NEW

(Firefox Build System :: General, enhancement)

Description

[Nathan Froyd [:froydnj]]

Got tired of seeing very non-parallel builds after compile was done. Not sure it's as simple as the patch I'm about to post, but maybe it can serve as a starting point.

Comment 1

[Nathan Froyd [:froydnj]]

Attachment: Bug 1641526 - run the libs tier in parallel; r=#build

Any concerns we have about bad actors messing with the libs tier are
pretty small nowadays; a quick skim through Makefile.in files with
libs rules indicates that libs is basically used for copying files
and doing Mac-specific application packaging, all of which can run in
parallel.

This change also solves the frustration in a local build of watching
libxul link, then the JS shell, then ipc/app, etc. etc. I suppose
it's possible that linking too many things all at once can cause
OOMs (especially if we're linking libxul and libxul-gtest
simultaneously), but I don't know if we have the tools to solve that in
the recursivemake backend.

Comment 2

[Mike Hommey [:glandium]]

Linkage doesn't happen in the libs tier anymore. It has been so for a long while. I'd rather move things out of the libs tier into a tier that runs in parallel (misc, I think, does), rather that force libs to be parallel and pay the (random) consequences.

Comment 3

[Nathan Froyd [:froydnj]]

(In reply to Mike Hommey [:glandium] from comment #2)

Linkage doesn't happen in the libs tier anymore. It has been so for a long while. I'd rather move things out of the libs tier into a tier that runs in parallel (misc, I think, does), rather that force libs to be parallel and pay the (random) consequences.

Do you think jar manifests can be moved into misc? That's probably 80%+ of the win of doing libs in parallel to start.

Comment 4

[Nathan Froyd [:froydnj]]

(Parallelizing libs cuts about 20s off a trivial mach build on my machine, so doing something here seems like a win.)

Comment 5

[Nathan Froyd [:froydnj]]

(In reply to Nathan Froyd [:froydnj] from comment #3)

(In reply to Mike Hommey [:glandium] from comment #2)

Linkage doesn't happen in the libs tier anymore. It has been so for a long while. I'd rather move things out of the libs tier into a tier that runs in parallel (misc, I think, does), rather that force libs to be parallel and pay the (random) consequences.

Do you think jar manifests can be moved into misc? That's probably 80%+ of the win of doing libs in parallel to start.

Oh, bother, we probably can't do this because of locale repacks, can we? Moving jar manifests into misc would entail a whole bunch of cascading changes there, wouldn't it?

Comment 6

[Mike Hommey [:glandium]]

(In reply to Nathan Froyd [:froydnj] from comment #5)

Oh, bother, we probably can't do this because of locale repacks, can we? Moving jar manifests into misc would entail a whole bunch of cascading changes there, wouldn't it?

That's bug 1240930, which now has a patch :)

There are a bunch of manual things happening in libs that I'm not too sure about parallelizing. It would be safer overall to move things out of libs than to change how libs works. With its current summary, I'd say this bug is WONTFIX, but we could repurpose it to track moving things out of libs.

Comment 7

[Mike Hommey [:glandium]]

FWIW, with bug 1240930, this is the kind of difference that can be seen on automation:

tier	before	after
misc	5.5s	6.9s
libs	23.7s	0.4s

I think we're good for now as far as libs is concerned.

Comment 8

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee didn't login in Bugzilla in the last 7 months.
:mhentges, could you have a look please?
For more information, please visit auto_nag documentation.