461395 - add support for PARALLEL_DIRS to build system, ||ize content/

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[(not currently active) Ted Mielczarek]

Attachment: add PARALLEL_DIRS support, use it in content

Currently, the directories in DIRS get built sequentially. If you have a lot of directories, like the content/ top-level directory, this means that even for a no-op build, you have to sequentially recurse into each one only to figure out that you have nothing to do. At least for content, (and probably for a lot of other places) we could in fact build these in parallel without any harm. This patch adds support for a PARALLEL_DIRS var to the build system. The directories in this var will be built in parallel before any directories in DIRS. The normal export/libs passes apply, so all PARALLEL_DIRS will be built export, then all DIRS, then all PARALLEL_DIRS will be built libs, then DIRS. On my dual-core Windows machine, a no-op -j2 build with this patch in content/ (no code changes, nothing to build) saves about 33% of the build time (19 seconds without the patch, 12 seconds with). It doesn't appear to win anything on a clobber build, since all the time is spent compiling, but I'm hoping that bug 428532 + this will help that. I'll post more numbers as I get them.

The approach of this patch does require us to manually opt-in directories into this approach, but I think that's the only way to do it without breaking things.

Comment 1

[(not currently active) Ted Mielczarek]

Here are some numbers for a stock build, build with this patch, and with this patch + bug 428532. This patch alone helps a no-op build pretty well, but doesn't do much for a clobber. However, this patch + the other patch helps quite a bit in the clobber case, and lets us go far more parallel. I was able to crank up to -j8 and still see wins, even on only a dual core system. I guess msys I/O is just that bad!

all times on Windows XP, Core 2 Duo, 2Gb ram
======================================

rebuild (nothing to do): (time make -C ../obj-firefox-opt-libxul/content/)

no patches:
-j1
real    0m19.297s
-j2 
real    0m19.078s
-j4
real    0m18.922s

with parallel dirs patch: (||izing almost everything under content/)
-j1
real    0m18.000s
-j2
real    0m12.047s
-j4
real    0m11.907s
-j8 (just for kicks)
real    0m11.562s

with parallel dirs patch + single compiler invocation:
-j1
real    0m18.781s
-j2
real    0m12.797s
-j4
real    0m12.187s
-j8
real    0m12.953s

======================================

full build (make -C ../obj-firefox-opt-libxul/content/ clean && time make -C ../obj-firefox-opt-libxul/content/)

no patches:
-j1
real    6m36.766s
-j2
real    4m6.859s
-j4
real    4m8.438s

with parallel dirs patch: (||izing almost everything under content/)
-j1
real    6m33.157s
-j2
real    3m56.000s
-j4
real    4m0.641s

with parallel dirs patch + single compiler invocation:
-j1
real    4m23.703s
-j2
real    3m40.438s
-j4
real    3m5.125s
-j8 (appears to be the sweet spot for my setup)
real    2m40.656s
-j10
real    2m44.812s

Comment 2

[(not currently active) Ted Mielczarek]

Linux results, not directly comparable because it's different hardware etc, but still interesting. Looks like this patch helps a bit, especially in the no-op case, but the bug 428532 patch either breaks even or performs worse than the normal case. I guess gcc doesn't do so well at compiling multiple files at once.

all times on Ubuntu 8.04 x86-64, Core 2 Quad, 4Gb ram
======================================

rebuild (nothing to do): (time make -C ../obj-firefox-opt-libxul/content/)

no patches:
-j1
real	0m2.706s
-j2 
real	0m2.664s
-j4
real	0m2.682s
-j8
real	0m2.740s
-j16
real	0m2.668s

with parallel dirs patch: (||izing almost everything under content/)
-j1
real	0m2.697s
-j2
real	0m1.531s
-j4
real	0m1.077s
-j8
real	0m0.986s
-j16
real	0m0.971s

with parallel dirs patch + single compiler invocation:
-j1
real	0m2.706s
-j2
real	0m1.528s
-j4
real	0m1.027s
-j8
real	0m0.970s
-j16
real	0m0.979s

======================================

full build (make -C ../obj-firefox-opt-libxul/content/ clean && time make -C ../obj-firefox-opt-libxul/content/)

no patches:
-j1
real	5m8.063s
-j2
real	2m45.116s
-j4
real	1m43.837s
-j8
real	1m44.881s

with parallel dirs patch: (||izing almost everything under content/)
-j1
real	5m10.261s
-j2
real	2m36.112s
-j4
real	1m28.544s
-j8
real	1m28.391s

with parallel dirs patch + single compiler invocation:
-j1
real	5m9.812s
-j2
real	2m37.949s
-j4
real	1m45.383s
-j8
real	1m39.458s
-j10
real	1m37.789s

Comment 3

[(not currently active) Ted Mielczarek]

Comment on attachment 344511 [details] [diff] [review]
add PARALLEL_DIRS support, use it in content

This patch is a net win on both Windows and Linux. We should get this in. The patch in bug 428532 needs some more work, but it's a separate issue.

I need to test this on OS X still, but I don't see it being wildly different than Linux.

Comment 4

[(not currently active) Ted Mielczarek]

Yeah, looks like OS X is comparable to Linux. This patch is still good, the other patch will need some work (and probably should only be enabled on Windows.)

all times on Macbook Pro, Core 2 Duo, 2Gb ram
======================================

rebuild (nothing to do): (time make -C ../obj-firefox-opt-libxul/content/)

no patches:
-j1
real	0m4.713s
-j2 
real	0m4.671s
-j4
real	0m4.644s

with parallel dirs patch: (||izing almost everything under content/)
-j1
real	0m4.765s
-j2
real	0m2.909s
-j4
real	0m2.781s
-j8 (just for kicks)
real	0m2.895s

with parallel dirs patch + single compiler invocation:
-j1
real	0m3.935s
-j2
real	0m2.381s
-j4
real	0m2.422s
-j8
real	0m2.436s

======================================

full build (make -C ../obj-firefox-opt-libxul/content/ clean && time make -C
../obj-firefox-opt-libxul/content/)

no patches:
-j1
real	6m22.019s
-j2
real	3m45.578s
-j4
real	3m45.685s

with parallel dirs patch: (||izing almost everything under content/)
-j1
real	6m29.834s
-j2
real	3m36.530s
-j4
real	3m37.213s

with parallel dirs patch + single compiler invocation:
-j1
real	6m28.274s
-j2
real	3m44.641s
-j4
real	3m49.319s

Comment 5

[(not currently active) Ted Mielczarek]

Pushed: http://hg.mozilla.org/mozilla-central/rev/80406c10f795