535897 - Building libxul builds on Mac takes forever

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Boris Zbarsky [:bzbarsky]]

I just compared the time it takes to make in layout/build (just layout/build, nothing else) after touching a single file in layout/base and making in layout/base in libxul and non-libxul debug builds on mac.  The non-libxul one takes about a minute over here.  The libxul one takes 8 minutes.

General time breakdown seems to be:

* rm of libgklayout.a is fast.
* rm of all the .o files takes about 4 seconds.
* ar x of every single .a around 1.5 minutes.
* ar cr to create libgklayout.a takes about 4 minutes.
* ranlib on libgklayout.a takes about 2.5 minutes.
* rm of all the .o files again takes about 6 seconds.

I just checked, and unfortunately doing:

  ar x ../base/libgkbase_s.a 
  ar d libgklayout.a *.o

takes about 2.5 minutes for the second step.  :(  It seems to run ranlib on the archive in the process of removing the files, for one thing.  Then doing:

  ar q libgklayout.a *.o

takes a long time too.

Tossing in the S option, however, makes things faster (maybe one minute for the 'd', almost immediate 'q').  Is it worth possibly looking into using the combination of dS, qS, ranlib to speed this up, maybe?  Or something to make  this sane?  8-minute layout link times, followed by libxul link, are just _painful_.

Comment 1

[(not currently active) Ted Mielczarek]

Finishing bug 522770 would probably make this better. Instead of producing static libs, it just produces text files with the names of the object files in them, then passes them to the linker with @filename, which both VC++ and GCC read the contents of and treat as if they were passed on the command line.

Comment 2

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

How about we just switch to "tar cf" and "tar x" instead of ar?

Comment 3

[(not currently active) Ted Mielczarek]

It's a little tricky because we do produce a few static libs in the tree that we actually want to ship as static libs in stuff like the SDK. I have a working patch in bug 522770, it's just stuck in bsmedberg review limbo. (That patch doesn't stop us from making real static libs, but we could do that as a pretty easy followup.)

Comment 4

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Yeah, not making real static libs would be a big io win.  But your patch in bug 522770 has no review requests on it?

Comment 5

[(not currently active) Ted Mielczarek]

Oh, bah!

Comment 6

[(not currently active) Ted Mielczarek]

bug 584474 is fixed on the build-system branch, that should have resolved these issues. If you feel like cloning yet another branch to try it out, it's here:
http://hg.mozilla.org/projects/build-system

Comment 7

[Boris Zbarsky [:bzbarsky]]

I tried that branch.  Takes the incremental build time (for a manual build in the relevant dir, layout/build, and toolkit/library) for a minor change to content/layout code from about 130 seconds to about 10 seconds on my hardware.  I can live with that!

Comment 8

[Boris Zbarsky [:bzbarsky]]

That was on Mac, of course.

Comment 9

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Woot!  Nice job Mike!

Comment 10

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Fixed by Bug 584474.