419344 - enable profile-guided optimization on mac

Status: RESOLVED WORKSFORME

(Firefox Build System :: General, enhancement, P3)

Description

[(not currently active) Ted Mielczarek]

Promised joduinn I would file this, and forgot.  This is blocked on a few things, I don't have bug numbers handy, but I know it at least needs XCode 3.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Tweaking summary, as bug 417045 will give us new machines rather than upgrading existing ones.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

People have been sounding much less keen on this. Should we close this out ?

Comment 3

[Jesse Ruderman]

Eh?  Who is less keen on turning on PGO and why?

Comment 4

[Simon Howes]

Nobody is less keen.  I would like it to be a blocker for Firefox 3.  

Comment 5

[(not currently active) Ted Mielczarek]

I'm less keen. Apple's gcc kind of sucks, so building with profiling currently hangs. We'd have to narrow down a list of modules to enable it on, and I haven't had time to do that yet. We're running out of time in the beta cycle to do this, so unless someone else wants to figure it out, it's probably going to fall by the wayside.

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Ted: Do you have any idea how many modules hang with the new gcc? 

RobSayer: from your comments in bug#417045, it seems you got this to work. Can you help? 

Note: we are upgrading macs from 10.4 to 10.5 *because* of this request to compile with PGO. If we are not going to compile with PGO on mac, I'd like to revisit the 10.4->10.5 upgrades work being done in bug#417045. One way or another, we need to decide quickly.

Note: There's also been reliability problems with Talos machines working fine on 10.4, but failing intermittently on 10.5. Dont know details, but Alice would know more.

Comment 7

[Robert Sayre]

> Rob: from your comments in bug#417045, it seems you got this to work. Can
> you help? 

I got it to work for SpiderMonkey only. 

> Note: we are upgrading macs from 10.4 to 10.5 *because* of this request to compile with PGO

It also fixed compliance bugs and seemed to generate faster code based on my testing, so it is worth doing even if we don't use PGO.

Comment 8

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #7)
> > Rob: from your comments in bug#417045, it seems you got this to work. Can
> > you help? 
> 
> I got it to work for SpiderMonkey only. 

Ah, interesting. Has *anyone* ever got compile with PGO on mac to work? Or is Ted "boldly going where noone has gone before"? 


> > Note: we are upgrading macs from 10.4 to 10.5 *because* of this request to compile with PGO
> It also fixed compliance bugs and seemed to generate faster code based on my
> testing, so it is worth doing even if we don't use PGO.
ok, good to know. Any ballpark of how much faster the code is with 10.5 yet without PGO?

Comment 9

[Simon Howes]

Some have said at Mozillazine they have got PGO on PPC Mac's to compile.

http://forums.mozillazine.org/viewtopic.php?t=630298&postdays=0&postorder=asc&postsperpage=15&start=30

I have also seen improvements in using GCC 4.2 from Apple.  I can't get the current trunk to compile on 4.2 now though, so can't come up with some accurate results.

Comment 10

[(not currently active) Ted Mielczarek]

(In reply to comment #6)
> Ted: Do you have any idea how many modules hang with the new gcc? 

No, because that would require me to enable it on a case-by-case basis, which I haven't done yet.

(In reply to comment #8)
> Ah, interesting. Has *anyone* ever got compile with PGO on mac to work? Or is
> Ted "boldly going where noone has gone before"? 

AFAIK, nobody has done a full PGO build on Mac yet. Rob did builds with a PGO'ed Spidermonkey, but that's it.

Comment 11

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Summary of quick call with damon, rsayer, and ted:

1) We're should enable PGO compiling for SpiderMonday *only*. We're going to compile all other modules without PGO.

2) We're staying with the gcc compiler on mac until we figure out bug#403224. 

3) bm-xserve16 is up and running with 10.5 on it. nthomas and ted are using it right now.

Comment 12

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

fyi: bm-xserve16 is being recycled and used for moz2 builds. Once we solve the dep.bug, we can come back and try again.

Comment 13

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 14

[Chris AtLee [:catlee]]

Please get this working somewhere before we automate it.

Comment 15

[Sam N]

Using clang 3.2 seems to correctly profile the application and put the GCDA's in the proper folders. 

Using 3.1 I did a coverage build, created a list of the GCNO locations, built firefox with profiling, and moved the incorrectly placed GCDA's in their correct place using the names/locations of the GCNO files.

Comment 16

[Dave-C]

Hi, I could not get a PGO build to work on OS X Yosemite, Instead I get the following error:

make[1]: Leaving directory '/Volumes/hd2/src/mozilla-release'
make -C /Volumes/hd2/src/mozilla-release/build_directory package MOZ_PGO_INSTRUMENTED=1 MOZ_INTERNAL_SIGNING_FORMAT= MOZ_EXTERNAL_SIGNING_FORMAT=
make[1]: Entering directory '/Volumes/hd2/src/mozilla-release/build_directory'
make[1]: *** No rule to make target 'package'.  Stop.
make[1]: Leaving directory '/Volumes/hd2/src/mozilla-release/build_directory'
client.mk:233: recipe for target 'profiledbuild' failed
make: *** [profiledbuild] Error 2
 
.mozconfig:
# Import the stock config for building the browser (Firefox)
. $topsrcdir/browser/config/mozconfig
. $topsrcdir/build/macosx/mozconfig.common
. $topsrcdir/build/macosx/universal/mozconfig



#mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/build_directory
mk_add_options MOZ_OBJDIR=/Volumes/hd2/src/mozilla-release/build_directory/
mk_add_options MOZ_MAKE_FLAGS="-j5 "
mk_add_options AUTOCONF=autoconf213


ac_add_options --enable-application=browser
ac_add_options --disable-debug
ac_add_options --disable-crashreporter
ac_add_options --disable-debug-symbols

ac_add_options --with-macos-sdk=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk
ac_add_options --with-unify-dist=../i386/dist

ac_add_options --enable-install-strip
ac_add_options --enable-strip
ac_add_options --enable-xterm-updates

ac_add_options --enable-llvm-hacks
ac_add_options --disable-jemalloc
ac_add_options --enable-static
ac_add_options --enable-static-libs
ac_add_options --disable-stdcxx-compat
ac_add_options --enable-media-plugins
ac_add_options --with-google-oauth-api-keyfile=/Users/dave/scripts/firefox_google_OAuth
mk_add_options PROFILE_GEN_SCRIPT='$(PYTHON) $(MOZ_OBJDIR)/_profile/pgo/profileserver.py'
mk_add_options PROFILE_GEN_SCRIPT_PAUSE='$(PYTHON) $(MOZ_OBJDIR)/_profile/pgo/stopmsg.py'
ac_add_options --enable-profile-guided-optimization
ac_add_options --disable-tests

Comment 17

[(not currently active) Ted Mielczarek]

You're doing a universal build, and the build system doesn't know how to do a universal PGO build.

Comment 18

[BugBot [:suhaib / :marco/ :calixte]]

The severity field for this bug is relatively low, S3. However, the bug has 11 votes.
:ahochheiden, could you consider increasing the bug severity?

For more information, please visit auto_nag documentation.

Comment 19

[BugBot (nomail) [:suhaib / :marco/ :calixte]]

The last needinfo from me was triggered in error by recent activity on the bug. I'm clearing the needinfo since this is a very old bug and I don't know if it's still relevant.