Closed Bug 448515 Opened 16 years ago Closed 14 years ago

Instill client.mk what profiledbuild means on comm-central

Categories

(MailNews Core :: Build Config, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
Thunderbird 3.1b2

People

(Reporter: Nomis101, Assigned: Nomis101)

References

Details

Attachments

(4 files, 1 obsolete file)

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_4; de-de) AppleWebKit/526.11 (KHTML, like Gecko) Version/3.1.2 Safari/525.20.1
Build Identifier: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; de; rv:1.9.0.2pre) Thunderbird/3.0a2pre

Profiledbuild doesn't work on comm-central (Thunderbird). If I try to use "make -f client.mk profiledbuild", the current response is "make: *** No rule to make target `profiledbuild'.  Stop."
It works as desired at mozilla-central (Firefox).

Reproducible: Always

Steps to Reproduce:
1. Do  "make -f client.mk profiledbuild" on comm-central
2.
3.
Actual Results:  
make: *** No rule to make target `profiledbuild'.  Stop.
Attached file my used patch, for testing only (obsolete) —
Now I've tried something to do Profiledbuilds on comm-central (for Shredder). And today I was a little bit successful. To do this I've done two things.

1. I copied the Profiledbuild related parts from /src/mozilla/client.mk into /src/client.mk
For testing purpose I created a patch that will do the same. So you can reproduce what I have done.

2. I changed in my mozconfig the line
mk_add_options PROFILE_GEN_SCRIPT='$(PYTHON) $(MOZ_OBJDIR)/_profile/pgo/profileserver.py'
into
mk_add_options PROFILE_GEN_SCRIPT='$(PYTHON) $(MOZ_OBJDIR)/mozilla/_profile/pgo/profileserver.py'


With this two modifications I was able to do PGO on comm-central.
BUT (I don't know why) the second pass (after the application launch, when -fprofile-use is used) was very quick, only two minutes. And the resulting application crash everytime if I quit them. And the filesize of the application (Shredder) is very big.
PGO on comm-central apps is not (yet) supported, deliberately.

I'd suggest anyone who would care to review this get an approval/comment from KaiRo first.
Ah, I didn't realize that profiledbuild was PGO - we have not looked into anything about that yet, the profile it generates and tries to optimize for is probably bogus. Because of that, this is not supported on any comm-central application yet.

If we ever enable PGO for those apps, we first need to decide what to optimize for and how to create a correct profile for that case.
Oh, sorry for the misunderstanding. Yes, I mean PGO. I didn't know that there is no correct profile for that case yet, because under CVS it was no problem to build Thunderbird with PGO. And so I didn't realize that this wasn't supported.
Than I will wait and hope PGO gets enabled in comm-central anytime, hopefully.
reso/invalid for now per c#3
Status: UNCONFIRMED → RESOLVED
Closed: 16 years ago
Resolution: --- → INVALID
I think you should put the targets back in, even if you're not using/officially supporting PGO. It's helpful to have community members like Nomis here trying out things like PGO for you so that some of the pitfalls can be known/addressed before you get to the point where you'd like to use it. In any case, it's a very tiny patch.

Just my 2 cents.
imo, if someone wanted to port the PGO code from m-c to c-c I'm not strictly opposed to it, but the initial thinking was "let PGO fail" and import (clean) PGO-related code from m-c as a straight patch to c-c when we are "ready" to support/implement it.

I'll have to try and get Kairo, Standard8 and the others together to look into this thought.
I still think that whatever your position about officially supporting it, the bug shouldn't be invalid. It's a valid request to have the build system support for PGO (that exists in mozilla-central, so you guys explicitly broke it in comm-central) available. We've had PGO support for gcc in the tree for ages and we weren't necessarily using it. I don't see any reason to not just take this patch, and even less reason why this bug should be invalid.
Given that I believe SM & TB developers aren't ready to look in detail at this yet, but someone else may wish to. I think we should reopen this bug and if someone wants to pick up the work to put it back in they are welcome to.

We have many build options that aren't "supported" as such (and we could always document it in configure).
Reopening, we have folks willing to work on this for TB.
Status: RESOLVED → UNCONFIRMED
Resolution: INVALID → ---
Status: UNCONFIRMED → RESOLVED
Closed: 16 years ago16 years ago
Resolution: --- → INVALID
I was able to successfully compile thunderbird using PGO.

I in the Makefile.in in the mozilla/modules/lcms/src directory I added the following line:

NO_PROFILE_GUIDED_OPTIMIZE = 1

before the MODULE_OPTIMIZE_FLAGS=-O2 line.

I did this because I received the following error:
LINK : fatal error LNK1000: Internal error during IMAGE::BuildImage

with this file:
f:\mozilla\thunderbird\mozilla\modules\lcms\src\cmscgats.c


I built Thunderbird with PGO with the following commands:

$(MAKE) -f $(TOPSRCDIR)/client.mk build MOZ_PROFILE_GENERATE=1
OBJDIR=${PGO_OBJDIR} /mozilla/dist/bin/thunderbird.exe ( I then did stuff and closed it )
$(MAKE) -f $(TOPSRCDIR)/client.mk build MOZ_PROFILE_USE=1

Of course this only uses the PGO already enabled in the core.
(In reply to comment #11)
> $(MAKE) -f $(TOPSRCDIR)/client.mk build MOZ_PROFILE_GENERATE=1
> OBJDIR=${PGO_OBJDIR} /mozilla/dist/bin/thunderbird.exe ( I then did stuff and
> closed it )
> $(MAKE) -f $(TOPSRCDIR)/client.mk build MOZ_PROFILE_USE=1

You did this on Windows, right? So I've tried to do it in a similar way on Mac OS X. First I've edited the Makefile.in in that way you described it. Than I've build Thunderbird with the commands:

$ make -f client.mk build MOZ_PROFILE_GENERATE=1
$ export NO_EM_RESTART=1
$ mkdir /temp/src/obj-i386-apple-darwin9.5.0/_profileprofile
$ /temp/src/obj-i386-apple-darwin9.5.0/mozilla/dist/Thunderbird.app/Contents/MacOS/thunderbird-bin -no-remote -profile /temp/src/obj-i386-apple-darwin9.5.0/_profileprofile
$ make -f client.mk build MOZ_PROFILE_USE=1

I've tried it with Apples GCC 4.0 and with Apples GCC 4.2. I don't no why, but the MOZ_PROFILE_USE step only takes 3 minutes. And if I use Apples GCC 4.0 I get a TB build that won't start, if I use Apples GCC 4.2 I get a TB build you can't quit after launching it. Maybe I do something wrong or it is a little bit tricky on Mac OS X.
Nomis: his instructions were on Windows. On Mac or Linux, after you build the first pass, and run the app for profiling, you'll need to "make clean", and then run the second build pass.
Status: RESOLVED → UNCONFIRMED
Resolution: INVALID → ---
Status: UNCONFIRMED → NEW
Ever confirmed: true
Thanks for your advice, Ted.
Now I build Thunderbird with the commands::

$ make -f client.mk build MOZ_PROFILE_GENERATE=1
$ export NO_EM_RESTART=1
$ mkdir /temp/src/obj-i386-apple-darwin9.5.0/_profileprofile
$ /temp/src/obj-i386-apple-darwin9.5.0/mozilla/dist/Thunderbird.app/Contents/MacOS/thunderbird-bin -no-remote -profile /temp/src/obj-i386-apple-darwin9.5.0/_profileprofile
$ make -f client.mk clean
$ make -f client.mk build MOZ_PROFILE_USE=1

This seems to work better for the second build pass now. But now I get after the MOZ_PROFILE_USE is running for a few minutes:

/temp/src/mozilla/xpcom/io/nsDirectoryService.cpp:942: warning: ‘PLstrcmp’ is deprecated (declared at /Developer/SDKs/MacOSX10.5.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/PLStringFuncs.h:78)
/temp/src/mozilla/xpcom/io/nsDirectoryService.cpp:942: warning: ‘PLstrcmp’ is deprecated (declared at /Developer/SDKs/MacOSX10.5.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/PLStringFuncs.h:78)
make[5]: *** [libs] Error 2
make[4]: *** [libs_tier_xpcom] Error 2
make[3]: *** [tier_xpcom] Error 2
make[2]: *** [default] Error 2
make[1]: *** [default] Error 2
make: *** [build] Error 2
> /temp/src/mozilla/xpcom/io/nsDirectoryService.cpp:942: warning: ‘PLstrcmp’ is
> deprecated (declared at
> /Developer/SDKs/MacOSX10.5.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/PLStringFuncs.h:78)
> /temp/src/mozilla/xpcom/io/nsDirectoryService.cpp:942: warning: ‘PLstrcmp’ is
> deprecated (declared at
> /Developer/SDKs/MacOSX10.5.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/PLStringFuncs.h:78)
> make[5]: *** [libs] Error 2
> make[4]: *** [libs_tier_xpcom] Error 2
> make[3]: *** [tier_xpcom] Error 2
> make[2]: *** [default] Error 2
> make[1]: *** [default] Error 2
> make: *** [build] Error 2

I just see warnings.  What's the error?  Can you paste the lines above this?
Attached file Error from Comment #14
(In reply to comment #15)
> I just see warnings.  What's the error?  Can you paste the lines above this?

Oh sorry, you are right. I closed the terminal, but I did it once again.
Aaaaah. I've got it! I've got it! I've tried it in the same way like in comment #14, but I also cleaned up my mozconfig a bit. And now I've got my first PGO'ed Thunderbird build. :) And the best thing is, this build doesn't crash, not on startup nor if you close the application.
Now I try to find out what in my mozconfig was the culprit for the error.
I did a few tests now and I found out:
• A few flags in enable-optimize="X" can produce the mentioned error in the second pass.
• I can use Apples GCC 4.0 instead of GCC 4.2 to avoid the error.
• Than I've tried what will happen If I allow PGO in the makefile of sqlite and I get an error similar to that in my attachment (#16). So I also added in mozilla/xpcom/io/Makefile.in a line "NO_PROFILE_GUIDED_OPTIMIZE = 1". And with this modification I can build PGO'ed Thunderbird on Mac OS X with every CFLAGs I want and with GCC 4.2 without any error.
• I also found out it's not necessary on Mac OS X to exclude mozilla/modules/lcms/src (Comment #11 ). I've forgotten this one time, but it builds nevertheless.
To get PGO to build for Thunderbird Nomis101 and I have been editing code from mozilla core source.

When building Firefox this source works fine with PGO enabled.  As part of my research I'm gonna look into what Thunderbird and Firefox do differently when compiling to see if instead of modifying the core, we can modify Thunderbird and get PGO enabled.

Also I'll try duplicating Nomis101's results in comment #18 on Windows.
FYI, yesterday I read at mozillazine about an similar error than mine while trying to build an PGO'ed Firefox. 
http://forums.mozillazine.org/viewtopic.php?p=4946455#p4946455
Chris: I wouldn't spend a lot of time on OS X, (no offense Nomis), as we're not PGOing Firefox builds there either. Getting Win32 working will be the easiest path to success.
I will be working on Windows.
On my Windows build of Thunderbird with PGO I removed the NO_PROFILE_GUIDED_OPTIMIZE = 1 from the mozilla/modules/lcms/src/Makefile.in and added it to the mozilla/xpcom/io/Makefile.in.

I received the same error as the attachment in comment #11.  I was unable to get Thunderbird to compile using PGO as Nomis was in Comment #18.  This was done on Windows.  Nomis acheived his build on Mac OS X.
Attached file Another OS X error
Maybe it is important. In some cases I get this error at the end of the second build pass. But at the moment I have no idea how I can suppress this.

And maybe it is important to mention that I had no problems with PGO on my old PPC-Mac. Now I have an Intel-Mac.
(In reply to comment #24)
> Created an attachment (id=347860) [details]
> Another OS X error
> 
> Maybe it is important. In some cases I get this error at the end of the second
> build pass.

Maybe this error has something to do with Bug 465339 ??
I found out, that some optimize options (like -funroll-loops, -ftree-loop-linear and -minline-all-stringops) can cause PGO to fail (on my Intel-Mac). But why and which kind of flags can cause this?
And is there a way to automatically close the launched application after the profile is generated? Then it wouldn't be necessary anymore to keep an eye on the build process.
I posted a Bug 465592 to start a conversation on how to patch Thunderbird so PGO doesn't fail on Thunderbird.

Currently I have to modify a file manually in the Mozilla core but I shouldn't have to.  I want to look into modifying the Thunderbird build process so it can build with PGO enabled.
Depends on: 465592
I have a tip for peoples trying to do a PGO build on Mac OS X. Update to the latest version of Apples GCC. Since I've updated to gcc version 4.2.1 "Apple Inc. build 5566" (included in Xcode 3.1.2 Developer Tools and iPhone_SDK_for_iphone_os_2.2_9m2621_final) I have no problems anymore with my PGO builds.
I've tested Thunderbird builds with the SunSpider JavaScript Test to compare the performance of a PGO'ed build with a non-PGO'ed build. Is seems PGO gives me 2.5 - 4.5% better performance. Furthermore, PGO reduces the application size about 2 - 4.5%. The benefit depends on the CFLAGS in the mozconfig.

Official 3.0b2 RC1:
Total:                 1211.0ms +/- 1.0%
Total:                 1204.4ms +/- 0.4%
Total:                 1203.0ms +/- 2.2%
(average 1206.1)

Own 3.1a1pre, without PGO (application size 30,8MB):
Total:                 1141.6ms +/- 1.4%
Total:                 1139.4ms +/- 0.9%
Total:                 1142.4ms +/- 0.8%
(average 1141.1)

Own 3.1a1pre, with PGO (application size 29,4MB):
Total:                 1117.2ms +/- 1.1%
Total:                 1111.2ms +/- 0.6%
Total:                 1110.2ms +/- 1.8%
(average 1112.9)



Is it possible to suppress the account wizzard at the step:
$ /temp/src/obj-i386-apple-darwin9.6.0/mozilla/dist/Thunderbird.app/Contents/MacOS/thunderbird-bin -no-remote -profile /temp/src/obj-i386-apple-darwin9.6.0/_profileprofile
?
Summary: Profiledbuild doesn't work on comm-central → Instill client.mk what profiledbuild means on comm-central
I build my own PGO'ed TB builds with this patch (Mac OSX). The above mentioned errors doesn't occurre anymore (why ever).

Who can I choose to review this?
Attachment #333100 - Attachment is obsolete: true
Comment on attachment 430602 [details] [diff] [review]
Copy the missing parts from m-c to c-c (updated to latest trunk)
[Checkin: Comment 34]

From a fast inspection, this looks OK though I'm not sure if we need more if we're doing real PGO (i.e. with usable profiles and with compilers that bring actual improvements, unlike gcc).
I'll let Callek do a real review, but consider this comment to be half the way to r+ as it look OK for me as a module owner. ;-)
Attachment #430602 - Flags: review?(bugspam.Callek)
Comment on attachment 430602 [details] [diff] [review]
Copy the missing parts from m-c to c-c (updated to latest trunk)
[Checkin: Comment 34]

This is good, sorry for the delayed review; is this checkin-needed or do you have access?
Attachment #430602 - Flags: review?(bugspam.Callek) → review+
Thanks, I don't have access, so I've added the checkin-needed keyword.
Keywords: checkin-needed
Comment on attachment 430602 [details] [diff] [review]
Copy the missing parts from m-c to c-c (updated to latest trunk)
[Checkin: Comment 34]


http://hg.mozilla.org/comm-central/rev/34948b9a2016
Attachment #430602 - Attachment description: Copy the missing parts from m-c to c-c (updated to latest trunk) → Copy the missing parts from m-c to c-c (updated to latest trunk) [Checkin: Comment 34]
Assignee: nobody → Nomis101
Status: NEW → RESOLVED
Closed: 16 years ago14 years ago
Flags: in-testsuite-
Keywords: checkin-needed
Product: Thunderbird → MailNews Core
QA Contact: build-config → build-config
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 3.1b2
Version: unspecified → Trunk
You need to log in before you can comment on or make changes to this bug.