Open Bug 420327 Opened 16 years ago Updated 5 years ago

Problems with PGO build of SeaMonkey v2.0

Categories

(SeaMonkey :: Build Config, defect)

Product:
SeaMonkey
Component:
Build Config
Platform:
x86
Windows 2000
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Milestone:
Future

People

(Reporter: swsnyder, Unassigned)

Details

(Keywords: perf) 

This is a catch-all bug to document problems building SeaMonkey 2.0 with Program-Guided Optimization (PGO).
Few functions profiled on Win32 executable.

Building with VS2005 from 27 Feb. trunk source I see suspiciously few functions being reported as having been profiled.  See below.  Only 4.68% coverage does not sound right given that I loaded up 12 web pages and manipilated the Bookmark and Preferences dialogs.

Disclaimer: I an doing a static build of SeaMonkey, i.e.:

  ac_add_options --enable-static --disable-shared
  ac_add_options --enable-extensions=default,-typeaheadfind

---------------------------------

link -NOLOGO -OUT:seamonkey.exe -PDB:seamonkey.pdb -SUBSYSTEM:WINDOWS -ENTRY:wmainCRTStartup -LTCG -NXCOMPAT  -DEBUG -OPT:REF -OPT:nowin98 -LTCG:PGUPDATE  -LTCG -OPT:REF,ICF,NOWIN98 /HEAP:0x40000 nsSuiteApp.obj nsStaticComponents.obj ./module.res ../../toolkit/xre/xulapp_s.lib -LIBPATH:../../staticlib ../../dist/lib/js3250.lib  ../../dist/lib/xpcom.lib ../../dist/lib/xpcom_core.lib ../../dist/lib/nspr4.lib ../../dist/lib/plc4.lib ../../dist/lib/plds4.lib    ../../staticlib/components/xppref32.lib ../../staticlib/components/uconv.lib ../../staticlib/components/ucvmath.lib ../../staticlib/components/i18n.lib ../../staticlib/components/necko.lib ../../staticlib/components/auth.lib ../../staticlib/components/xpc3250.lib ../../staticlib/components/chardet.lib ../../staticlib/components/zipwriter.lib ../../staticlib/components/mork.lib ../../staticlib/components/cookie.lib ../../staticlib/components/perms.lib ../../staticlib/components/strgcmps.lib ../../staticlib/components/rdf.lib ../../staticlib/components/caps.lib ../../staticlib/components/gkparser.lib ../../staticlib/components/gkgfxthebes.lib ../../staticlib/components/imgicon.lib ../../staticlib/components/imglib2.lib ../../staticlib/components/gkplugin.lib ../../staticlib/components/gkwidget.lib ../../staticlib/components/txmgr.lib ../../staticlib/components/composer.lib ../../staticlib/components/gklayout.lib ../../staticlib/components/docshell.lib ../../staticlib/components/embedcomponents.lib ../../staticlib/components/webbrwsr.lib ../../staticlib/components/appshell.lib ../../staticlib/components/xmlextras.lib ../../staticlib/components/universalchardet.lib ../../staticlib/components/oji.lib ../../staticlib/components/chrome.lib ../../staticlib/components/mozfind.lib ../../staticlib/components/intlapp.lib ../../staticlib/components/windowds.lib ../../staticlib/components/xpautoc.lib ../../staticlib/components/appcomps.lib ../../staticlib/components/tkautoc.lib ../../staticlib/components/satchel.lib ../../staticlib/components/cmdlines.lib ../../staticlib/components/tkitcmps.lib ../../staticlib/components/spellchk.lib ../../staticlib/components/pipboot.lib ../../staticlib/components/pipnss.lib ../../staticlib/components/pippki.lib ../../staticlib/components/autoconfig.lib ../../staticlib/components/wallet.lib ../../staticlib/components/wlltvwrs.lib ../../staticlib/mozreg_s.lib ../../staticlib/unicharutil_s.lib ../../staticlib/ucvutil_s.lib ../../staticlib/thebes.lib ../../staticlib/gfxshared_s.lib ../../staticlib/gkgfx.lib ../../staticlib/jsj3250.lib  ../../modules/libimg/png/png.lib ../../jpeg/jpeg3250.lib ../../modules/zlib/src/mozz.lib ../../dist/lib/mozlcms.lib  ../../dist/lib/crmf.lib ../../dist/lib/smime3.lib ../../dist/lib/ssl3.lib ../../dist/lib/nss3.lib ../../dist/lib/nssutil3.lib ../../dist/lib/softokn3.lib  ../../gfx/cairo/cairo/src/mozcairo.lib ../../gfx/cairo/libpixman/src/mozlibpixman.lib ../../dist/lib/sqlite3.lib comctl32.lib comdlg32.lib uuid.lib shell32.lib ole32.lib oleaut32.lib version.lib winspool.lib imm32.lib usp10.lib kernel32.lib user32.lib gdi32.lib winmm.lib wsock32.lib advapi32.lib comctl32.lib comdlg32.lib uuid.lib shell32.lib ole32.lib oleaut32.lib version.lib winspool.lib msimg32.lib
PGOMGR : warning PG0188: No .PGC files matching 'seamonkey!*.pgc' were found.
   Creating library seamonkey.lib and object seamonkey.exp
Generating code
3367 of 72020 (  4.68%) profiled functions will be compiled for speed

All final links complain about lack of VS2005 *.pgc file.

For every executable file generated I get a complaint that the *.pgc file wasn't found (see comment above and text below).  Is this a problem with the build or just poor phrasing on microsoft's part?

----------------------

link -NOLOGO -DLL -OUT:js3250.dll -PDB:js3250.pdb -SUBSYSTEM:WINDOWS  jsapi.obj jsarena.obj jsarray.obj jsatom.obj jsbool.obj jscntxt.obj jsdate.obj jsdbgapi.obj jsdhash.obj jsdtoa.obj jsemit.obj jsexn.obj jsfun.obj jsgc.obj jshash.obj jsinterp.obj jsiter.obj jslock.obj jslog2.obj jslong.obj jsmath.obj jsnum.obj jsobj.obj jsopcode.obj jsparse.obj jsprf.obj jsregexp.obj jsscan.obj jsscope.obj jsscript.obj jsstr.obj jsutil.obj jsxdrapi.obj jsxml.obj prmjtime.obj    js3240.res -LTCG -NXCOMPAT  -DEBUG -OPT:REF -OPT:nowin98 -LTCG:PGUPDATE  -LTCG -OPT:REF,ICF,NOWIN98 -OPT:NOICF  ../../dist/lib/nspr4.lib ../../dist/lib/plc4.lib ../../dist/lib/plds4.lib  kernel32.lib user32.lib gdi32.lib winmm.lib wsock32.lib advapi32.lib
PGOMGR : warning PG0188: No .PGC files matching 'js3250!*.pgc' were found.
   Creating library js3250.lib and object js3250.exp
Generating code
1859 of 1859 (100.00%) profiled functions will be compiled for speed
1859 of 1859 functions (100.0%) were optimized using profile data
2204372249 of 2204372249 instructions (100.0%) were optimized using profile data

re comment 1: we see that while building Firefox, it only says something like 3% compiled for speed. I don't know that that means "had profiling coverage", but I can't find any documentation to prove or disprove that. Compiling spidermonkey generally shows 100% there.

re comment 2: that's an artifact of our build process, don't worry about it. There's a Python script that gets called during the export pass--pgomerge.py--that merges the .pgc files into the .pgd file and then deletes them, since they wind up in dist/bin instead of next to the .pgd file where the linker expects them. The linker is stupid and doesn't realize that the .pgd file already has this info merged in, but it uses it anyway.

Second link (after profiling) fails to complete.  It stops with this error:

c:\mozbuild\obj-seamonkey\dist\include\string\nstsubstring.h(294) : error C2220: warning treated as error - no 'executable' file generated

Yeah, I need to file a followup, we added some extra CFLAGS/CXXFLAGS to get around this:
http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/tools/tinderbox-configs/firefox/win32/mozconfig
export CFLAGS="-GL -wd4624 -wd4952"
export CXXFLAGS="-GL -wd4624 -wd4952"

We need to get these into configure for the time being. Can you file that for me?

Full log of attempted PGO build of Win32/SeaMonkey v2.0a1pre:

    http://developer.mozilla.org/wiki-images/en/7/7c/pgo-sm-build-log.zip

This log documents more fully comments #1, #2, and #4 above.

The ZIPped log is 417KB in size; 8573KB when extracted.

FYI.

We hit all of those in bug 418865, and worked around them (or they weren't important). Nothing new here yet.

I rebuilt (same 27 Feb source) with debug symbols.  When SM crashes on start-up the call stack in VS2005 shows the actual crash in NTDLL.DLL's realloc().

The last SM function to be shown is sqlite3_mutex_enter().

I've read that sqlite has been a problem in profiled Firefox builds, so this likely won't come as a surprise.

We're explicitly disabling PGO in sqlite though, so that's puzzling:
http://lxr.mozilla.org/mozilla/source/db/sqlite3/src/Makefile.in#69

re: c#0 leaving this bug open.

But SeaMonkey does not currently support PGO builds, so problems are likely.
Just noting that PGO is still not enabled for SM 2.0a3 (19 Feb 2009).
With the new static builds working so well it sure would be nice to get PGO working for beta1 (hint, hint).

With the back-end of Firefox v3.5 and SeaMonkey v2.0 now in sync, it shouldn't be that hard to get PGO working on SM.

$ make -C mozilla/ -f client.mk profiledbuild
make: Entering directory `/c/mozbuild/mozilla'
make: *** No rule to make target `profiledbuild'.  Stop.
make: Leaving directory `/c/mozbuild/mozilla'
QA Contact: build-config → build-config
Target Milestone: --- → Future
Given that the 2nd (and final) beta of SM has been tagged, I guess PGO builds won't be supported for v2.0?
Building with PGO could be supported (not sure, I currently don't care much), but we will not do PGO builds by default, as we simply lack the build machine power to make every Windows build take twice as long.
I tried to build the SeaMonkey 2.3.1 with PGO for Windows.
The result is below:

32bit(x86) : success. see http://htguard.island.ac/x86/
64bit(x64) : failure. link.exe is hung up, xul.dll can't be linked.

Environment:
Compiler: Microsoft Visual Studio 2010(SP1)
SDK: Windows SDK 7.1
DirectX: Direct X SDX(June 2010)

Error message:(Sorry, this message contains Japanese.)
comm-release\mozilla\security\manager\ssl\src\
nsPKCS11Slot.cpp : fatal error C1001: コンパイラで内部エラーが発生しました。
(コンパイラ ファイル 'f:\dd\vctools\compiler\utc\src\p2\main.c[0x000007FEE727688
2:0x0000000000166056]'、行 183)
この問題を回避するには、上記の場所付近のプログラムを単純化するか変更してくださ
い。
詳細については、Visual C++ ヘルプ メニューのサポート情報コマンドを
選択してください。またはサポート情報 ヘルプ ファイルを参照してください。

LINK : fatal error LNK1000: Internal error during IMAGE::BuildImage

  Version 10.00.40219.01

  ExceptionCode            = C0000005
  ExceptionFlags           = 00000000
  ExceptionAddress         = 000007FEE7276882 (000007FEE71D0000) "c:\Program Fil
es (x86)\Microsoft Visual Studio 10.0\VC\BIN\amd64\c2.dll"
  NumberParameters         = 00000002
  ExceptionInformation[ 0] = 0000000000000000
  ExceptionInformation[ 1] = 0000000000166056

CONTEXT:
  Rax    = 0000000000166056  R8     = 0000000000010000
  Rbx    = 000000010018B3C8  R9     = 0000000000000002
  Rcx    = 0000000000000000  R10    = 0000000000000002
  Rdx    = 0000000000000000  R11    = 00000000A03C7CE8
  Rsp    = 00000000002BD500  R12    = 0000000086474CE0
  Rbp    = 00000000002BD600  E13    = 000007FEE71D0000
  Rsi    = 00000000A03C7CE8  R14    = 0000000000000000
  Rdi    = 00000000A05333F0  R15    = 0000000000000004
  Rip    = 000007FEE7276882  EFlags = 0000000000010202
  SegCs  = 0000000000000033  SegDs  = 000000000000002B
  SegSs  = 000000000000002B  SegEs  = 000000000000002B
  SegFs  = 0000000000000053  SegGs  = 000000000000002B
  Dr0    = 0000000000000000  Dr3    = 0000000000000000
  Dr1    = 0000000000000000  Dr6    = 0000000000000000
  Dr2    = 0000000000000000  Dr7    = 0000000000000000
make[5]: *** [xul.dll] Error 232
make[5]: *** Deleting file `xul.dll'
make[5]: Leaving directory `/obj/mozilla/toolki
t/library'
make[4]: *** [libs_tier_platform] Error 2
make[4]: Leaving directory `/obj/mozilla'
make[3]: *** [tier_platform] Error 2
make[3]: Leaving directory `/obj/mozilla'
make[2]: *** [default] Error 2
make[2]: Leaving directory `/obj/mozilla'
make[1]: *** [default] Error 2
make[1]: Leaving directory `/obj'
make: *** [build] Error 2
(In reply to Atsushi Okawa from comment #15)
> I tried to build the SeaMonkey 2.3.1 with PGO for Windows.
> The result is below:
> 
> 32bit(x86) : success. see http://htguard.island.ac/x86/
> 64bit(x64) : failure. link.exe is hung up, xul.dll can't be linked.

Yes, that's a common problem with contemporary Mozilla code.

Using VS2008 I too can successfully do 32-bit PGO builds of Firefox and Thunderbird, and successfully do non-PGO 64-bit builds of FF & TB, but 64-bit PGO builds of both applications fail with similar compiler faults.
That's an internal compiler error, FWIW. The only thing you can do is try to make a reproducible testcase and submit it to Microsoft at connect.microsoft.com.
See: http://support.microsoft.com/kb/974229
(In reply to Ted Mielczarek [:ted, :luser] from comment #17)
> That's an internal compiler error, FWIW. The only thing you can do is try to
> make a reproducible testcase and submit it to Microsoft at
> connect.microsoft.com.
> See: http://support.microsoft.com/kb/974229

I didn't mean to imply that it was Mozilla's code that was the source of the problem.  Rather, it is that Mozilla's code is big and complex enough to expose bugs in Microsoft's 64-bit compiler/linker.

Microsoft has zero interest now in addressing VS2008 bugs.  (And, in mid-2011, likely VS2010 bugs as well.)  Their standard answer seems to be to invite the user to spend another $800 for a newer version of Visual Studio.
Hm, doesn't Microsoft offer for free all the command-line tools necessary to compile Mozilla applications? See https://developer.mozilla.org/En/Developer_Guide/Build_Instructions/Windows_Prerequisites and among others the link to "Visual C++ 2008 Express Edition with SP1" (under "Microsoft Visual C++ (MSVC)", below the pink box), where the first sentence says "The Microsoft Visual Studio 2008 Express Editions with SP1 are a free set of tools that are simple, fun and easy to learn." Of course the sentence just quoted is Microsoft propaganda, but advertising something as "free" and then requiring a $800 payment for it would be grossly "false or misleading advertising", wouldn't it?
PGO is not included in the express edition.
(In reply to Ted Mielczarek [:ted, :luser] from comment #20)
> PGO is not included in the express edition.

Not even if you add the Windows SDK? (see also https://developer.mozilla.org/En/Windows_SDK_versions which says to add the Windows 7 SDK except with the non-Express version of Visual C++ 2010)
(In reply to Tony Mechelynck [:tonymec] from comment #21)
> (In reply to Ted Mielczarek [:ted, :luser] from comment #20)
> > PGO is not included in the express edition.
> 
> Not even if you add the Windows SDK? (see also
> https://developer.mozilla.org/En/Windows_SDK_versions which says to add the
> Windows 7 SDK except with the non-Express version of Visual C++ 2010)

I have ever misunderstood about the compiler which Win7 SDK contains, too.

PGO is not included in that compiler.

If someone want to compiler a Windows program with PGO, he(she) must use non-Express version of Visual C++ or Intel C++ Compiler.
I tried to build the SeaMonkey 2.8 with PGO for Windows.
The result is below:

32bit(x86) : failure, pgocvt internal error, xul.dll can be linked, seamonkey binary will crash.

Environment:
Compiler: Microsoft Visual Studio 2010(SP1)
SDK: Windows SDK 7.1
DirectX: Direct X SDX(June 2010)

Error message:(Sorry, this message contains Chinese.)

t:\comm\mozilla\config\rules.mk:1087:0$ d:/moz-build/python25/python2.5.exe t:/comm/mozilla/config/pythonpath.py -I../../config t:/comm/mozilla/config/expandlibs_exec.py --uselist -- link -NOLOGO -DLL -OUT:xul.dll -PDB:xul.pdb -SUBSYSTEM:WINDOWS dlldeps-xul.obj nsStaticXULComponents.obj nsDllMain.obj nsGFXDeps.obj dlldeps-zlib.obj nsUnicharUtils.obj nsBidiUtils.obj nsRDFResource.obj ./module.res -LARGEADDRESSAWARE -NXCOMPAT -DYNAMICBASE -SAFESEH -DEBUG -DEBUGTYPE:CV -DEBUG -OPT:REF -LTCG:PGINSTRUMENT -LIBPATH:../../dist/lib -NODEFAULTLIB:msvcrt -NODEFAULTLIB:msvcrtd -NODEFAULTLIB:msvcprt -NODEFAULTLIB:msvcprtd -DEFAULTLIB:mozcrt ../../toolkit/xre/xulapp_s.lib ../../staticlib/components/necko.lib ../../staticlib/component 正在建立程式庫 xul.lib 和物件 xul.exp
正在產生程式碼
t:\comm\mozilla\gfx\cairo\libpixman\src\pixman-mmx.c(166) : warning C4799: 函式 'to_uint64' 沒有 EMMS 指令
t:\comm\mozilla\gfx\cairo\libpixman\src\pixman-mmx.c(317) : warning C4799: 函式 'store8888' 沒有 EMMS 指令
t:\comm\mozilla\gfx\cairo\libpixman\src\pixman-mmx.c(437) : warning C4799: 函式 'combine' 沒有 EMMS 指令
已完成程式碼產生
PGOCVT : 嚴重錯誤 PG0001: 在原始程式檔 'f:\dd\vctools\compiler\utc\src\tools\pogo\cvtpgd\cvtpgd.cpp'中的第 800 行偵測到未預期的內部錯誤。
PGOCVT : 嚴重錯誤 PG0001: 在原始程式檔 'f:\dd\vctools\compiler\utc\src\tools\pogo\cvtpgd\cvtpgd.cpp'中的第 858 行偵測到未預期的內部錯誤。
You need to log in before you can comment on or make changes to this bug.