1190607 - Illegal instruction during startup on pre-SSE2 Windows systems when builds haven't been PGO'ed

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Seth Fowler [:seth] [:s2h]]

Currently on pre-SSE2 Windows system, non-PGO'ed builds fail to start with a "Couldn't load XPCOM" message. Investigating with windbg reveals the following failure:

[bc8,88] LDR: Real INIT LIST for process C:\fftest\firefox\firefox.exe pid 3016 0xbc8
[bc8,88] LDR: odbcbcp.dll loaded - Calling init routine at 711A2652
[bc8,88] LDR: pdh.dll loaded - Calling init routine at 740014D6
[bc8,88] LDR: Recursive DLL load
[bc8,88]   Previous DLL being loaded: "C:\fftest\firefox\xul.dll"
[bc8,88]   DLL being requested: "rpcrt4.dll"
[bc8,88]   DLL whose initializer was currently running: "C:\WINDOWS\system32\pdh.dll"
LDR: LdrLoadDll, loading rpcrt4.dll from C:\fftest\firefox;C:\WINDOWS\system32;C:\WINDOWS\system;C:\WINDOWS;C:\Program Files\Debugging Tools for Windows (x86)\winext\arcade;C:\Python27\;C:\Python27\Scripts;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Documents and Settings\Administrator\Application Data\Python\Scripts
LDR: Refcount ADVAPI32.dll (cc)
LDR: Refcount RPCRT4.dll (41)
LDR: Refcount Secur32.dll (2e)
LDR: Refcount ADVAPI32.dll (cd)
[bc8,88] LDR: USERENV.dll loaded - Calling init routine at 769C15E4
[bc8,88] LDR: xul.dll loaded - Calling init routine at 029D11CD
LDR: LdrGetProcedureAddress by NAME - InitializeCriticalSectionEx
(bc8.88): Illegal instruction - code c000001d (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=00c02600 ebx=7c812fc9 ecx=0000001c edx=00b00044 esi=00c025e0 edi=00e7be80
eip=00e1fa5f esp=0012ee5c ebp=0012ee74 iopl=0         nv up ei pl nz na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010206
nss3!_PR_Getfd+0x9f:
00e1fa5f 660fd64010      movq    mmword ptr [eax+10h],xmm0 ds:0023:00c02610=5a5a5a5a5a5a5a5a
0:000> k
ChildEBP RetAddr  
0012ee5c 00e2062d nss3!_PR_Getfd+0x9f [c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\nsprpub\pr\src\io\prfdcach.c @ 99]
0012ee74 00e3284d nss3!_PR_InitIO+0x3d [c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\nsprpub\pr\src\io\prio.c @ 43]
0012eeb0 00e22791 nss3!_PR_InitStuff+0x10d [c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\nsprpub\pr\src\misc\prinit.c @ 203]
0012eeb8 01009c1d nss3!PR_NewLogModule+0x11 [c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\nsprpub\pr\src\io\prlog.c @ 352]
0012eec0 002acc79 xul!`dynamic initializer for 'gGetAddrInfoLog''+0xb [c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\netwerk\dns\getaddrinfo.cpp @ 26]
0012eed4 029d1186 MSVCR120!_initterm+0x29 [f:\dd\vctools\crt\crtw32\startup\crt0dat.c @ 954]
0012eef8 029d1259 xul!_CRT_INIT+0x1b0 [f:\dd\vctools\crt\crtw32\dllstuff\crtdll.c @ 295]
0012ef3c 029d11e9 xul!__DllMainCRTStartup+0x69 [f:\dd\vctools\crt\crtw32\dllstuff\crtdll.c @ 502]
0012ef50 7c90118a xul!_DllMainCRTStartup+0x1c [f:\dd\vctools\crt\crtw32\dllstuff\crtdll.c @ 472]
0012ef70 7c91c4da ntdll!LdrpCallInitRoutine+0x14
0012f078 7c916351 ntdll!LdrpRunInitializeRoutines+0x344
0012f324 7c9164b3 ntdll!LdrpLoadDll+0x3e5
0012f5cc 100024c3 ntdll!LdrLoadDll+0x230
0012f6b0 7c801bbd mozglue!`anonymous namespace'::patched_LdrLoadDll+0x42a [c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\mozglue\build\windowsdllblocklist.cpp @ 702]
0012f718 00402248 kernel32!LoadLibraryExW+0x18e
0012f93c 0040242d firefox!ReadDependentCB+0x57 [c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\xpcom\glue\standalone\nsxpcomglue.cpp @ 228]
0012fb64 00402630 firefox!XPCOMGlueLoad+0x15b [c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\xpcom\glue\standalone\nsxpcomglue.cpp @ 383]
0012fb70 00401756 firefox!XPCOMGlueStartup+0x29 [c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\xpcom\glue\standalone\nsxpcomglue.cpp @ 485]
0012fea4 004018f7 firefox!InitXPCOMGlue+0xe3 [c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\browser\app\nsbrowserapp.cpp @ 301]
0012ff44 004021d4 firefox!NS_internal_main+0x4f [c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\browser\app\nsbrowserapp.cpp @ 367]
0012ff78 00402cbd firefox!wmain+0x11e [c:\builds\moz2_slave\try-w32-0000000000000000000000\build\src\toolkit\xre\nswindowswmain.cpp @ 138]
0012ffc0 7c817067 firefox!__tmainCRTStartup+0xfe [f:\dd\vctools\crt\crtw32\startup\crt0.c @ 255]
0012fff0 00000000 kernel32!BaseProcessStart+0x23

Comment 1

[Seth Fowler [:seth] [:s2h]]

As I understand the problem, Visual Studio 2012 and above default to an -arch flag value that assumes SSE2 support. We need to manually add an -arch flag to ensure that code compatible with non-SSE2 machines is generated, but somewhere there's a bug in the build system and we're failing to do so for prfdcach.c (and NSPR generally, presumably). prfdcach.c has the following build command line, which doesn't contain -arch anywhere:

python2.7 c:/builds/moz2_slave/try-w32-0000000000000000000000/build/src/sccache/sccache.py cl -Foprfdcach.obj -c      -Oy- -W3 -nologo -GF -Gy -MD -O2 -Zi  -DlibVersionPoint=libVersionPoint -UDEBUG -U_DEBUG -UWINNT -DPACKAGE_NAME=\"\" -DPACKAGE_TARNAME=\"\" -DPACKAGE_VERSION=\"\" -DPACKAGE_STRING=\"\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DMOZILLA_CLIENT=1 -DNDEBUG=1 -DXP_PC=1 -DWIN32=1 -DWIN95=1 -D_PR_GLOBAL_THREADS_ONLY=1 -D_X86_=1 -DFORCE_PR_LOG -D_NSPR_BUILD_ -Ic:/builds/moz2_slave/try-w32-0000000000000000000000/build/src/obj-firefox/dist/include/nspr -Ic:/builds/moz2_slave/try-w32-0000000000000000000000/build/src/nsprpub/pr/include -Ic:/builds/moz2_slave/try-w32-0000000000000000000000/build/src/nsprpub/pr/include/private  "c:/builds/moz2_slave/try-w32-0000000000000000000000/build/src/nsprpub/pr/src/io/prfdcach.c"

Comment 2

[Seth Fowler [:seth] [:s2h]]

glandium, you're our best hope here. Any idea where the build system is going wrong?

Comment 3

[Mike Hommey [:glandium]]

Turns out it's bug 1160125.

Comment 4

[Seth Fowler [:seth] [:s2h]]

Ted, it looks like the fix from bug 1160125 never ended up in our tree because there's no NSPR release that contains the fix. Do you think we can get an NSPR update containing this fix so we can pull it into our tree?

Comment 5

[(not currently active) Ted Mielczarek]

(In reply to Seth Fowler [:seth] [:s2h] from comment #4)
> Ted, it looks like the fix from bug 1160125 never ended up in our tree
> because there's no NSPR release that contains the fix. Do you think we can
> get an NSPR update containing this fix so we can pull it into our tree?

I can easily tag NSPR trunk and we can import that into our tree. I've never done an actual NSPR release, and it looks like wtc is no longer going to be active in NSPR development, so if we need a release we'll need to sort that out.

Comment 6

[Seth Fowler [:seth] [:s2h]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #5)
> I can easily tag NSPR trunk and we can import that into our tree. I've never
> done an actual NSPR release, and it looks like wtc is no longer going to be
> active in NSPR development, so if we need a release we'll need to sort that
> out.

That sounds like a reasonable approach. Just looking for whatever the easiest path is to fixing the bug. =)

Comment 7

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

If we do need a new NSPR release for this bug, I already filed bug 1187049 for that because I'll need it for something else.

Comment 8

[Seth Fowler [:seth] [:s2h]]

It may be that we can resolve this now. We need to verify the fix, though.

Comment 9

[(not currently active) Ted Mielczarek]

I'm assuming this is fixed, since we've definitely updated NSPR since then, and also bug 1230117 landed a while ago which made us stop using NSPR's build system.