1099339 - Firefox Nightly 32-bit crashes on startup on Windows XP 64

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Syd Polk :sydpolk]

Running Firefox Nightly on our Windows XP 64-bit VM (fully updated), it crashes. Here is the Crash Report details. Note that this is on first launch creating a new profile.

AdapterDeviceID: 0x0000
AdapterSubsysID: 00000000
AdapterVendorID: 0x0000
Add-ons: %7B972ce4c6-7e08-4474-a285-3208198ce6fd%7D:36.0a1
AvailablePageFile: 3707445248
AvailablePhysicalMemory: 1570312192
AvailableVirtualMemory: 3941904384
BIOS_Manufacturer: Phoenix Technologies LTD
BlockedDllList: 
BreakpadReserveAddress: 80674816
BreakpadReserveSize: 41943040
BuildID: 20141114030206
CrashTime: 1415996959
EMCheckCompatibility: true
FramePoisonBase: 00000000f0de0000
FramePoisonSize: 65536
InstallTime: 1415978658
Notes: AdapterVendorID: 0x0000, AdapterDeviceID: 0x0000, AdapterSubsysID: 00000000, AdapterDriverVersion: 
D3D11 Layers? D3D11 Layers- D3D9 Layers? D3D9 Layers- 
ProductID: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
ProductName: Firefox
ReleaseChannel: nightly
SecondsSinceLastCrash: 29
StartupTime: 1415996952
SystemMemoryUsePercentage: 26
Theme: classic/1.0
Throttleable: 1
TotalPageFile: 4156772352
TotalPhysicalMemory: 2146787328
TotalVirtualMemory: 4294836224
URL: https://www.mozilla.org/en-US/firefox/nightly/firstrun/?oldversion=33.0.2
Vendor: Mozilla
Version: 36.0a1
Winsock_LSP: MSAFD Tcpip [TCP/IP] : 2 : 1 :  
 MSAFD Tcpip [UDP/IP] : 2 : 2 : %SystemRoot%\system32\mswsock.dll 
 MSAFD Tcpip [RAW/IP] : 2 : 3 :  
 RSVP UDP Service Provider : 6 : 2 : %SystemRoot%\system32\mswsock.dll 
 RSVP TCP Service Provider : 6 : 1 :  
 VMCI sockets DGRAM : 0 : 2 : %windir%\system32\vsocklib.dll 
 VMCI sockets STREAM : 0 : 1 : 
useragent_locale: en-US

This report also contains technical information about the state of the application when it crashed.

Comment 1

[Syd Polk :sydpolk]

Exposed by platform WebRTC testing.

Comment 2

[Syd Polk :sydpolk]

The first build that shows this behavior is this:

http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-11-14-03-02-06-mozilla-central/firefox-36.0a1.en-US.win32.installer.exe

The 11/13 build launches fine.

Comment 3

[Syd Polk :sydpolk]

The crash report is at:

https://crash-stats.mozilla.com/report/index/b0a86aaa-2aa8-4f12-af87-97aaf2141114

Comment 4

[Syd Polk :sydpolk]

Luke, this is crashing in your patch. Do you have any light to shed on this?

#if defined(XP_WIN)
// On Windows, we can simply suspend the main thread and work directly on
// its context from this thread.
    HANDLE thread = (HANDLE)rt->ownerThreadNative();
    if (SuspendThread(thread) == -1)
        MOZ_CRASH("Failed to suspend main thread");

Comment 5

[Luke Wagner [:luke]]

Hmm, I wonder if this is a WOW64 problem, in which case we should call Wow64SuspendThread (http://msdn.microsoft.com/en-us/library/windows/desktop/ms687400(v=vs.85).aspx).  By any chance can you modify your build to print GetLastError() before crashing?

dmajor: do you have any experience with WOW64?  If that is indeed the problem, is this a thing we can #ifdef or is there a dynamic WOW64 query?

Comment 6

[Syd Polk :sydpolk]

I am not running out of a source tree; I am downloading nightly builds. Not likely I'll be able to get a useable build for this change in a timely manner.

Comment 7

[Luke Wagner [:luke]]

Do you have Visual Studio installed?  If so, you could attach and then execute GetLastError() in the debug immediate window.

Comment 8

[Syd Polk :sydpolk]

This crashes when running tests via launching via firefox.exe command-line on Windows XP 64 and on Win 8.1 64. It crashes when I double-click on Firefox on XP, but on 8.1, that works.

Comment 9

[Syd Polk :sydpolk]

I do NOT have VS on my Windows XP 64-bit VM. It was a pain to build, and I was hoping to keep it as close to an end-user system as possible.

Comment 10

[Syd Polk :sydpolk]

You can RDC into 10.252.73.236 if you are on VPN. User name mozilla, password mozilla. Feel free to install whatever you want.

Comment 11

[Syd Polk :sydpolk]

I cloned the VM reproducing the problem.

Comment 12

[Luke Wagner [:luke]]

This SO answer (http://stackoverflow.com/a/24153578) would suggest that WOW64 isn't relevant.  OTOH, re-reading msdn docs (http://msdn.microsoft.com/en-us/library/aa914551.aspx) shows that SuspendThread can spuriously fail if the target thread is making a kernel call (good chance it is).

We can depend on InterruptRunningJitCode being called again in 1s, so the simple fix is to just 'return;' instead of crashing.

Comment 13

[Luke Wagner [:luke]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/d1903414198f

Comment 14

[Luke Wagner [:luke]]

Cancelling need-info, but feel free to weigh in.

Comment 15

[mcdavis941 (sporadically reading bugmail)]

I'm having what appears to be the same crash.  I say that because the symptoms are the same and this bug number has been attached to the crash report, like https://crash-stats.mozilla.com/report/index/b97b8835-f0bc-4724-ac52-5c9d42141114 for example.  I'm on 64-bit Vista, running 32-bit Firefox 36 nightlies.  If there's anything I can do (that doesn't take a whole day), feel free to needinfo me.

Comment 16

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/d1903414198f

Comment 17

[(Away)]

Oops, I'm late to the party, but I'm still curious why it originally failed. I'll take a look at that RDC tomorrow. (WinDbg can GetLastError with "!gle")

Comment 18

[Luke Wagner [:luke]]

(In reply to David Major [:dmajor] (UTC+13) from comment #17)
I'd be happy to hear what you see.  The msdn documentation says the SuspendThread call can fail if the thread is in a kernel syscall,

Comment 19

[(Away)]

So this int 3 is actually folded together with a bunch of others. Given the register state, it couldn't have been the SuspendThread call. A debug build says: Hit MOZ_CRASH(Failed to get suspended thread context)

I couldn't get !gle working on this ancient OS :) but I re-pushed the parameters and ran GetThreadContext again.

0:010> !error c0000022
Error code: (NTSTATUS) 0xc0000022 (3221225506) - {Access Denied}  A process has requested access to an object, but has not been granted those access rights.

Comment 20

[(Away)]

http://msdn.microsoft.com/en-us/library/windows/desktop/ms679362%28v=vs.85%29.aspx
"WOW64:  The handle must also have THREAD_QUERY_INFORMATION access."

JSRuntime::init says:
    size_t openFlags = THREAD_GET_CONTEXT | THREAD_SET_CONTEXT | THREAD_SUSPEND_RESUME;

Comment 21

[Luke Wagner [:luke]]

Wow, nice find!  I can't believe I didn't pick that up in my multiple passes of those docs.  Do you by chance know if THREAD_QUERY_INFORMATION will succeed by default on these systems?

Comment 22

[(Away)]

I don't have any direct experience with it, but I would imagine that if you're allowed to SET_CONTEXT then you should be allowed to QUERY_INFORMATION.

Comment 23

[Luke Wagner [:luke]]

Attachment: tweak-openthread

Comment 24

[(Away)]

Comment on attachment 8524173 [details] [diff] [review]
tweak-openthread

I or'ed 0x40 into the stack value on the XP64 VM, and it works!

Comment 25

[(Away)]

Oh and DXR says the other occurrences of THREAD_GET_CONTEXT in the codebase are fine.

Comment 26

[Luke Wagner [:luke]]

Thanks again for digging in further.
https://hg.mozilla.org/integration/mozilla-inbound/rev/f185bda1e516

Comment 27

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/f185bda1e516