677043 - Firefox keeps crashes in CreateFIle

Status: NEW

(Core :: XPCOM, defect)

Description

[Sandy Canetti]

User Agent: Mozilla/5.0 (Windows NT 6.0; WOW64; rv:5.0.1) Gecko/20100101 Firefox/5.0.1
Build ID: 20110707182747

Steps to reproduce:

Running Firefox 5.01 on Windows XP 64-bit.  Everytime I leave Firfox open and come back to my computer, it has crashed. Sometimes it happens when I'm working in Firefox at the time, but most of the time I'm not at my computer when it happens, so I only have the crash report to help,


Actual results:

Crash report IDs:
bp-a9b8005c-d7a4-4498-adf6-731a12110806
bp-9acf70b6-df9e-4e8c-bf76-adca22110806


Expected results:

It should not have crashed!

Comment 1

[Thomas Ahlblom]

0 	kernel32.dll 	CreateFileW 	
1 	kernel32.dll 	BasepGetVolumeNameFromReparsePoint 	
2 	kernel32.dll 	GetVolumePathNameInternalW 	
3 	kernel32.dll 	GetVolumePathNameW 	
4 	feclient.dll 	RemoteFile 	
5 	feclient.dll 	EfsClientFileEncryptionStatus 	
6 	xul.dll 	nsLocalFile::CopySingleFile 	xpcom/io/nsLocalFileWin.cpp:1475
7 	xul.dll 	nsLocalFile::CopyMove 	xpcom/io/nsLocalFileWin.cpp:1609

Comment 2

[Thomas Ahlblom]

Does the issue still occur if you start Firefox in Safe Mode?
https://support.mozilla.com/en-US/kb/Safe+Mode

Does your computer enter some kind of power save mode when you leave it alone?

Comment 3

[(mostly gone) XtC4UaLL [:xtc4uall]]

Seems to happen also in Firefox 7: https://crash-stats.mozilla.com/report/index/ac9dc1ea-4dcf-4c0e-84c0-b20882110804 per Crashstats http://bit.ly/qWk0oE.

Comment 4

[Matthias Versen [:Matti]]

That looks like a windows bug

Comment 5

[Sandy Canetti]

(In reply to Thomas Ahlblom from comment #2)
> Does the issue still occur if you start Firefox in Safe Mode?
> https://support.mozilla.com/en-US/kb/Safe+Mode

I will try running in Safe Mode and report back.


> Does your computer enter some kind of power save mode when you leave it
> alone?

No. I've had it crash on me while Firefox was open, but not in focus and I was in working in another app.  It's also crashed while I had focus on Firefox and was browsing.

Comment 6

[Tracy Walker [:tracy]]

This bug has been around for some time. However just recently it has risen sharply in volume to the #3 topcrasher on Nightly at 5% of all crashes.  It is a start up crasher with 100% of these crashes occurring in less than one minute of up time. 100% Win 7 with AMD64

The notable increase in volume began with builds of 2014020303.

Comment 7

[Tracy Walker [:tracy]]

@CreateFileW: https://crash-stats.mozilla.com/report/list?product=Firefox&range_value=7&range_unit=days&date=2014-02-10&signature=CreateFileW&version=Firefox%3A30.0a1

@CreateFileInternal: https://crash-stats.mozilla.com/report/list?product=Firefox&range_value=7&range_unit=days&date=2014-02-10&signature=CreateFileInternal&version=Firefox%3A30.0a1#tab-reports

Comment 8

[Benjamin Smedberg]

The stacks are at least somewhat busted: dmajor can you load a few and see what's actually going on? I see a bunch of EXCEPTION_ACCESS_VIOLATION_EXEC with crash address 0x202. This feels like monkeypatching, bad symbols, or external software problems.

Comment 9

[(Away)]

The code had intended to do:
call    qword ptr [KERNELBASE!_imp_NtCreateFile (000007fe`baa781a8)]

...but wound up at 0x202. We don't have the memory for kernelbase so it's hard to say what happened. It could be that the import table got clobbered. Or perhaps the code at the call destination had some problems. I'll dig some more...

Comment 10

[Lukas Blakk [:lsblakk] use ?needinfo]

Not tracking since it's 64bit, which we don't officially support yet.

Comment 11

[(Away)]

Crashing thread's stack:
0x202
KERNELBASE!CreateFileW
kernel32!GetVolumeNameForRoot
kernel32!BasepGetVolumeNameForVolumeMountPoint
kernel32!GetVolumeNameForVolumeMountPointW
shell32!CGetVolumeNameForVolumeMountPointParams::Call
shell32!CCallWithTimeoutParams::~CCallWithTimeoutParams
ntdll!RtlpTpWorkCallback
ntdll!TppWorkerThread
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Meanwhile, the main thread was doing:
ntdll!ZwWaitForSingleObject
ntdll!RtlpWaitOnCriticalSection
ntdll!RtlEnterCriticalSection
shell32!CMountPoint::GetMountPoint
shell32!CDrivesFolder::FillDriveID
shell32!CDrivesFolder::ParseDisplayName
shell32!CRegFolder::ParseDisplayName
shell32!CDesktopFolder::ParseDisplayName
shell32!CRegFolder::ParseDisplayName
shell32!SHParseDisplayName
shell32!kfapi::_CreateIDListFromPath
shell32!kfapi::CFolderIDListBuilder::GetIDList
shell32!kfapi::CFolderCache::GetIDList
shell32!kfapi::CKFFacade::GetFolderIDList
shell32!SHGetKnownFolderIDList_Internal
shell32!SHGetFolderLocation
shell32!SHGetSpecialFolderLocation
xul!GetShellFolderPath

This started on build 20140203030203, which had these changes: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=3e40f7389d1b&tochange=44ba69cacd7e

Possibly this?
	61a591c422b7	Aaron Klotz — Bug 902587 - Part 2C: Interpose NtCreateFile, NtReadFile, NtReadFileScatter, NtFlushBuffersFile, and NtQueryFullAttributesFile; r=ehsan

Comment 12

[(No longer employed by Mozilla) Aaron Klotz]

That patch added several new hooks using WindowsDllInterceptor. I suspect that we're hitting a corner case that is causing this to be patched incorrectly.

Comment 13

[(No longer employed by Mozilla) Aaron Klotz]

We will need to either find an AMD64 machine to reproduce this on and/or take a look at the WOW64 ntdll.dll from such an installation.

Comment 14

[(Away)]

I couldn't reproduce on my Win7sp1 machine, and my NtCreateFile didn't seem out of the ordinary. There may be external factors involved (like bug 951827 comment 15) or maybe I'm overlooking something.

Comment 15

[Wesley Hardman]

Just hit this a few minutes ago. bp-e470cf6e-1e29-4c0b-8915-fed4c2140303

(In reply to Aaron Klotz [:aklotz] from comment #13)
> We will need to either find an AMD64 machine to reproduce this on and/or
> take a look at the WOW64 ntdll.dll from such an installation.

This was on an Intel machine so I don't think it is AMD specific.
Unfortunately, I can't seem to reproduce it again.

I was updating my theme, and had just saved browser.css and was restarting Nightly when it crashed instantly on start.  Not sure if that provides any clues.

Comment 16

[Tracy Walker [:tracy]]

(In reply to Lukas Blakk [:lsblakk] from comment #10)
> Not tracking since it's 64bit, which we don't officially support yet.

I understand we don't "officially" support 64bit yet. However this is currently our top volume crasher on Nightly at 7%+ of all crashes. It is a start up crash as well, making it even more critical.  If this can be figured out, I hope we'll take the fix before 30 goes to release next quarter.

Comment 17

[Robert Kaiser]

(In reply to Aaron Klotz [:aklotz] from comment #13)
> We will need to either find an AMD64 machine to reproduce this

"amd64" only means the standard x86_64 architecture. The name is for historic reasons, as this exact 64bit spec was initially created by AMD, but AFAIK all Intel "Core" processors of the last few years support the same architecture and all 64bit Windows is running that architecture.

All you should need to repro is being on a 64bit Windows and using a Nightly/trunk build compiled as a 64bit binary.
(That said, I do not think the current spike has anything to do with WOW64, i.e. 32bit-Firefox-on-64bit-Windows, it's on pure 64bit builds, and therefore we hijacked this bug with something unrelated to comment #0 - but it had no activity for >2 years so I guess we can just go with it.)

Comment 18

[(No longer employed by Mozilla) Aaron Klotz]

(In reply to [:tracy] Tracy Walker - QA Mentor from comment #16)
> (In reply to Lukas Blakk [:lsblakk] from comment #10)
> > Not tracking since it's 64bit, which we don't officially support yet.
> 
> I understand we don't "officially" support 64bit yet. However this is
> currently our top volume crasher on Nightly at 7%+ of all crashes. It is a
> start up crash as well, making it even more critical.  If this can be
> figured out, I hope we'll take the fix before 30 goes to release next
> quarter.

The good news is that the feature that we suspect is causing the spike is only enabled on Nightly -- the same code exists in 29 Aurora as well, but as you can see this signature is not as prominent on that channel.

Comment 19

[(No longer employed by Mozilla) Aaron Klotz]

(In reply to Aaron Klotz [:aklotz] from comment #18)
> The good news is that the feature that we suspect is causing the spike is
> only enabled on Nightly -- the same code exists in 29 Aurora as well, but as
> you can see this signature is not as prominent on that channel.

And by "Nightly" I am referring to the actual release channel; when 30 is promoted to Aurora the feature will become inactive.

Comment 20

[Robert Kaiser]

(In reply to Aaron Klotz [:aklotz] from comment #18)
> The good news is that the feature that we suspect is causing the spike is
> only enabled on Nightly -- the same code exists in 29 Aurora as well, but as
> you can see this signature is not as prominent on that channel.

Well, we only create Win64 builds for Nightly as we continue to call that build architecture "experimental". That said, a good portion of our Nightly users are on those builds as they eliminate some OOM (actually out-of-virtual-memory) problems for some patterns of heavy usage.

Comment 21

[(No longer employed by Mozilla) Aaron Klotz]

Since bug 972577 has landed this signature has dropped off.

Comment 22

[BugBot [:suhaib / :marco/ :calixte]]

Since the crash volume is low (less than 5 per week), the severity is downgraded to S3. Feel free to change it back if you think the bug is still critical.

For more information, please visit auto_nag documentation.