1579758 - Firefox frequently crashes under Windows 2019 [Version 10.0.17763.652]

Status: RESOLVED FIXED

(External Software Affecting Firefox :: Other, defect)

Description

[sergei]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0

Steps to reproduce:

I'm just browsing the internet and crash happens sporadically even if I'm just reading something.

Actual results:

Firefox crashes, no traces in about:crashes.
Cursor spins for a while and Windows reports "bad_module_info" error and application shuts down.
I happens with FF version 70b4 as well as with latest stable.

EventLog has the following:
Version=1
EventType=APPCRASH
EventTime=132124671664707788
ReportType=2
Consent=1
UploadTime=132124671706037874
ReportStatus=268435456
ReportIdentifier=050942b0-b189-4005-8855-d050af99a027
IntegratorReportIdentifier=3696fdf4-b3af-4b0e-b54c-67ec0072dc20
Wow64Host=34404
NsAppName=bad_module_info
AppSessionGuid=00001ef8-0001-001a-b46a-5d07a466d501
TargetAppId=W:0000b97c6437386509f9408ae6d9e78116510000ffff!0000e763845f3215f95a0947b914c94ac0854d52fc9d!firefox.exe
TargetAppVer=2019//09//05:19:42:23!960ab!firefox.exe
BootId=4294967295
TargetAsId=432051
IsFatal=1
EtwNonCollectReason=1
Response.BucketId=f4099093f56e03f2adc7bc24018064be
Response.BucketTable=4
Response.LegacyBucketId=2145890610295366846
Response.type=4
Sig[0].Name=Application Name
Sig[0].Value=bad_module_info
Sig[1].Name=Application Version
Sig[1].Value=0.0.0.0
Sig[2].Name=Application Timestamp
Sig[2].Value=00000000
Sig[3].Name=Fault Module Name
Sig[3].Value=StackHash_db16
Sig[4].Name=Fault Module Version
Sig[4].Value=0.0.0.0
Sig[5].Name=Fault Module Timestamp
Sig[5].Value=00000000
Sig[6].Name=Exception Code
Sig[6].Value=c0000005
Sig[7].Name=Exception Offset
Sig[7].Value=PCH_F2_FROM_ucrtbase+0x0000000000023875
DynamicSig[1].Name=OS Version
DynamicSig[1].Value=10.0.17763.2.0.0.272.7
DynamicSig[2].Name=Locale ID
DynamicSig[2].Value=1033
DynamicSig[22].Name=Additional Information 1
DynamicSig[22].Value=db16
DynamicSig[23].Name=Additional Information 2
DynamicSig[23].Value=db1656fa4277de70b0faab06aae39e9b
DynamicSig[24].Name=Additional Information 3
DynamicSig[24].Value=f5dc
DynamicSig[25].Name=Additional Information 4
DynamicSig[25].Value=f5dc67b387d3eb93c529c24d903a300d
UI[2]=bad_module_info
UI[3]=bad_module_info has stopped working
UI[4]=Windows can check online for a solution to the problem the next time you go online.
UI[5]=Check online for a solution and close the program
UI[6]=Check online for a solution later and close the program
UI[7]=Close the program
State[0].Key=Transport.DoneStage1
State[0].Value=1
OsInfo[0].Key=vermaj
OsInfo[0].Value=10
OsInfo[1].Key=vermin
OsInfo[1].Value=0
OsInfo[2].Key=verbld
OsInfo[2].Value=17763
OsInfo[3].Key=ubr
OsInfo[3].Value=652
OsInfo[4].Key=versp
OsInfo[4].Value=0
OsInfo[5].Key=arch
OsInfo[5].Value=9
OsInfo[6].Key=lcid
OsInfo[6].Value=1049
OsInfo[7].Key=geoid
OsInfo[7].Value=244
OsInfo[8].Key=sku
OsInfo[8].Value=7
OsInfo[9].Key=domain
OsInfo[9].Value=0
OsInfo[10].Key=prodsuite
OsInfo[10].Value=272
OsInfo[11].Key=ntprodtype
OsInfo[11].Value=3
OsInfo[12].Key=platid
OsInfo[12].Value=10
OsInfo[13].Key=sr
OsInfo[13].Value=0
OsInfo[14].Key=tmsi
OsInfo[14].Value=50701
OsInfo[15].Key=osinsty
OsInfo[15].Value=3
OsInfo[16].Key=iever
OsInfo[16].Value=11.615.17763.0-11.0.135
OsInfo[17].Key=portos
OsInfo[17].Value=0
OsInfo[18].Key=ram
OsInfo[18].Value=65423
OsInfo[19].Key=svolsz
OsInfo[19].Value=953
OsInfo[20].Key=wimbt
OsInfo[20].Value=0
OsInfo[21].Key=blddt
OsInfo[21].Value=180914
OsInfo[22].Key=bldtm
OsInfo[22].Value=1434
OsInfo[23].Key=bldbrch
OsInfo[23].Value=rs5_release
OsInfo[24].Key=bldchk
OsInfo[24].Value=0
OsInfo[25].Key=wpvermaj
OsInfo[25].Value=0
OsInfo[26].Key=wpvermin
OsInfo[26].Value=0
OsInfo[27].Key=wpbuildmaj
OsInfo[27].Value=0
OsInfo[28].Key=wpbuildmin
OsInfo[28].Value=0
OsInfo[29].Key=osver
OsInfo[29].Value=10.0.17763.652.amd64fre.rs5_release.180914-1434
OsInfo[30].Key=buildflightid
OsInfo[30].Value=f40c70ac-8456-45b0-9e32-c76417858a5c
OsInfo[31].Key=edition
OsInfo[31].Value=ServerStandard
OsInfo[32].Key=ring
OsInfo[32].Value=Retail
OsInfo[33].Key=expid
OsInfo[34].Key=containerid
OsInfo[35].Key=containertype
OsInfo[36].Key=edu
OsInfo[36].Value=0
FriendlyEventName=Stopped working
ConsentKey=APPCRASH
AppName=bad_module_info
AppPath=bad_module_info
NsPartner=windows
NsGroup=windows8
ApplicationIdentity=A01ED07A810B640A20A150D57CC9C420
MetadataHash=-1988655147

Comment 1

[Liviu Seplecan]

Hi @sergei, tested this issue on Windows 10 using several FF versions: beta 70.0b4, 70.0b5, 70.0b6 - cannot reproduce the issue. No crash in my end. From where did you get the above data (EventLog) ? You said that no crash report was present in about:crashes- due this is difficult to handle it without the signature of the crash.
Additionally, there are specific links, app/s that you've accessed ?
I guess this issue could be related(from the point of view of the Window's update): 1521370
Further, I will set a component and someone from dev's team could give us a hand.
Regards,
Liviu

Comment 2

[sergei]

Hello there, I started getting this kind of issue since I upgraded to build Windows 2019 build 1809.
Google says there are similar issues other people having but none of the suggested solutions helped me to resolve the issue.
E.g. https://support.mozilla.org/en-US/questions/1249865

The above data was retrieved from the file that Windows create when app crashes, e.g.
c:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_bad_module_info_9259481a3a5c7a7f71a55b08650e436c3dc0_85207d7d_c0ee68f6\Report.wer

Again, I haven't did anything special at a time of a crash, as usual - google for something, reading MSDN or looking at my Jira tickets. It could happen anytime, even if I'm away from the computer.

Probably would help if I have a debug version here with Windows PDB file.

Regards,
Sergei

Comment 3

[:Gijs (out for now; he/him)]

(In reply to sergei from comment #2)

Probably would help if I have a debug version here with Windows PDB file.

You can download debug versions from https://tools.taskcluster.net/index/gecko.v2.mozilla-release.latest.firefox/win64-debug - you probably want the target.zip link on the right.

I'm not sure what this bad_module_info thing is. Maybe our crash reporter folks have ideas about why our crash reporter wouldn't run here or how to get more info - Gabriele?

Comment 4

[:Gijs (out for now; he/him)]

(In reply to :Gijs (he/him) from comment #3)

(In reply to sergei from comment #2)

Probably would help if I have a debug version here with Windows PDB file.

You can download debug versions from https://tools.taskcluster.net/index/gecko.v2.mozilla-release.latest.firefox/win64-debug - you probably want the target.zip link on the right.

Oh, though you might also just want the crashreporter symbols from that link, or the ones that got produced for your build, which you should be able to find with the artifact browser using the release date and/or revision... they'll be in the crashreporter-symbols zipfiles, I think.

Comment 5

[Gabriele Svelto [:gsvelto]]

I haven't heard of this bad_module_info issues yet so I had to look it up. This is a Windows component failing while you run Firefox. The reason you don't get a Firefox crash report is that Firefox hasn't crashed, it's just been closed by Windows in response to that failure. I couldn't find information on this problem from Microsoft itself but it seems that the problem is common to applications leveraging the GPU, which is something that Firefox is doing more and more.

Sergei, could you try upgrading your graphics drivers and see if the problem goes away?

Comment 6

[:Gijs (out for now; he/him)]

(In reply to Gabriele Svelto [:gsvelto] from comment #5)

I haven't heard of this bad_module_info issues yet so I had to look it up. This is a Windows component failing while you run Firefox. The reason you don't get a Firefox crash report is that Firefox hasn't crashed, it's just been closed by Windows in response to that failure. I couldn't find information on this problem from Microsoft itself but it seems that the problem is common to applications leveraging the GPU, which is something that Firefox is doing more and more.

Sergei, could you try upgrading your graphics drivers and see if the problem goes away?

+ni for this.

Comment 7

[sergei]

(In reply to Gabriele Svelto [:gsvelto] from comment #5)

I haven't heard of this bad_module_info issues yet so I had to look it up. This is a Windows component failing while you run Firefox. The reason you don't get a Firefox crash report is that Firefox hasn't crashed, it's just been closed by Windows in response to that failure. I couldn't find information on this problem from Microsoft itself but it seems that the problem is common to applications leveraging the GPU, which is something that Firefox is doing more and more.

Sergei, could you try upgrading your graphics drivers and see if the problem goes away?

I already have all latest nVidia drivers installed

Comment 8

[Gabriele Svelto [:gsvelto]]

(In reply to sergei from comment #7)

I already have all latest nVidia drivers installed

I see, have you experienced instability in other applications too? I have looked for more information on the issue but the only things I could find point either to GPU drivers or faulty hardware. Since we can discount the drivers would you be able to run a memory test on your machine using a tool like memtest86? If that comes out clean then we know we have to look elsewhere for the problem.

Comment 9

[sergei]

(In reply to Gabriele Svelto [:gsvelto] from comment #8)

(In reply to sergei from comment #7)

I already have all latest nVidia drivers installed

I see, have you experienced instability in other applications too? I have looked for more information on the issue but the only things I could find point either to GPU drivers or faulty hardware. Since we can discount the drivers would you be able to run a memory test on your machine using a tool like memtest86? If that comes out clean then we know we have to look elsewhere for the problem.

No, everything else works very well and no issues. That's a machine I'm using for development and lot of apps running here - but only FF crashes frequently.
Sometimes I'm able to see the stack trace at a time of the failure, no symbols, but failure goes form from xul.dll into ntdll

Comment 10

[:Gijs (out for now; he/him)]

(In reply to sergei from comment #9)

(In reply to Gabriele Svelto [:gsvelto] from comment #8)

(In reply to sergei from comment #7)

I already have all latest nVidia drivers installed

I see, have you experienced instability in other applications too? I have looked for more information on the issue but the only things I could find point either to GPU drivers or faulty hardware. Since we can discount the drivers would you be able to run a memory test on your machine using a tool like memtest86? If that comes out clean then we know we have to look elsewhere for the problem.

No, everything else works very well and no issues. That's a machine I'm using for development and lot of apps running here - but only FF crashes frequently.
Sometimes I'm able to see the stack trace at a time of the failure, no symbols, but failure goes form from xul.dll into ntdll

We run a symbol server that you can use with windbg or msvc if that's how you're looking at stacks for a release build of Firefox - see https://developer.mozilla.org/en-US/docs/Mozilla/Using_the_Mozilla_symbol_server ; TL;DR: you want https://symbols.mozilla.org/ . A symbolicated stack here might shed some light on what's happening...

Comment 11

[sergei]

Some stack trace

 	mozglue.dll!arena_t::MallocSmall(unsigned __int64 aSize, bool) Line 2792	
        mozglue.dll!je_malloc(unsigned __int64 arg1) Line 38	C++
 	mozglue.dll!operator new(unsigned __int64 size) Line 33	C++
 	[External Code]	
 	mozglue.dll!patched_BaseThreadInitThunk(int aIsInitialThread, void * aStartAddress, void * aThreadParam) Line 603	C++
 	[External Code]	

Comment 12

[:Gijs (out for now; he/him)]

(In reply to sergei from comment #11)

Some stack trace

 	mozglue.dll!arena_t::MallocSmall(unsigned __int64 aSize, bool) Line 2792	
        mozglue.dll!je_malloc(unsigned __int64 arg1) Line 38	C++
 	mozglue.dll!operator new(unsigned __int64 size) Line 33	C++
 	[External Code]	
 	mozglue.dll!patched_BaseThreadInitThunk(int aIsInitialThread, void * aStartAddress, void * aThreadParam) Line 603	C++
 	[External Code]	

This looks like it's just an allocation, but there's no info in the stack what we're allocating... It's also odd because comment #9 mentions xul.dll which isn't in this trace - is the stack/failure different each time?

I wonder if there's some kind of dll hijacking going on. Gabriele, anything else we could suggest?

Comment 13

[sergei]

I will post other stack traces soon, it crashes at different points, sometimes it is just a long stack trace without any symbols at all.

Comment 14

[Gabriele Svelto [:gsvelto]]

It could be an interaction between an injected DLL and our code. We hook up the allocation functions and we've had bugs in the past where external code would call into our allocator and then crash because it made assumptions that weren't valid anymore. For example we had similar odd crashes when external code had an object it had allocated on its own (with the system allocator for example) and tried to free it with our allocator (because it used a hooked free() or delete call).

Sergei having a list of the DLLs that are loaded with Firefox on your machine could be useful, here's what you can do:

• Crash a tab by browsing to the about:crashcontent page, then submit the crash report
• Crash the whole browser by going to the about:crashparent page and submit the crash report
• After restarting Firefox go to the about:crashes page and find the links to the last two submitted crashes. Post them here

The crash reports will have the full list of DLLs in Firefox' processes so we can figure out if there's something suspicious.

Comment 15

[sergei]

https://crash-stats.mozilla.org/report/index/c6ba5b72-d69b-4f4e-b386-aa9b20190927
https://crash-stats.mozilla.org/report/index/e28cc796-dfa5-4c61-902d-5772c0190927

Comment 16

[sergei]

Attachment: stack trace

More stack trace but no symbols :(
```

```

Comment 17

[Gabriele Svelto [:gsvelto]]

OK, there's two DLLs that stand out: onepin-opensc-pkcs11.dll and CDAKEYMonitor64.dll. The former is from OpenSC and a Google search reveals that the latter is from a Samsung piece of software called CDA Custom Monitor for Smart Capture. Could you try and uninstall both and see if the crashes go away? The best would be to uninstall one of them and wait a while, if Firefox still crashes then remove the other too and try again.

Comment 18

[sergei]

So cool, I uninstalled onepin-opensc-pkcs11.dll and Firefox do not crash anymore for a week!

Comment 19

[:Gijs (out for now; he/him)]

Gabriele, can/should we try to blocklist the DLL? (If so, what's the appropriate component for that work?)

Comment 20

[Gabriele Svelto [:gsvelto]]

Let's file this in external software affecting Firefox. I'll add this particular DLL version to the blocklist.

Comment 21

[Gabriele Svelto [:gsvelto]]

Attachment: Bug 1579758 - Block a nightly build of OpenSC which crashes Firefox; r=aklotz

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4508caa91027
Block a nightly build of OpenSC which crashes Firefox; r=aklotz

Comment 23

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/4508caa91027