Closed Bug 1672523 Opened 4 years ago Closed 4 years ago

firefox.exe version 82.0.0.7592 crash with Digital Guardian

Categories

(External Software Affecting Firefox :: Other, defect, P5)

Product:
External Software Affecting Firefox
Component:
Other
Platform:
Unspecified
Windows
Type:
defect

Tracking

(firefox81 unaffected, firefox82 affected, firefox83 ?, firefox84 ?)

Status:
RESOLVED DUPLICATE of bug 1672367
Tracking Flags:
Tracking Status
firefox81 --- unaffected
firefox82 --- affected
firefox83 --- ?
firefox84 --- ?

People

(Reporter: ccooper, Unassigned)

References

Details

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4287.0 Safari/537.36 Edg/88.0.673.0

Steps to reproduce:

Installed today's update.

Actual results:

Firefox is no longer starting and Event Viewer displays:

Faulting application name: firefox.exe, version: 82.0.0.7592, time stamp: 0x5f8738dd
Faulting module name: ntdll.dll, version: 10.0.17134.1425, time stamp: 0x4c780f2c
Exception code: 0xc0000409
Fault offset: 0x000000000009f7a0
Faulting process id: 0x2530
Faulting application start time: 0x01d6a7f7eb722eb7
Faulting application path: C:\Program Files\Mozilla Firefox\firefox.exe
Faulting module path: C:\WINDOWS\SYSTEM32\ntdll.dll
Report Id: 090b00c5-cafa-4c29-9d20-db0674d99ef6
Faulting package full name:
Faulting package-relative application ID:

Expected results:

It should have started

hi craig, which security software is in use in the affected environment?

Hi,

Not the submitter but seeing the same issue with the same Firefox build.

OS is Windows 10 build 1809, it's a corporate environment so there may be some customizations to the build.

Security software is Windows Defender, definitions last updated 10/22/2020 5:04 AM. (Don't know if that timestamp is UTC or not, but I am in EDT right now.) Also have Digital Guardian installed, not sure how to get version or defs. (installed and controlled by corporate, but hasn't been a problem before now.)

it would probably also be very helpful for developers looking into the issue, if we could come up with an exact regression range of when this issue got introduced into the codebase. would you be able to run the tool from https://mozilla.github.io/mozregression/ to narrow it down? thank you

Flags: needinfo?(quinn_jones)

I've run the tool, the complete log is at https://pastebin.com/0wfnzZ7D and the last few lines are below:

2020-10-22T12:41:05.359000: INFO : Narrowed integration regression window from [95c978d7, 7801d63e] (4 builds) to [0f0b1fa8, 7801d63e] (2 builds) (~1 steps left)
2020-10-22T12:41:05.368000: DEBUG : Starting merge handling...
2020-10-22T12:41:05.368000: DEBUG : Using url: https://hg.mozilla.org/integration/autoland/json-pushes?changeset=7801d63e36f8ab85953f99c26f3aeee3b8655a14&full=1
2020-10-22T12:41:05.368000: DEBUG : redo: attempt 1/3
2020-10-22T12:41:05.368000: DEBUG : redo: retry: calling _default_get with args: ('https://hg.mozilla.org/integration/autoland/json-pushes?changeset=7801d63e36f8ab85953f99c26f3aeee3b8655a14&full=1',), kwargs: {}, attempt #1
2020-10-22T12:41:05.371000: DEBUG : urllib3.connectionpool: Resetting dropped connection: hg.mozilla.org
2020-10-22T12:41:06.341000: DEBUG : urllib3.connectionpool: https://hg.mozilla.org:443 "GET /integration/autoland/json-pushes?changeset=7801d63e36f8ab85953f99c26f3aeee3b8655a14&full=1 HTTP/1.1" 200 None
2020-10-22T12:41:06.385000: DEBUG : Found commit message:
Bug 1662204 - Prevent all printed documents, not just print preview, from getting a regular non-print presentation. r=jwatt

Before bug 1636728 this couldn't happen because print documents weren't
hosted in an <browser>. The presentation of documents that are being
printed should be managed by the print job.

We should, in fact, probably just make mDocument->IsStaticDocument() the
condition, or such.

Differential Revision: https://phabricator.services.mozilla.com/D88901

2020-10-22T12:41:06.385000: DEBUG : Did not find a branch, checking all integration branches
2020-10-22T12:41:06.388000: INFO : The bisection is done.
2020-10-22T12:41:06.402000: INFO : Stopped

Flags: needinfo?(quinn_jones)

(In reply to Quinn Jones from comment #2)

Not the submitter but seeing the same issue with the same Firefox build.

Specifically, a crash that is preventing Firefox from even starting?

(In reply to Jonathan Watt [:jwatt] from comment #5)

Specifically, a crash that is preventing Firefox from even starting?

Correct. No window appears, no errors, no startup notifications. I cannot even start with -ProfileManager to choose a different profile.

The only indication that anything occurred is in the Windows Event Viewer. The original submitted posted a representative log.

In a comment #4 I posted results from moz-regression, which found the point where Firefox starts not-starting on my machine.

we're also getting this problem reported multiple times on sumo: https://support.mozilla.org/en-US/questions/firefox?tagged=bug1672523&show=all&owner=all

Status: UNCONFIRMED → NEW
status-firefox81: --- → unaffected
status-firefox82: --- → affected
status-firefox83: --- → ?
status-firefox84: --- → ?
Ever confirmed: true
Keywords: regression

I confirm what appears to be the same problem in firefox83 (83.0.0.7597).

Additional stuff:
Fault bucket 1466154697534873915, type 5
Event Name: BEX64
Response: Not available
Cab Id: 0

Problem signature:
P1: firefox.exe
P2: 83.0.0.7597
P3: 5f8dce98
P4: ntdll.dll
P5: 10.0.18362.1049
P6: b5beef21
P7: 00000000000a1170
P8: c0000409
P9: 000000000000000a
P10:

Attached files:
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER5F72.tmp.dmp
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER5FE1.tmp.WERInternalMetadata.xml
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER6011.tmp.xml
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER6020.tmp.csv
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER6050.tmp.txt

These files may be available here:
\?\C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_firefox.exe_9da2ae4a2d816fe266927f581e1b98a15ab31_44287bd4_cb4af7ed-d517-460c-8b7e-3494e93e1916

Analysis symbol:
Rechecking for solution: 0
Report Id: 7f35fc72-850a-488a-8c9b-0508ae5f198a
Report Status: 268435456
Hashed bucket: 76387b27e96dc29fd458d3edec56213b
Cab Guid: 0

(In reply to [:philipp] from comment #1)

hi craig, which security software is in use in the affected environment?

We have Cisco AMP on our Cisco provided (corporate) laptops.

Sorry, due to my time zone I've been a bit slow catching up. Do you need anything further from me?

Tentatively setting as regressed by bug 1662204 based on comment 4.

It might be helpful for other affected users to make sure they're actually seeing the same thing, and not a crash with a different cause, using mozregression (see comment 3).

Flags: needinfo?(jwatt)
Flags: needinfo?(emilio)
Regressed by: 1662204

I'm also somewhat confused as bug 1662204 was in 81 already.

Yeah, that code path is also printing specific, so short of something messing up with Firefox badly it seems unlikely to be the real cause... Is there some crash reports we could look at to try to analyze the dumps?

Flags: needinfo?(emilio)

Gabriele, do you see anything in the WER (windows crash thing) that looks this?
thanks

Flags: needinfo?(gsvelto)

Not from the dashboard but the reporter could help us here. Craig could you gather a minidump of the crash for us to analyze it? You would have to follow this procedure:

  • Using regedit.exe add the following key to your Windows registry:
    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps
  • Launch Firefox as usual and wait for it to crash
  • Go into the C:\Users\<your_username>\AppData\Local\CrashDumps folder
  • You should find a file with the .dmp extension in there. Please send it to me via e-mail. Do not attach it to this bug. It's a small snapshot of the program memory at the moment of the crash so while it's unlikely to contain sensitive information it may, so it's better to not expose it publicly.
  • Delete the key you created previously to restore Windows Error Reporting default behavior
Flags: needinfo?(gsvelto) → needinfo?(ccooper)

(In reply to Gabriele Svelto [:gsvelto] from comment #15)

Not from the dashboard but the reporter could help us here. Craig could you gather a minidump of the crash for us to analyze it? You would have to follow this procedure:

  • Using regedit.exe add the following key to your Windows registry:
    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps
  • Launch Firefox as usual and wait for it to crash
  • Go into the C:\Users\<your_username>\AppData\Local\CrashDumps folder
  • You should find a file with the .dmp extension in there. Please send it to me via e-mail. Do not attach it to this bug. It's a small snapshot of the program memory at the moment of the crash so while it's unlikely to contain sensitive information it may, so it's better to not expose it publicly.
  • Delete the key you created previously to restore Windows Error Reporting default behavior

Hi Gabriele, the registry key was already there (presumably it's been used for other things in the past) and—with the number of times I've tried to get it working—there were 10 dumps.

I've re-installed the latest version of Firefox from mozilla.org and will email you the latest .dmp file now.

Flags: needinfo?(ccooper)

(In reply to Gabriele Svelto [:gsvelto] from comment #15)

Not from the dashboard but the reporter could help us here. Craig could you gather a minidump of the crash for us to analyze it? You would have to follow this procedure:
...

Please confirm that you've received it.

Thanks,
Craig

I've received the minidump and analyzed it, this is the stack trace of the crash:

Thread 0 (crashed)
 0  ntdll.dll!RtlFailFast2 + 0x0
 1  ntdll.dll!RtlpHandleInvalidUserCallTarget + 0x5c
 2  ntdll.dll!LdrpHandleInvalidUserCallTarget + 0x38
 3  0x7ffce1e50292
 4  DgApi64.dll + 0x1a6bbd
 5  xul.dll!CrashReporter::GetOrInit(nsIFile*, nsTSubstring<char> const&, nsTSubstring<char>&, nsresult (*)(nsTSubstring<char>&)) [nsExceptionHandler.cpp:bbdea0acf29a60ac9500439691337f3e0e96eb2f : 2219 + 0x9e]
 6  0xdeadbeefdeadbeef
 7  xul.dll!CrashReporter::GetOrInit(nsIFile*, nsTSubstring<char> const&, nsTSubstring<char>&, nsresult (*)(nsTSubstring<char>&)) [nsExceptionHandler.cpp:bbdea0acf29a60ac9500439691337f3e0e96eb2f : 2219 + 0x9e]

Note the DgApi64.dll module on the stack. That's a piece of antivirus software (Digital Guardian) which injected itself into Firefox and probably inserted some filters in the file access functions. It seems that it's causing a stack overflow that leads to the crash. Can you check if you have the latest version of that software installed? This kind of issue usually happens when old AV software is involved. If the AV software is recent though we might have to block this specific version to prevent it from crashing Firefox.

There is however something else that's suspicious about this crash. I found two more DLLs in the memory map:

0x7ffd5a380000 - 0x7ffd5a5f2fff  q_yexaxsvi.dll  10.0.17134.1726
0x7ffd5bef0000 - 0x7ffd5bfa0fff  a_schaxq.dll  10.0.17134.1425

The version numbers appear to be similar to Microsoft Windows DLLs but the names are obviously randomly generated and I found no trace of them on the internet. This could be a piece of malware present on your machine. I suggest looking up these files in your filesystem and checking where they came from.

As soon as I saw the original poster mention Digital Guardian I knew that was the culprit.

I can confirm the details mentioned above. DG is currently update to date and was up to date prior to the install. The previous version of Firefox was working properly.

Component: Untriaged → Other
OS: Unspecified → Windows
Product: Firefox → External Software Affecting Firefox
Version: Firefox 82 → unspecified

I sent DG an email and got a reply saying their devs are aware of this issue and working on a fix.

Flags: needinfo?(jwatt)

For customers with Digital Guardian agents installed, the solution is to upgrade to the latest version of DG Agent for Windows version 7.6.1. Please reach out to your Digital Guardian representative if you have any more questions, enter a ticket with Digital Guardian Support, or contact your organization's IT department.

https://digitalguardian.support.com

(In reply to Gabriele Svelto [:gsvelto] from comment #18)

I've received the minidump and analyzed it, this is the stack trace of the crash:

Thread 0 (crashed)
 0  ntdll.dll!RtlFailFast2 + 0x0
 1  ntdll.dll!RtlpHandleInvalidUserCallTarget + 0x5c
 2  ntdll.dll!LdrpHandleInvalidUserCallTarget + 0x38
 3  0x7ffce1e50292
 4  DgApi64.dll + 0x1a6bbd
 5  xul.dll!CrashReporter::GetOrInit(nsIFile*, nsTSubstring<char> const&, nsTSubstring<char>&, nsresult (*)(nsTSubstring<char>&)) [nsExceptionHandler.cpp:bbdea0acf29a60ac9500439691337f3e0e96eb2f : 2219 + 0x9e]
 6  0xdeadbeefdeadbeef
 7  xul.dll!CrashReporter::GetOrInit(nsIFile*, nsTSubstring<char> const&, nsTSubstring<char>&, nsresult (*)(nsTSubstring<char>&)) [nsExceptionHandler.cpp:bbdea0acf29a60ac9500439691337f3e0e96eb2f : 2219 + 0x9e]

Note the DgApi64.dll module on the stack. That's a piece of antivirus software (Digital Guardian) which injected itself into Firefox and probably inserted some filters in the file access functions. It seems that it's causing a stack overflow that leads to the crash. Can you check if you have the latest version of that software installed? This kind of issue usually happens when old AV software is involved. If the AV software is recent though we might have to block this specific version to prevent it from crashing Firefox.

There is however something else that's suspicious about this crash. I found two more DLLs in the memory map:

0x7ffd5a380000 - 0x7ffd5a5f2fff  q_yexaxsvi.dll  10.0.17134.1726
0x7ffd5bef0000 - 0x7ffd5bfa0fff  a_schaxq.dll  10.0.17134.1425

The version numbers appear to be similar to Microsoft Windows DLLs but the names are obviously randomly generated and I found no trace of them on the internet. This could be a piece of malware present on your machine. I suggest looking up these files in your filesystem and checking where they came from.

Yes, it appears I have Digital Guardian Agent version 7.5.3.0018 installed on my company laptop. Thanks Gabriele!

Keywords: regression
No longer regressed by: 1662204
Summary: firefox.exe version 82.0.0.7592 crash in ntdll.dll version 10.0.17134.1425 → firefox.exe version 82.0.0.7592 crash with Digital Guardian
Hello,

I run dumpchk.exe on dump file produced by the crash and I found following interesting output:

Ldr.InInitializationOrderModuleList: 000001c372a038c0 . 000001c372bd0700
Ldr.InLoadOrderModuleList: 000001c372a03a30 . 000001c372bd06e0
Ldr.InMemoryOrderModuleList: 000001c372a03a40 . 000001c372bd06f0
Cannot read ntdll!_LDR_DATA_TABLE_ENTRY at 000001c372a03a30
SubSystemData: 0000000000000000
ProcessHeap: 000001c372a00000
ProcessParameters: 000001c372a02e80
CurrentDirectory: 'C:\Users\212544016\applis_portables\FirefoxPortableEnglish\App\firefox'
WindowTitle: 'C:\Users\212544016\applis_portables\FirefoxPortableEnglish\App\firefox64\firefox.exe'
ImageFile: 'C:\Users\212544016\applis_portables\FirefoxPortableEnglish\App\firefox64\firefox.exe'
CommandLine: 'C:\Users\212544016\applis_portables\FirefoxPortableEnglish\App\firefox64\firefox.exe -profile C:\Users\212544016\applis_portables\FirefoxPortableEnglish\Data\profile'
DllPath: '< Name not readable >'
Environment: 000001c372a00fe0

See Also: → 1672367

The severity field is not set for this bug.
:gcp, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(gpascutto)

Low priority. The vendor has recommended users update in comment 22. https://bugzilla.mozilla.org/show_bug.cgi?id=1672523#c22

We block old versions in https://bugzilla.mozilla.org/show_bug.cgi?id=1672367, perhaps this can be closed?

Severity: -- → S3
Flags: needinfo?(gpascutto) → needinfo?(tkikuchi)
Priority: -- → P5

We're closing this bug as we blocked DG <v7.6 to workaround a crash of bug 1672367. If your company did not have a chance to upgrade DG to 7.6, please download the latest Nightly from this page and see it launches normally or not? If Nightly still fails to launch, it may be caused by an application other than Digital Guardian, then we reopen the bug and continue investigation.

Status: NEW → RESOLVED
Closed: 4 years ago
Flags: needinfo?(tkikuchi)
Resolution: --- → DUPLICATE

(In reply to Toshihito Kikuchi [:toshi] from comment #28)

We're closing this bug as we blocked DG <v7.6 to workaround a crash of bug 1672367. If your company did not have a chance to upgrade DG to 7.6, please download the latest Nightly from this page and see it launches normally or not? If Nightly still fails to launch, it may be caused by an application other than Digital Guardian, then we reopen the bug and continue investigation.

*** This bug has been marked as a duplicate of bug 1672367 ***

Just tested with nightly build 84.0a1 on my Windows 10 work machine that's running DG version 7.5.3.0018 and it launched without issue.

Thanks to all contributors for their hard work!

(In reply to Chris Vecchio from comment #29)

Just tested with nightly build 84.0a1 on my Windows 10 work machine that's running DG version 7.5.3.0018 and it launched without issue.

Thanks to all contributors for their hard work!

Glad to hear that! Thank you for verifying with Nightly.

You need to log in before you can comment on or make changes to this bug.