Closed Bug 1553776 Opened 1 year ago Closed 1 year ago

Firefox cannot start with Ivanti Endpoint Security installed

Categories

(External Software Affecting Firefox :: Other, defect, P1)

Desktop
Windows 7
defect

Tracking

(firefox-esr6868+ fixed, firefox67+ wontfix, firefox68+ fixed, firefox69 fixed, firefox70 fixed)

RESOLVED FIXED
Tracking Status
firefox-esr68 68+ fixed
firefox67 + wontfix
firefox68 + fixed
firefox69 --- fixed
firefox70 --- fixed

People

(Reporter: userbs, Assigned: aklotz)

References

(Blocks 1 open bug)

Details

Attachments

(2 files)

Attached image sxwmon.png

Cannot start Firefox 67.0 on Windows 7 machine.

Following error is present in Application Eventlog :

Faulting application name: firefox.exe, version: 67.0.0.7075, time stamp: 0x5cdded92
Faulting module name: sxwmon64.dll, version: 4.6.432.0, time stamp: 0x569d072d
Exception code: 0xc0000005
Fault offset: 0x000000000004a14a
Faulting process id: 0x1378
Faulting application start time: 0x01d51146fe44e450
Faulting application path: C:\Program Files\Mozilla Firefox\firefox.exe
Faulting module path: C:\windows\system32\sxwmon64.dll
Report Id: 3c4dd20d-7d3a-11e9-ab11-6c0b840368eb

On Windows 10 machine Firefox starts normally.

For sxwmon64.dll file details see attachement.

[Tracking Requested - why for this release]:
Startup crash that (apparently) didn't show the crash reporter, correlated with a security product, potentially going to affect more people who won't know how to diagnose.

Thanks for the report! This looks like an issue with Lumension / Ivanti security, based on web search for sxwmon64.

https://forums.ivanti.com/s/article/Resolving-Driver-Conflicts-in-Lumension-Endpoint-Security may be helpful in resolving this for you locally?

:aklotz/:dmajor, anything we can/should do about this from our side?

Group: firefox-core-security → mozilla-employee-confidential
Flags: needinfo?(dmajor)
Flags: needinfo?(aklotz)

Yes it's Ivanti Endpoint Security issue. I can confirm. On a Windows 7 machine without HEAT EMSS installed Firefox 67 is working fine. HEAT EMSS is required i.e. must be installed so no workaround is possible.

Pinging Pascal as well to get this on someone's radar...

Flags: needinfo?(pascalc)
Summary: Firefox cannot start → Firefox 67 cannot start with Ivanti Security Endpoint installed
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(pascalc)

This is another one of those enterprise products that you need to engage with sales in order to obtain a demo. If I can figure out how to get into our Microsoft WER account to look at crash reports, I might be able to examine the one in this bug.

In the meantime, I think we should try to open a channel with the vendor. Romain?

Flags: needinfo?(rtestard)
Flags: needinfo?(dmajor)
Flags: needinfo?(aklotz)
Component: General → Other
Product: Firefox → External Software Affecting Firefox
Version: 67 Branch → unspecified

I don't see contacts there on the AV list - Peter or Adam maybe you have contacts? Unsure what the standard way to reach out if we have no contacts - go through BD or ping directly on LinkedIn?

Flags: needinfo?(stpeter)
Flags: needinfo?(rtestard)
Flags: needinfo?(astevenson)

I don't have contacts with them unfortunately. As stpeter mentioned previously:

let's do the outreach we need to do in order to solve
problems, but let's also check in with Christopher and Mika so all the
relevant folks are in the loop.

I pinged Christopher Arnold and they do not have contacts there either. But he is going to start messaging people on LinkedIn for us.
Romain - it's good to try the standard communication channels (support emails etc) at the same time.
I will give Mika a heads up on this bug as well.

Flags: needinfo?(astevenson)

I'll try to reach out to Ivanti Business Development (https://www.linkedin.com/in/reddleman/)

New "incremental version 67.01 does not work either.

  • I have checked with the 4.6.3 and 5.1.3 in my lab and Firefox is failing during install (although it looks like it finish installation) and then crushes once trying to open in the same way.

  • I have tried to whitelist the application in IDAC but this don’t seem to work. https://forums.ivanti.com/s/article/How-to-exclude-specific-applications-from-IDAC-LES-protection

  • As far as I have checked this is not happening with current stable version of Firefox 67.0.1 (64-bit) (installer version 18.5.0.0)

  • But when I try to install 67.0.0.7075 which displays in Program Files as 68.0 (x64) (installer version 18.5.0.0)

so apparently there was a change between 67 and 68 that is causing it.

There is an access violation error 0xc0000005 “The thread tried to read from or write to a virtual address for which it does not have the appropriate access.”

The full dump is available here https://landeskinc-my.sharepoint.com/:u:/g/personal/bohdan_podlaski_ivanti_com/ETh7EadlikZHiPnglVIuXvsBn9BrNWUqCqMd3vCRQfn8LQ?e=TgTn1f

Could we confirm that the current stable build of Firefox has resolved this bug?

Flags: needinfo?(userbs)

That dump is useful, thanks.

From what I can see, they are injecting a DLL that hooks into the loader the same way that the launcher process does. Unfortunately their code assumes that they're the only ones hooking that function, so their hook corrupts ntdll!NtMapViewOfSection.

Since our hook is in first, we can probably block them from loading their bad code. If they insist on continuing to inject, they're going to need to start playing nicely with our code in order for us to lift the block.

(Of course, the better course of action is to find out what they're really trying to do, and recommend a course of action that doesn't require them to inject and hook into us in the first place.)

I've got a test build pushed to try containing a block for their DLL. Once this builds, let's see if that fixes it.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=249071dfba6eaf327765e06a169eea6f51a96028&selectedJob=250980121

(In reply to Aaron Klotz [:aklotz] from comment #12)

I've got a test build pushed to try containing a block for their DLL. Once this builds, let's see if that fixes it.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=249071dfba6eaf327765e06a169eea6f51a96028&selectedJob=250980121

This try build is now ready to test.

Flags: needinfo?(carnold)

Thanks Aaron. If you need help from Ivanti, please let me know.

Flags: needinfo?(carnold)

Chris, were you able to verify that Aaron's patch worked? It's not clear where we stand in this bug at this point, but unfortunately we've probably missed 68 at this point if there's still a patch that needs to land :(

Flags: needinfo?(carnold)

Thank you Ryan!

Flags: needinfo?(carnold)
Group: mozilla-employee-confidential
Priority: -- → P3
Summary: Firefox 67 cannot start with Ivanti Security Endpoint installed → Firefox cannot start with Ivanti Security Endpoint installed
Duplicate of this bug: 1564535
Blocks: 1564546

Changing the priority to p1 as the bug is tracked by a release manager for the current release.
See What Do You Triage for more information

Priority: P3 → P1

Myrlin1, can you please try this experimental build and see whether it is able to start successfully?

Flags: needinfo?(myrlin1)

It works correctly. Installed and ran with no issue.

Flags: needinfo?(myrlin1)

Thanks! Would you mind trying the 32-bit version as well?

Flags: needinfo?(myrlin1)
Assignee: nobody → aklotz
Status: NEW → ASSIGNED

Same story. No issues presented. I didn't bother with a clean install, though; just opted for an upgrade install.

Flags: needinfo?(myrlin1)

Great, thanks! I'll submit this patch for review.

Summary: Firefox cannot start with Ivanti Security Endpoint installed → Firefox cannot start with Ivanti Endpoint Security installed
Flags: needinfo?(userbs)
Pushed by aklotz@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6fc15315a29b
Add DLL blocklist entries for injected Ivanti Security Endpoint DLLs; r=dmajor
Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED

Comment on attachment 9076930 [details]
Bug 1553776: Add DLL blocklist entries for injected Ivanti Security Endpoint DLLs; r=dmajor!

Beta/Release Uplift Approval Request

  • User impact if declined: Users running Ivanti Endpoint Security won't be able to start Firefox. That product is an enterprise AV, so ESR68 is probably more important than release, but I think this would be a good ride-along.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Simple addition to DLL blocklist, fix verified by affected user
  • String changes made/needed: None

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: Firefox won't be able to start at Enterprises running Ivanti Endpoint Security.
  • User impact if declined: Users won't be able to start Firefox
  • Fix Landed on Version: 70
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Simple addition to DLL blocklist, fix verified by affected user
  • String or UUID changes made by this patch: None
Attachment #9076930 - Flags: approval-mozilla-release?
Attachment #9076930 - Flags: approval-mozilla-esr68?
Attachment #9076930 - Flags: approval-mozilla-beta?

Comment on attachment 9076930 [details]
Bug 1553776: Add DLL blocklist entries for injected Ivanti Security Endpoint DLLs; r=dmajor!

Blocklists Ivanti DLLs causing startup crashes. Approved for 69.0b4.

Attachment #9076930 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

Comment on attachment 9076930 [details]
Bug 1553776: Add DLL blocklist entries for injected Ivanti Security Endpoint DLLs; r=dmajor!

dll blocklist addition, approved for 68.0.1 and 68.1esr

Attachment #9076930 - Flags: approval-mozilla-release?
Attachment #9076930 - Flags: approval-mozilla-release+
Attachment #9076930 - Flags: approval-mozilla-esr68?
Attachment #9076930 - Flags: approval-mozilla-esr68+
Duplicate of this bug: 1566491
Duplicate of this bug: 1566815

Per discussion with jcristau, we're uplifting this to 68.0.1esr also to maintain parity with the non-ESR 68.0.1 release and hopefully avoid some confusion.

(In reply to Aaron Klotz [:aklotz] from comment #28)

  • Is this code covered by automated tests?: Yes
  • Needs manual test from QE?: No

Marking this as qe-verify- per comment 28.

Flags: qe-verify-

I can confirm Firefox version 68.0.1 now starts correctly on machines with Windows 7 OS and Ivanti Endpoint security installed.
I want to thank you all for your support.

Flags: needinfo?(stpeter)
You need to log in before you can comment on or make changes to this bug.