1762576 - Firefox is not allowing Symantec DLP to inject DLL into the browser for Data Loss Prevention software

Status: RESOLVED FIXED

(Core :: DLL Services, defect, P1)

Description

[Sumeet Agrawal]

Steps to reproduce:

With Firefox Beta 99.0 and earlier Beta versions, we are seeing that Symantec DLP DLL injection into Firefox process is getting failed. DLLs that we use: clpbm.dll/ clpbm64.dll, prntm.dll/prntm64.dll and ffm.dll / ffm64.dll.

Actual results:

We are seeing that Symantec DLP DLL injection into Firefox process is getting failed. DLLs that we use: clpbm.dll/ clpbm64.dll, prntm.dll/prntm64.dll and ffm.dll / ffm64.dll.

Comment 1

[Daniel Veditz [:dveditz]]

. Dll injection is a malware technique that we indeed try to block. Even "good" injection is done without coordination with us and has historically caused a huge number of stability problems as Firefox development changes the code around (some potentially exploitable, ironically). Most recently this kind of injection has prevented us from properly sandboxing web content.

Comment 2

[Sumeet Agrawal]

Symantec Data Loss Prevention helps organizations enhance their security infrastructure. Is there a way for Symantec to work side-by-side with Mozilla Firefox team so that our DLLs are not blocked from loading?

Comment 3

[Sumeet Agrawal]

Over the last few years we had encountered very few instances of stability problems with FF due to DLP usage and in those instances, the cases were investigated and we resolved issues on the DLP side. While we appreciate the intention behind blocking DLL injection, we think that it will be more enterprise-friendly if we collaborated on an alternate viable solution before putting blocking in place. This will ensure that Enterprises which monitor browser usage for data loss are not affected and that they can maintain the same strong security posture as before including being able to protect sensitive data from leaving their Enterprise. We are open and willing to working with the FF team in transitioning over to an alternate viable approach for integrating with the FF browser to continue to provide data loss monitoring solution to our customers. The current blockage of our DLP injection will cause disruption to our Customers' Enterprises due to a sudden breakage in our data loss coverage.

Our urgent request to the Mozilla FF team is to unblock us for the near term so we can let current customers' leverage existing DLP coverage with the latest FF 99 update and without any interruption to DLP monitoring. In parallel we intend to engage with FF team to put a plan in place for our transition over to an alternate monitoring technique over a reasonable time period. Towards that please also let us know PM and Architect team contacts that we could reach out to for continuing our discussions.

Comment 4

[Daniel Veditz [:dveditz]]

Tagging :gcp to respond to the above.

Comment 5

[Gian-Carlo Pascutto [:gcp]]

It's with the right triage owner already. Haik owns this and our third-party blocking policies.

the cases were investigated and we resolved issues on the DLP side... it will be more enterprise-friendly if we collaborated on an alternate viable solution before putting blocking in place

As far as I can tell, several bugs about Symantec DLP crashing Firefox are still open, see for example:
https://bugzilla.mozilla.org/show_bug.cgi?id=1649485
https://bugzilla.mozilla.org/show_bug.cgi?id=1705042

I checked and both of these issues were reported to you through the mailing list, typically months/years ago. You can see that despite this it's still crashing Firefox users every day.

But that aside, see the next point.

With Firefox Beta 99.0 and earlier Beta versions, we are seeing that Symantec DLP DLL injection into Firefox process is getting failed.

So, there is no specific block for Symantec DLLs at this point. There indeed used to be one for this specific product, but it got removed: https://hg.mozilla.org/mozilla-central/rev/a2466922a9d64479388392e90f309a3b7225c8ef and a replacement wasn't put in place yet.

So if your product is failing now that means it's falling foul of a generic security mitigation we did. I wonder if it may be this, which was in Nightlies for 4 years, then in beta 98 and beta 99, and finally rolled out to release 99 today:
https://bugzilla.mozilla.org/show_bug.cgi?id=1481454
https://bugzilla.mozilla.org/show_bug.cgi?id=1757487

My understanding is that this injection technique is error-prone, predominantly used by malware, and has caused major incidents in the past where other injected software completely blocked Firefox from starting.

Because this isn't a Symantec specific block, I'm not sure how to roll this back, especially as it replaced a mitigation that caused a dot release in Firefox 97.0.1. Disabling the block completely would risk breakage again? We'd probably want to be able to check that thoroughly. And note Firefox 99 hit release with this security mitigation, so it's too late for that.

Comment 6

[Donal Meehan [:dmeehan]]

Fixed in 99 by backing out bug 1757487 from release

Comment 7

[Haik Aftandilian [:haik]]

The release backout of bug 1757487 will address this problem for now on Release. We will use this bug to land a patch to again limit this blocking to Nightly and early Beta builds while we evaluate how to proceed.

Comment 8

[BugBot [:suhaib / :marco/ :calixte]]

:toshi, since you are the author of the regressor, bug 1757487, could you take a look?
For more information, please visit auto_nag documentation.

Comment 9

[Toshihito Kikuchi [:toshi]]

(In reply to Release mgmt bot [:suhaib / :marco/ :calixte] from comment #8)

:toshi, since you are the author of the regressor, bug 1757487, could you take a look?
For more information, please visit auto_nag documentation.

So DLP injects their modules through CreateRemoteThread+LoadLibrary. We prevented that technique on purpose because WebRoot, which uses the same technique, caused a deadlock (bug 1757487). Even before that, that prevention had been enabled for four years in Nightly (bug 1435816 and bug 1481454).

(In reply to Sumeet Agrawal from comment #2)

Symantec Data Loss Prevention helps organizations enhance their security infrastructure. Is there a way for Symantec to work side-by-side with Mozilla Firefox team so that our DLLs are not blocked from loading?

This is great suggestion. If DLP really needs to inject modules, we'd like to expect Broadcom to change the injection technique of DLP. We can help that. (Well, since I left Mozilla recently, Haik's team will take care of this, but I still can help personally.)

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:haik, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 11

[Haik Aftandilian [:haik]]

Attachment: Bug 1762576 - Firefox is not allowing Symantec DLP to inject DLL into the browser for Data Loss Prevention software r?mhowell

Move the LoadLibrary injection blocking back to Nightly-only.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2bb13139aa79
Firefox is not allowing Symantec DLP to inject DLL into the browser for Data Loss Prevention software r=mhowell

Comment 13

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/2bb13139aa79

Comment 14

[Dianna Smith [:diannaS]]

:haik do you want to nominate this for beta uplift or do we need to backout bug 1757487 in 100 instead?

Comment 15

[Haik Aftandilian [:haik]]

(In reply to Dianna Smith [:diannaS] from comment #14)

:haik do you want to nominate this for beta uplift or do we need to backout bug 1757487 in 100 instead?

I'll nominate this for uplift. The fix is slightly different (in that it reverts to Nightly instead of early beta builds) so it's not equivalent to a backout of bug 1757487.

Comment 16

[Haik Aftandilian [:haik]]

Comment on attachment 9272527 [details]
Bug 1762576 - Firefox is not allowing Symantec DLP to inject DLL into the browser for Data Loss Prevention software r?mhowell

Beta/Release Uplift Approval Request

• User impact if declined: Firefox will not allow third party products (such as enterprise Symantec data loss prevention (DLP)) to inject DLLs into Firefox rendering those products unable to be used with Firefox. Note: we are exploring how to continue blocking this method of DLL injection in a way that allows Firefox to work with DLP products for enterprise use.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): It reverts the fix to Nightly-only. The fix was only enabled on Beta for a few months and doesn't have dependencies.
• String changes made/needed: None

Comment 17

[Dianna Smith [:diannaS]]

Comment on attachment 9272527 [details]
Bug 1762576 - Firefox is not allowing Symantec DLP to inject DLL into the browser for Data Loss Prevention software r?mhowell

Approved for 100.0b8

Comment 18

[Dianna Smith [:diannaS]]

https://hg.mozilla.org/releases/mozilla-beta/rev/e40fff526e74

Comment 19

[Haik Aftandilian [:haik]]

@Sumeet, this fix is now in Firefox 100 Beta Build 8, could you verify the problem is no longer reproducible for you with that build?

Comment 20

[Sumeet Agrawal]

Yes, we've verified that all's well for Symantec DLP with Firefox GA v99.0.1 and Firefox Beta 100.0b8. Thank you all for the quick assistance so far.