1868531 - Symantec DLP detection of File Upload and Folder Upload has been broken in Firefox 121 beta

Status: RESOLVED INVALID

(External Software Affecting Firefox :: Other, defect, P2)

Description

[Sumeet Agrawal]

Steps to reproduce:

1. Install Symantec DLP Endpoint agent and configure it to use a policy to detect for sensitive data loss through Firefox based on keyword type policy.
2. Upload a sensitive file through a website like 'https://dlptest.com/https-post' . Sensitive file is lost.
3. Similarly performing Folder Upload to a website like Google Drive is not detected.
4. Drag-n-drop the same sensitive file to a supported website like Google Drive and it is detected.

Actual results:

Symantec DLP endpoint agent detects file and folder uploaded to various websites through Firefox browser using a in-house developed Firefox module - ffm.dll and ffm64.dll.
With Firefox 121 Beta, the File and Folder Upload detection of DLP is not functional anymore.

Expected results:

This functionality has been intact till Firefox 120 which is in the GA market right now and even till Firefox 120 Beta channel.

Comment 1

[Greg Stoll :gstoll]

Hoping mozregression will help pinpoint this

Comment 2

[Haik Aftandilian [:haik]]

@Sumeet, on bug 1837079, we introduced code for moving the file dialog out of the main process on Windows. The change in behavior is not enabled in Beta at this time, but the code changes and refactoring are and might have affected DLL injection function hooking. For example, see the most recent changes in this log https://hg.mozilla.org/releases/mozilla-beta/log/tip/widget/windows/filedialog/WinFileDialogCommands.h

Comment 3

[Ray Kraesig [:rkraesig]]

If that's the issue, there will probably need to be additional revision for Fx122, as we've performed some further refactoring in bug 1858225 to fix a number of longstanding issues with the file dialog. You may need to take particular care if so, as this was accomplished in part by moving the file dialog to a separate COM apartment.

Comment 4

[Sumeet Agrawal]

@haik, I checked the 'mozgression' tool and it seems that tool helps to find which date / version range of Firefox release may have introduced the issue. So I realized that perhaps we may not need to run this tool.

We run our Automation tests on each Nightly Beta Build of Firefox for testing compatibility with our DLP Endpoint Agent. Our last successful tests are from around 11 November with Firefox 120.0b9 version. Starting with Firefox 121.0b1 our tests have been consistently failing.

I hope this helps.

Comment 5

[Haik Aftandilian [:haik]]

@Sumeet, given those version numbers, it would definitely be helpful to run mozregression. 121.0b1 contains thousands of fixes that are not in 120.0b9 and mozregression, in most cases, is able to identify a small handful of pushes that caused the change in behavior in the Nightly build of Firefox before Beta.

Comment 6

[Sumeet Agrawal]

Thanks Haik. I am trying to use the tool. I have watched the video from the link you had provided. However, I am facing a few issues. Our problem was found in Firefox Beta. However, am not able to find the right options for the tool. When I had started to execute the tool Build type selected was 'Shippable' and Repository was left blank. But with these options even after running the various FF windows, I was not able to find the exact build so the exercise didn't yield fruits yet. I will continue to experiment around these flags and try to find out the logs for changes which caused this issue. Please stay tuned.

Comment 7

[Sumeet Agrawal]

I again executed the mozgression tool and this time I selected 'Build type' as 'Shippable' and Repository as 'mozilla-beta'. Preferences / Add-ons was left blank. With this configuration, I was able to execute the tool to narrow down the list of builds where the issue is seen. A reference snapshot of the mozgression tool is attached with this ticket. The 2 faulty builds had the below details:

Beta build: f2fa5dbe
build_date: 2023-11-21 17:37:39.134000
https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=a1334e8e601add3495509f96d677cff89b4693b1&tochange=f2fa5dbe8e76996e929aebba1ccfe170e0a231ed

Beta build: d6c8c84a
build_date: 2023-11-20 18:05:00.115000
https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=a1334e8e601add3495509f96d677cff89b4693b1&tochange=d6c8c84a26388612d432fd27aadcc8cdc1bc4e07

I am also attaching the report generated at the bottom of the tool for details about this exercise. Please let us know if this is what you were looking for.

Comment 8

[Sumeet Agrawal]

Attachment: mozgression_tool_final_stage.png

Comment 9

[Sumeet Agrawal]

Attachment: mozgression_log_output.txt

Comment 10

[Ray Kraesig [:rkraesig]]

Attachment: bug-1837079.diff

(In reply to Sumeet Agrawal from comment #7)

I again executed the mozgression tool and this time I selected 'Build type' as 'Shippable' and Repository as 'mozilla-beta'. Preferences / Add-ons was left blank. With this configuration, I was able to execute the tool to narrow down the list of builds where the issue is seen. A reference snapshot of the mozgression tool is attached with this ticket. The 2 faulty builds had the below details:

This seems to call for an explanation of mozregression.

Mozregression selects various builds across Firefox's history to check for being "good" or "bad", working on the assumption that any commit before a "good" commit is also "good", and that any commit after a "bad" commit is also "bad". At each step, it selects a build in the middle of the range being considered: if the build is "good", then all builds before it are "good" and can be disregarded, and if it is "bad", then all builds after it are "bad", and can be disregarded. This allows mozregression to reduce a range of thousands of changes to a range of very few, while testing only 10 to 13 individual builds. (This is the same fundamental principle used by git bisect, if you're acquainted with that.) As such, the intermediate builds' information isn't particularly important or useful: ordinarily, only the final range produced is important.

However.

If you've limited the repository to mozilla-beta, that's not going to check most builds — only beta builds. That's not very helpful here, for the reasons that :haik explained in comment 5. Your initial attempt, wherein the Repository field was left blank, is the expected procedure; this would start on mozilla-central and then proceed to autoland, to narrow down which patchset introduced the incompatibility...

... but based on the conversation at bug 1762576, my understanding is that Symantec's DLP plugin relies on a DLL-injection mechanism that has not worked on Nightly versions of Firefox for quite some time. As such, mozregression will not be of much help, unless you can adjust your DLL's injection mechanism to avoid the blocking circumscribed by the #ifdefs in D143843. (If you can, I would strongly suggest doing that as a first step.)

---

Otherwise, I'm not really sure how much help we can be without some knowledge of how exactly your plugin does what it does. Under the assumption that it hooks certain of our file-dialog functions, you might want to look at the Phabricator diffs linked at the top of bug 1837079, to see if you can identify the relevant changes by eye. I'm also attaching a unified diff — probably much less readable on its own, but potentially useful if you have some particular identifiers you can just grep for.

Comment 11

[Sumeet Agrawal]

[:rkraesig]
I am hoping to answer most of your queries here:

1. The issue we discovered is with Firefox BETA 121. Firefox 121 is not GA yet. Firefox last GA version 120 is working fine. THAT is the reason I selected beta repository. Do you still think the repo I selected is incorrect?
2. At a high level, I can say that we have a callback registered for 'onUpload' function / Open button from a File Upload dialog box. We are in process of gathering more information about which exact Firefox API we use that is failing in this scenario. Once we have that information, we will provide that too and hoping that helps to further the investigation.
3. Bug 1762576 was very old and that has been resolved long time back. Symantec Firefox monitoring for Enterprise data loss is working fine now for quite some time, as I also mentioned in my point 1 above.

You mentioned in your earlier comment about bug 1858225 involving refactoring of the File Upload related code and that indeed therefore seems to be the cause of our issue.

Based on all above inputs and the mozgression output, do you think we could pin-point the single change that affected this?

Comment 12

[Ray Kraesig [:rkraesig]]

1. The issue we discovered is with Firefox BETA 121. Firefox 121 is not GA yet. Firefox last GA version 120 is working fine. THAT is the reason I selected beta repository. Do you still think the repo I selected is incorrect?

Yes. Running it using the beta repository will give no more information than you gave in comment 4. :haik explained why this was not sufficient in comment 5.

3. Symantec Firefox monitoring for Enterprise data loss is working fine now for quite some time, as I also mentioned in my point 1 above.

Only with Firefox Beta and Release. As explained in comment 10 — and contrary to both :haik's initial impression and mine — we have reason to believe that your plugin does not work with Firefox Nightly. Unless this has changed since bug 1762576, there is no point in running mozregression with the correct repository. Having the plugin work with Firefox Nightly is a hard requirement for using mozregression to pinpoint the relevant change(s).

You mentioned in your earlier comment about bug 1858225 involving refactoring of the File Upload related code and that indeed therefore seems to be the cause of our issue.

That bug's patches haven't made it out to Beta yet. As :haik noted in comment 2, the bug most likely to be immediately relevant is bug 1837079.

Based on all above inputs and the mozgression output, do you think we could pin-point the single change that affected this?

No. As explained above and previously, despite our initial hopes, the mozregression output will not be helpful. If you believe that refactoring on our side is likely to have caused it, then I would advise, as suggested in comment 10, that your next step be looking at the changes made in bug 1837079.

Comment 13

[Sumeet Agrawal]

Ray,
I checked with Firefox nightly today and indeed our plugin doesn't load into Firefox Nightly. I am attaching a snapshot of the same.
Is there a page similar to 'about:third-party' that would show the list of blocked DLLs from loading into Firefox? Something like 'about:blocked-modules'?

We have a different email thread where we are discussing this issue and Haik has helped by providing a potential workaround for now which we will be trying over the next few days. At the same time, from long term perspective, is there a suggestion from the Mozilla team to help us resolve this issue so that our modules are loaded into Firefox nightly too?

Comment 14

[Sumeet Agrawal]

Attachment: FirefoxBlockedModules.png

Comment 15

[Haik Aftandilian [:haik]]

@Sumeet, we have filed bug 1869397 to remove the Firefox Nightly LoadLibrary blocking and I expect that to be available in a Firefox Nightly build within the next few days.

Comment 16

[Ray Kraesig [:rkraesig]]

(In reply to Sumeet Agrawal from comment #13)

Ray,
I checked with Firefox nightly today and indeed our plugin doesn't load into Firefox Nightly. I am attaching a snapshot of the same.
Is there a page similar to 'about:third-party' that would show the list of blocked DLLs from loading into Firefox? Something like 'about:blocked-modules'?

There isn't, I'm afraid. about:third-party does show modules which have been blocked via about:third-party itself, but to my knowledge we don't keep track of modules blocked by that lower-level mechanism.

As :haik has noted, it's moot for now, but it's a reasonable suggestion for the future. I'll reraise it if and when we start (conditionally) enabling that block again, assuming :gstoll hasn't already made note of it.

Comment 17

[Haik Aftandilian [:haik]]

We've heard from the Broadcom team that this problem is resolved and will be addressed in Symantec DLP Endpoint. Closing the bug.