Closed Bug 1677633 Opened 4 years ago Closed 4 years ago

Antivirus software corrupts universal macOS Nightly builds

Categories

(External Software Affecting Firefox :: Other, defect, P1)

Product:
External Software Affecting Firefox
Component:
Other
Platform:
Unspecified
macOS
Type:
defect

Tracking

(firefox84blocking fixed, firefox85+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox84 blocking fixed
firefox85 + fixed

People

(Reporter: RyanVM, Unassigned)

References

Details

Spinning off the report from bug 1675740 into its own bug.

(In reply to JonnyRobbie from bug 1675740 comment #13)

The commits linked with this bug are causing false positive flagging with antivirus software (Cylance protect).
The nightly version https://ftp.mozilla.org/pub/firefox/nightly/2020/11/2020-11-12-09-44-41-mozilla-central/firefox-84.0a1.en-US.mac.dmg works fine. But the next nightly version - https://ftp.mozilla.org/pub/firefox/nightly/2020/11/2020-11-13-04-14-59-mozilla-central/firefox-84.0a1.en-US.mac.dmg (as far as I can tell, it's the commits 84480987c9a7f832986085c71b0fd0a71955c9f2 and 74442fd27dfa81129105fe2de0ebea5dbdd85c85) is causing Cylance protect to quarantine several essential files in the bundle, basically corrupting Nightly and leaving it unupdateable without admin rights. I'll try to attach a screenshot with the event log.

Macos Mojave 10.14.6

(In reply to JonnyRobbie from bug 1675740 comment #14)

Created attachment 9187936 [details]
Screenshot 2020-11-15 at 21.04.34.png

list of files quarantined

(In reply to Mike Hommey [:glandium] from bug 1675740 comment #15)

All the files in your screenshot are Universal binaries. If your AV doesn't like those, it won't like any of the upcoming applications meant to work on both new and old macs. I don't think there's anything we can do here.

This sounds like a possible blocking issue to me if the end result for macOS users is a corrupted Firefox install. Can we get some data about what AV usage on macOS looks like so we can get a sense of impacted users?

If needed, we can hold back 84.0b1 from macOS users for the time-being while we decide on a plan going forward, but if we don't have a good solution soon or fixed AV software, we may need to consider backing out the universal installer from 84.

Flags: needinfo?(rtestard)

AFAICT our understanding of antivirus usage is limited to Windows on 8.1+ versions for AVs that do register as a system AV (we leverage a Windows API). I don't think we can collect AV info on macOS but NIing aklotz in case I'm missing something.

Flags: needinfo?(rtestard) → needinfo?(aklotz)

We could use the modules from one of the modules pings.

Indeed, did not realize we had telemetry.modules, this includes data from normalized_os='Mac'.
I tried for a bit but did not manage to get some kind of module name distribution (I assume we want to assess share of users having Cylance as a module name), I'm not familiar enough with this telemetry and would suggest raising this with DS.
Also worth noting is that Cylance may not inject into the Firefox processes and in this case this would not get reported on modules telemetry ?

We didn't investigate it on non-Windows platforms.

Flags: needinfo?(aklotz)

I made this query to see data of the modules ping on Mac. I didn't read the code, but it seems the version or the signature of a module is not available on Mac. As Romain mentioned, we need a hint of the module's name. At least there was no module like '%cyl%'.

If there is a field about an installed AV in the metadata or environment column as we have on Windows, we can use it, but I could not find such a field.

If I can help with a bit of diagnostics, just let me know (though reverting the offending commits would be best short term workaround). But it is a company work laptop, so my admin rights are limited. It has multiple layers of corporate security, but the cylance seems to be the only one (so far) which is throwing tantrums.

When I can, I'll try to update the av, as its policies may be a few days old. in the meantime: CylancePROTECT Version 2.1.1550.514

We found CyMemDef.dylib in the module list, and a Google search confirms this is likely Cylance. From our statistics, we see it in 376 out of 2.6M pings, or 0.014% of the macOS users.

Our anti-virus Telemetry on macOS is obviously not as advanced as the one on Windows, but based on this, I don't think we have evidence this needs to block Beta.

We reached out to Cyclance about this, and managed to get a contact, but that wasn't very productive. Their engine is a machine learning "black box" so they can't really fix it nor do they understand how it works, and they have to whitelist by using SHA256 of the binaries. That's not manageable given the amount of releases, updates, localized builds we do.

This is the answer I received from Cylance regarding this. I don't have a "BlackBerry myAccount" so I have no idea how to proceed here.

Hello ,

Thank you for contacting Cylance Smart AV!

My apologies.  You have reached the support team for the consumer
product Cylance Smart Antivirus.  To create a case to be received by the
correct team, you will need to use your BlackBerry myAccount.  Here is
an article on using that account.  Please let me know if you are still
having issues in creating a case as I may be able to assist with that.
I will be closing this case as it is outside my ability to troubleshot.
http://support.blackberry.com/kb/articleDetail?articleNumber=000068846

I was able to find contact info for a Cylance engineer who has been able to assist us. They are fetching all relevant Firefox builds and feeding them to their machine learning model. I'm not sure how long it will take for that result to roll out, so we'll also try to make sure they have direct inputs from the next (few?) beta releases.

We heard from Cylance today that this should be resolved in the very near future:

I just wanted to update you and let you know we are pushing the whitelist centroids to resolve this issue to production today. This will be sent to all endpoints running Cylance products. While we are already monitoring things on our end, please let us know if you hear of any new reports on 12/3 or afterward.

The severity field is not set for this bug.
:gcp, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(gpascutto)
Severity: -- → S2
Flags: needinfo?(gpascutto)
Priority: -- → P1

We received another update from Cylance suggesting that their latest update from Friday the 11th should resolve this going forward:

Data Science has deployed an additional white centroid to endpoints last Friday to help with the false positives involving specific multi-architecture Macho files (Macho Fat binaries). This includes Firefox. The new white centroid appears to be very performant. I installed 84.0 myself and I didn't have any items convicted.

Please let us know if you continue to see this after installing their latest update.

Given comment 16 and the lack of comments since, I'm going to go ahead and close this out as fixed by Cylance. Feel free to reopen if you're still seeing these crashes with their current centroid endpoints and we can reach out again.

Status: NEW → RESOLVED
Closed: 4 years ago
status-firefox84: affected → fixed
status-firefox85: affected → fixed
Resolution: --- → FIXED

This is happening again on 86a1...

This is happening again on 86a1...

Tracking in bug 1677633.

See Also: → 1690695
You need to log in before you can comment on or make changes to this bug.