1549624 - Kaspersky antivirus is associated with poor uptake of the addon certificate hotfix

Status: NEW

(Firefox :: Normandy Client, defect, P2)

Description

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

:wbeard discovered that certain Kaspersky antivirus products (as reported in environment.system.sec.antivirus; see bug 1418131) are associated with poor takeup of the [e-mail] extension pushed out by Normandy to address the addon intermediate certificate expiration (bug 1548973).

The symptoms are that client_ids that "should have" received the hotfix (i.e. have at least 10 minutes of session time after 8am PT Sat May 4) have never sent a main ping listing [e-mail] in environment.addons.activeAddons.

Affected products have uptake rates below 20%. Uptake rates are above 90% for many popular products.

Affected:
Kaspersky Internet Security
Kaspersky Total Security
Kaspersky Free
Kaspersky Anti-Virus
Kaspersky Security Cloud
Kaspersky Small Office Security
カスペルスキー インターネット セキュリティ

Not affected:
Kaspersky Endpoint Security 10 for Windows
Kaspersky Endpoint Security for Windows

A table broken out by all antivirus products: https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/113658/dashboard/113775

Comment 1

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

Does bug 1549249 resolve this and bug 1549627?

Comment 2

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

No; bug 1549249 ships the fixup cert in-tree, which will help affected users reenable their addons after they update, but does not address the failure to take up the hotfix addon through Normandy.

Comment 3

[Michael Cooper [:mythmon]]

This is likely caused by AV products installing MITM proxies that intercept our traffic and re-encrypt it with different keys. For add-on installation, we only trust expected keys, and so the MITM proxy breaks it. You can see this with Avast in this Twitter thread.

I believe many AV providers have a list of possible add-on sources, so as not to break normal add-on installation. However, Normandy hosts its own add-ons, on a domain that AV products don't expect. Because of this, add-on installation is broken by AV products.

This likely affects all add-on studies, not just the hotfix.

Comment 4

[Romain Testard [:RT]]

Just back from parental leave, is the situation unchanged? Is the only way forward to outreach to these vendors to discuss a custom solution?

Comment 5

[Michael Cooper [:mythmon]]

The situation is unchanged. Reaching out to AV vendors is one option here.

It may be possible to change our security posture on this issue to avoid the problem as well. I believe we already do something like this for browser updates, even going so far as to fallback to HTTP if HTTPS connections fail entirely (This is safe because the update payloads are specially signed).

Making this change to Normnady may be necessary for other reasons in the future, so I'd like to consider it.

Comment 6

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

I'm also curious about how much breakage we face from enterprise MitM middleboxes, but we don't have a signal for those in telemetry (do we have relationships with those vendors?).

Comment 7

[Michael Cooper [:mythmon]]

Normandy sends enrollFailed events when we fail to install. I suspect that in the case of HTTPS middleboxes, this would be presented as reason=download-failure, detail=ERROR_NETWORK_FAILURE. This error is surely used for other issues too, like interrupted connections. It would be good to verify that middleboxes' meddling does get reported this way in Telemetry.

For some studies this is pretty uncommon, and for others it is nearly equal the number of enrollment events. I made a query that looks at the number of enroll and enrollFailed for recent studies, broken out by event reason and details where appropriate.

https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/122026/command/122028

I don't know if these failures usually are from clients that never enroll, or from clients that try many times and eventually succeed. I do notice that studies that are sent to more users tend to have a higher proportion of network errors.

Comment 8

[BugBot [:suhaib / :marco]]

The priority flag is not set for this bug.
:mythmon, could you have a look please?

For more information, please visit auto_nag documentation.