1829925 - AV software Avira breaks websites

Status: RESOLVED FIXED

(External Software Affecting Firefox :: Other, defect, P2)

Description

[Sören Hentzschel]

In the last four days we have received six independent reports from users in the German Firefox support that several websites broke for them. It seems that scripts were blocked for them.

In all cases it was caused by the AV software Avira and uninstalling it solved it. Other browsers were not affected.

This screencast was created by one of the users:
https://www.mobirise-tutorials.com/videos/2023-04-24-NO-Links.mp4

Comment 1

[Dianna Smith [:diannaS]]

This was reported on Reddit as well https://www.reddit.com/r/firefox/comments/12yfj1a/some_websites_twitter_meganz_fail_to_load_problem/ One of the workarounds mentioned was adding Firefox to trusted apps in Avira settings.

Comment 3

[Gian-Carlo Pascutto [:gcp]]

We sent an email to our contacts asking what's up.

It might be useful to know if this was our update triggering an existing bug in Avira (for which it might have been possible to get advance warning), or Avira just pushing out a broken update.

Comment 4

[Gian-Carlo Pascutto [:gcp]]

Bad bot.

Comment 5

[Gian-Carlo Pascutto [:gcp]]

Analysis from Yannis so far indicates Avira is doing Man-In-The-Middle interception of secure traffic (with "Web Protection" enabled) and the fake certificate from their MITM proxy is broken somehow in Firefox 112.

Comment 6

[Gian-Carlo Pascutto [:gcp]]

Setting to S3 for now. Avira is fairly popular but this requires the paid version, so likely only a fraction of those users are having the problem.

Comment 7

[Yannis Juglaret [:yannis] (PTO -> Aug 31)]

Toggling HTTPS traffic analysis on/off in Avira's Web Protection feature enables/disables the issue. The Web Protection feature requires the paid version. Only users using this feature are affected, but very likely all of them. The issue doesn't reproduce in Firefox 111.

Here are profiles recorded with the feature enabled, in 111 (no bug) and 112 (bug).

Network preset

111 - https://share.firefox.dev/42igzej
112 - https://share.firefox.dev/3n5INtD

Firefox preset

111 - https://share.firefox.dev/3L8LdQi
112 - https://share.firefox.dev/3AwvLIK

(Update: see comment 15.)

Comment 8

[Yannis Juglaret [:yannis] (PTO -> Aug 31)]

I don't manage to reproduce anymore. It would be good to ask users if it works for them as well now -- Avira may have acted upon it?

Before, in 112, Twitter's certificate would show with Avira Security NetProtection Root 2 as issuer. Now it shows with DigiCert Inc as issuer.

Comment 9

[Benjamin Beurdouche [:beurdouche]]

I just tried to push a build reverting NSS to version 3.88 (Fx111), here is a link to try:
https://treeherder.mozilla.org/jobs?repo=try&revision=576cbdc18149d6136e6dd1746fcb6a414bc32392

If someone manages to repro with 112 and can use that build to see if it works or not that would give us
some indication on a potential NSS involvement.
B.

Comment 10

[Dana Keeler (she/her) [:keeler]]

(In reply to Yannis Juglaret [:yannis] from comment #8)

I don't manage to reproduce anymore. It would be good to ask users if it works for them as well now -- Avira may have acted upon it?

Before, in 112, Twitter's certificate would show with Avira Security NetProtection Root 2 as issuer. Now it shows with DigiCert Inc as issuer.

That makes it sound like Avira has disabled TLS interception altogether.

Comment 11

[Yannis Juglaret [:yannis] (PTO -> Aug 31)]

Attachment: twitter-111.png

I am still able to reproduce by reinstalling and delaying updates. Attached is what things look like in 111 with the faulty Avira. Which is also what things now look like in 112 with the fixed Avira.

Comment 12

[Yannis Juglaret [:yannis] (PTO -> Aug 31)]

Attachment: twitter-112.png

Attached is what things look like in 112 with the faulty Avira.

Comment 13

[Yannis Juglaret [:yannis] (PTO -> Aug 31)]

I can still reproduce after manually updating. I'm not sure what fixed the issue the first time in comment 8.

Comment 14

[Yannis Juglaret [:yannis] (PTO -> Aug 31)]

On my computer, re-installing Firefox 112 using the official installer seems to fix the issue. The issue seems to reproduce only after updating from 111 to 112 in the browser.

Edit: Actually that only fixes the issue temporarily. Restarting and clearing cache (at least) seem to restore it.

Here are some profiles for huggingface.co:

Firefox preset
OK https://share.firefox.dev/3AtIoEv
KO https://share.firefox.dev/3L86SYU

Network preset
OK https://share.firefox.dev/3oCHTVM
KO https://share.firefox.dev/3Vg7Oiw

Comment 15

[Yannis Juglaret [:yannis] (PTO -> Aug 31)]

Update: Previous messages are wrong. This is not tied to a particular Firefox version. I can reproduce in 108, 109, 110, 111, 112. Right after installation, the first launch will work flawlessly (unless the cache contains the bad result from a previous test). But starting with the second launch, the problem appears (unless the cache contains the good result from a previous test).

Comment 16

[Yannis Juglaret [:yannis] (PTO -> Aug 31)]

During the first launch, a new folder containing a new file appears under C:\Program Files\Mozilla Firefox, namely distribution/policies.json, with contents:

{
  "policies": {
    "Certificates": {
      "ImportEnterpriseRoots": true
    }
  }
}

Deleting this folder and file temporarily removes the issue just like reinstalling. This is the only difference in C:\Program Files\Mozilla Firefox file contents between the first and second launch.

Comment 17

[Yannis Juglaret [:yannis] (PTO -> Aug 31)]

Attachment: avira.cer

Attached is the certificate that Avira installs in the Trusted Root Certification Authorities.

Comment 18

[Yannis Juglaret [:yannis] (PTO -> Aug 31)]

Attachment: no-cert.PNG

Attached is what loading the websites listed in the Reddit thread looks like if I uninstall Avira's certificate (this is not what users experience). Other websites load without problem as shown on the right. To me this suggests that Avira only does MITM on some specific websites, and that the issue is that the data sent back by Avira's MITM is sometimes corrupt or incomplete.

Comment 19

[Yannis Juglaret [:yannis] (PTO -> Aug 31)]

Setting network.http.http2.enabled to false in about:config fixes the issue (thanks [:gijs]).

Comment 20

[Yannis Juglaret [:yannis] (PTO -> Aug 31)]

Network logs (too big for Bugzilla attachments) here.

All taken with network.http.http2.enabled set to true.

{huggingface,twitter}-ko.txt: With distribution/policies.json from comment 16.
{huggingface,twitter}-ok.txt: Without distribution/policies.json from comment 16.

Comment 21

[Yannis Juglaret [:yannis] (PTO -> Aug 31)]

The investigation by [:kershaw] and [:valentin] suggests that this is caused by Avira rewriting some of the content-length headers when doing MITM (somehow only with HTTP/2 enabled in Firefox?) and adding commas to separate thousands, e.g. content-length: 15,336 for 15336, content-length: 70,033 for 70033. Firefox parses those as 15 and 70 respectively which explains the issue.

Comment 22

[Frederik Braun [:freddy]]

FWIW a comma in a header value could also mean that there were two headers of the same name and the values have been combined. E.g.

Content-Length: 10
Content-Length: 20

becomes Content-Length: 10,20

Comment 23

[Yannis Juglaret [:yannis] (PTO -> Aug 31)]

For extra clarity, loading the same websites without Avira yielded content-length: 15336 and content-length: 70033 headers, hence why we are talking about rewriting here.

Comment 25

[Yannis Juglaret [:yannis] (PTO -> Aug 31)]

The tests from above were run on a Windows 10 22H2 machine, winver shows: OS Build 19045.2846. Avira version: 1.1.87.3.

Comment 27

[Yannis Juglaret [:yannis] (PTO -> Aug 31)]

(In reply to Mihai Popa [:papermario] from comment #26)

And Firefox?

See comment 15.

Comment 28

[Yannis Juglaret [:yannis] (PTO -> Aug 31)]

So this issue is a combination of:

• Avira rewriting content-length: 70033 to content-length: 70,033;
• Firefox parsing content-length: 70,033 as content-length: 70 and then trashing the remaining 69963 bytes it received in the HTTP/2 stream (but keeping the 70 first bytes).

Avira has acknowledged the problem on their side and is preparing to ship a patch. On our side the follow-up is bug 1830359.

Comment 32

[Yannis Juglaret [:yannis] (PTO -> Aug 31)]

Avira has started shipping an update yesterday which addresses this issue. I have received the update today and confirm that the problem not longer reproduces. As far as I can tell the update does not change Avira's version number, so it is hard to tell how to check if you have received it. However Avira expects that most users should have received it by Monday.