1590898 - filterResponseData completely corrupting webpages content encoding

Status: VERIFIED FIXED

(WebExtensions :: Request Handling, defect, P1)

Description

[particlecore]

Attachment: test.zip

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0

Steps to reproduce:

This issue is present in Nightly 72.0a1 (2019-10-23)

Implement filterResponseData with and without any modification to the stream itself.

Example test extension example attached to verify the issue.

Install it and then open youtube.com.

Actual results:

The page Content Encoding gets completely corrupted

See the following issue for more reports from extension users: https://github.com/ParticleCore/Iridium/issues/788

Expected results:

The page should not become corrupted and any modifications done to the stream should go through without issues.

Comment 1

[Miruna Curtean]

Managed to reproduce using STR on Windows 10 Pro x 64 bit and macOS catalina 10.15 only On Nightly browser, tested on Nightly 72.0a1 (20191024214023 & 20191025095546).
Also tested using the link from 1587960, https://www.youtube.com/watch?v=YzbARjL1o-s, and the a "Content encoding error" is shown.

Comment 2

[Shane Caraveo (:mixedpuppy)]

Miruna, could you validate whether this is the same problem with the info I posted in bug 1587960 comment 6? I'm actually unable to reproduce this bug right now.

Comment 3

[Shane Caraveo (:mixedpuppy)]

I updated an now I can reproduce the problem. This is not related to bug 1587960.

When I updated I noticed an update to ICU 65 in Bug 1583269, I'm not certain if that is the cause, but it seems it could be related.

Miruna, can you do a mozregression on this?

Comment 4

[Thomas Wisniewski [:twisniewski]]

I just ran mozregression, and got bug 1583700 as the culprit.

Comment 6

[Matt Woodrow (:mattwoodrow)]

Attachment: Bug 1590898 - Forward stream filter creation requests across PDocumentChannel. r?kmag

Comment 8

[Matt Woodrow (:mattwoodrow)]

Shane, could you please help me with writing a test for this?

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a4c51f3f0bb1
Forward stream filter creation requests across PDocumentChannel. r=kmag

Comment 10

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/a4c51f3f0bb1

Comment 11

[Miruna Curtean]

Verified fixed with Windows 10 Pro x 64 bit and macOS catalina 10.15 on Firefox 72.0a1 (20191104214406) and now after loading the test extension and Youtube can be opened successfully.

Comment 12

[Shane Caraveo (:mixedpuppy)]

Bug 1591809 indicates it's simple to produce using https://github.com/mdn/webextensions-examples/blob/master/http-response/background.js

My guess is that it is the replacing of the data that caused the problem, but I'm not sure. A simple test using a background script similar to that should produce it.

I tried something I thought should fail, but it doesn't. So I'm missing some aspect of what is going on.

https://gist.github.com/mixedpuppy/8698a7d40bda0dd461793a3533011a43

Comment 13

[Guido Berhoerster]

The actual problem is that gzip encoded content is no longer decompressed before being fed into the ondata handler of the filter which then later triggers an exception when it is being fed into the TextDecoder.

Comment 14

[Shane Caraveo (:mixedpuppy)]

Attachment: Bug 1590898 test encoding in webRequest.filterResponseData

Comment 15

[Shane Caraveo (:mixedpuppy)]

Guido, thanks!

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8ff3fc3e27c5
test encoding in webRequest.filterResponseData r=mattwoodrow

Comment 17

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/8ff3fc3e27c5