1560525 - GZIP usage in 67.0.3 compresses json

Status: RESOLVED WORKSFORME

(Core :: Networking: HTTP, defect, P5)

Description

[katja.suvajac]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0

Steps to reproduce:

Our webpage (run by Bloomreach CMS) uses enterprise forms to receive data from visitors, that wish to apply for jobs. A while back we implemented GZIP, to make our webpage run faster.
Our original configuration was

•    <Connector port="8080" protocol="HTTP/1.1"
  

•               connectionTimeout="20000"
  

•               redirectPort="8443" compression="on" />
  

For some reason .SVG files were not compressed with this configuration, so we added

• compression="force".

Actual results:

After form completion and upon pressing the submission button, the form goes through a check. FF ziped the result of our CMS integrated validation. The response was in zipped form, instead of the expected JSON. Because of that, the validation failed. The user experienced this as an unability to submit the form.

Expected results:

We asume that in some casses, the browser fails to uncompress data.
We have tried to reproduce said bug in Chrome v75.0.3770.100 and Safari v12.1.1 (12607.2.6.1.2) - in said browsers we did not encounter the same problem.

Comment 1

[Hani Yacoub, Desktop QA]

I am going to assign the "Firefox: File Handling" component for it and hopefully someone with more knowledge in this area will a look over this.

Comment 2

[:Gijs (he/him)]

(In reply to katja.suvajac from comment #0)

Our webpage (run by Bloomreach CMS) uses enterprise forms to receive data from visitors, that wish to apply for jobs. A while back we implemented GZIP, to make our webpage run faster.
Our original configuration was

•    <Connector port="8080" protocol="HTTP/1.1"
  

•               connectionTimeout="20000"
  

•               redirectPort="8443" compression="on" />
  

For some reason .SVG files were not compressed with this configuration, so we added

• compression="force".

After form completion and upon pressing the submission button, the form goes through a check. FF ziped the result of our CMS integrated validation. The response was in zipped form, instead of the expected JSON. Because of that, the validation failed. The user experienced this as an unability to submit the form.

Can you link to a live testcase, or more information to reproduce the issue? When you say "zipped" do you mean "gzipped"? What does the form look like, and what does the request Firefox makes to your server look like (when intercepted with wireshark or similar) ? Is it using http2? How does the request compare with the request other browsers make?

Comment 3

[katja.suvajac]

Hello,
thank you for your quick response.
Unfortunately, we can't give you a link to a live test case, as this issue was causing trouble in our production environment. So currently, the forced compression is turned off.
Yes, when I wrote "zipped", I meant "gzipped". We are using https. I can give you a link to our form in our production environment, so you can see, what kind of forms we are talking about.
https://www.petrol.si/obrazec/povprasevanje-ogrevalne-toplotne-crpalke#daikin-altherma-lt
I'm not sure if this is important - we are using reCAPTCHA v3 in our forms (https://developers.google.com/recaptcha/docs/v3).
I can have more data for you in the beggining of next week. We will try to recreate this issue localy and provide you with the additional information you asked for.

Kind regards,
Katja

Comment 4

[katja.suvajac]

Attachment: Eforms-firefox-bug.pdf

Hello,

we have some additional information you have asked for, please check the pdf attached.
If you have additional questions, don't hesitate to ask.

Comment 5

[:Gijs (he/him)]

Thanks, the pdf helps. I'm afraid that I'm not familiar with the CMS on the server side of this, but from the headers reported by the devtools, it looks like the server claims that it's sending gzip. Not just Content-Encoding: gzip and Content-Type: application/json, but it's sending gzip for both of those (ie "application/x-gzip" for the Content-Type header). I believe that's what's going wrong. For instance, when using the REST API on this bugzilla instance, the response headers look like this:

HTTP/1.1 200 OK
Access-control-allow-headers: accept, content-type, origin, user-agent, x-requested-with, x-bugzilla-token, x-bugzilla-password, x-bugzilla-login, x-bugzilla-api-key
Access-Control-Allow-Origin: *
Content-Encoding: gzip
Content-Security-Policy-Report-Only: default-src 'self'; worker-src 'none'; connect-src 'self' https://product-details.mozilla.org https://www.google-analytics.com https://treeherder.mozilla.org/api/failurecount/ https://crash-stats.mozilla.org/api/SuperSearch/; font-src 'self' https://fonts.gstatic.com; img-src 'self' data: blob: https://secure.gravatar.com; object-src 'none'; script-src 'self' 'nonce-VURPbP1QmgTsShdkD5nIaDto3g4NZgGdl8KparSxrtiRA7bS' 'unsafe-inline' https://www.google-analytics.com; style-src 'self' 'unsafe-inline'; frame-src https://crash-stop-addon.herokuapp.com; frame-ancestors 'self'; form-action 'self' https://www.google.com/search https://github.com/login/oauth/authorize https://github.com/login
Content-Type: application/json; charset=UTF-8
Date: Tue, 02 Jul 2019 14:07:41 GMT
Referrer-policy: same-origin
Set-Cookie: <omitted>
Status: 200
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-content-type-options: nosniff
X-frame-options: SAMEORIGIN
X-xss-protection: 1; mode=block
transfer-encoding: chunked
Connection: keep-alive

Note how Content-Encoding is set to gzip but the Content-Type still clarifies that it's sending JSON.

When you run the XHR to your website in Chrome (or Edge/Safari or any other browser that does work), does the server send different headers or does the browser manage to make sense of what's going on and interpret the response as JSON despite the header saying it's just gzip'd content?

Comment 6

[:Gijs (he/him)]

Given that this seems more about networking than local file handling, moving over. Katja, if you have more information / answers to the questions in comment 5 that would still be helpful. Thank you!

Comment 7

[katja.suvajac]

Hello,
we will provide the additional data you asked for in the beggining of the next week.
Thank you!

Kind regards,
Katja

Comment 8

[katja.suvajac]

Attachment: FirefoxChromeResponseHeaders.pdf

We have tried the problematic POST call again in Firefox and Chrome and here are the outputs. Firefox updated this morning and is now setting Content Type to application/x-gzip (same as Chrome). Outputs for both browsers will be added in PDF.

Kind regards,
Katja

Comment 9

[:Gijs (he/him)]

Dragana, looking at the compared headers/responses in the latest attachment, any idea what's going wrong here? Maybe https://searchfox.org/mozilla-central/rev/b9041f813de0a05bf6de95a145d4e25004499517/netwerk/protocol/http/nsHttpChannel.cpp#5779 ?

Comment 10

[katja.suvajac]

Hello,
just checking, if there is any progress on this issue?

Kind regards,
Katja

Comment 11

[Honza Bambas (:mayhemer)]

Been 6 months. Let's move this back to the triage pool.

Comment 12

[Andy Grover [:grover]]

Valentin do you have any bandwidth to look at this?

Comment 13

[Valentin Gosu [:valentin]]

This was probably caused by ClearBogusContentEncodingIfNeeded, but that has been removed now (bug 1030660).
@Reporter, please let us know if you're still seeing this issue.