Content Encoding Error received for a file with brotli content-encoding
Categories
(Core :: Networking, defect, P2)
Tracking
()
People
(Reporter: ksenia, Assigned: manuel)
References
(Blocks 1 open bug, )
Details
(Whiteboard: [necko-triaged][necko-priority-queue] )
Attachments
(2 files)
STR:
Open https://catering.panerabread.com/app.126cf6644357ca5f5c12.js and observe the page.
Expected:
File contents displayed on the page
Actual:
Content Encoding Error received
Response headers for this file have content-encoding: br. The file loads successfully in Chrome.
It's probably worth mentioning that content-type is missing in response headers, wonder if that could be the reason of the error?
|
||
Comment 1•3 years ago
•
|
||
Interesting. A different number of bytes is transferred every time I try. It's gotten close to transferring all data (~500 out of 540kB) but never made it all the way.
panerabread.com is working, but catering.panerabread.com (which is linked directly in the site's landing page) does not load, due to this issue.
It's possible that we are depending on a content-type being available here and not defaulting to e.g. text/plain, but this does not seem to be mandated or suggested or otherwise covered in the spec - so I think we should probably follow Chrome for webcompat here.
edit: so if we ignore the brotli compression due to this, it's possible that we misinterpret some encoded bytes as an EOF for example.
|
||
Comment 2•3 years ago
|
||
If we do not have a content type we do sniffing (unknownDecoder). Before sniffing we decode data (remove brotli decoding).
This all should just work.
Nihanth, do you want to make a log and try to figure out what is happening? Look for what is canceling the nsHttpChannel. You can try with debugging as well since there will not be too many requests.
|
|
||
Comment 3•3 years ago
|
||
I can look into this this week, keeping the needinfo for tracking.
|
|
||
Updated•3 years ago
|
|
||
Comment 4•3 years ago
|
||
I think this is not a P1, since I can reproduce this on ESR 78.
|
|
||
Updated•3 years ago
|
|
|
||
Comment 5•3 years ago
|
||
New URL to reproduce: https://catering.panerabread.com/app.b24c00b7d6636877d6ab.js
|
|
||
Comment 6•3 years ago
|
||
I can confirm that injecting a Content-Type: text/plain (or text/javascript) response header fixes the issue.
I'm working on figuring out what's different when this response header is absent when it comes to brotli decoding - we seem to be trying to decode only a chunk of size 512 and then bailing out.
In the meantime, I wonder if we can make an assumption or guess for the Content-Type when it is absent.
|
|
||
Comment 7•3 years ago
|
||
(In reply to Nihanth Subramanya [:nhnt11] from comment #6)
I can confirm that injecting a Content-Type: text/plain (or text/javascript) response header fixes the issue.
I'm working on figuring out what's different when this response header is absent when it comes to brotli decoding - we seem to be trying to decode only a chunk of size 512 and then bailing out.
When Content-Type is not present nsUnknownDecoder is involved. The unknown-decode uses only first 512bytes (up to 512 byte) to sniff the content and decide what it is. If content is encoded it decodes the content. There is a test for gzip you can try to make one for brotli as well.
In the meantime, I wonder if we can make an assumption or guess for the Content-Type when it is absent.
nsUnknownDecoder is doing the guessing and I am wondering why it is failing.
|
Reporter | |
Updated•3 years ago
|
|
|
||
Comment 8•3 years ago
|
||
Tried to pick this back up but can't reproduce anymore because the app.foo.js file is now gzip encoded. Interestingly the Content-Type header is still absent.
I tried manually removing the Content-Type header from another file that is served brotli-encoded: https://pagecdn.io/lib/vue/2.6.11/vue.min.js
In this case, the Save File dialog opened and I was able to download the file (i.e. Firefox did not try to display the content) but the downloaded file seemed fine - no decoding errors.
So... not sure if there's any point of keeping this open :-)
|
||
Comment 9•2 years ago
|
||
It would be good to have a test case for this.
James suggests to set a test case on WPT with a compressed file.
|
|
||
Comment 10•2 years ago
|
||
On the other hand the site is right now working. so removing the Webcompat priority
|
|
||
Updated•2 years ago
|
|
|
||
Updated•2 years ago
|
|
||
Comment 11•1 year ago
|
||
From the description this sounds like the bug I just ran into when making a .NET 7 Blazor WASM application, so can confirm it's still here in Firefox 107.0 - it fails at decoding most of the DLLs if a Content-Type header is not present, but succeeds if any content-type header is present.
I'm attaching a caddyserver based repro "project" - it will listen on https://localhost/ with a self signed cert by default, and browsing to it will be enough to trigger the issue.
|
|
||
Comment 12•1 year ago
|
||
|
Assignee | |
Updated•1 year ago
|
|
||
Comment 14•1 year ago
|
||
The severity field for this bug is set to S3. However, the following bug duplicate has higher severity:
- Bug 1818005: S2
:manuel, could you consider increasing the severity of this bug to S2?
For more information, please visit auto_nag documentation.
|
|
Assignee | |
Comment 15•1 year ago
|
||
I would keep S3, because the workaround of sending a Content-Type: plain/text-header with the response exists.
|
|
Assignee | |
Comment 16•1 year ago
•
|
||
More observations:
- The error resulting in the channel to be cancelled is introduced in nsHTTPCompressConv::OnStopRequest. The state from decoding is still BROTLI_DECODER_RESULT_NEEDS_MORE_INPUT:
mBrotli->mState.error_code == 2. - The code was discussed here on what the statement should be and what to return: https://bugzilla.mozilla.org/show_bug.cgi?id=1242904#c15 (permalink to the dead github link: https://github.com/google/brotli/blob/d4f0cb98411491ade114158f363d8693e89ab473/dec/decode.c#L1843)
I think either the statement or the return value should be changed to fix this bug. E.g. adding mBrotli->mState.error_code != 2 to the if statement makes the channel not be cancelled anymore. The detected Content-Type would be octet-stream.
I'll look into what fix is the correct one (keeping all tests passing).
The one time it gets decompressed, it asks for more data and decompresses 0 bytes: https://searchfox.org/mozilla-central/rev/ad732108b073742d7324f998c085f459674a6846/netwerk/streamconv/converters/nsHTTPCompressConv.cpp#215,223
2023-04-12 13:49:41.025037 UTC - [Parent 206166: Main Thread]: D/nsHttp converter removed 'br' content-encoding
2023-04-12 13:49:41.025104 UTC - [Parent 206166: Main Thread]: D/nsHttp nsHttpCompresssConv 7f7a04543c00 onstart
2023-04-12 13:49:41.025183 UTC - [Parent 206166: Main Thread]: D/nsHttp nsHttpCompressConv 7f7a04543c00 OnDataAvailable 512
2023-04-12 13:49:41.025334 UTC - [Parent 206166: Main Thread]: D/nsHttp nsHttpCompresssConv 7f7a04543c00 brotlihandler decompress 512
2023-04-12 13:49:41.025490 UTC - [Parent 206166: Main Thread]: D/nsHttp nsHttpCompresssConv 7f7a04543c00 brotlihandler decompress rv=2 out=0
2023-04-12 13:49:41.025616 UTC - [Parent 206166: Main Thread]: D/nsHttp nsHttpCompresssConv 7f7a04543c00 onstop 0
2023-04-12 13:49:41.025679 UTC - [Parent 206166: Main Thread]: D/nsHttp nsHttpCompresssConv 7f7a04543c00 onstop brotlihandler rv 804b001b
2023-04-12 13:49:41.025796 UTC - [Parent 206166: Main Thread]: D/nsHttp HttpBaseChannel::InterceptFailedOnStop 7f7a1f451200 seting status 804b001b
|
|
Assignee | |
Comment 17•1 year ago
|
||
Highly compressed brotli files contain meta blocks that don't produce
output from the first 512 Bytes. That is because the meta block header
might be longer than 512 Bytes.1 NBLTYPESL, NBLTYPESI, NBLTYPESD,
NTREESL, NTREESD all can blow up the header size. Each at least by
2*256 bits.
With this solution the Content-Type would be always detected as
octet-stream. That might not be the correct one. If a different
Content-Type is required, the server can still specify it with the
Content-Type-Header.2
This solution looks directly at the error code to determine invalid
encoding instead of relying on number of output bytes.
|
||
Comment 19•11 months ago
|
||
Tried to pick this back up but can't reproduce anymore because the app.foo.js file is now gzip encoded. Interestingly the Content-Type header is still absent.
Maybe a better self-contained test case can be found in https://bugzilla.mozilla.org/show_bug.cgi?id=1829309 . Might have been better to close this issue as duplicate of 1829309 given that has a reproducible test case.
Currently we have a continuous influx of Unity game developers bitten by this bug, since we encourage all developers to compress their Unity game sites with Brotli. Some developers miss setting the Content-Type field since after all it is not strictly needed for anything, and then they get a cryptic error in Firefox that they have no ability to debug to think that Firefox would be failing to uncompress Brotli.
I would argue raising this to be a high priority issue if there is bandwidth to tackle it.
|
||
Comment 20•10 months ago
|
||
Pushed by mbucher@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/d65747d25190 Fix highly compressed brotli responses with unknown content-type being rejected r=necko-reviewers,valentin
|
||
Comment 21•10 months ago
|
||
| bugherder | ||
|
||
Updated•10 months ago
|
|
||
Comment 22•9 months ago
|
||
I can not reproduce the issue using the links from description and comment#5 (they return 404), I also tried the attached files from bug 1829309 but same issue page did not load.
Reporter, can you please confirm issue is not reproducing using latest build https://archive.mozilla.org/pub/firefox/candidates/116.0-candidates/build1/win64/
Thank you.
|
|
Reporter | |
Comment 23•9 months ago
|
||
Looks like the site has changed and the file in question (https://catering.panerabread.com/app.b7e58e7a91fbb3381bb6.js) no longer gets served with brotli content-encoding, but with gzip.
We can close this as fixed based on the comment 4 from bug1829309 I believe.
Description
•