1832779 - gzipped text content in network monitor's request pane is not readable

Status: NEW

(DevTools :: Netmonitor, defect, P3)

Description

[:Gijs (he/him)]

What were you doing?

1. open Firefox (I tested on current nightly on macOS)
2. open browser toolbox
3. switch to network monitor
4. switch to new tab page in Firefox
5. click any pocket recommended story
6. in the network monitor, inspect any of the POST requests to incoming.telemetry.mozilla.org
7. in the right hand pane, switch to "request"

What happened?

The request pane displays binary unreadable things as text.

What should have happened?

It should show the text content - the content type for the request header is application/json; charset=UTF-8, but content-encoding is gzip, so presumably this is the gzipped request content instead of the un-gzipped request body text.

Comment 1

[Julian Descottes [:jdescottes]]

We attempt to decompress response bodies, but we are not doing the same thing for request bodies. We can check if we could share some logic between the two codepaths.

Comment 2

[Julian Descottes [:jdescottes]]

It seems that compression for requests is very rarely supported in the wild because it could expose servers to receive malicious payloads which are highly compressed. So in practice it's very rarely enabled or used, which is probably why this was never on the roadmap.

I could not reproduce with the STRs here, but you can simply craft a fetch request such as:

fetch("https://example.com/", {
      method: 'POST',
      mode: 'cors',
      headers: {
        'Content-Encoding': 'gzip',
        'Content-Type': 'application/json'
      },
      body: "eJwtjktuwzAMBe/CtRGQlESKvkoQBIwltSqaD2I7myB3jxt0O/MweE9IE0/eUknmlcW1WOHSaBJypVrNY8gpFIFx/4ReYCRFpqQcaYC5f118We/1+Gc4YDCKA/xcTx8QSYOo4ba8rfP3hxHHTLqRe330uV8vMIK5cmqiSUtTD3JK1pAwG0sMGLMhs+cc4L+z9HOdFz/ftpxs7xKjyQAP/10rjLiT+Dq83lphPTY=",
});

I tried porting our response logic

  readAndConvertFromStream(stream, charset, request) {
    let text = null;
    try {
      if (request) {
        const channel = request.QueryInterface(Ci.nsIChannel);
        const encodingHeader = channel.getRequestHeader("Content-Encoding");
        const scs = Cc["@mozilla.org/streamConverters;1"].getService(
          Ci.nsIStreamConverterService
        );
        const encodings = encodingHeader.split(/\s*\t*,\s*\t*/);
        const acceptedEncodings = [
          "gzip",
          "deflate",
          "br",
          "x-gzip",
          "x-deflate",
        ];
        for (const i in encodings) {
          // There can be multiple conversions applied
          const enc = encodings[i].toLowerCase();
          if (acceptedEncodings.indexOf(enc) > -1) {
            stream = scs.convert(stream, enc, "uncompressed", null);
          }
        }
      }

      text = lazy.NetUtil.readInputStreamToString(stream, stream.available());

      return this.convertToUnicode(text, charset);
    } catch (err) {
      return text;
    }
  },

But I can't get scs.convert to work, I keep getting Uncaught NS_ERROR_NOT_IMPLEMENTED: Component returned failure code: 0x80004001 (NS_ERROR_NOT_IMPLEMENTED) [nsIStreamConverterService.convert]

jesup: I could not find examples for using nsIStreamConverterService.convert from chrome js, only for the async variant. But here I already have the stream so I would like to simply use convert. Do you have any example or do you know what could be wrong?

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

There are no converters that implement ::Convert() -- nsDeflateConverter, nsHttpCompressConv, nsIndexedToHTML, nsMultiMixedConv, nsUnknownDecoder don't support ::Convert(), and mozTXTToHTMLConv doesn't support either Convert or AsyncConvertData().

So I think you'll have to use the async variant. Can you file bugs on the others, or at least one bug on the lot?

Comment 4

[Julian Descottes [:jdescottes]]

(In reply to Randell Jesup [:jesup] (needinfo me) from comment #3)

There are no converters that implement ::Convert() -- nsDeflateConverter, nsHttpCompressConv, nsIndexedToHTML, nsMultiMixedConv, nsUnknownDecoder don't support ::Convert(), and mozTXTToHTMLConv doesn't support either Convert or AsyncConvertData().

So I think you'll have to use the async variant. Can you file bugs on the others, or at least one bug on the lot?

Thanks I filed Bug 1835924. This is not necessarily a blocker though I imagine we can workaround and use the async version if we want to push for this.