1424543 - Allow StreamFilter to get and alter charset and contentType

Status: RESOLVED DUPLICATE of bug 1425479

(WebExtensions :: Request Handling, enhancement)

Description

[Oriol Brufau [:Oriol]]

The JSON Viewer always decodes the JSON using UTF-8 even if the server specified something else. This is what the RFC says but not what the user wants, so I created a webextension which fixes it. While doing so I noticed the API lacks some features.

 - There is no direct way to check the contentType. I only want to alter the response if it's application/json or application/token+json. I had to parse the beginning of the "Content-Type" header, this was relatively easy and foolproof.

 - There is no easy way to check the charset parameter specified by the server. I had to parse the parameters of the "Content-Type" header. But parameter values can be quoted and then contain arbitrary strings, so if you want to do it correctly you can't use a simple regex, you need a big parser (see e.g. https://github.com/jshttp/content-type/blob/master/index.js), and it's so easy to do it wrong.

 - While I was testing my add-on, I accidentally installed it twice. The result was so bad, e.g. if the server specifies latin1 charset and sends 0xE0 (i.e. "à"), then
    - The first add-on correctly transforms it into 0xC3 0xA0, which decodes into "à" in UTF-8.
    - The second add-on transforms 0xC3 0xA0 into 0xC3 0x83 0xC2 0xA0, which decodes into "Ã " in UTF-8.
   The second add-on should be able to know that the request was no longer latin1 and it was already converted to UTF-8.

So I think that webextensions should be able to check from which contentCharset and contentType they are converting, and specify to which contentCharset and contentType they are converting to.

The API could work like this:

 - Provide the contentCharset and contentType as custom event properties in the StreamFilter.onstart event listener.

 - For the first webextension handling that request, these would be the values specified by the server. For subsequent ones, the values would be the specified by the previous webextension.

 - Add a new StreamFilter.open method. It would accept an object parameter where a webextension can optionally specify the resulting contentCharset and contentType. If not specified, they won't be altered.

 - The StreamFilter.onstart event listener of the subsequent webextension won't run until StreamFilter.open is called in the current webextension.

 - Calling StreamFilter.write, StreamFilter.close or StreamFilter.suspend would implicitly call StreamFilter.open with no values, if it hasn't been called already.

 - StreamFilter.open should not be called more than once, otherwise it should throw or be ignored.

 - Note a webextension may need to accumulate some data before deciding the resulting contentCharset (e.g. check if there is a BOM), so it should be possible delay the StreamFilter.open call to the ondata or even the onstop event listeners. This could be a breaking change, I think currently all StreamFilter.onstart listeners are called together before waiting the first one to write some data.

 - If no StreamFilter.onstart listener is provided, it can be reasonable to have a default one which calls StreamFilter.open under the hood.

Comment 1

[Rob Wu [:robwu]]

I am marking this as a duplicate of bug 1425479 because that bug has a broader description (besides the issues mentioned here, it also acknowledges the issues caused by content sniffing).

Our API proposals are similar, except I suggest to specify the MIME and charset in the "write" method, whereas you suggest a new "open" method. If you have a preference for one over another, please comment on the other bug :)


PS. Until the API is implemented, if you are looking for a Content-Type parser that parses the type in the same way as Firefox, see https://github.com/Rob--W/open-in-browser/commit/a6b926ea9522b35298632e5e6a2c89ddb456c5d9 (using the Content-Type header from webRequest.onHeadersReceeived ).