Closed Bug 1250255 Opened 4 years ago Closed 2 years ago

Dev Tools > Network > Response, on JSON responses, shows error and base64 encodes content

Categories

(DevTools :: Netmonitor, defect, P2)

46 Branch
defect

Tracking

(firefox57 fix-optional, firefox58 fixed)

RESOLVED FIXED
Firefox 58
Tracking Status
firefox57 --- fix-optional
firefox58 --- fixed

People

(Reporter: lucas.werkmeister, Assigned: Honza)

Details

Attachments

(1 file)

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0
Build ID: 20160220004011

Steps to reproduce:

Open https://foaas.com/operations or any other URL that will yield a JSON response.

Open Dev Tools, go to Network Panel, reload the page to get the request, open Response tab on the right


Actual results:

Above the Response content, the following is shown:

> SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data

The Response text itself is not shown as plain JSON, but instead encoded in base64 format. The content type is also not application/json, as returned by the server, but instead application/vnd.mozilla.json.view.

If the JSON contains characters from Latin-1 Supplement (U+0080 - U+00FF) – example: http://lucaswerkmeister.de/files/2016/02/umlaut.json – they are embedded as-is in the base64, not UTF-8 encoded. That is to say, the base64 content is ISO 8859-1 encoded.

If the JSON contains characters outside of the ISO 8859-1 range – example: http://lucaswerkmeister.de/files/2016/02/snowman.json – the Response is completely empty, the SyntaxError is gone, and the Type, Transferred and Size fields are also empty.


Expected results:

Firefox Dev Tools should show the plain JSON text. No transcoding should happen. Responses should never appear to be completely empty when they are not.
WFM in Fx45b6 46a2 47a1 on Win10.
Component: Untriaged → Developer Tools: Netmonitor
Do you have a special text encoding chosen for these pages?  As in, is View menu -> Text Encoding set to something other than "Unicode"?

Another way to check is to try a clean Firefox profile.  Does the issue occur there?

https://support.mozilla.org/en-US/kb/profile-manager-create-and-remove-firefox-profiles#w_creating-a-profile
Flags: needinfo?(lucas.werkmeister)
I do not have any special encoding set, and I was able to reproduce this both in my main profile and a fresh, clean profile.

However, there is a difference between those profiles. In my main profile, the main web content “frame” is blank (white). In the fresh profile, prettified JSON showed up instead, with proper Unicode and everything. But in both cases, Dev Tools Network view didn’t show the request properly.
Flags: needinfo?(lucas.werkmeister)
Priority: -- → P2
I'm too facing the same issue on master@cc67c49 (Git)

Can I pick this?
This bug still occurs for me in nightly 58. I looked at the raw data in wireshark (after turning off gzip and ssl) and it clearly is plain text and not base64.

STR
- Open Network monitor in new tab
- Navigate to http://jsonplaceholder.typicode.com/posts
- Click on the /posts request to open it
- Select the 'Response' tab
- Scroll past the json section
- Look at Response Payload section

Expected
- Raw response payload

Actual
- Base64 (?) encoded data

The JSON view Raw Data tab does correctly show raw data.
@Oriol: is JSON Viewer doing some special decoding?

Honza
Flags: needinfo?(oriol-bugzilla)
This problem does not happen in non-e10s.

(In reply to Jan Honza Odvarko [:Honza] from comment #6)
> @Oriol: is JSON Viewer doing some special decoding?

No, currently the JSON Viewer just spits the received input as it is: http://searchfox.org/mozilla-central/rev/298033405057ca7aa5099153797467eceeaa08b5/devtools/client/jsonview/converter-child.js#68-70
Flags: needinfo?(oriol-bugzilla)
Interesting is that if I disable JSON Viewer (in about:config) it works.

Also, the mime type is `application/json; charset=utf-8` if JSON Viewer disabled.
It's `application/vnd.mozilla.json.view; charset=utf-8` when JSON Viewer is enabled.

Also encoding is properly set to "base64" if JSON Viewer is enabled so, it could be relatively simple to fix in Net panel.

But, I don't understand why JSON Viewer affects the response.

Honza
Assignee: nobody → odvarko
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Comment on attachment 8914611 [details]
Bug 1250255 - Properly decode response body;

https://reviewboard.mozilla.org/r/185948/#review190928

::: devtools/client/netmonitor/src/connector/firefox-data-provider.js:155
(Diff revision 1)
>        if (mimeType.includes("image/")) {
>          payload.responseContentDataUri = formDataURI(mimeType, encoding, response);
>        }
>  
> +      if (encoding === "base64") {
> +        response = b64DecodeUnicode(response);

Since decode takes time and we don't need to show response Content while load requests.

For performance concern I'd suggest doing the actual decoding in `response-panel`, which also has `encoding` info.
Attachment #8914611 - Flags: review?(gasolin)
Comment on attachment 8914611 [details]
Bug 1250255 - Properly decode response body;

https://reviewboard.mozilla.org/r/185948/#review190930

::: devtools/client/netmonitor/src/utils/request-utils.js:120
(Diff revision 1)
> + * See also: https://developer.mozilla.org/en-US/docs/Web/API/WindowBase64/Base64_encoding_and_decoding
> + *
> + * @param {string} url - a string
> + * @return {string} decoded string
> + */
> +function b64DecodeUnicode(str) {

I know the function name is copied from MDN, but `base64DecodeUnicode` will be more clear while reading.
(In reply to Fred Lin [:gasolin] from comment #10)
> For performance concern I'd suggest doing the actual decoding in
> `response-panel`, which also has `encoding` info.
Done

(In reply to Fred Lin [:gasolin] from comment #11)
> I know the function name is copied from MDN, but `base64DecodeUnicode` will
> be more clear while reading.
I renamed it to `decodeUnicodebase64`, so it's more inline with `decodeUnicodeUrl`

Also, the patch decodes only "application/vnd.mozilla.json.view" responses since that's the original reported problem. Responses for standard requests/mime-types should stay as is, so the user can see the raw data as it's been received from the server.

Honza
Comment on attachment 8914611 [details]
Bug 1250255 - Properly decode response body;

https://reviewboard.mozilla.org/r/185948/#review191738

::: devtools/client/netmonitor/src/utils/request-utils.js:118
(Diff revision 2)
> + * to original string.
> + *
> + * @param {string} url - a string
> + * @return {string} decoded string
> + */
> +function decodeUnicodebase64(string) {

nit: `decodeUnicodeBase64` (cap B)

::: devtools/client/netmonitor/src/utils/request-utils.js:120
(Diff revision 2)
> + * @param {string} url - a string
> + * @return {string} decoded string
> + */
> +function decodeUnicodebase64(string) {
> +  try {
> +    return decodeURIComponent(atob(string));

The explaination seems incorrect since there's no percent-encoding here (Decode base64 string. From bytestream, to percent-encoding)

Previously the function comes from https://developer.mozilla.org/en-US/docs/Web/API/WindowBase64/Base64_encoding_and_decoding

```
decodeURIComponent(atob(str).split('').map(function(c) {
        return '%' + ('00' + c.charCodeAt(0).toString(16)).slice(-2);
    }).join(''));
```

Not sure why it's simplified.
Attachment #8914611 - Flags: review?(gasolin) → review+
(In reply to Fred Lin [:gasolin] from comment #14)
> nit: `decodeUnicodeBase64` (cap B)
Ah, yes fixed.

> ::: devtools/client/netmonitor/src/utils/request-utils.js:120
> The explaination seems incorrect since there's no percent-encoding here
Fixed

> Not sure why it's simplified.
I could make the http://lucaswerkmeister.de/files/2016/02/umlaut.json
to work with the previous version.

Thanks for the review!

Honza
(In reply to Jan Honza Odvarko [:Honza] from comment #16)
> I could make the http://lucaswerkmeister.de/files/2016/02/umlaut.json
> to work with the previous version.
Sorry, I meant "I could not make ..."
Honza
Pushed by jodvarko@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/64fe29d5b248
Properly decode response body; r=gasolin
https://hg.mozilla.org/mozilla-central/rev/64fe29d5b248
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 58
Product: Firefox → DevTools
You need to log in before you can comment on or make changes to this bug.