1159078 - Display a useful message when a response body is truncated

Status: VERIFIED FIXED

(DevTools :: Netmonitor, defect)

Description

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

We limit response logging to 1 MB.  http://mxr.mozilla.org/mozilla-central/source/toolkit/devtools/webconsole/network-monitor.js?force=1#32  When we hit this limit we should display a useful message rather than e.g. failing to parse a JSON blob.  Something like "response too large to display", etc.

Comment 1

[Sebastian Zartner [:sebo]]

In Firebug the response limit is controlled through extensions.firebug.netDisplayedResponseLimit. If that value is exceeded, the following message is shown:

"Firebug response size limit has been reached. Click [here] to open the entire response in a new Firefox tab."

Clicking on 'here' opens the response as data URL in a new browser tab.

Sebastian

Comment 2

[nadir merouane]

Still not really solved. How do we use the ability to bypass the limit in a javascript code. Actually I am loading multiples JSON from ADWORDS XML->JSON through php and cache files exposing the JSON in gzip compression to the javascript in a worker process using XHR object. Then I uncompress the zip content into variable using packo library and vue-worker. Then I transmit the datas to the parent process doing the JSON.parse. But with the 1MB limit my json is truncated at runtime. It is statistical data from AdWords about keywords and the size of the DATA in JSON format is 9MB. 

In Chrome and EDGE the JSON.parse on the 9MB structure to publish the data into boostrap vue tables is working fine and quick without many memory consomption

How can I achevied that with Firefox? I would not like to recommand to my 40 000 companies clients to not use firefox because it can't display their adwords datas in their web interface... 

PS: the whole loading time without caching is around 4 sec and 0.8s with caching under chrome for 5 adwords reports in real time campaign, account, adgroups, ad, keywords (the big one).

Regards.

Comment 3

[Sebastian Zartner [:sebo]]

Attachment: Error message for truncating big JSON

Nadir, I think your issue is a little different to the one covered here. Yours rather sounds like bug 1281490. This one here is to display a proper message within the response preview of the Network panel when a response is clipped, independent of whether the response is JSON data or not.

Nonetheless, I've attached a screenshot of what's currently displayed within the response preview when selecting a request.

Sebastian

Comment 4

[Sebastian Zartner [:sebo]]

Attachment: JSON file larger than 1MB

And to be able to quickly reproduce this issue, I've attached a JSON file which is a bit bigger than 1MB.

Steps to reproduce the issue:
1. Open the Network panel
2. Load the attached JSON
3. Select the request
4. Switch to the "Response" tab

Sebastian

Comment 5

[Martin Koroknay]

Hi, I'd be happy to take this on as long as I could get some pointers please. The code the original reporter is referencing is no longer applicable, so it would be great to know where the code responsible for limiting response logging to 1MB lives.

Comment 6

[Sebastian Zartner [:sebo]]

I am not the best person to ask about the related code, as I am currently not much involved in DevTools development, but as far as I can see the limitation of the code is happening in /devtools/server/actors/network-monitor/network-response-listener.js controlled by the devtools.netmonitor.responseBodyLimit preference and the preparation of the data and its display is done in /devtools/client/netmonitor/src/components/ResponsePanel.js. I redirected your request to Honza for further information on what exactly needs to be done, as he's the "master of the Network Monitor". :-)

Sebastian

Comment 7

[Jan Honza Odvarko [:Honza] (always need-info? me)]

@Martin: assigned to you.
@Sebastian: thanks for the pointers.

---

Some more info:

1. Sebastian's links are correct

2. Here is the place where the current error message is rendered
   https://searchfox.org/mozilla-central/rev/e00ea598e52bbb35f8c45abf9c2eade17962bb5e/devtools/client/netmonitor/src/components/ResponsePanel.js#236

3. I think we could display an error message (instead of the current error) saying e.g. "Response has been truncated" (any other better suggestions for the message?)

4. The code can use the following condition to figure out whether the response has been truncated.

// In the render method
const limit = Services.prefs.getIntPref("devtools.netmonitor.responseBodyLimit");
if (limit <= request.responseContent.content.size) { ... }

Honza

Comment 8

[Martin Koroknay]

Attachment: Bug 1159078 - Display a useful message in network monitor response tab when response body is truncated. r=Honza

Previously when a JSON response's size was over the limit specified in pref devtools.netmonitor.responseBodyLimit, when inspecting this response in the Network Monitor one simply saw a JSON parse error message. This was due to the fact that the response was truncated in this view and thus no longer a valid JSON, even if the actual response was a valid JSON. Of course the actual response could have been invalid in the first place -- there was no way to know.

This patch will ensure that in such cases a truncation warning message is displayed instead of a parse error message, thus avoiding a likely misleading parse error message.

Comment 9

[Martin Koroknay]

Thanks for the pointers Sebastian & Honza. I have submitted my patch for review, for now with the text "Response has been truncated." displaying.

Comment 10

[Jan Honza Odvarko [:Honza] (always need-info? me)]

@Martin: please look at my comments in Phabricator

Thanks,
Honza

Comment 11

[Martin Koroknay]

Hi Honza, I have updated the diff in Phabricator.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/24a48c472242
Display a useful message in network monitor response tab when response body is truncated. r=Honza

Comment 13

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/24a48c472242

Comment 15

[Anca Soncutean, Desktop QA]

I reproduced this issue with Fx 66.0a1 (2018-12-17) on Windows 10 x64.
The desired message ( "Response has been truncated") is displayed with Fx 67.0b8 on Windows 10 x64, Ubuntu 18.04 x64 and macOS 10.13.