1557795 - Request headers/cookies/params is not visible until request is finished

Status: NEW

(DevTools :: Netmonitor, enhancement, P3)

Description

[Eugen Konkov]

Attachment: Kazam_screencast_00001.mp4

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0

Steps to reproduce:

I send post request to a slow server

Actual results:

I can not see POSTed data

Expected results:

I should see POSTed data immediately after request is sent

Comment 1

[Daniel Bodea [:danibodea]]

Can you provide clear and simple steps to reproduce? I need to reproduce the issue myself in order to confirm it. Thank you!

Comment 2

[Eugen Konkov]

just send request to server which responds after 30s for example.
Thus you have time to notice that posted data is not visible at dev.tools

Comment 3

[Daniel Bodea [:danibodea]]

I have attempted to reproduce the issue using the following steps:

1. Took this server from: http://biercoff.com/simple-http-server-that-will-give-serve-simple-files-and-can-emulate-delay-before-giving-response/
2. Modified the code to delay the server response with 15-30-60 seconds.
3. Ran it with Python 2.
4. Opened a page from the server with Firefox and navigated around the local server.
   -> Could not reproduce the behavior seen in the video attached, could not see the "Params" section of the post event being populated at all.
   I do not have the necessary technical knowledge to reproduce that. I had to take quite some time to do "trial and error" until I reproduced this much. Please help me further. Thanks.

Comment 4

[Eugen Konkov]

I create the server with 7sec timeout. Try: http://193.151.89.164:8080/ff

Comment 5

[Eugen Konkov]

Attachment: Kazam_screencast_00005.mp4

Here is the demo for URL above

Comment 6

[Daniel Bodea [:danibodea]]

I can confirm this issue using the server link in comment 4. It occurs on Nighty v69.0a1 (2019-07-01), Beta v68.0b14 and Release v67.0.4.
I am setting component to (Core) Networking.
Reporter: Thank you for help. the server could be closed if it troubles in any way, but I will need it working again if the bug gets fixed. Thanks!

Comment 7

[Eugen Konkov]

would be nice if it will be fixes ASAP, because, probably, I will do not remember details later. or have no access to server etc.
I have already such reports where after a year or so I was asked additional info.
I am glad to help, but memory, resources etc =(

And thank you to confirm it +1

Comment 8

[Daniel Bodea [:danibodea]]

Awesome. You could share the server code here and I will have it when verification is needed. Thanks!

Comment 9

[Eugen Konkov]

something like this:

sub ff_post {
sleep 7;
shift->render( 'response text is here' );
}

Comment 10

[Kershaw Chang [:kershaw]]

I am not familiar with devtool/netmonitor code.

Honza, could you take a look? Thanks.

Comment 11

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Thanks for the report.

Analysis

STR:

1. Load http://193.151.89.164:8080/ff
2. Press the button on the page
3. See new entry/request in the panel content
4. Select the Request and see the Params panel
5. Post data are available when the request finishes -> BUG
   Expected: post data can be visible in the Params panel since the requests is sent.

• The back-end is collecting the POST data here:
  https://searchfox.org/mozilla-central/rev/15be167a5b436b57fef944b84eef061d24c1af8c/devtools/server/actors/network-monitor/network-observer.js#491

It sends a notification about the POST data being available to the client/Network panel immediately

• The Network panel handles the notification/update events here:
  https://searchfox.org/mozilla-central/rev/15be167a5b436b57fef944b84eef061d24c1af8c/devtools/client/netmonitor/src/connector/firefox-data-provider.js#390

• The update event (informing the Network panel that POST data body is available) is stored in a queue
  https://searchfox.org/mozilla-central/rev/15be167a5b436b57fef944b84eef061d24c1af8c/devtools/client/netmonitor/src/connector/firefox-data-provider.js#433

Important is that the ${updateType}Available flag is stored in the queue - indicating that specific data/updateType (like e.g. requestPostData) are available on the backend for pickup (the data itself needs to be fetched explicitly).

The Params panel does request the requestPostData explicitly here (when the panel is created):
https://searchfox.org/mozilla-central/rev/15be167a5b436b57fef944b84eef061d24c1af8c/devtools/client/netmonitor/src/components/ParamsPanel.js#67

using fetchNetworkUpdatePacket util function

• The fetchNetworkUpdatePacket is implemented here:
  https://searchfox.org/mozilla-central/rev/15be167a5b436b57fef944b84eef061d24c1af8c/devtools/client/netmonitor/src/utils/request-utils.js#91

And it only fetches the data if request[${updateType}Available] == true, which isn't => BUG

This means that we set the flag (about request post data being available) in the FirefoxDataProvider, but it isn't propagated to Requests reducer and thus fetchNetworkUpdatePacket has no effect.

Honza

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

The priority flag is not set for this bug.
:Honza, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 17

[Sebastian Zartner [:sebo]]

As the previous test case links don't work anymore, here is a new one (GET request):
http://www.mocky.io/v2/5e84bfbb300000bd0597abe8?mocky-delay=10000ms

If that link breaks after some time, a new request can easily be created from the homepage of Mocky again.

Note that this affects all request methods, not only POSTs. What should be available at the time of the request are at least headers and payload information, parameters (already shown), the call stack (probably already shown, not possible to test with Mocky and can't test it locally right now), and partially the timings.

Sebastian

Comment 19

[:Harald Kirschner :digitarald]

This also makes SSE debugging feel really awkward, as most tabs stay empty while data is coming in.

Comment 23

[jtmiller]

I would love for this to be resolved. Was sharing my screen the other day working through a thing, and my coworker mentioned that you can see the request right away in Chrome, where as I expected this was just normal behavior in all browsers.

No real details needed I'd think, it's simply that working in FF devtools, having your webapp fire an XHR request, you do not see anything in the Request/Headers tab until the request finishes.

Comment 24

[javolek]

Just to complement Sebastians comment, it would be meaningfull to update any request detail as soon as it is available. E.g. HTTP-Status and Response-Headers should also be displayed as soon as they are available to browser, without waiting for the request to complete. And as already mentioned by Harald - to be able to debug SSE, it is essential to see all messages as they come without waiting for the connection to close.

It may have already been working, haven't it? https://firefox-source-docs.mozilla.org/devtools-user/network_monitor/inspecting_server-sent_events/index.html

I was always preferring Firefox Dev Tools over Chromes but the fact that this bug is still not fixed and I need to debug SSE is forcing me to switch to Chrome. It's a pitty.