704983 - Make request and response body logging more discoverable in the Web Console

Status: RESOLVED FIXED

(DevTools :: Console, enhancement)

Description

[Michael Coates [:mcoates] (acct no longer active)]

Issue

The Web Console is a very helpful developer tool. It also can be used for security inspection of data exchanged in requests and responses.  However, one big missing piece of data is the POST data.  Currently web console includes the request URL, Method, headers, cookies and response headers but not the actual data sent as part of the POST message.

Steps to Demonstrate
1. Browse to any site with a form that POSTs (e.g. login page https://addons.mozilla.org/en-US/firefox/users/login)
2. Enable Web Console
3. Submit the form 
4. Click on the post link within the Web Console window and observe the POST entry 

Recommended Remediation
Include POST data within the detailed data captured within the Web Console. It would be fine if this opened another window due to the size of the POST'ed data. Or, perhaps displayed in the normal window until a max size and then included a link.  Either way, being able to see this data would make the tool more usable.

Comment 1

[Mihai Sucan [:msucan]]

Please right-click in the Web Console and enable logging of network request and response bodies. You should get the POST data. If not, then it's a bug. Please retest. Thank you!

Comment 2

[Michael Coates [:mcoates] (acct no longer active)]

(In reply to Mihai Sucan [:msucan] from comment #1)
> Please right-click in the Web Console and enable logging of network request
> and response bodies. You should get the POST data. If not, then it's a bug.
> Please retest. Thank you!

You are correct. After right-clicking a menu screen appeared that provided an option to log precisely this data.  Thanks!

I've been using Web Console for a variety of things and I never encountered this menu nor thought to right click in that space, that's not intuitive to me at all.  Should I file a separate bug with a recommendation for a UI change (e.g. maybe add a visible options button to the left of the "Position" dropdown) or should I morph this one to this new request?

Comment 3

[Mihai Sucan [:msucan]]

(In reply to Michael Coates [:mcoates] from comment #2)
> (In reply to Mihai Sucan [:msucan] from comment #1)
> > Please right-click in the Web Console and enable logging of network request
> > and response bodies. You should get the POST data. If not, then it's a bug.
> > Please retest. Thank you!
> 
> You are correct. After right-clicking a menu screen appeared that provided
> an option to log precisely this data.  Thanks!
> 
> I've been using Web Console for a variety of things and I never encountered
> this menu nor thought to right click in that space, that's not intuitive to
> me at all.  Should I file a separate bug with a recommendation for a UI
> change (e.g. maybe add a visible options button to the left of the
> "Position" dropdown) or should I morph this one to this new request?

Either way is fine for me (rename the bug or open a new bug).

Your concern is valid, about the option being too hidden - it's not easy to find the checkbox.

Maybe we should add a checkbox to the network panel where users could toggle the logging of request and response bodies. That would be much more discoverable. Thoughts?

Comment 4

[Paul Rouget [:paul]]

(In reply to Mihai Sucan [:msucan] from comment #3)
> Maybe we should add a checkbox to the network panel where users could toggle
> the logging of request and response bodies. That would be much more
> discoverable. Thoughts?

I think it's a very good idea.

Comment 5

[Rob Campbell [:rc] (:robcee)]

My one concern with moving this option into the Network filter button menu is that it lends the option to be storable via prefs (since the new filter settings preferences just landed). I'm worried about people forgetting about this setting and having it impact their browser performance.

Still, I'd like this option to be more discoverable so my concern is probably outweighed by the usability win.

Comment 6

[Mihai Sucan [:msucan]]

(In reply to Rob Campbell [:rc] (robcee) from comment #5)
> My one concern with moving this option into the Network filter button menu
> is that it lends the option to be storable via prefs (since the new filter
> settings preferences just landed). I'm worried about people forgetting about
> this setting and having it impact their browser performance.

I suggested having the checkbox to enable the logging of request and response bodies inside the network panel - when the user inspects network requests and sees there are no bodies. Would this be better?

Putting the checkbox in the Network filter button dropdown menu sounds acceptable as well.

Comment 7

[Girish Sharma [:Optimizer]]

May be we can do this : Have a text saying that Response Body was not logged. Click <a>here</a> to start logging.

And also we will have an option in the Network filter menu popup.

I am ready to take this bug, if its okay ?

Comment 8

[Mihai Sucan [:msucan]]

(In reply to Girish Sharma [:Optimizer] from comment #7)
> May be we can do this : Have a text saying that Response Body was not
> logged. Click <a>here</a> to start logging.

Instead of "Click <a>here</a> to start logging." you can put a checkbox "Enable network request and response body logging".

> And also we will have an option in the Network filter menu popup.
> 
> I am ready to take this bug, if its okay ?

Sure! Please let me know if you have any questions!

Thank you!

Comment 9

[Girish Sharma [:Optimizer]]

So before starting work on this, I wanted to confirm the final decision.

There will be two places where the "Log Request and Response Bodies" will be placed:

1) In the Network Panel, which is opened by the user upon clicking the HTTP line in the Console.

2) Under the "Net" Filter on the top filter toolbar.

Would it be better if it is under the "Logging" Filter instead of "Net" Filter? It will make more sense there.

Comment 10

[Mihai Sucan [:msucan]]

Girish: this is something the UX experts need to tell us. I don't know what's best for us to do.

Shorlander: can you please give us your input? Please see comment 9. Thank you!

Comment 11

[Paul Rouget [:paul]]

(In reply to Girish Sharma [:Optimizer] from comment #9)
> 2) Under the "Net" Filter on the top filter toolbar.

What about what Rob said in Comment 5?

Comment 12

[Girish Sharma [:Optimizer]]

(In reply to Paul Rouget [:paul] from comment #11)
> (In reply to Girish Sharma [:Optimizer] from comment #9)
> > 2) Under the "Net" Filter on the top filter toolbar.
> 
> What about what Rob said in Comment 5?

Yes, users can forget it checked and that will affect the performance. One things that can be done, is that not to store this preference under a pref, and let it be disabled on every web console awakening.

Since we are making it more discoverable, it should be easily reachable and the user will explicitly switch it on if he needs.

Comment 13

[Girish Sharma [:Optimizer]]

Shorlander: ping for ideas.

Comment 14

[Rob Campbell [:rc] (:robcee)]

(In reply to Girish Sharma [:Optimizer] from comment #12)
> (In reply to Paul Rouget [:paul] from comment #11)
> > (In reply to Girish Sharma [:Optimizer] from comment #9)
> > > 2) Under the "Net" Filter on the top filter toolbar.
> > 
> > What about what Rob said in Comment 5?
> 
> Yes, users can forget it checked and that will affect the performance. One
> things that can be done, is that not to store this preference under a pref,
> and let it be disabled on every web console awakening.

I thought about this a bit and this could be a good solution. I was initially thinking we should update and respect the preference, but on further discussion with Mihai, we decided this might make more sense to be an on-demand thing.

> Since we are making it more discoverable, it should be easily reachable and
> the user will explicitly switch it on if he needs.

Maybe our link in the network panel should say something like, "Click here to enable request and response logging and reload the page". This would result in the network panel closing, the setting updated and then the page reloads with bodies intact.

What do we think about this?

Comment 15

[Girish Sharma [:Optimizer]]

(In reply to Rob Campbell [:rc] (:robcee) from comment #14)
> Maybe our link in the network panel should say something like, "Click here
> to enable request and response logging and reload the page". This would
> result in the network panel closing, the setting updated and then the page
> reloads with bodies intact.
> 
> What do we think about this?

Sounds good. So finally it should be in which all places?

1) In the Network Panel
2) ... ? (Net filter or Logging filter ?

Comment 16

[Rob Campbell [:rc] (:robcee)]

marking as unassigned. No activity.

Comment 17

[Girish Sharma [:Optimizer]]

(In reply to Rob Campbell [:rc] (:robcee) from comment #16)
> marking as unassigned. No activity.

I am free to take this, if a final decision is made. My last comment asks for a decision only :) .

Comment 18

[Girish Sharma [:Optimizer]]

So I am going forward with this design:

1) the Net drop-down menu popup contains a new entry below Error and Log saying "Request & Response bodies" (trying to be as short as possible.
2) it has a tool-tip text explaining that on clicking this, page will reload.
3) this is unchecked by default on every web console start (not sure about this yet, but this is what I got from comments).

Comment 19

[Girish Sharma [:Optimizer]]

(In reply to Rob Campbell [:rc] (:robcee) from comment #14)
> Maybe our link in the network panel should say something like, "Click here
> to enable request and response logging and reload the page". This would
> result in the network panel closing, the setting updated and then the page
> reloads with bodies intact.
> 
> What do we think about this?

Why should it reload the page?. The current behavior is not like this.

Comment 20

[Mihai Sucan [:msucan]]

(In reply to Girish Sharma [:Optimizer] from comment #19)
> (In reply to Rob Campbell [:rc] (:robcee) from comment #14)
> > Maybe our link in the network panel should say something like, "Click here
> > to enable request and response logging and reload the page". This would
> > result in the network panel closing, the setting updated and then the page
> > reloads with bodies intact.
> > 
> > What do we think about this?
> 
> Why should it reload the page?. The current behavior is not like this.

I wouldn't like the page to reload. Enabling request and response body logging should have no direct impact on the web app state. A reload can be very disruptive.

Comment 21

[Girish Sharma [:Optimizer]]

(In reply to Mihai Sucan [:msucan] from comment #20)
> I wouldn't like the page to reload. Enabling request and response body
> logging should have no direct impact on the web app state. A reload can be
> very disruptive.

I agree, dropping the reload part then.

Comment 22

[Girish Sharma [:Optimizer]]

Attachment: Patch with changes to existing tests so that they pass

I was also thinking of reducing the length of the label from "Log resp..." to just "Save Bodies" or something similar but shorter than existing one.
Will do if you feel it right.

Comment 23

[Mihai Sucan [:msucan]]

Comment on attachment 659326 [details] [diff] [review]
Patch with changes to existing tests so that they pass

Review of attachment 659326 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks for the patch Girish! Finally we'll fix this bug!

One problem I noticed: the menuitem is disabled when I turn off network > log, but I keep network > errors enabled. Maybe you don't really need to disable the menu item at all - it's an option that affects network logging irrespective of filters. When you toggle the filters back on, you see the option still had an impact on network request that were logged, during the time network messages were hidden.

Also, one concern: should we remove the existing context menu item? It breaks user's habit... Rob/Paul, what do you think? I'd say we don't remove the menuitem from the context menu, at least two releases.

Giving the patch an r+, but I want the menuitem fix and please get a second opinion for the contextmenu item removal or not. I vote "no". :)

Comment 24

[Girish Sharma [:Optimizer]]

Attachment: Added the context menu entry back

Carry forward r+ and pushing to try.

Comment 25

[Nick Fitzgerald [:fitzgen] [⏰PST; UTC-8]]

https://tbpl.mozilla.org/?tree=Try&rev=b9981f6b99df

Comment 26

[Girish Sharma [:Optimizer]]

Try is (almost) green , with only some XPCshellTest failing on win Opt and WinXP Debug. I don't think that it is this patch, so adding the whiteboard.

Comment 27

[Paul Rouget [:paul]]

https://hg.mozilla.org/integration/fx-team/rev/305673036867

Comment 28

[Joe Walker [:jwalker]]

https://tbpl.mozilla.org/?rev=8b46964e55c9

Comment 29

[M Spreij]

Using this feature a *lot*, webdev/startup yaddayadda, and would love for the setting to stick..

I guess I'm 2 years late, but is there any remote possibility that this will be ever be something you can toggle via about:config to have the default be "on"?
As for performance impact, maybe there could be a setting that switched it to "on" for the current window whenever the user opens a webconsole? And the setting could be visible in the webconsole somewhere, when you open it, so (hopefully) you don't forget about it.