Closed Bug 1278923 Opened 8 years ago Closed 8 years ago

Exported HAR format is missing Content-Type and Content-Length headers

Categories

(DevTools :: Netmonitor, defect, P2)

Product:
DevTools
Component:
Netmonitor
Version:
47 Branch
Type:
defect

Tracking

(firefox51 fixed)

Status:
RESOLVED FIXED
Milestone:
Firefox 51
Tracking Flags:
Tracking Status
firefox51 --- fixed

People

(Reporter: huseyint, Assigned: rickychien)

References

Details

(Whiteboard: [good taipei bug])

Attachments

(2 files)

Test case to reproduce
219 bytes, text/html
Details
Bug 1278923 - Export Content-Type and Content-Length to HAR headers
58 bytes, text/x-review-board-request
Honza
: review+
Details 
User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36

Steps to reproduce:

Submitted a form with POST method and using Network tab of F12 developer tools, I have copied the request as HAR format.


Actual results:

The following data is copied:

{
  "log": {
    "version": "1.1",
    "creator": {
      "name": "Firefox",
      "version": "47.0"
    },
    "browser": {
      "name": "Firefox",
      "version": "47.0"
    },
    "pages": [
      {
        "startedDateTime": "2016-06-08T17:58:25.773+03:00",
        "id": "page_1",
        "title": "http://ht-laptop/multipartencoding/",
        "pageTimings": {
          "onContentLoad": -1,
          "onLoad": -1
        }
      }
    ],
    "entries": [
      {
        "pageref": "page_1",
        "startedDateTime": "2016-06-08T17:58:25.773+03:00",
        "time": 97,
        "request": {
          "bodySize": 86,
          "method": "POST",
          "url": "http://ht-laptop/multipartencoding/process.aspx",
          "httpVersion": "HTTP/1.1",
          "headers": [
            {
              "name": "Host",
              "value": "ht-laptop"
            },
            {
              "name": "User-Agent",
              "value": "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0"
            },
            {
              "name": "Accept",
              "value": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
            },
            {
              "name": "Accept-Language",
              "value": "en-US,en;q=0.5"
            },
            {
              "name": "Accept-Encoding",
              "value": "gzip, deflate"
            },
            {
              "name": "Referer",
              "value": "http://ht-laptop/multipartencoding/"
            },
            {
              "name": "Connection",
              "value": "keep-alive"
            }
          ],
          "cookies": [],
          "queryString": [],
          "postData": {
            "mimeType": "application/x-www-form-urlencoded",
            "params": [],
            "text": "Content-Type: application/x-www-form-urlencoded\r\nContent-Length: 15\r\n\r\nfoo=123&bar=456"
          },
          "headersSize": 359
        },
        "response": {
          "status": 200,
          "statusText": "OK",
          "httpVersion": "HTTP/1.1",
          "headers": [
            {
              "name": "Cache-Control",
              "value": "private"
            },
            {
              "name": "Content-Length",
              "value": "3"
            },
            {
              "name": "Content-Type",
              "value": "text/html; charset=utf-8"
            },
            {
              "name": "Date",
              "value": "Wed, 08 Jun 2016 14:58:25 GMT"
            },
            {
              "name": "Server",
              "value": "Microsoft-IIS/10.0"
            },
            {
              "name": "X-AspNet-Version",
              "value": "4.0.30319"
            }
          ],
          "cookies": [],
          "content": {
            "mimeType": "text/html; charset=utf-8",
            "size": 3,
            "text": "123"
          },
          "redirectURL": "",
          "headersSize": 196,
          "bodySize": 3
        },
        "cache": {},
        "timings": {
          "blocked": 0,
          "dns": 0,
          "connect": 1,
          "send": 0,
          "wait": 96,
          "receive": 0
        },
        "serverIPAddress": "fe80::600d:85b4:9543:6a4b",
        "connection": "80"
      }
    ]
  }
}


Expected results:

Expected Content-Type and Content-Length request headers exist on "request.headers" array. They exist (though) on postData.text as first 2 lines, strangely.
Component: Untriaged → Networking
Product: Firefox → Core
Component: Networking → Developer Tools
Product: Core → Firefox
Component: Developer Tools → Developer Tools: Netmonitor
The content is put after the HTTP headers. The format of an HTTP POST is to have the HTTP headers, followed by a blank line, followed by the request body. The POST variables are stored as key-value pairs in the body.

In any case, can you please attach a test case I can use to check this out on my machine?

Honza
Flags: needinfo?(huseyint)
Priority: -- → P3
Attached file Test case to reproduce — 
Here you go, please find it attached.
Flags: needinfo?(huseyint)
Thanks!

So, as mentioned in comment #1, Content-Type and Content-Length are part of the upload stream (POST request body). But I agree that they should be presented in the `request.headers` array of generated HAR.

I general, request headers from upload stream should be inserted into request.headers array.

Honza
Status: UNCONFIRMED → NEW
Ever confirmed: true
Priority: P3 → P2
Whiteboard: [good taipei bug]
Assignee: nobody → rchien
Status: NEW → ASSIGNED
New Content-Type and Content-Length will be added in her-builder.js and patch is still WIP
Patch was ready and I'm going to submit r?Honza request once he comes back from PTO on 8/8.
Comment on attachment 8778831 [details]
Bug 1278923 - Export Content-Type and Content-Length to HAR headers

https://reviewboard.mozilla.org/r/68506/#review68690

See my comment inline.

Thanks for fixing this!
Honza

::: devtools/client/netmonitor/har/har-builder.js:209
(Diff revision 1)
> +    if (!file.requestPostData) {
> +      return input;
> +    }
> +
> +    this.fetchData(file.requestPostData.postData.text).then(value => {
> +      let contentType = value.match(/Content-Type: (.+)/);

The regex should check for `;` character that's marking end of the content-type name. Otherwise you'll end up with content type name containing also `boundary=-` indentifier.

Honza
Attachment #8778831 - Flags: review?(odvarko)
What I saw for postData.text is 

"Content-Type: application/x-www-form-urlencoded\r\nContent-Length: 15\r\n\r\nfoo=123&bar=456"

There is no any ';' character appear within the text. Or do I misunderstand you?
Flags: needinfo?(odvarko)
(In reply to Ricky Chien [:rickychien] from comment #9)
> What I saw for postData.text is 
> 
> "Content-Type: application/x-www-form-urlencoded\r\nContent-Length:
> 15\r\n\r\nfoo=123&bar=456"
> 
> There is no any ';' character appear within the text. Or do I misunderstand
> you?
Try `multipart/form-data` content type (simple form submit)

Honza
Flags: needinfo?(odvarko)
Thanks for your information! 

Patch has been updated and content type of `multipart/form-data` is able to be displayed properly now.
Comment on attachment 8778831 [details]
Bug 1278923 - Export Content-Type and Content-Length to HAR headers

https://reviewboard.mozilla.org/r/68504/#review69076
Comment on attachment 8778831 [details]
Bug 1278923 - Export Content-Type and Content-Length to HAR headers

Both 'application/x-www-form-urlencoded' and 'multipart/form-data' work for me now.

Thanks!
Honza
Attachment #8778831 - Flags: review?(odvarko) → review+
Pushed by ryanvm@gmail.com:
https://hg.mozilla.org/integration/fx-team/rev/c73cad6f6773
Export Content-Type and Content-Length to HAR headers. r=Honza
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/c73cad6f6773
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
status-firefox51: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → Firefox 51
Honza, should the HAR export really extract only the Content-Type and Content-Length headers from the POST data and ignore any others? There can be Content-Disposition, Content-Transfer-Encoding etc.

In NetMonitorView [1], there is code that does the same thing, i.e., extracts the headers from the upload streams and shows them in the request details, but it doesn't filter them and shows them all.

I'd like to remove the duplicate code from har-builder.js and use the already existing methods from Curl.jsm.

[1] http://searchfox.org/mozilla-central/source/devtools/client/netmonitor/netmonitor-view.js#1660
Flags: needinfo?(odvarko)
(In reply to Jarda Snajdr [:jsnajdr] from comment #17)
> Honza, should the HAR export really extract only the Content-Type and
> Content-Length headers from the POST data and ignore any others? There can
> be Content-Disposition, Content-Transfer-Encoding etc.
Good point, these headers should be exported too.

> In NetMonitorView [1], there is code that does the same thing, i.e.,
> extracts the headers from the upload streams and shows them in the request
> details, but it doesn't filter them and shows them all.
> 
> I'd like to remove the duplicate code from har-builder.js and use the
> already existing methods from Curl.jsm.
Sounds good to me.

Please, file a new bug and cc me.

Thanks!
Honza
Flags: needinfo?(odvarko)
Depends on: 1297082
(In reply to Jan Honza Odvarko [:Honza] from comment #18)
> Please, file a new bug and cc me.

OK, fixing in bug 1297082.
Product: Firefox → DevTools
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: