1248454 - Developer Tools: Network tab - HTTP Archive (HAR) files not properly tracking pages

Status: RESOLVED FIXED

(DevTools :: Netmonitor, defect, P3)

Description

[Michael Dunagan]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.97 Safari/537.36

Steps to reproduce:

I'm on Mac OS (El Capitan), and did the following:
1. Open Firefox (44.0.2)
2. Open a new tab
3. Right-click some whitespace on the page
4. Choose "Inspect Element"
5. Switch to the Network tab of the developer tools
6. Navigate to http://httpbin.org
7. Then navigate to https://www.google.com/
8. Finally navigate to https://www.mozilla.org/
9. Right-click the request log in the Network tab
10. Choose "Save All as HAR"


Actual results:

Viewing the HAR file in a text editor reveals one page object in the pages array. Each request in the entries array is associated with the single page object.

Current Firefox Example:

    "pages": [
      {
        "startedDateTime": "2016-02-15T10:31:07.291-07:00",
        "id": "page_1",
        "title": "Mozilla Firefox Start Page",
        "pageTimings": {
          "onContentLoad": -1,
          "onLoad": -1
        }
      }
    ],
    "entries": [
      {
        "pageref": "page_1",
        ...other parts of the requests were removed for brevity...
      },
      ...pages was shortened to a single request for brevity...
      {
        "pageref": "page_1",
      },
      {
        "pageref": "page_1",
      }
    ]


Expected results:

The pages array should contain three objects. One object for each home page that was navigated to.

Current Chrome Example:

    "pages": [
      {
        "startedDateTime": "2016-02-15T18:14:00.456Z",
        "id": "page_1",
        "title": "http://httpbin.org/",
        "pageTimings": {
          "onContentLoad": 534.1219999827445,
          "onLoad": 2035.769000183791
        }
      },
      {
        "startedDateTime": "2016-02-15T18:14:05.921Z",
        "id": "page_2",
        "title": "http://www.google.com/",
        "pageTimings": {
          "onContentLoad": 905.0619998015463,
          "onLoad": 904.8279998824
        }
      },
      {
        "startedDateTime": "2016-02-15T18:14:13.535Z",
        "id": "page_3",
        "title": "http://www.firefox.com/",
        "pageTimings": {
          "onContentLoad": 1544.1089998930693,
          "onLoad": 1649.2790002375841
        }
      }
    ],
    "entries": [
      {
        "pageref": "page_1",
        ...other parts of the requests were removed for brevity...
      },
      ...pages was shortened to a single request for brevity...
      {
        "pageref": "page_2",
      },
      {
        "pageref": "page_3",
      }
    ]

Oddly enough, Microsoft's Edge browser has a similar issue, but it is much worse. They're completely missing the pages array and the pageref for each request.

Comment 1

[Joe Walker [:jwalker]]

Doing bug triage with Emma.
Filter on LOBSTER-THERMADOR

Comment 2

[Michael Dunagan]

This is still an issue in the newest version of Firefox. (v46.0.1) Has anyone had a chance to follow my steps to reproduce the issue?

Comment 3

[turkeburgr]

Similar behavior was reported in the following bugs as well.  Just wanted to reference them here.

bugs 1190213, 1238345

Comment 4

[Ricky Smith]

Attachment: single pageref.har.json

It appears to be even worse in v53 on OSX: I am seeing a single pageref for multiple sites. In this HAR, I navigated to yahoo, then google, then bing. All of the entries in the HAR have the same pageref (page_3).

Comment 5

[jmk]

I was trying to understand tracking behavior with some sites and just discovered this issue which lead me to read this spec (I'm not sure this is the last one) :
https://w3c.github.io/web-performance/specs/HAR/Overview.html#sec-object-types-pages

According to this doc the page and the pageref sections should be empty when the Har creator is not able to group request by pages, as it is the case now in firefox.

Maybe it would be better to follow the spec and remove the said section/references until this bug is closed ?

Failure to do so could lead har tool creators to think their code is buggy while the problem stands with the firefox exporter ...

Comment 6

[Julian Descottes [:jdescottes]]

Attachment: Bug 1248454 - [devtools] Handle multiple navigations in devtools har builder

Just a WIP to keep track of several pages in devtools har builder

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f059a6b74120
[devtools] Handle multiple navigations in devtools har builder r=devtools-reviewers,bomsy

Comment 8

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/f059a6b74120