Closed Bug 1174100 Opened 9 years ago Closed 5 years ago

Cache information is not exposed to HAR

Categories

(DevTools :: Netmonitor, defect, P3)

Product:
DevTools
Component:
Netmonitor
Type:
defect

Tracking

(firefox69 fixed)

Status:
RESOLVED FIXED
Milestone:
Firefox 69
Tracking Flags:
Tracking Status
firefox69 --- fixed

People

(Reporter: sebo, Assigned: lalas)

References

(Blocks 2 open bugs,
URL
)

Details

(Whiteboard: [webpagetest])

Attachments

(1 file, 1 obsolete file)

Bug 1174100 - Cache information is not exposed to HAR. r=honza
47 bytes, text/x-phabricator-request
Details | Review
Browser cache information for network requests is not correctly exported to HAR. It is either exposed as empty like this: "cache": {} or like this: "cache": { "afterRequest": null } Sebastian
Can I pick this one up? It seems fairly straightforward after 859051.
Yes, assigned to you! Honza
Assignee: nobody → dmccurry
Status: NEW → ASSIGNED
Product: Firefox → DevTools
Whiteboard: [webpagetest]

No activity on this bug over the past 9 months when it was claimed.
Unassigning now.
Feel free to pick it up again if you did intend to work on it still.
This is just to clean our backlog of bugs and make things available for others when they're not being worked on.

Assignee: dmccurry → nobody
Status: ASSIGNED → NEW

Hi all,

I would like to try this, if it's available :)

thanks!

Is this a bug someone is already working on? If Mellina doesn't want to take it on any longer, I would like to take this one on.

Thanks!

Flags: needinfo?(odvarko)

Done!

Honza

Assignee: nobody → lloanalas
Status: NEW → ASSIGNED
Flags: needinfo?(odvarko)

Honza,

Tried a few different approaches and haven't been able to figure this one out. Will work on a few other and put this one on hold. Perhaps we can put it back to unassigned until I circle back in case someone more familiar with the HAR component wants to work on it in the meantime.

Thanks!

Flags: needinfo?(odvarko)
Status: ASSIGNED → NEW
Status: NEW → ASSIGNED

Some pointers:

This is why e.g. buildRequest methdo is async (see in the previous link)
entry.request = await this.buildRequest(file);
... but this.buildCache is not

  • Fetching data from the backend is done through this._options.requestData(file.id, "<data type>");

See an example for fetching "requestHeaders"
https://searchfox.org/mozilla-central/rev/7556a400affa9eb99e522d2d17c40689fa23a729/devtools/client/netmonitor/src/har/har-builder.js#175

So, this needs to be used: this._options.requestData(file.id, "responseCache");

Does that help?

Honza

Flags: needinfo?(odvarko) → needinfo?(lloanalas)

(In reply to Jan Honza Odvarko [:Honza] (always need-info? me) from comment #8)

Some pointers:

This is why e.g. buildRequest methdo is async (see in the previous link)
entry.request = await this.buildRequest(file);
... but this.buildCache is not

  • Fetching data from the backend is done through this._options.requestData(file.id, "<data type>");

See an example for fetching "requestHeaders"
https://searchfox.org/mozilla-central/rev/7556a400affa9eb99e522d2d17c40689fa23a729/devtools/client/netmonitor/src/har/har-builder.js#175

So, this needs to be used: this._options.requestData(file.id, "responseCache");

Does that help?

Honza

It does. I know I did tinker with a few of these files but not the latter functions. I'll tinker with those - thanks Honza!

Flags: needinfo?(lloanalas)

Pushing an update for you to check Honza. Will post questions on the Diff. comments.

Flags: needinfo?(odvarko)

Comments posted to Phab
Honza

Flags: needinfo?(odvarko)

I can also reproduce the problem with undefined cache

Here are the places which should be investigated:

  1. The Cache panel works and it fetches the data here
    https://searchfox.org/mozilla-central/rev/0da35261b6789eec65476dbdd4913df6e235af6d/devtools/client/netmonitor/src/components/CachePanel.js#42-49

In the end it uses the same API as the HAR Builder. But, note that the panel expects the store to be updated with returned data from the backend, which causes the panel to rerender

  1. The HAR Builder uses:
    const responseCache = await this._options.requestData(file.id, "responseCache");

It ends up using FirefoxDataProvider.requestData (just like the CachePanel):
https://searchfox.org/mozilla-central/rev/0da35261b6789eec65476dbdd4913df6e235af6d/devtools/client/netmonitor/src/connector/firefox-data-provider.js#462

But, HARBuilder expects a return value, perhaps it's broken for the cache data?
Another difference is that this.actionsEnabled is set to false during the HAR export
https://searchfox.org/mozilla-central/rev/0da35261b6789eec65476dbdd4913df6e235af6d/devtools/client/netmonitor/src/har/har-exporter.js#205

... to make HAR export faster. This could also be part of the problem...?

It's used in requestData and it means that actions are not fired and reducer not updated:
https://searchfox.org/mozilla-central/rev/0da35261b6789eec65476dbdd4913df6e235af6d/devtools/client/netmonitor/src/connector/firefox-data-provider.js#477

  1. The backend is collecting cache data here:
    https://searchfox.org/mozilla-central/rev/0da35261b6789eec65476dbdd4913df6e235af6d/devtools/server/actors/network-monitor/network-response-listener.js#393

And loading the cache data from the Browser cache here:
https://searchfox.org/mozilla-central/rev/0da35261b6789eec65476dbdd4913df6e235af6d/devtools/server/actors/network-monitor/network-response-listener.js#313

You should:

  1. Create console.logs around the places above
  2. Use Browser debugger to debug the places above

Mainly figure out why the CachePanel works well and HARBuilder doesn't

Honza

Flags: needinfo?(lloanalas)

Honza,

Those are great ideas. I've made some progress, going to keep looking at the CachePanel, feels like I might be taking longer than I should on this one, lol.

Thanks!

Flags: needinfo?(lloanalas)
Pushed by jodvarko@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/b4f0ee392b97 Cache information is not exposed to HAR. r=Honza

Backed out changeset b4f0ee392b97 (bug 1174100) for failing at browser_net_har_import.js on a CLOSED TREE.

Backout link: https://hg.mozilla.org/integration/autoland/rev/bef3fa175d99de7fef6e1c34e94d6eba50719f35

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&selectedJob=251899940&revision=b4f0ee392b9771796370b8f2b3194abba3599ff6

Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=251899940&repo=autoland&lineNumber=7744

Log snippet:

[task 2019-06-14T15:17:38.601Z] 15:17:38 INFO - TEST-PASS | devtools/client/netmonitor/src/har/test/browser_net_har_import.js | There must be the same number of keys for: log/entries/2 -
[task 2019-06-14T15:17:38.602Z] 15:17:38 INFO - TEST-PASS | devtools/client/netmonitor/src/har/test/browser_net_har_import.js | There must be the same keys for: log/entries/2 -
[task 2019-06-14T15:17:38.602Z] 15:17:38 INFO - Buffered messages finished
[task 2019-06-14T15:17:38.606Z] 15:17:38 INFO - TEST-UNEXPECTED-FAIL | devtools/client/netmonitor/src/har/test/browser_net_har_import.js | There must be the same number of keys for: log/entries/2/cache - Got 1, expected 0
[task 2019-06-14T15:17:38.607Z] 15:17:38 INFO - Stack trace:
[task 2019-06-14T15:17:38.607Z] 15:17:38 INFO - chrome://mochikit/content/browser-test.js:test_is:1324
[task 2019-06-14T15:17:38.608Z] 15:17:38 INFO - chrome://mochitests/content/browser/devtools/client/netmonitor/src/har/test/browser_net_har_import.js:compare:86
[task 2019-06-14T15:17:38.610Z] 15:17:38 INFO - chrome://mochitests/content/browser/devtools/client/netmonitor/src/har/test/browser_net_har_import.js:compare:128
[task 2019-06-14T15:17:38.611Z] 15:17:38 INFO - chrome://mochitests/content/browser/devtools/client/netmonitor/src/har/test/browser_net_har_import.js:compare:128
[task 2019-06-14T15:17:38.611Z] 15:17:38 INFO - chrome://mochitests/content/browser/devtools/client/netmonitor/src/har/test/browser_net_har_import.js:compare:118
[task 2019-06-14T15:17:38.611Z] 15:17:38 INFO - chrome://mochitests/content/browser/devtools/client/netmonitor/src/har/test/browser_net_har_import.js:null:71
[task 2019-06-14T15:17:38.612Z] 15:17:38 INFO - chrome://mochikit/content/browser-test.js:Tester_execTest/<:1115
[task 2019-06-14T15:17:38.613Z] 15:17:38 INFO - chrome://mochikit/content/browser-test.js:Tester_execTest:1143
[task 2019-06-14T15:17:38.614Z] 15:17:38 INFO - chrome://mochikit/content/browser-test.js:nextTest/<:1004
[task 2019-06-14T15:17:38.614Z] 15:17:38 INFO - chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<:803
[task 2019-06-14T15:17:38.615Z] 15:17:38 INFO - Not taking screenshot here: see the one that was previously logged

Flags: needinfo?(lloanalas)

I'll take a look at that test and update it - thanks!

Flags: needinfo?(lloanalas)
Attached patch har-cache.patch (obsolete) — — Splinter Review

Some analysis in the attached patch.
Honza

One more thing. The HAR spec says that custom fields need to start with underscore
http://www.softwareishard.com/blog/har-12-spec

So, all additional cache fields need to follow that rule.

  • _dataSize
  • _lastModified
  • _device

Honza

Pushed by jodvarko@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/49ba71fb412d Cache information is not exposed to HAR. r=Honza

https://hg.mozilla.org/mozilla-central/rev/49ba71fb412d

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
status-firefox69: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → Firefox 69
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: