1740116 - HAR file export results in an empty file

Status: VERIFIED FIXED

(DevTools :: Netmonitor, defect, P2)

Description

[Kuba Orlik]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0

Steps to reproduce:

1. Open https://wiadomosci.onet.pl/bialystok/na-granicy-z-bialorusia-sa-tysiace-migrantow-na-zywo/sbkd6j8
2. Open Devtools, navigate to network panel
3. Refresh the page
4. Click the cog -> Save all as HAR
5. Save the file to disk

Actual results:

The file is empty (0B)

Expected results:

The file should be a JSON containing the information on all requests performed by the website

Comment 1

[Kuba Orlik]

The stack in the browser console says:

Error: Error while calling method getRequestHeaders: No such actor for ID: server0.conn6.netEvent81
    _requestData resource://devtools/client/netmonitor/src/connector/firefox-data-provider.js:588
    requestData resource://devtools/client/netmonitor/src/connector/firefox-data-provider.js:505
    requestData resource://devtools/client/netmonitor/src/connector/index.js:605
    buildRequest resource://devtools/client/netmonitor/src/har/har-builder.js:177
    buildEntry resource://devtools/client/netmonitor/src/har/har-builder.js:113
    build resource://devtools/client/netmonitor/src/har/har-builder.js:64
    buildHarData resource://devtools/client/netmonitor/src/har/har-exporter.js:211
    fetchHarData resource://devtools/client/netmonitor/src/har/har-exporter.js:159
    save resource://devtools/client/netmonitor/src/har/har-exporter.js:78
    saveAllAsHar resource://devtools/client/netmonitor/src/har/har-menu-utils.js:39
    onClick resource://devtools/client/netmonitor/src/components/Toolbar.js:519
    React 38
    componentDidMount resource://devtools/client/shared/components/menu/MenuButton.js:126
    componentDidMount resource://devtools/client/shared/components/menu/MenuButton.js:122
    React 19
    bootstrap resource://devtools/client/netmonitor/src/app.js:77
    open resource://devtools/client/netmonitor/panel.js:21
    onLoad resource://devtools/client/framework/toolbox.js:2534
har-exporter.js:183:17

Comment 2

[u695718]

I'm also seeing this symptom with large sessions (~1gb). Although I think it's been an issue for a very long time, if not forever.

Comment 3

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Thank for the report!

I was able to reproduce the problem in 94

Error: Error while calling method getRequestHeaders: No such actor for ID: server0.conn0.netEvent7
    _requestData resource://devtools/client/netmonitor/src/connector/firefox-data-provider.js:588
har-exporter.js:183:17

But, I am not seeing it in 95/96

@Kuba, could you try Firefox Nightly?
https://www.mozilla.org/cs/firefox/channel/desktop/

Comment 4

[Kuba Orlik]

I've just checked in 96.0a1 (2021-11-11). The issue still persists. The error message is the same:

Error: Error while calling method getEventTimings: No such actor for ID: server0.conn0.netEvent4637 firefox-data-provider.js:600:15
Error: Error while calling method getEventTimings: No such actor for ID: server0.conn0.netEvent4684 firefox-data-provider.js:600:15
Error: Error while calling method getEventTimings: No such actor for ID: server0.conn0.netEvent4637 firefox-data-provider.js:600:15
Error: Error while calling method getEventTimings: No such actor for ID: server0.conn0.netEvent4684 firefox-data-provider.js:600:15
Error: Error while calling method getRequestHeaders: No such actor for ID: server0.conn0.netEvent4549
    _requestData resource://devtools/client/netmonitor/src/connector/firefox-data-provider.js:600
har-exporter.js:183:17
    onError resource://devtools/client/netmonitor/src/har/har-exporter.js:183
    (Async: promise callback)
    fetchHarData resource://devtools/client/netmonitor/src/har/har-exporter.js:182
    save resource://devtools/client/netmonitor/src/har/har-exporter.js:78
    saveAllAsHar resource://devtools/client/netmonitor/src/har/har-menu-utils.js:39
    onClick resource://devtools/client/netmonitor/src/components/Toolbar.js:519
    React 19

Comment 5

[Kuba Orlik]

I should add that it happens on other websites, as well - most often when the tracking protection is disabled

Comment 6

[Jan Honza Odvarko [:Honza] (always need-info? me)]

I still have troubles to reproduce the problem. Did you try it in fresh new Firefox profile?
(no extensions installed)

@Bomsy, can you please try this on your machine?

Comment 7

[Kuba Orlik]

Hi, I tried it on Firefox Nightly in Safe Mode. Still the same. Here's a video of me reproducing this issue. It doesn't happen every time, but usually when I "accept" the cookies on this website and then refresh without cache and wait a while and then attempt to generate the HAR file, it happens on 2nd or 3rd try

https://video.kuba-orlik.name/videos/watch/808a142f-fc26-46b7-a67a-cf2333897c19

Comment 8

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Thank you, Kuba!

I was able to reproduce the issue on my machine (Fx Nightly, Win10).

My STRs:

1. Open Firefox in Private mode
2. Load https://wiadomosci.onet.pl/bialystok/na-granicy-z-bialorusia-sa-tysiace-migrantow-na-zywo/sbkd6j8
3. Accept the Cookies, click on "Przejdź do serwisu"
4. Open DevTools, select the Network panel
5. Reload the page and wait till it's loaded
6. Right click on the list of request and pick "Save All As HAR"
7. Check out the size of the file, it's empty -> BUG

There is an exception in the Browser Console

Error: Error while calling method getRequestHeaders: No such actor for ID: server0.conn6.netEvent9135
    _requestData resource://devtools/client/netmonitor/src/connector/firefox-data-provider.js:600
har-exporter.js:183:17

Honza

Comment 9

[Kuba Orlik]

What does the regressionwindow-wanted tag signify and how can I help with that?

Comment 10

[Jan Honza Odvarko [:Honza] (always need-info? me)]

It's a request for finding regression range that would help us to identify specific bug (patch) that caused this reported bug (I believe this worked before, so I think it's a regression).

There is a 'mozregression' tool that is helping to find that range. It supports auto downloading various Firefox builds and allowing you to test the feature on those builds quickly (by bisecting) and narrowing the range between: the feature works / the feature is broken.

The source of the tool + links to user docs: https://github.com/mozilla/mozregression
User docs: https://mozilla.github.io/mozregression/documentation/

It would be great help if you could find the culprit bug/patch!

Comment 11

[Peter Hedenskog]

I'm also having problem with HAR generation in Firefox 94, I wonder if its related. I use the https://github.com/firefox-devtools/har-export-trigger and the problem is described here: https://github.com/sitespeedio/browsertime/issues/1671

For me it's like this: I use Browsertime to start the browser with the HAR exporter installed. When I navigate to a page I can see in the network tools that requests happens but when the HAR is generated it is a JSON without any page and requests.

I've been able to verify that it worked fine in 93 so this happens in 94. However when I use the har-export-trigger test page, it's always work.

I found a workaround: If I open Firefox, wait some time (on my local machine 7 seconds is the magic number) and then navigate to the URL, the HAR file is generated correctly.

I'm having trouble using mozregression when I automate things with webdriver (as in Browsertime) but maybe there's a workaround?

Should I create a separate issue for this or do you think it's the same thing Jan?

Comment 12

[Peter Hedenskog]

Just want to clarify that this makes it impossible for us to upgrade to 94 for our WebPageReplay tests at Wikimedia + for Browsertime/sitespeed.io users their performance tests will fail when they run with 94.

Comment 13

[Kuba Orlik]

Ok, I ran mozregression and the result is:

60:41.58 INFO: Last good revision: 9e713e49c00c14cc49032e828ad245f04be6447c
60:41.58 INFO: First bad revision: c1ba43b65415d33eeb716a191c2bb8395aed6224
60:41.58 INFO: Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=9e713e49c00c14cc49032e828ad245f04be6447c&tochange=c1ba43b65415d33eeb716a191c2bb8395aed6224

Comment 14

[Jan Honza Odvarko [:Honza] (always need-info? me)]

(In reply to Peter Hedenskog from comment #11)

I'm also having problem with HAR generation in Firefox 94, I wonder if its related. I use the https://github.com/firefox-devtools/har-export-trigger and the problem is described here: https://github.com/sitespeedio/browsertime/issues/1671

This feels like a different problem, but let's see if fix for this report helps or not.

Just want to clarify that this makes it impossible for us to upgrade to 94 for our WebPageReplay tests at Wikimedia +
for Browsertime/sitespeed.io users their performance tests will fail when they run with 94.

I am increasing priority for this bug

(In reply to Kuba Orlik from comment #13)

Ok, I ran mozregression and the result is:

Thank you Kuba, this is great help!

@Alex, it looks like this is a regression caused by Bug 1439509 - [dt-leak] NetworkEvent actors leak on reload
Can you please look at it, thank you.

Honza

Comment 15

[Alexandre Poirot [:ochameau]]

This exception seems to be related to old request made by the previous page load.
The netmonitor frontend receives these request a bit late, display them and try to fetch extra data about them.
As the HAR export is based on netmonitor frontend state, it also tries to manipulate these old, now defunct requests.

You may workaround these exception by toggling devtools.netmonitor.persistlog to true, but that may record more requests than you expect in the HAR file.

Another approach compared to the submitted patch would be to strengthen HAR codebase to be more resilient against unexpected errors with one request or one particular extra data fetching. So that, in case of exception we don't break the complete HAR export but only the one request or attribute we fail fetching.

Comment 16

[Alexandre Poirot [:ochameau]]

Attachment: Bug 1740116 - [devtools] Fix fetching extra data about cancelled navigation request.

These exception may happen when some pending request from the previous page
are received lated by the netmonitor, which tries to fetch extra data about these
old requests.
On the server side, these requests are destroyed when devtools.netmonitor.persistlog=false.
(which is the default behavior)

Comment 17

[Peter Hedenskog]

I can confirm that setting the preference to devtools.netmonitor.persistlog:true makes it work for me, that is great! Couldn't see any extra requests being picked up when I briefly looked,

What I cannot understand is what request that could be pending? In my use case, the browser opens, then navigates to data:text/html;charset=utf-8,<html><body></body></html> and then to the page that is measured (where we get the HAR). Could it be backend request (made by the browser) that was introduced in 94?

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8bde59d048c8
[devtools] Fix fetching extra data about cancelled navigation request. r=nchevobbe

Comment 19

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/8bde59d048c8

Comment 20

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Thank you Alex for fixing this!

Comment 21

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:ochameau, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

Comment 22

[Peter Hedenskog]

I just upgraded to Firefox 95 (from 94) and then the devtools.netmonitor.persistlog to true fix doesn't fix the problem anymore?

Comment 23

[Alexandre Poirot [:ochameau]]

(In reply to Peter Hedenskog from comment #22)

I just upgraded to Firefox 95 (from 94) and then the devtools.netmonitor.persistlog to true fix doesn't fix the problem anymore?

Now, you might be hitting yet another issue specific to Fission.
Could you try disabling Fission, by turning fission.autostart preference to false?

Otherwise, in Firefox 97, I'm still able to reproduce comment 0 issues.
It looks like I only fixed one very particular failure and more are still reproducing.
I opened bug 1747804 to keep fixing HAR export code.

Finally, regarding the uplift request. Yes, it is worth uplifting, but I should figure out bug 1747804 first.

Comment 24

[Ryan VanderMeulen [:RyanVM]]

We're in RC week for 96, so too late for uplift.

Comment 25

[Simona Badau, Desktop QA]

Reproduced the initial issue on Firefox 96 on macOS Big Sur 11.6.

Verified as fixed on the latest Nightly 98.0a1 and Firefox 97 beta 5 - used the STR from the Description and from Comment 8 - the Har file is not empty anymore. Tested on macOS Big Sur 11.6, Windows 10 x64, and Ubuntu 20.04.

Comment 26

[Peter Hedenskog]

I verified that this work in Firefox 97.

Comment 27

[Jan Honza Odvarko [:Honza] (always need-info? me)]

🎉🎉🎉