1845076 - Missing "network.responseCompleted" event if "responseStarted" is not received

Status: NEW

(Remote Protocol :: WebDriver BiDi, defect, P3)

Description

[Nik]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36

Steps to reproduce:

I run this Puppeteer test should fail when navigating to bad SSL after redirects https://github.com/puppeteer/puppeteer/blob/main/test/src/navigation.spec.ts#L187
I expect the BiDi event network.responseCompleted to always have a header list.
As per spec https://w3c.github.io/webdriver-bidi/#get-the-response-data
Step 6 into step 13

Actual results:

The headers parameter is missing on the JSON reponse

Expected results:

The headers property should be empty list

Comment 1

[Julian Descottes [:jdescottes]]

Thanks for filing! Looking at the code, the only way this could happen would be if for some reason we missed the response started event internally, which is where we fetch headers...

Do you have logs for this failure?

Comment 2

[Nik]

Attachment: firefox-headers.log

Comment 3

[Nik]

It seems like that is the case I see network.beforeRequestSent and network.responseCompleted events. Attached full logs with tracing.

Comment 4

[Julian Descottes [:jdescottes]]

Thanks, looking at the payload, it's definitely a case where we missed the response started event, although I don't really see anything in the logs which would explain why.

Do I need a specific version of puppeteer to reproduce the issue? I was trying with 20.9.0 and it seemed ok?

Comment 5

[Nik]

We have a fix for that in place for some time, but forgot to file a bug.
https://github.com/puppeteer/puppeteer/blob/main/packages/puppeteer-core/src/common/bidi/HTTPResponse.ts#L61
You will need to manually build a puppeteer version.

Comment 6

[Julian Descottes [:jdescottes]]

Thanks, I can confirm the issue.

So as suspected it's a case where the NetworkObserver we use skips the responseStarted step. Quickly checked our CDP version, we only emit Network.requestWillBeSent for this scenario.

Will need to take a closer look at the request which fails, but at the very least, we should probably try to emit sensible defaults... or to skip the event if we missed the responseStarted (which would be consistent with our current CDP implementation and also seems to "fix" the test after removing the workaround).

Comment 7

[Julian Descottes [:jdescottes]]

For later investigations, here's a browser mochitest which only use the network observer and reproduces the "missing response start" issue: https://phabricator.services.mozilla.com/D184385.

And for reference, a patch to skip responseCompleted when responseStarted was missing: https://phabricator.services.mozilla.com/D184384 (but this probably needs more investigations)

Comment 8

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:whimboo, could you have a look please?

For more information, please visit BugBot documentation.

Comment 9

[Julian Descottes [:jdescottes]]

Seems like we have another case where responseStarted was missing at https://github.com/firefox-devtools/bidi-har-export/pull/23

Comment 10

[Julian Descottes [:jdescottes]]

Note that the symptoms for this issue changed a bit since it was reported.
We will now fail to emit the responseCompleted event because our logic to build the responseCompleted event throws early with:

JavaScript error: chrome://remote/content/shared/listeners/NetworkEventRecord.sys.mjs, line 328: TypeError: can't access property "setResponseSizes", this[#response] is null

This is because this.#response is created when we prepare the responseStarted event. We might want to guard this at least to write a proper error message, referencing the skipped URL.