1561604 - webRequest.filterResponseData sometimes breaks load event in HTML documents

Status: NEW

(WebExtensions :: Request Handling, defect, P3)

Description

[Rob Wu [:robwu]]

Attachment: webRequest-replace-response.zip

There is a bug where the "load" event is not fired at all, when webRequest.filterResponseData is used. This issue is not consistently reproducible; yesterday I managed to reproduce this bug 5 times early on (in Firefox Nightly on Linux and macOS, and in Firefox 64.0.4 on Linux). But thereafter (including today), I was not able to reproduce it in dozens of attempts.

This issue was initially reported in a comment at bug 1410755, but unlike the other bug, the load event here has not been triggered at all (in the other bug it was triggered too early).

STR:

1. Load the attached extension (which is based on bug 1410755#c26).
2. Visit https://example.com/
3. If the expected result happens, open the developer tools and reload. (not sure if this step is needed, but I was only able to reproduce with this step).

Expected:

• The page should be blank and show "replacement content"

Actual:

• Sometimes, the page stays blank. When I inspect the page, I see that the expected HTML is present, but that the window.onload handler has not been triggered. The page is stuck at readyState "interactive", and the "load" event has never fired, as seen by the reported timings below (via the developer tools in the affected tab).

This is part of the reported timings that stand out:

domComplete: 0
domContentLoadedEventEnd: 1561455537083
loadEventEnd: 0

Based on the non-zero domContentLoadedEventEnd value, Document::DispatchContentLoadedEvents has been called (at least at this point).
At the end of the method, Document::UnblockOnload is called. But this did unexpectedly not move the readyState to "complete" (which is also reflected as zero in the above timings).

If I manually call tabs[0].contentViewer.loadComplete(Components.results.NS_OK), then the expected result happens. So somehow nsDocShell::EndPageLoad is not called, or called with an error code in aStatus (if I use NS_ERROR_FAILURE instead of NS_OK, then the test page is still stuck at the readyState).

Full output:

document.readyState
"interactive"

performance.timing
PerformanceTiming
connectEnd: 1561455536852
connectStart: 1561455536852
domComplete: 0
domContentLoadedEventEnd: 1561455537083
domContentLoadedEventStart: 1561455537079
domInteractive: 1561455537077
domLoading: 1561455537065
domainLookupEnd: 1561455536852
domainLookupStart: 1561455536852
fetchStart: 1561455536852
loadEventEnd: 0
loadEventStart: 0
navigationStart: 1561455536852
redirectEnd: 0
redirectStart: 0
requestStart: 1561455536918
responseEnd: 1561455537015
responseStart: 1561455537015
secureConnectionStart: 1561455536852
unloadEventEnd: 1561455537066
unloadEventStart: 1561455537065

Output of the content toolbox, in case it's relevant:

tabs[0].docShell.busyFlags is 0 (BUSY_FLAGS_NONE).
tabs[0].docShell.currentDocumentChannel.status = 0 (NS_OK).

Comment 1

[Rob Wu [:robwu]]

kernp, I filed this report based on your comment on bug 1410755 (at https://bugzilla.mozilla.org/show_bug.cgi?id=1410755#c26).
In its current form, it's difficult to debug. Could you provide any more information that may help with reproducing or debugging this issue?

Comment 2

[kernp25]

I don't really know how to reproduce the issue (it sometimes fires the load event and sometimes not).

I was seeing this bug, while i was playing with the webRequest.filterResponseData api (needed for an add-on).

If i use the DOMContentLoaded event instead, then everything works as expected.

Like this add-on: https://github.com/tymofij/fb2reader/blob/master/webext-src/streamfilter.js

Comment 3

[kernp25]

It also happens with xml documents!

https://www.oasis-open.org/docbook/xml/4.3b3/test.xml
https://google.github.io/styleguide/Rguide.xml

I used this code for testing:

window.onload = function() { document.documentElement.textContent = "replacement content"; };

Comment 4

[kernp25]

I think you need to update the bug title?

Comment 5

[Rob Wu [:robwu]]

Did you only use the stream filter, or did you also rewrite the response headers?

If I modify the original test case to match XML documents, then I can reproduce the issue at a 100% rate in Firefox 69.0a1 buildID 20190624213657: the document is forever stuck at readyState="interactive". It's not clear whether it's the same bug; if it is not then I would file a separate bug.

If I however change the Content-Type to HTML, then the issue does not occur any more.

browser.webRequest.onHeadersReceived.addListener(function({responseHeaders}) {
  let head = responseHeaders.find((n) => n.name.toLowerCase() === "content-type");
  if (!head) {
    head = {name: "Content-Type", value: ""};
    responseHeaders.push(head);
  }
  head.value = "text/html; charset=utf-8";
  return {responseHeaders};
}, {urls: ["*://*/*.xml"], types: ["main_frame"]},
  ["blocking", "responseHeaders"]
);

Comment 6

[kernp25]

(In reply to Rob Wu [:robwu] from comment #5)

Did you only use the stream filter, or did you also rewrite the response headers?

I only used the stream filter!

Comment 7

[Rob Wu [:robwu]]

If you only used the stream filter, then the originally provided test case renders the result as XML, not HTML. Therefore the script would not execute at all. If relevant for the reproduction, could you attach a new test case? And please mention whether the issue is 100% reproducible, or only sometimes.

Comment 8

[kernp25]

I used this code:

<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<script>
  var rs = document.readyState;
  var loadEventEnd = performance.timing;

console.log(rs, loadEventEnd);
  window.onload = function() { document.documentElement.textContent = "replacement content"; };
</script>
</head>
<body>
</body>
</html>

Comment 9

[sycxyc+mozilla]

Attachment: test_streamfilter-1.zip

Comment 10

[sycxyc+mozilla]

Attachment: media.html

This test case should be easy to reproduce. If not, you may need to replace the video source in "media.html" with a larger and slower one.

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0

Steps to reproduce:

1. Load the "test_webnavigation-1.zip" temporary extension.
2. Open "media.html", close the "LOADED" prompt.
3. Click the "Load Video" button, and wait for the video to load.
4. Click the "Reload Page" button.

Expected:

• A "LOADED" prompt will pop up.

Actual:

• The page never finishes loading.
• document.readyState==="interactive"