1437748 - webRequest.onBeforeRequest fires two different requestIds after redirect when network cache is filled

Status: RESOLVED FIXED

(WebExtensions :: Request Handling, defect, P2)

Description

[stoically]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0
Build ID: 20180131010234

Steps to reproduce:

Registered webRequest listeners with WebExtension:

browser.webRequest.onBeforeRequest.addListener((details) => {
  console.log('incoming onBeforeRequest', details);
}, {
  urls: ['<all_urls>'],
  types: ['main_frame']
}, [
  'blocking',
  'requestBody'
]);
browser.webRequest.onBeforeRedirect.addListener((details) => {
  console.log('incoming onBeforeRedirect', details);
},  {
  urls: ['<all_urls>']
}, [
  'responseHeaders'
]);
browser.webRequest.onCompleted.addListener((details) => {
  console.log('incoming onCompleted', details);
},  {
  urls: ['<all_urls>'],
  types: ['main_frame']
}, [
  'responseHeaders'
]);


Actual results:

Clean profile, no network cache, navigating to http://youtube.com - all requests have the same requestId

incoming onBeforeRequest Object { requestId: "5503", url: "http://youtube.com/" .. }
incoming onBeforeRedirect Object { requestId: "5503", url: "http://youtube.com/" .. }
incoming onBeforeRequest Object { requestId: "5503", url: "https://youtube.com/" .. }
incoming onBeforeRedirect Object { requestId: "5503", url: "https://youtube.com/" .. }
incoming onBeforeRequest Object { requestId: "5503", url: "https://www.youtube.com/" .. }
incoming onCompleted Object { requestId: "5503", url: "https://www.youtube.com/" .. }

---

Second time navigating to http://youtube.com with filled cache. After redirecting the requestId changes

incoming onBeforeRequest Object { requestId: "5586", url: "http://youtube.com/" .. }
incoming onBeforeRequest Object { requestId: "5586", url: "https://youtube.com/" .. }
incoming onBeforeRedirect Object { requestId: "5586", url: "https://youtube.com/" .. }
incoming onBeforeRequest Object { requestId: "5586", url: "https://www.youtube.com/" .. }
incoming onBeforeRequest Object { requestId: "5587", url: "https://www.youtube.com/" .. }
incoming onCompleted Object { requestId: "5587" .. }


Expected results:

The requestId during the second navigation with filled cache should not change.

Couldn't find any other information regarding this and since the first navigation doesn't change the requestId and the second navigation doesn't fire onCompleted for the requestId 5586 I'm assuming this is not intended behavior. Also the requestId doesn't change in Chromium under the same circumstances.

This popped up while fixing a "two tabs randomly opening"-Issue with Multi-Account Containers (https://github.com/mozilla/multi-account-containers/pull/1120).

Comment 1

[stoically]

Just to add, webRequest.onErrorOccurred doesn't see the request 5586 either - and without being able to correlate the two requests, it's afaik not possible to reliably cancel all requests that originate from the first 5586 request without checking for a combination of tabId and url, which could also affect other requests. In case of Multi-Account Container that would mean canceling clicks on the same link on the originating website.

Also tested with Firefox Nightly 60.0a1

Comment 2

[Kris Maglione [:kmag]]

This is probably due to race cache with network, which creates separate requests to the cache service and to the network host. Someone should confirm whether this still happens with RCWN disabled.

Comment 3

[stoically]

Tested with Firefox Developer Edition 66.0b10 (64-bit) on Linux and network.http.rcwn.enabled set to false. The issue is still present: after youtube.com redirects to www.youtube.com an additional onBeforeRequests with a new requestId is fired and the original request never fires onCompleted.

Comment 4

[mehrmor]

Hello,

I can confirm this bug is still present and it is breaking our web extension from working properly, whereas in Chrome it doesn't occur.

From what I have gathered so far, it seems to be a problem with those websites that redirect with 301 status codes rather than 307.
An example of a website with 307 redirection would be : http://www.badoo.com
An example of a website with 301 redirection would be : http://alibaba.ir

Youtube also uses 301 redirection, so it led me to believe that is the cause of the changing RequestId.

Can someone please confirm this? And if yes, can it be classified as a bug? Or is it by design?

Comment 5

[mehrmor]

Just to add. I tested it with the Disable Cache check-box marked in the Network tab of the Developers Tools.

Another related note: The extension broke even when the cache was empty. I use the web-ext module to debug my extension and it clears the cache every time it is run.
My observation was that onBeforeRequest event was raised with two different RequestIds of which the second one happens after the redirection made by the first request.

Comment 6

[Marc Streckfuß [:MeFisto94]]

Attachment: testcase-webext.zip

So according to recent testing this issue doesn't happen with nightly, but does so with all other releases (dev and beta).
Furthermore when using a self-compiled version based on an older commit, which happened before the current nightly was released, the issue is NOT present either.

So it seems there is some different side effect playing into this here. Can someone help testing and/or has a clue why this happens?
The attached zip contains a simple web extension to test it. Go to about:debugging, install it and click on Inspect to open the Console.
Then browse to http://youtube.com (no www no https) a few times and see if the IDs always match up.
Use the main_frame filter in the console to not be spammed.

Note that this web extension will also put a red border around every webpage (borderify example), so that it's really clear whether the web extension is currently active or not.

Comment 7

[Marc Streckfuß [:MeFisto94]]

Okay, using mozregression I was able to find the root of this problem or rather: where it has been fixed as "regression":
"Bug 1456995 (be9a6289)" did fix it and the reason we saw differences between nightly and beta is the presence of the pref dom.serviceWorkers.parent_intercept which is only enabled on nightly currently.

So this is basically solved, the pref will be included as a result of Bug 1588154.

Comment 8

[Marc Streckfuß [:MeFisto94]]

Closing as this issue is fixed as of Bug 1588154 and should land with Firefox 73.

Comment 14

[kernp25]

Restrict comments here to prevent spam?

Comment 15

[:Gijs (out for now; he/him)]

Thanks for the heads-up; done.