1770392 - webRequest.onBeforeRequest is sometimes being fired multiple times

Status: NEW

(WebExtensions :: Request Handling, defect, P3)

Description

[Booteille]

Steps to reproduce:

Hi!

I am intercepting URL's from YouTube and log a message when I do it.
To test it, I go on https://duckduckgo.com/?t=ffab&q=joueur+du+grenier&iax=videos&ia=videos&iai=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DQrgAgh46Ct8&pn=1 and click on any video.

I check my logs to see what happens.

Here is the code :

import Browser from "webextension-polyfill"

const a = (r) => {
  console.log(r)
  return {}
}

Browser.webRequest.onBeforeRequest.addListener(a, {
    urls: [ '*://*.youtube.com/*' ],
    types: [ 'main_frame' ]
}, [ 'blocking' ])

The browser is opened through web-ext cli tool, on Firefox v100.0.1 64 bits, on Pop_OS! 22.04

Actual results:

In logs, multiple requests appear to be blocked sometimes.

All requests have different ID and timestamp but are exactly the same:

{
  "requestId": "2519",
  "url": "https://www.youtube.com/watch?v=0thLCQHtLP4",
  "originUrl": "https://duckduckgo.com/?t=ffab&q=joueur+du+grenier&iax=videos&ia=videos&pn=1&iai=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DQrgAgh46Ct8",
  "method": "GET",
  "type": "main_frame",
  "timeStamp": 1653043239174,
  "tabId": 2,
  "frameId": 0,
  "parentFrameId": -1,
  "incognito": false,
  "thirdParty": false,
  "cookieStoreId": "firefox-default",
  "proxyInfo": null,
  "ip": null,
  "frameAncestors": [],
  "urlClassification": {
    "firstParty": [
      "tracking_content",
      "any_strict_tracking"
    ],
    "thirdParty": []
  },
  "requestSize": 0,
  "responseSize": 0
}

{
  "requestId": "2521",
  "url": "https://www.youtube.com/watch?v=0thLCQHtLP4",
  "originUrl": "https://duckduckgo.com/?t=ffab&q=joueur+du+grenier&iax=videos&ia=videos&pn=1&iai=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DQrgAgh46Ct8",
  "method": "GET",
  "type": "main_frame",
  "timeStamp": 1653043239183,
  "tabId": 2,
  "frameId": 0,
  "parentFrameId": -1,
  "incognito": false,
  "thirdParty": false,
  "cookieStoreId": "firefox-default",
  "proxyInfo": null,
  "ip": null,
  "frameAncestors": [],
  "urlClassification": {
    "firstParty": [
      "tracking_content",
      "any_strict_tracking"
    ],
    "thirdParty": []
  },
  "requestSize": 0,
  "responseSize": 0
}

{
  "requestId": "2522",
  "url": "https://www.youtube.com/watch?v=0thLCQHtLP4",
  "originUrl": "https://duckduckgo.com/?t=ffab&q=joueur+du+grenier&iax=videos&ia=videos&pn=1&iai=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DQrgAgh46Ct8",
  "method": "GET",
  "type": "main_frame",
  "timeStamp": 1653043239228,
  "tabId": 2,
  "frameId": 0,
  "parentFrameId": -1,
  "incognito": false,
  "thirdParty": false,
  "cookieStoreId": "firefox-default",
  "proxyInfo": null,
  "ip": null,
  "frameAncestors": [],
  "urlClassification": {
    "firstParty": [
      "tracking_content",
      "any_strict_tracking"
    ],
    "thirdParty": []
  },
  "requestSize": 0,
  "responseSize": 0
}

Expected results:

The callback should have been fired only once.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'WebExtensions::Request Handling' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Alex Cornestean]

Hello Booteille,

I’m from QA and I’m attempting to reproduce the issue in order to confirm it, however I am having some trouble in doing so.

Would you be so kind as to attach the extension you used, as well as a video of the issue occurring so I’ll know exactly what to look for?

Thank you !

Comment 3

[Tomislav Jovanovic :zombie]

I wanna investigate, but not in the immediate future.

Comment 4

[Booteille]

(In reply to Alex Cornestean from comment #2)

Hello Booteille,

I’m from QA and I’m attempting to reproduce the issue in order to confirm it, however I am having some trouble in doing so.

Would you be so kind as to attach the extension you used, as well as a video of the issue occurring so I’ll know exactly what to look for?

Thank you !

Hi!

Here is the code not using typescript (which I used in my previous example, with webextension-polyfill):

manifest.json

{
  "manifest_version": 2,
  "name": "WebRequest Redirection Test",
  "version": "0.0.1",
  "description": "WebRequest Redirection Test",
  "permissions": [
    "<all_urls>",
    "webRequest",
    "webRequestBlocking"
  ],
  "background": {
    "scripts": [
      "background.js"
    ]
  }
}

background.js

const a = (r) => {
  console.log(r)
  return {}
}

browser.webRequest.onBeforeRequest.addListener(a, {
    urls: [ '*://*.youtube.com/*' ],
    types: [ 'main_frame' ]
}, [ 'blocking' ])

And you can find a video on it running and the result of the console (on your left) on this link: https://send.stolon.fr/download/2e8f4714669f280d/#j0JgClo6eAUuljyslFolMQ

If you need other informations, just ask!

Thanks!

Comment 5

[Alex Cornestean]

Hello Booteille and thank you for the info !

I have gone about testing this in 2 ways:

1. Launch Firefox and then load the extension via about:debugging
2. Launch Firefox via web-ext cli

In both cases, after I click on any video from the suggested link in Comment 0 (the option to watch on YouTube has been set prior to this, as per what I saw in the video from Comment 4), the console will log only 1 request.
Now, the YT video page is accessed and a “Before you continue to YouTube” pop-up is displayed (as in my case it was the first YT visit on that profile), with several options, but the most important ones are “REJECT ALL” and “ACCEPT ALL”.
Clicking on “ACCEPT ALL” will cause another request to be logged to console, almost identical to the first one (only the ID and timestamp will differ, of course). This is the only case a second request has been logged.
Clicking on “REJECT ALL” will not log another request.

All this can be seen in the attached video.

Tested on the latest Nightly (102.0a1/20220523213527), Beta (101.0/20220523135024) and Release (100.0.2/20220519220738) under Windows 10 x64 and Ubuntu 16.04 LTS.

Please note that on Ubuntu I was able to only test the first scenario (manually loading the extension) as for some reason I cannot install the web-ext tool due to some errors.

Comment 6

[Alex Cornestean]

Attachment: 2022-05-24_09h54_33.mp4

Comment 7

[Booteille]

Hi.

I can reproduce your behavior when clicking on ACCEPT ALL or REJECT ALL.

I also wanted you to know that it seems to fire the event multiple times randomly. Sometimes it fires 3 of it, most of the time just one.

In the next video, pay attention to timestamps where you'll notice most of my clicks fired the event only once except the last one.

Comment 8

[Booteille]

Attachment: Capture d’écran vidéo de 25-05-2022 00:24:45.webm

Comment 9

[Alex Cornestean]

Hi Booteille !

The last video you attached made it clear what I was doing wrong in the first attempts, but it was hard to spot as you had your cursor hidden in the videos. I was initially clicking on the “Watch on YouTube” button from the “YouTube Privacy Warning” panel and not on the name of the video as it seems you are doing.

Clicking on the video name will indeed cause multiple requests to be fired.

I’ve reproduced the behavior on the latest Nightly (102.0a1/20220524214448), Beta (101.0/20220523135024) and Release (100.0.2/20220519220738) under Windows 10 x64 and Ubuntu 16.04 LTS.

For more details, see the attached video (Windows).

NOTE:

• On Ubuntu, it seems that only the first accessing of a video will fire multiple requests. Each following accessing of the same video will no longer fire multiple requests, but only a single one. I’ve tried this maybe 80 times on a video and each time I accessed that video after the initial attempt, only one request was fired.
• On Windows, multiple requests were still fired randomly when accessing the same video, even after the initial accessing of the video.

Comment 10

[Alex Cornestean]

Attachment: 2022-05-25_13h37_35.mp4

Comment 11

[Booteille]

Hi Alex!

Thanks for your investigation and sorry for the missing cursor!

Tell me if I can do more for you.

Comment 12

[Alex Cornestean]

Hi Booteille,

Thanks to the information you provided so far, the bug has been confirmed, so there in nothing more you can do for now. Thank you !

Now we just have to wait until one of our developers picks this up and further investigates the underlying causes.