<a class="header-button" href="https://bugzilla.mozilla.org/home" title="Go to home page"> Bugzilla

Reporter

Comment 2

•

7 months ago

Just noticed that UA is completely wrong, it should be Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:127.0) Gecko/20100101 Firefox/127.0.

Comment 3

•

7 months ago

Hello,

Would you be willing to share your extension by attaching it to the report, so I can check the issue out? Or point me to an extension that will suit this need?

Thank you !

Flags: needinfo?(xlucas4956)

Reporter

Comment 4

•

7 months ago

I first noticed it while working on https://addons.mozilla.org/en-US/firefox/addon/chutils/, but it happens with every browser extension.

Flags: needinfo?(xlucas4956)

Comment 5

•

7 months ago

Hello and thank you for the additional info !

Checked with the latest Nightly (128.0a1/20240516214828) under Windows 10, where I installed the linked extension along with several others and tried using them on various sites while monitoring the “Network” tab (with a performance analysis started). Three of the extensions I installed beside “CHUtils” register requests, while “CHUtils” does not. See the attached screenshot.

At this point I’m inclined to say that the issue might not be on the browser side, but with the extension itself.

Did the requests get logged at any point while developing the extension? Do you remember a Firefox version where the requests were logged? I’m asking this as I tested Firefox Developer Edition 124.0b9/20240308091626 too under the same conditions and the results were the same as on Nightly.

Comment 6

•

7 months ago

Attached image 2024-05-17_10h06_07.png — Details

Reporter

Comment 7

•

7 months ago

I installed the linked extension along with several others and tried using them on various sites

CHUtils runs exclusively on the social media site https://cohost.org/, so it would not log requests on any other site. It's clearly sending the intended requests, as it functions as expected on the site and my custom helper for the site's API still logs debug messages containing request response data.

Did the requests get logged at any point while developing the extension? Do you remember a Firefox version where the requests were logged?

I believe I should clarify; the scripts generating network requests are running inside the page, and not in a background script, so the requests were previously logged to the regular Developer tab and not the extension toolbox. Requests were logged as expected until May 9, and neither Nightly nor Developer updated in the window between when requests were logged and when they stopped being logged.

Comment 8

•

6 months ago

I tried again using CHUtils on https://cohost.org/ and checking the Network tab from Web Developer Tools. Indeed, there are no web requests created by add-ons shown.

Tested on the latest Nightly (128.0a1/20240519213431), Beta (127.0b3/20240517091915) and Release (126.0/20240509170740) under Windows 10. I also checked a Nightly from before May 9 (127.0a1/20240501214803 from May 1) and encountered the same lack of add-on requests being shown in the Network tab.

Status: UNCONFIRMED → NEW

status-firefox126: --- → affected

status-firefox127: --- → affected

status-firefox128: --- → affected

Ever confirmed: true

Rob Wu [:robwu]

Updated

•

6 months ago

Component: Untriaged → Developer Tools

Comment 9

•

6 months ago

Hi Hubert,
is there any change we may have applied to the network panel internals that may have had a chance to introduce this kind of side effect for the network panel from the Addon Debugging window?

Flags: needinfo?(hmanilla)

Comment 10

•

6 months ago

Hey Alex,
given that you seem to be able to reproduce this issue, would you mind to try to bisect the regressing change using mozregression?

Flags: needinfo?(acornestean)

Updated

•

6 months ago

Keywords: regressionwindow-wanted

Julian Descottes [:jdescottes]

Comment 11

•

6 months ago

I took a brief look to the extension internals (from the github repo: https://github.com/enchanted-sword/ch-utils) and based on that it seems the background page and the action popup panel aren't originating any network request, while I see some code in the part of the extension that it seems to be meant to be injected into the cohost.org website (with the purpose of augmenting the webapp UI, main feature that the extension seems to provide).

Based on that I'd like to confirm a couple of details around the steps to reproduce and expected behaviors with you:

which developer toolbox you where using to monitor the network requests?
what I see in the source code of the extensions seems to suggest the network requests may be meant to be visible
in the network tool tab from the developer toolbox associated to the tab where the cohost.org webpage is being used from
(if the scripts that are originating the requests are injected into the page itself, the requests is expected to look like being
originated from the website itself)
would you mind to comment with one simple sequence of steps to reproduce a specific network request that used to be visible and it is not anymore?
(a screencast may also be helpful if describing what to do on the website may be tricky to be described just using words)

(Alternatively you may also give it a try to use the mozregression tool https://mozilla.github.io/mozregression/ to try to bisect the regressing change).

Flags: needinfo?(xlucas4956)

Comment 12

•

6 months ago

•

Edited

I looked a bit at the extension, and detailed STRs would certainly be helpful. I feel like the requests which are missing might be moz-extension:// requests, which I'm not sure we are logging in the netmonitor?

Edit: I also tried on FF121, which is the first version supported by the extension and I can't see any request either. But it would be nice to confirm which requests are missing first, because I've been focusing on the moz-extension:// ones so far.

Hubert Boma Manilla (:bomsy)

Comment 13

•

6 months ago

•

Edited

Hi Luca,
Nothing i can think of. i reached out to Julian to get his thoughts as well, as he did some network stuff recently.

Tested the extension and could not see any requests.

We'll wait for response on the STR request, as it would really help us with investigating?

Thanks

Flags: needinfo?(hmanilla)

Julian Descottes [:jdescottes]

Reporter

Comment 14

•

6 months ago

Attached video A screen capture of a user clicking on a button to open a popup, which results in several network requests being logged to the browser console — Details

I was using the developer toolbox associated with the tab displaying cohost.org, as the extension does inject content scripts into the page, and previously, the web requests created by the extension would show up alongside the web requests created by the site's web app itself.

This screen capture is an example of a series of web requests that would have previously appeared in the network tab, which are being logged to the console by the extension's API helper. When switching over to the network tab, the only XHR entries visible are from the script client.61abb74d628c9a711286.js, which is part of the site's app and not from the extension.

Flags: needinfo?(xlucas4956)

Comment 15

•

6 months ago

I've been trying to reproduce using the extension CHUtils but I'm having a lot of troubles using it (seems to throw early because browser.storage.local.get("preferences") returns undefined?). After fiddling a bit with it, I manage to see requests going to projects.searchByHandle in the BrowserToolbox, but they don't show in the regular toolbox.

Their channel.loadInfo does not link to the browsing context id of the debugged tab, and I can see that the loadingPrincipal is an extended addon principal [Expanded Principal [https://cohost.org, moz-extension://80d324db-678c-4dcd-b0be-f8b3be16b49f]] with isAddonOrExpandedAddonPrincipal set to true.

In any case, we completely fail to match the channel with the web toolbox because there is no property on the channel which links back to the debugged browsing context.

Luca do you know if there has been any change to the way scripts loaded in content pages by webextensions are sandboxed? Other than this I can't really see another explanation as to why this would have regressed.

Maybe we should build a simple webextension which simply loads a content script and performs a fetch to make the investigation easier.

Flags: needinfo?(lgreco)

Comment 16

•

6 months ago

Tried running a regression range spanning 2024-04-01 to 2024-06-04 and using the developer toolbox associated with the tab displaying cohost.org, switching add-on options on and off and clicking buttons on the page.

I do see requests in the console tab which as per Comment 14 are logged by the extension’s API helper, but nothing shows up in the network tab except XHR entries from client.61abb74d628c9a711286.js.

The issue is, I checked builds from before May 9 (down to April 2, 2024) which according to Comment 7 should be showing requests from the extension in the network tab, but they don’t.

Flags: needinfo?(acornestean)

Reporter

Comment 17

•

6 months ago

May 9 is just when I noticed the change, but I don't think I updated either Nightly or Developer between when it was logging requests and when it stopped.

Comment 18

•

5 months ago

(In reply to Julian Descottes [:jdescottes] from comment #15)

I've been trying to reproduce using the extension CHUtils but I'm having a lot of troubles using it (seems to throw early because browser.storage.local.get("preferences") returns undefined?). After fiddling a bit with it, I manage to see requests going to projects.searchByHandle in the BrowserToolbox, but they don't show in the regular toolbox.

Their channel.loadInfo does not link to the browsing context id of the debugged tab, and I can see that the loadingPrincipal is an extended addon principal [Expanded Principal [https://cohost.org, moz-extension://80d324db-678c-4dcd-b0be-f8b3be16b49f]] with isAddonOrExpandedAddonPrincipal set to true.

In any case, we completely fail to match the channel with the web toolbox because there is no property on the channel which links back to the debugged browsing context.

Luca do you know if there has been any change to the way scripts loaded in content pages by webextensions are sandboxed? Other than this I can't really see another explanation as to why this would have regressed.

Maybe we should build a simple webextension which simply loads a content script and performs a fetch to make the investigation easier.

Thanks for digging into the network monitor side Julian.

Manifest V3 vs Manifest V2 content scripts

The only change on the webextensions internals side that I can think of is the slightly different sandbox config used for the content scripts related to a Manifest V3 extension, but the outcome of that would be actually the opposite:

In a Manifest V3 extension cross-origin XHR and fetch requests originated from content scripts are deprecated and so the requests triggered from an MV3 content scripts through fetch and XMLHttpRequest will then be exactly the same as requests triggered from the webpage and so they are expected to be listed in the network monitor just fine (because they will belong to the Browsing Context ID that the tab toolbox is watching).

Manifest V2 can also “opt-in” to use the same kind of fetch and XMLHttpRequest that belongs to the attached webpage (and so also limited in terms of allowed cross-origin requests as the webpage would be) by using content.fetch and content.XMLHttpRequest explicitly (and those are expected to also be visible in the tab toolbox network panel for the same reasons described above), as also briefly mentioned inside this section of the MDN docs: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Content_scripts#xhr_and_fetch.

Minimal Test Extensions and attempt to bisecting the regressing change

I’m attaching to this bug two small test extensions to more easily trigger fetch requests from a content script, one as Manifest V3 just to be used to confirm that fetch requests originated from an Manifest V3 content script are been shown as expected) and one as Manifest V2 that I used to try to bisect the regressing change.

Unfortunately I didn't manage to be able to go too far in the past because on very old versions the tab loading webpage was always crashing the tab right away, that looked like an issue unrelated to this one, but prevented mozregression from bisecting the regressing point.

If anyone wants to give that another try, the following is the mach mozregression command I used locally:

./mach mozregression --addon path/to/test-contentscript-fetch.xpi \
             --pref "xpinstall.signatures.required:false" -a=https://example.org -a="--devtools" \
             --good 90

The STR is then just:

Make sure to have a tab on https://example.org and the developer toolbox opened on that tab
Select the devtools network panel and open the split console
Click on the body on the page
Confirm logs related to the fetch request starting and completing are logged in the console
Confirm if the request has been logged in the network panel

Without mozregression successfully bisecting a regressing change, and based on what you mentioned in comment 15 and what I see in the matchRequest function defined in NetworkUtils.sys.mjs here, I’d expect this behavior to have been started around the time when we switched to filter the requests based on the Browsing Context (and I honestly don’t recall how that used to work before that) and so it doesn’t look like something that may have regressed only recently.

Potential approaches

I took a look into what is available in the channel loadInfo, we can definitely determine from the triggeringPrincipal properties if the request is originated from a content script sandbox and even which is the addon id of the related extension, but we don’t have any property that would help us to correlate the channel with a given Browsing Context ID, at least not without changes specifically introduced to let us trace that back.

I haven’t digged deeply enough to determine how much work that be, on the WebExtensions side we could consider adding the browser context ID of the attached webpage into the sandbox metadata, but then we would still have to propagate that across more than a few internals to then have it available when matchRequest will be filtering the channel.

Flags: needinfo?(lgreco)

Updated

•

5 months ago

Summary: Web requests created by extensions are not visible in the Developer Toolbox → Web requests created by extensions content scripts are not visible in the Developer Toolbox

Updated

•

4 months ago

Flags: needinfo?(valentin.gosu)

Comment 19

•

4 months ago

(In reply to Luca Greco [:rpl] [:luca] [:lgreco] from comment #18)

I’m attaching to this bug two small test extensions to more easily trigger fetch requests from a content script

Hi Luca,

I don't see the extensions attached.
Maybe you forgot to add them?

Unfortunately I didn't manage to be able to go too far in the past because on very old versions the tab loading webpage was always crashing the tab right away, that looked like an issue unrelated to this one, but prevented mozregression from bisecting the regressing point.

If anyone wants to give that another try, the following is the mach mozregression command I used locally:
./mach mozregression --addon path/to/test-contentscript-fetch.xpi \
             --pref "xpinstall.signatures.required:false" -a=https://example.org -a="--devtools" \
             --good 90

Past a certain point I think it's the sandboxing that's causing the crash.
You can add security.sandbox.content.level:0 and that should avoid the crash.

In any case, thank you for the detailed investigation. Initially I wasn't convinced it was a regression, but looking at the code you linked, it seems that this should have worked at some point
https://searchfox.org/mozilla-central/rev/058b47cf516776bc95f938a1aaf3a80466b7e39e/toolkit/components/extensions/ExtensionContent.sys.mjs#914-920

// This metadata is required by the Developer Tools, in order for
// the content script to be associated with both the extension and
// the tab holding the content page.
let metadata = {
  "inner-window-id": this.innerWindowID,
  addonId: extensionPrincipal.addonId,
};

Searching for inner-window-id, I got to this code
Looking through the code, it seems the inner-window-id was introduced in https://hg.mozilla.org/mozilla-central/rev/81c63efd1278db830ac6dbbaedcee3f959e27259 - a long time ago. Most likely a regression - we don't seem to have a test for it.
If you manage to run mozregression again we might find the issue. I can also try to do it if you attach the extensions.
Thanks!

Flags: needinfo?(valentin.gosu) → needinfo?(lgreco)

Comment 20

•

3 months ago

Attached file test-contentscript-fetch.xpi — Details

Reduced "Manifest V2" Test Extension, which attaches a content script to example.org webpages and triggers a fetch request from the content script on mouse clicks triggered on the webpage body element.

Comment 21

•

3 months ago

Attached file test-contentscript-fetch-mv3.xpi — Details

Reduced "Manifest V3" test extension, which does the same thing as the "Manifest V2" test extension, but the fetch requests are expected to be actually visible (because for "Manifest V3" content scripts there is no fetch global that belongs to the ExpandedPrincipal Sandbox, and so fetch is inherited from the webpage and it works exactly like the webpage fetch).

Comment 22

•

3 months ago

(In reply to Valentin Gosu [:valentin] (he/him) from comment #19)

(In reply to Luca Greco [:rpl] [:luca] [:lgreco] from comment #18)

I’m attaching to this bug two small test extensions to more easily trigger fetch requests from a content script

Hi Luca,

I don't see the extensions attached.
Maybe you forgot to add them?

Sorry about that, it looks that I have definitely missed to attach them :-(
I've attached both of them (the MV2 one that is expected to be hitting this issue and the MV3 one that is expected to not be hitting it).

Unfortunately I didn't manage to be able to go too far in the past because on very old versions the tab loading webpage was always crashing the tab right away, that looked like an issue unrelated to this one, but prevented mozregression from bisecting the regressing point.

If anyone wants to give that another try, the following is the mach mozregression command I used locally:
./mach mozregression --addon path/to/test-contentscript-fetch.xpi \
             --pref "xpinstall.signatures.required:false" -a=https://example.org -a="--devtools" \
             --good 90
Past a certain point I think it's the sandboxing that's causing the crash.
You can add security.sandbox.content.level:0 and that should avoid the crash.

Thanks! that was it, I managed to go further in the past versions, but couldn't find any older version showing the request, and so it looks I was wrong and we may have never shown content script requests in the tab network panel.

At this point I'm not even sure that matter that much, given that in the meantime we have also changed the way we filter the request to be shown.

While digging a big deeper into the fetch internals I did notice that we seem to have an approach used for the Workers to make sure the loadInfo has a browsingContextID, which seems to have been introduced as part of Bug 1819570, would it be reasonable to introduce something along those lines to associate the browsingContextID related to the webpage the content script's Sandbox is associated with? wdyt?

Flags: needinfo?(lgreco)

Comment 23

•

3 months ago

(In reply to Luca Greco [:rpl] [:luca] [:lgreco] from comment #22)

While digging a big deeper into the fetch internals I did notice that we seem to have an approach used for the Workers to make sure the loadInfo has a browsingContextID, which seems to have been introduced as part of Bug 1819570, would it be reasonable to introduce something along those lines to associate the browsingContextID related to the webpage the content script's Sandbox is associated with? wdyt?

I think pursuing a similar approach to workers makes a lot of sense.
Please let me know if you require any assistance with that.

Comment 24

•

3 months ago

(In reply to Valentin Gosu [:valentin] (he/him) from comment #23)

(In reply to Luca Greco [:rpl] [:luca] [:lgreco] from comment #22)

While digging a big deeper into the fetch internals I did notice that we seem to have an approach used for the Workers to make sure the loadInfo has a browsingContextID, which seems to have been introduced as part of Bug 1819570, would it be reasonable to introduce something along those lines to associate the browsingContextID related to the webpage the content script's Sandbox is associated with? wdyt?

I think pursuing a similar approach to workers makes a lot of sense.
Please let me know if you require any assistance with that.

Thanks for confirming that following the approach used for the Workers sounds like one that would make sense.

Based on a brief look to Bug 1819570 stack of patches, it looks the change may likely be largely in the "Core :: Networking" / "DOM: Networking" components area (e.g. for changes to be applied to LoadInfo and for changes to DOM Fetch internals needed to propagate the browsingContextID?) and I'd guess maybe also with some changes that may end up in Sandbox.cpp's SandboxFetch (which is part of "Core :: XPConnect" component).

Could you or someone on your team help us getting a more complete picture about what those changes may look like?
Either by working on those changes or on some draft patches that would help us to complete the work using those drafts as a starting point.

Flags: needinfo?(valentin.gosu)

Comment 25

•

3 months ago

These seem to be the parts of devtools that decide when a request is shown in devtools
matchRequest
NetworkEventWatcher.shouldIgnoreChannel

If I make them return true, and look at the parent process browsing console I see an error here:
NetworkEventActor._createResource

So we probably need to account for this in getChannelBrowsingContextID - either by somehow extracting the window ID from the sandbox in devtools (there's a test that does it, I think) or we need to add it to the loadInfo and make the fetch, XHR and WS code check the sandbox metadata to determine the correct browsingContextID. I think we might need to do the second one, since I'm not sure if devtools is able to get the sandbox metadata for a particular channel. @bomsy - do you know if that's true?

In any case, for fetch I think we should be able to query the sandbox in mozilla::dom::Request::Constructor and set it on the load info in mozilla::dom::FetchDriver::HttpFetch.

In summary, unless devtools can already get the sandbox info for a channel opened in the content process, we'll need to:

change XHR, fetch and WS to get the browsing-context-id of the triggering page and stick it into load info
figure out if we need a new loadInfo field or if we can use the existing browsingContextId (would that be OK for the sandbox).
possibly change devtools to extract the correct browsingContextId from the channel's loadinfo.

Flags: needinfo?(valentin.gosu) → needinfo?(hmanilla)