1898490 - DevTools stop responsing

Status: RESOLVED FIXED

(DevTools :: General, defect, P2)

Description

[Matic Mercina]

Attachment: Screenshot 2024-05-23 141030.png

This has been happening after I updated to Firefox 126 on my Windows 11 laptop.

When running a locally hosted web application, the DevTools will often not open. The first time this happens, a blank window will open instead. After I close the blank window, neither F12 or "Inspect element" will do anything in the affected tab anymore. Opening a new tab will temporarily allow the use of DevTools again, but each time I close them, there is a chance they wont open anymore.

What I have already tried

My first thought was that my Firefox profile was broken. I therefore deleted the profile and created a new one (but I did login with my Firefox account, it's not a completely fresh profile). On day 1 after doing this I was certain I have solved the issue, since I had no problems the entire day. The day after that, the bug started re-appearing.

What can I do?

Is there any way for me to get more information about the issue? Does Firefox store logs somewhere on the hard drive, or can I configure it to do so?

Comment 1

[Matic Mercina]

Attachment: Devtools.png

A screenshot of the blank DevTools window.

Comment 2

[Nicolas Chevobbe [:nchevobbe]]

Thanks for the report Matic(In reply to Matic Mercina from comment #0)

What can I do?

Is there any way for me to get more information about the issue?

If you can share the local application code you're seeing this on, that would be great.
Also, are you experiencing that consistently?
Do you had a lot of other opened tabs when this happens? We know there's an issue with slow to initialize DevTools (Bug 1898463), so maybe you're hitting this, but it never settles?
You may also check that it's not a performance issue, recording a profile with https://profiler.firefox.com/ and sharing it here.

Does Firefox store logs somewhere on the hard drive, or can I configure it to do so?

You can open the Browser Console https://firefox-source-docs.mozilla.org/devtools-user/browser_console/index.html and copy all messages here when this happens.

Comment 3

[Matic Mercina]

Attachment: Error log - blank DevTools.png

Comment 4

[Matic Mercina]

Attachment: Error log - F12 unresponsive.png

Comment 5

[Matic Mercina]

(In reply to Nicolas Chevobbe [:nchevobbe] from comment #2)

If you can share the local application code you're seeing this on, that would be great.

Unfortunately it is part of a large legacy application, which is company code I am not allowed to share.

Also, are you experiencing that consistently?

The error usually happens at most an hour into a debug sesssion.

Do you had a lot of other opened tabs when this happens? We know there's an issue with slow to initialize DevTools (Bug 1898463), so maybe you're hitting this, but it never settles?

I usually have 2-5 tabs open.

You can open the Browser Console https://firefox-source-docs.mozilla.org/devtools-user/browser_console/index.html and copy all messages here when this happens.

I've attached two screenshots.

The first (Error log - blank DevTools.png) is when I pressed F12 and a blank window opened. Unfortunately, I am not entirely certain which message is the one that showed the moment I pressed F12, since having the Browser Console open seemingly makes the bug impossible (or at least way lesss likely) to trigger. Consequently, I had to trigger the bug with the Browser Console closed, and then open the console to see the output.

The second (Error log - F12 unresponsive.png ) is after I closed the blank window and tried to open the DevTools again with F12 (typeError: this.topWindow is null) and "Inspect element" (Error: can't select tool....).

Comment 6

[Julian Descottes [:jdescottes]]

When you reproduce the issue can you open about:processes and share the information there?
Maybe you can also give a try to Firefox ESR (https://www.mozilla.org/en-US/firefox/enterprise/) which is slightly older, it would help verify this is a Firefox 126 regression.

Comment 7

[Matic Mercina]

Attachment: about:process

Comment 8

[Matic Mercina]

I've attached a screenshot of the about:process page after reproducing the issue on tab 3 (the one with 202MB memory usage on the screenshot).

It may be of note that at least on this occasion, it was only the "Network" tab of the DevTools that was blank at first. Closing and reopening the tools then first caused the entire window to be blank, after which the tools no longer opened, as previously observed.

Comment 9

[Alexandre Poirot [:ochameau]]

Unfortunately, there is no actionable data out of about:processes.
So far the most useful data is attachment 9404054 [details] with the failure around watchTargets request.

Do you reproduce DevTools issues only against your company application?
Does that reproduce against a fresh instance of firefox with only one tab for that application?

Recording and sharing a performance profile recorded while trying to open DevTools the first time could help.
https://profiler.firefox.com/ Helps enabling the profiler in Firefox (nothing to install, this page will just help you enable it)

Comment 10

[Alexandre Poirot [:ochameau]]

I tried to break watchTargets with some extreme scenarios.

• Using a page with a worker having an infinite loop doesn't break DevTools:
  http://techno-barje.fr/fission/worker-busy/
  You are able to spawn DevTools and evaluate JS against the blocked workers (DevTools interrupt the infinite loop to execute the console input).

• Having a page paused doesn't prevent the other tabs from opening DevTools.

• Using a page doing an infinite loop... does prevent opening DevTools in any other page:
  data:text/html,<script>let i = 0; while(true) {i++; " ".repeat(100000);};</script>
  With such extreme scenario. DevTools would be left blank, and you would get similar exception as in comment 3:

JavaScript error: resource://devtools/server/connectors/js-process-actor/ContentProcessWatcherRegistry.sys.mjs, line 394: NotFoundError: No such JSProcessActor 'DevToolsProcess'
JavaScript error: resource://devtools/server/connectors/js-process-actor/ContentProcessWatcherRegistry.sys.mjs, line 394: NotFoundError: No such JSProcessActor 'DevToolsProcess'
JavaScript error: resource://devtools/server/connectors/js-process-actor/ContentProcessWatcherRegistry.sys.mjs, line 394: NotFoundError: No such JSProcessActor 'DevToolsProcess'
console.error: "Failed to sendQuery in DevToolsProcessParent" "DevToolsProcessParent:watchTargets"
console.error: "AbortError: Actor 'DevToolsProcess' destroyed before query 'DevToolsProcessParent:watchTargets' was resolved"
console.error: "Failed to sendQuery in DevToolsProcessParent" "DevToolsProcessParent:addOrSetSessionDataEntry"
console.error: "AbortError: Actor 'DevToolsProcess' destroyed before query 'DevToolsProcessParent:addOrSetSessionDataEntry' was resolved"
console.error: "Error while calling actor 'watcher's method 'watchTargets'" "Actor 'DevToolsProcess' destroyed before query 'DevToolsProcessParent:watchTargets' was resolved"

But looking at comment 7 data with about:processes, it doesn't seem related to an infinite loop.
It isn't clear how a Content process could possibly be stuck and not respond to JSProcess actor messages.

Comment 11

[Matic Mercina]

(In reply to Alexandre Poirot [:ochameau] from comment #9)

Do you reproduce DevTools issues only against your company application?

So far I have only ever reproduced the error against our company application, but I almost never use the DevTools anywhere else.

Does that reproduce against a fresh instance of firefox with only one tab for that application?

So far I haven't been able to immediately reproduce the error on a fresh instance with a single tab open. I have been able to reproduce the issue with a single tab open, but it was in an old instance of firefox in which i had opened and closed multiple tabs/DevTools before that.

Recording and sharing a performance profile recorded while trying to open DevTools the first time could help.

I am currently running firefox ESR, and will continue to do so for a few days (or until I manage to reproduce the error), just to confirm the issue is related to the current release. I will try getting more data after that.

Comment 12

[Alexandre Poirot [:ochameau]]

Attachment: example of browser console output

Here is a snippet to execute in the browser console, just after reproducing a blank devtools, without trying to close the tab or doing anything else:
ChromeUtils.getAllDOMProcesses().map(p=>p.getActor("DevToolsProcess")).map(async a=>{try{await a.sendQuery("DevToolsProcessParent:packet");}catch(e){console.error(a.manager.name, a.manager.childID, a.manager.remoteType, a.manager.canSend, "exception", e)}})

It should print something like the attached screenshot.
It will help understand which particular process if failing and causing this issue.

Comment 13

[Alexandre Poirot [:ochameau]]

I managed to reproduce with a script like this:

const ProcessTools = Cc["@mozilla.org/processtools-service;1"].getService(
  Ci.nsIProcessToolsService
);

setInterval(()=>{
  ProcessTools.kill(ChromeUtils.getAllDOMProcesses().filter(r=>r.remoteType=="privilegedabout")[0].osPid);
  gBrowser.addTab("about:home", {triggeringPrincipal:Services.scriptSecurityManager.getSystemPrincipal()})
});

When opening devtools on another tab.
This force creating and destroying content processes for about:home, which is privilegedabout.
So this confirms that if you are opening DevTools while a content process is being destroyed, it will break DevTools opening.

It is a bit surprising as the failing code looks like this:
https://searchfox.org/mozilla-central/rev/23efe2c8c5b3a3182d449211ff9036fb34fe0219/devtools/server/actors/watcher.js#251-267

    const domProcesses = ChromeUtils.getAllDOMProcesses();
    const promises = [];
    for (const domProcess of domProcesses) {
      if (domProcess == topLevelTargetProcess) {
        continue;
      }
      promises.push(
        domProcess.getActor(this._jsActorName).watchTargets({
          watcherActorID: this.actorID,
          targetType,
        })

In theory getAllDOMProcesses would only return valid processes, but somehow it ends up being destroyed which calling watchTargets.
It is being destroyed while processing this request in that content process.

All watcher code dispatching to all the content processes should probably avoid throwing when one content process throws while being destroyed.

Comment 14

[Alexandre Poirot [:ochameau]]

Attachment: Bug 1898490 - [devtools] Ignore closing content processes when trying to open DevTools.

The issue is that some calls to JSProcess Actor's would throw when trying to reach
content processes that get destroyed while DevTools are opening.
We have to reach out all the content processes as navigations may spawn
document in a process other than the currently's debugged tab's process.

We should ignore these failures. There isn't any explicit flag on DOM Process object
to identify destroyed processes, but canSend is a good fallback.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bd636e92feb0
[devtools] Ignore closing content processes when trying to open DevTools. r=devtools-reviewers,nchevobbe

Comment 16

[Sandor Molnar[:smolnar]]

Backed out for causing dt failures @ browser_toolbox_many_toggles.js

Backout link: https://hg.mozilla.org/integration/autoland/rev/135ac60877f55bc07e7f70030cb320d00a6f1d3f

Push with failures

Failure log -> TEST-UNEXPECTED-FAIL | devtools/client/framework/test/browser_toolbox_many_toggles.js | uncaught exception - NS_ERROR_FAILURE: Component returned failure code

Comment 17

[Alexandre Poirot [:ochameau]]

It looks like the killing API I'm using is easily throwing on Windows.
It sounds fine running this only on linux given that race condition is OS-agnostic.
https://treeherder.mozilla.org/jobs?repo=try&revision=c7b03e32f722a40b819b2b851ba9a2452c1b268c

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9b314490c4bf
[devtools] Ignore closing content processes when trying to open DevTools. r=devtools-reviewers,nchevobbe

Comment 19

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/9b314490c4bf

Comment 20

[Alexandre Poirot [:ochameau]]

Matic, Thanks a lot for your report. I think I found the culprit behind your issue and fixed it in Firefox Nightly.
Would you be able to test your broken scenario against Firefox Nightly (https://www.mozilla.org/firefox/all/#product-desktop-nightly)
and confirm that DevTools now work fine for you?

Comment 21

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1866814

Comment 22

[Alexandre Poirot [:ochameau]]

Comment on attachment 9404791 [details]
Bug 1898490 - [devtools] Ignore closing content processes when trying to open DevTools.

Beta/Release Uplift Approval Request

• User impact if declined: Some users may have broken devtools when opening them. The only way to open them is to close the tab and reopen the same URL in a new tab.
  This breakage reproduces when opening devtools while one content process is being destroyed while opening devtools. It is a race condition, but not so hard to reproduce.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce: No reliable steps to reproduce without a browser console script (comment 13).
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Low risk as this only add try/catch block to ignore this race condition.
• String changes made/needed:
• Is Android affected?: No

Comment 23

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9404791 [details]
Bug 1898490 - [devtools] Ignore closing content processes when trying to open DevTools.

127 is in RC now, moving the approval request accordingly.

Comment 24

[Pascal Chevrel:pascalc]

Comment on attachment 9404791 [details]
Bug 1898490 - [devtools] Ignore closing content processes when trying to open DevTools.

We have an RC2, taking as a ride-along.

Comment 25

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-release/rev/d31aa3980a10

Comment 26

[Matic Mercina]

(In reply to Alexandre Poirot [:ochameau] from comment #20)

Matic, Thanks a lot for your report. I think I found the culprit behind your issue and fixed it in Firefox Nightly.
Would you be able to test your broken scenario against Firefox Nightly (https://www.mozilla.org/firefox/all/#product-desktop-nightly)
and confirm that DevTools now work fine for you?

Sorry for the late reply, I am currently on vacation so I cannot test at this time. I will be sure to install Firefox Nightly (or beta, if it already contains the fix?) build when I get back.

Comment 27

[Matic Mercina]

Attachment: Browser console 129.0a1 (2024-06-11) (64-bit).png

Comment 28

[Matic Mercina]

Unfortunately the bug still appears in the latest nightly version.

As instructed, I tried to execute this snippet:

ChromeUtils.getAllDOMProcesses().map(p=>p.getActor("DevToolsProcess")).map(async a=>{try{await a.sendQuery("DevToolsProcessParent:packet");}catch(e){console.error(a.manager.name, a.manager.childID, a.manager.remoteType, a.manager.canSend, "exception", e)}})

But I don't know how to execute a code snippet in the browser console, since, unlike the regular DevTools console, there does not appear to be a text entry field anywhere.

Comment 29

[Nicolas Chevobbe [:nchevobbe]]

(In reply to Matic Mercina from comment #28)

Unfortunately the bug still appears in the latest nightly version.

As instructed, I tried to execute this snippet:

ChromeUtils.getAllDOMProcesses().map(p=>p.getActor("DevToolsProcess")).map(async a=>{try{await a.sendQuery("DevToolsProcessParent:packet");}catch(e){console.error(a.manager.name, a.manager.childID, a.manager.remoteType, a.manager.canSend, "exception", e)}})

But I don't know how to execute a code snippet in the browser console, since, unlike the regular DevTools console, there does not appear to be a text entry field anywhere.

Thanks for the feedback Matic!

For the Browser Console input, in the main Firefox window, go to about:config, search for devtools.chrome.enabled and set it to true. Then, next time you open the Browser Console, you should have the input visible and be able to execute snippet (see https://firefox-source-docs.mozilla.org/devtools-user/browser_console/index.html#browser-console-command-line)

Comment 30

[Matic Mercina]

Attachment: Script result.png

Comment 31

[Matic Mercina]

I've added a screenshot of the browser console output of the snippet.

It does not seem to be failing in the way you'd probably expected - all of the promises are fulfilled. There is however an error in the console, but I'm not sure it's related, as I am sometimes getting that same error when running the snippet even when the DevTools are seemingly working.

Comment 32

[Alexandre Poirot [:ochameau]]

Thanks Matic for testing, I think I found something in bug 1892411 which may relate to similar issue again.
I'll try to get the fix landed soon and ask you to test again on Nightly once it is fixed.

Comment 33

[Alexandre Poirot [:ochameau]]

Matic, would you mind testing again with the very latest nightly build?
I landed a fix in bug 1903980 yesterday, which fixes yet another case of blank devtools.

Comment 34

[Matic Mercina]

(In reply to Alexandre Poirot [:ochameau] from comment #33)

Matic, would you mind testing again with the very latest nightly build?
I landed a fix in bug 1903980 yesterday, which fixes yet another case of blank devtools.

I've been using Nightly for the last couple of weeks, but I haven't been developing client-side code for a while, so I haven't been using the DevTools much. I'll stay on Nightly for now, and will report back if I still have issues.

Comment 35

[Matic Mercina]

I've not had any issues using the Nightly in well over 2 weeks now, so I assume whatever was causing the issue is fixed.

Comment 36

[Julian Descottes [:jdescottes]]

Should be in RESOLVED FIXED, a patch landed here.