Closed Bug 1197789 Opened 9 years ago Closed 7 years ago

When I close devtools and then open inspector (occasional scenario), Nightly hangs, consumes all available RAM

Categories

(DevTools :: General, defect, P2)

43 Branch
defect

Tracking

(firefox48 wontfix, firefox49 wontfix, firefox50 wontfix, firefox51 wontfix)

RESOLVED INCOMPLETE
Tracking Status
firefox48 --- wontfix
firefox49 --- wontfix
firefox50 --- wontfix
firefox51 --- wontfix

People

(Reporter: arni2033, Unassigned)

References

(Blocks 2 open bugs)

Details

(Keywords: hang, regression, Whiteboard: [MemShrink:P2])

Attachments

(3 files)

STR:   (Win7, Nightly 43.0a1 (2015-08-23))  
1. Open devtools (Ctrl+Shift+I)
2. Click Inspector tab
3. Click Debugger tab
4. Wait ~5 seconds for them to load (sic)
5. Quickly press Ctrl+Shift+I and then Ctrl+Shift+C

Result:       Nightly consumes more and more RAM and the browser hung
Expectations: Inspector should open, like in DevEdition, for example

[I can reprocude it since ~18 August; in Nightly of 4 August everything was OK]
* This is happening in non-e10s as well.
Any exceptions show in the browser console (ctrl+shift+J) ?
I am not seeing this behavior on my local build (updated today).
Just to be sure, you open the inspector, then the debugger, wait a bit, then close with ctrl+shift+I then quickly after press ctrl+shift+C, and the inspector doesn't open?
(In reply to Patrick Brosset [:pbrosset] [:pbro] from comment #2)
Console freezes with the browser and doesn't display anything
> ctrl+shift+I then quickly after press ctrl+shift+C
Yes, I make no pause between ctrl+shift+I and ctrl+shift+C.
I couldn't reproduce on windows 7 or 10 or even on Mac.
I could not reproduce either.

Just to eliminate some possible variables, can you retest from a new Firefox profile to see if it occurs there too?

From the video, it looks like you are testing on about:newtab, is that correct?
Flags: needinfo?(arni2033)
I actually recorded this video in new profile. The issue remains in Nightly 43.0a1 (2015-08-24) in safe mode, with all plugins "never activate", on this page:
data:text/html,<body>
Just checked Nightly 43.0a1 (2015-08-04), and there's no issues there in same profile.
(I won't be available for ~2 days; then I'll try to use mozregression, but some guide would be welcome)
Flags: needinfo?(arni2033)
I used Mozregression GUI to find the regression window, and when bisection was over, I got this as last message in Log
> 2015-09-06T16:40:43: INFO : Narrowed inbound regression window from [6379ad07, 12434b4b] (3 revisions) to [c43b8d74, 12434b4b] (2 revisions) (~1 steps left)
So I think that the last good is c43b8d74, and the first bad is 12434b4b. There also was the link:
https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=c43b8d7471a8702e9fa7d46c283b905a09e3b3f7&tochange=12434b4b24a907979e5d70c7bf2c0d27095cf61e

I personally think it's caused by bug 1161072. Here's my info:
> Win7_64, Nightly 64bit, new profile
// I actually use 32bit firefox, but for this Mozregression testing I used 64big version since I found out that it exposes the same behavior
Blocks: 1161072
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Keywords: regression
I just thought that you may be not fast enough to perform my STR, so I made more reliable STR:

1. Open New Tab page   or   data:text/html,<body>   - it doesn't matter since it's devtools issue
2. Open devtools (Ctrl+Shift+I)
3. Click Inspector tab in devtools, wait until it's loaded
4. Click Debugger  tab in devtools, wait until it's loaded
5. Open Scratchpad (Shift+F4), execute the following 2 lines of code in browser context (without ">"s)
>   gDevToolsBrowser.toggleToolboxCommand(gBrowser);           // open/close devtools
>   gDevToolsBrowser.selectToolCommand(gBrowser, "inspector"); // initiate element picker

It works fine in Aurora 42, and causes memory leak in Nightly 43. You may also set timeout for 2nd line. (The functions in STR should be fixed if the functions for opening/closing devtools and initiating element picker will be changed in the browser)
Update: I downloaded "debug build" and saw only 1 repeating message when the bug happens. Here it is:

INFO : [1772] WARNING: No active window: file c:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/src/js/xpconnect/src/XPCJSRuntime.cpp, line 1432
>>>   My Info:   Win7_64, Nightly 51, 32bit, ID 20160825030226  (2016-08-25)
I still sometimes loose all my work and opened tabs while using devtools.
This bug is 100% reproducible to this day.
It's a very painful bug, and yet nobody tried to reproduce/fix it. During the year.   v_v
Keywords: hang
Summary: I'm able to cause memory leak in Nightly and hang the browser by opening Inspector and Debugger and then closing them → When I close devtools and then open inspector (occasional scenario), Nightly hangs, consumes all available RAM
Version: Trunk → 43 Branch
Thanks arni, sorry about that. We don't want you to lose your work and your tabs!
Andrei, can someone from your team try to reproduce this issue?
This should be at the very least a P2. Pinging bgrins for triage.
The STR in comment 10 didn't work for me exactly, but by just opening and closing/reopening the inspector, I was able to see the leak.

In fact, I got curious and did various tests:
1 - open about:home and measure memory for 4 minutes
2 - on about:home open the inspector, leave it open and measure memory for 4 minutes
3 - on about:home open the inspector then close it and measure memory for 4 minutes
4 - on about:home open the console, leave it open and measure memory for 4 minutes
5 - on about:home open the console then close it and measure memory for 4 minutes

In test 1, the memory used by Firefox stagnated at around 100MB.
In all other tests, whether this was the inspector or the console, and whether they stayed opened or were closed, the memory used by Firefox increased by ~2.5MB per second.
So after 4 minutes, I reached ~700MB.

I did another test where I just let Firefox run with the console open for 8 minutes and reached 1GB. So the rate of increase wasn't exactly constant, but close to it.

So it appears that we have a bad memory leak somewhere in the toolbox, and have had it for a long while.
Unfortunately, we never found the time to work on this bug so far. STRs in comment 0 and comment 10 may or may not be directly related to this exact leak, maybe there are several at play here. But investigation is required.

Brian: when you have a chance, can you make sure this goes to someone's radar? I think Alex would be a good person to help and he's working on bug fixing. But, as the component triage lead, I wanted to make sure this went to you first.
Flags: needinfo?(bgrinstead)
(In reply to Patrick Brosset <:pbro> from comment #14)
> The STR in comment 10 didn't work for me exactly, 
As of 2016-08-25, it requires web console to be opened (I remember it). See STR_2. Please test reported versions to avoid false negative results. (I can't download huge files currently, and only have Nightly 2016-05-26, so I can't test later versions)
If you use hack from comment 10, then you need to set timeout, as said in comment 10 - see [1]. 

STR_2:   (Win7_64, Nightly 51 (2016-08-25))  
0. Open url  data:text/html,<body></body>
1. Open web console (Ctrl+Shift+K)
2. Click inspector tab
3. Click debugger tab
4. Wait ~5 seconds for them to load
5. Quickly press Ctrl+Shift+I and then Ctrl+Shift+C

AR:  Hang
ER:  No hang

[1]
gDevToolsBrowser.toggleToolboxCommand(gBrowser);           // open/close devtools
setTimeout(function(){
  gDevToolsBrowser.selectToolCommand(gBrowser, "inspector"); // initiate element picker
},0);  // (I remember this too)
Blocks: 1313888
Whiteboard: [MemShrink:P2]
(In reply to Patrick Brosset <:pbro> from comment #14)
> This should be at the very least a P2. Pinging bgrins for triage.
> The STR in comment 10 didn't work for me exactly, but by just opening and
> closing/reopening the inspector, I was able to see the leak.
> 
> In fact, I got curious and did various tests:
> 1 - open about:home and measure memory for 4 minutes
> 2 - on about:home open the inspector, leave it open and measure memory for 4
> minutes
> 3 - on about:home open the inspector then close it and measure memory for 4
> minutes
> 4 - on about:home open the console, leave it open and measure memory for 4
> minutes
> 5 - on about:home open the console then close it and measure memory for 4
> minutes
> 
> In test 1, the memory used by Firefox stagnated at around 100MB.
> In all other tests, whether this was the inspector or the console, and
> whether they stayed opened or were closed, the memory used by Firefox
> increased by ~2.5MB per second.
> So after 4 minutes, I reached ~700MB.
> 
> I did another test where I just let Firefox run with the console open for 8
> minutes and reached 1GB. So the rate of increase wasn't exactly constant,
> but close to it.
> 
> So it appears that we have a bad memory leak somewhere in the toolbox, and
> have had it for a long while.
> Unfortunately, we never found the time to work on this bug so far. STRs in
> comment 0 and comment 10 may or may not be directly related to this exact
> leak, maybe there are several at play here. But investigation is required.
> 
> Brian: when you have a chance, can you make sure this goes to someone's
> radar? I think Alex would be a good person to help and he's working on bug
> fixing. But, as the component triage lead, I wanted to make sure this went
> to you first.

Thanks arni and Patrick for the details.  Do you have to be running a debug build?  When following these steps (using about:memory to measure usage) I don't see memory build up over time (in a local opt build).  Also when trying to reproduce the hang in Comment 10 and 15, I've been unable to so far.  This does sound bad so going to needinfo Alex and see if he has some cycles to look into this.
Flags: needinfo?(poirot.alex)
Flags: needinfo?(bgrinstead)
Flags: needinfo?(arni2033)
Priority: -- → P2
A possible suspect is the issue from bug 1297525.  Since we start the network monitor whenever the toolbox is open these days, it means we're holding the data for every network response in memory until you close.

The STR in comment 15 says you have to open the console though, so perhaps it's not that bug.
(In reply to J. Ryan Stinnett [:jryans] (use ni?) from comment #17)
> A possible suspect is the issue from bug 1297525.  Since we start the
> network monitor whenever the toolbox is open these days, it means we're
> holding the data for every network response in memory until you close.
> 
> The STR in comment 15 says you have to open the console though, so perhaps
> it's not that bug.

I believe the console is storing request/response logs for the network listener to show when expanding requests (at least in some cases).  But most of these STR have to do with loading data URIs, so I don't expect any network traffic to be captured (at least not for that tab)
(In reply to Brian Grinstead [:bgrins] from comment #16)
> Do you have to be running a debug build?
No, I was using a standard nightly build, built from fx-team locally, FF52, dating from end of October.
> When following these steps (using about:memory to measure usage) I
> don't see memory build up over time (in a local opt build).
I opened the same build from back then and was able to reproduce it immediately. On a static page (about:home), no user events, no network calls, no console activity. Just opened devtools on the inspector panel, and waited for a few minutes, looking at the memory consumption in the Windows process explorer app.
I can also see the leak in about:memory (n the main process explicit allocations section). It's typically always ~50Meg below what the process explorer reports, but it goes up at the same rate and never stops going up.
In about:memory, if I click on "minimize memory usage", I see the memory go down by ~100Meg, but then it quickly goes back up again.
> Also when trying to reproduce the hang in Comment 10 and 15, I've been unable to so
> far.
Me neither.
I re-tested with a more recent nightly: 52.0a1 (2016-11-13) on Windows 10 and 52.01a (2016-11-14) on Mac and I can confirm that the leak I was seeing seems to be now gone. I must have tested last week when Nightly was in a broken state.
I'm not able to reproduce the various STRs either (Tried them all).

But I saw bug 1317347 while opening a new tab after the various given STRs.
It ended up throwing tons of exceptions and bumping firefox memory size a lot.

Also all arni's STR involve element picker. I fixed really bad things happening around it in bug 1154645.


arni2033, I imagine you don't have any add-on installed? Or any special preference set?
Would you mind telling us what's your CPU and memory size?
Given the state of arni's account and the lack of ability of anyone else to reproduce, I'm closing this as incomplete. If anybody is able to reproduce this still, feel free to reopen.
Status: REOPENED → RESOLVED
Closed: 9 years ago7 years ago
Flags: needinfo?(poirot.alex)
Flags: needinfo?(arni2033)
Resolution: --- → INCOMPLETE
Product: Firefox → DevTools
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: