1635681 - Huge memory leak even while sleeping

Status: RESOLVED INCOMPLETE

(Firefox :: Untriaged, defect)

Description

[boite.pour.spam]

Attachment: before_sleeping-memory-report.json.gz

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:75.0) Gecko/20100101 Firefox/75.0

Steps to reproduce:

Since Firefox 74, it's consuming a large about of memory and never release it on MacOS. With my usual browsing experience, before 74, I never had to restart Firefox because of its memory impact, but now I need to.
As usual, with memory based report, there are so many factor that could cause this, it's hard to reproduce, so I simplified the test to the most simple case I could do, that is:

1. Started as safe mode without any extension
2. Loaded a bunch of tabs (my previous browsing session)
3. Saved a memory report
4. Set the computer to sleep during the night by closing lid
5. Wake up the computer
6. Saved memory report, memory increased by a lot

Actual results:

The memory increased by at least 300MB from the process manager.

Expected results:

The memory should more or less stay the same while time goes on. Please notice that I've tried to minimize the memory usage (in about:memory) but it's useless ?

Comment 1

[boite.pour.spam]

Attachment: after-sleep.memory-report.json.gz

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 3

[Marcela (Please NI request to Brindusa Tot)]

Hi!
Thank you for providing us those memory reports files.

Please reproduce the issue on the latest FF Nightly, it can be downloaded from here: https://nightly.mozilla.org/
Capture a performance profile using the Cleopatra add-on. You can get more info on how to install and use the Cleopatra add-on (that helps you get the performance profile) by going to:
https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Profiling_with_the_Built-in_Profiler
https://perf-html.io/

Mike Conley (:mconley) could you check the memory report and advise what component it should have?

Comment 4

[Firefox Bug Husbandry Bot]

Because this bug's Severity is normal and has not been changed, and this bug's priority is -- (none,) indicating it has has not been previously triaged, the bug's Severity is being updated to -- (default, untriaged.)

Comment 5

[Mike Conley (:mconley) (:⚙️)]

There's quite a lot under "heap-unclassified" in the parent process, which doesn't unfortunately tell us much. Hey mccr8, do you see anything else that stands out?

Comment 6

[boite.pour.spam]

I'm not able to find Cleopatra add on. The only thing related to Cleopatra is a theme, but I'm almost sure it's not that ;-)

The link you've given allows to profile performance issues, but anything memory related (like stack traces / allocations) is disabled for Mac OS (it says).
Do you really need this ?

I'll install FF nightly tomorrow and re-run the same test with the ~ same browsing session.

Comment 7

[Andrew McCreight [:mccr8]]

Given that this is on OSX, it sounds a little similar to bug 1625590, though the heap unclassified here isn't quite as high as there.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I have some questions to get an idea if it is the same as bug 1625590:

• When you created the memory dump how long was Firefox already running?
• Also how long do you keep Firefox open until you close and start it again?
• How often do you put your MBP into sleep mode?

Comment 9

[boite.pour.spam]

@Andrew: I'm not cycling through tabs via Ctrl+tab (I didn't even know about this shortcut).
@Henrik:

When you created the memory dump how long was Firefox already running?

For this report, I've started it maybe 2h before (since I wanted a clean dump). But in typical usage, with Firefox 73 and previous, I never stopped using it (it was running for weeks with almost constant memory usage). Since Firefox 74, and also in Firefox 75, it can't survive more than 2 days, since after a single day, it's consuming ~5GB of memory (all process included). Since it's almost impossible to debug a long running process, I've tried to over-simplify the experiment with just a single night, with no browsing whatsoever.

Also how long do you keep Firefox open until you close and start it again?

For the experiment above, I've stopped it around ~17:00 and restarted in debug mode at the time. By ~19:00, I captured the dump above, then closed the lid of the MBP. The morning, by ~09:00, I opened the lid and unlocked the screen then captured the second report (without browsing in between).

How often do you put your MBP into sleep mode?

By closing the lid: every night. By letting it sleep by itself, probably 3 to 4x a day. In the capture above, the "only" sleep is the lid closed sleep (it's not hibernating, just deep sleep). BTW, I was connected via a Ethernet adapter, so there is no Wifi reconnection going on for this test.

Comment 10

[boite.pour.spam]

Attachment: before-sleep.memory-report.json.gz

This is with Firefox Nightly

Comment 11

[boite.pour.spam]

Attachment: after-sleeping-memory-report.json.gz

This is with firefox nightly

Comment 12

[boite.pour.spam]

Same test with Firefox Nightly as requested. I had to copy and paste some tab URL to kind of replicate my initial test, and I only created 10 tabs instead of my initial ~30 tabs.

Before sleeping, the memory for the main process was ~260MB, after sleeping it's ~300MB, without any browsing in between.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Thanks for the additional information! But to clarify if this bug is a dupe of bug 1625590 would you mind running a DMD build of Firefox? I created one based on the most recent mozilla-central revision which is close to a Nightly build. If you want to test it maybe you should better use a separate profile.

The build can be downloaded from:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9ef764c270def8744b847847cee6f0efb82873db&selectedTaskRun=FW9zD4NHQLGBqOP3Ty2NOw-0

Click the Job Details pane below and find the target.dmd artifact link.

To start the build use the following command from the terminal (only update the path to Firefox):

DMD=1 Firefox\ Nightly\ DMD.app/Contents/MacOS/firefox --mode=dark-matter --stack=full

When DMD is enabled about:memory should not longer grey out the Save DMD output button.

Once you see an increase of memory save the DMD output. There will be files for each of the processes. Most important will be the file for the parent process (find its PID via the activity monitor - but mostly it will be the one with the lowest number). You can send me that tar.gz file, or pre-process it yourself. If you want to do the latter follow these steps:

• Clone the unified repository which can take quite a bit
• Run ./mach bootstrap and select option 1 for an artifact desktop build (when asked to configure mercurial you can answer with no)
• Create a mozconfig file with the output as printed at the end of the bootstrap command
• Run ./mach build
• Pre-process the DMD file with `obj/<path>/dist/bin/dmd.py --max-frames 24 -a dmd-<timestamp-pid>.json.gz >dmd-parent.txt
• Share the generated text file with us, or search for unreported with the highest amount of memory blocks

Thanks!

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2]]

As I just learned there is no need to actually use such a DMD build. You can also use the following command with your regular Nightly build of Firefox:

DMD=1 /Applications/Firefox\ Nightly.app/Contents/MacOS/firefox --mode=dark-matter --stack=full

The only caveat is that not all the symbol information will be available but we will get (some) function names.

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+2]]

boite.pour.spam would you have the time to do the above mentioned measurement and checks? We would kinda appreciate it.

Comment 16

[boite.pour.spam]

I'm trying to follow your instructions and will send you both file once it's done.
Is there a way to capture the list of the tabs opened in Firefox so I can directly open them in FF nightly without copying and pasting all URL by hand ?

BTW, this also happens on Windows (since version 74, Firefox will use all memory and finally crash its process tabs and then its main process, if left running for days).
I've tested on a low end computer with "only" 4GB of memory, so the effect appears very quickly with only 20 tabs or so.

It's strange you don't see this, it very obvious to spot if you browse for a little bit.

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to boite.pour.spam from comment #16)

Is there a way to capture the list of the tabs opened in Firefox so I can directly open them in FF nightly without copying and pasting all URL by hand ?

You can pick the sessionstore-backup folder from your profile and copy it over to the other one and start Firefox afterward. Or you completely duplicate this affected profile, which can then be opened with Firefox Nightly.

BTW, this also happens on Windows (since version 74, Firefox will use all memory and finally crash its process tabs and then its main process, if left running for days).
I've tested on a low end computer with "only" 4GB of memory, so the effect appears very quickly with only 20 tabs or so.

It's strange you don't see this, it very obvious to spot if you browse for a little bit.

I don't have a machine with that low memory, so I cannot test it myself. As such if you could provide the files it might help us already to further isolate the problem. Thanks.

Comment 18

[boite.pour.spam]

Attachment: dmd-before-sleeping.json.gz

Capture of DMD before sleeping

Comment 19

[boite.pour.spam]

Attachment: dmd-after-sleeping.json.gz

DMD capture after sleeping (FF Nightly went from 130MB of memory used to 440MB if I trust the process monitor output).

Comment 20

[boite.pour.spam]

Honestly, I don't see why the size of the system's RAM should change the behavior of FF. What I'm seeing seems to happen on all my machine (whether it's PC or Mac) so I expect it does also happens on yours. You can easily spot the behavior as I've described it.

Run FF, open some tabs (which are javascript hogs like Gmail / Slack / Github / ...) and capture the memory. Let your computer stays idle (for example, when you go to sleep) and when you wake up, observe the memory consumption. It'll have increase by a lot. It evens happens when the machine is sleeping (which is kind of strange, if not explainable).

Just from yesterday night to this morning FF nightly went from 130MB to 440MB with no user-interaction whatsoever and the machine sleeping, I can't believe you don't observe this behavior on your side.

Obviously, the memory usage never decrease by itself, I have to kill FF to claim the memory back.

Comment 21

[boite.pour.spam]

By the way, I've used "78.0a1 (2020-05-12) (64 bits)" FF Nightly version you've linked above.

Comment 22

[koumarin]

Noticed the same on

• Windows 10 2004
• Ryzen 5 1600
• 16 GB DDR4 3000 MHz RAM
• GTX 1660 Ti
• Firefox 78.0b4 (64-bit)

Right after Firefox launches, it takes about 2 GB RAM and everything works smoothly. Then, I open a couple of Reddit tabs, play a YouTube video, visit Facebook, and even after I close all those tabs, memory usage is at about 10 GB. I have to wait for every action to happen. Even now, I type this text in mostly blind, and wait until the content of the texbox catches up to what I wrote.

True, I do have 800 tabs open, but I never had those issues even at my peak of 2500 tabs.

Comment 23

[koumarin]

Also, just noticed, that yeah – RAM usage does increase over time, no matter what I do. It was 10 GB at the moment of writing this comment, but I remember it being about 5 GB a while before.

Comment 24

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to boite.pour.spam from comment #19)

Created attachment 9155222 [details]
dmd-after-sleeping.json.gz

DMD capture after sleeping (FF Nightly went from 130MB of memory used to 440MB if I trust the process monitor output).

Sorry that it has been taken a bit to check your uploaded DMD report. But I did it right now...

Please note that Firefox creates a couple of those files. One for each process. The file you attached doesn't show anything helpful. The largest unreported block here is 13MB. As such I'm fairly sure that you uploaded the wrong file. Please make sure to really upload the file for the parent process. Thanks.

Comment 25

[boite.pour.spam]

I'm 95% sure I've uploaded the file that was written in the about:memory "report saved here :" section (I did a copy and paste here, since the path is awful like "/var/xuihfsdiguhezogiuzhdpvoiuzehobzeiuhbozeiuhbozeiuhbzoiuh/").
Also, I only generated 2 DMD reports and the 2 files above have different size so that'd match my observation.

Are you sure FF is actually capturing all memory allocation here ?

Right now, I don't have the report anymore since I've rebooted my machine, I'll try to generate a new set tomorrow.

Comment 26

[Henrik Skupin [:whimboo][⌚️UTC+2]]

When you start Firefox from the command line as mentioned above you will see output like the following for each and every Firefox process:

opened /var/folders/c2/q_47wqh13n902fj9_zxczclr0000gn/T/dmd-<timestamp>-<process>.json.gz for writing

If you only get 2 files, did you restrict Firefox to only use 2 processes by modifying dom.ipc.processCount?

Whatever the first file should always be the parent (unless you have a process id overrun).

Comment 27

[Marcela (Please NI request to Brindusa Tot)]

Hi boite.pour.spam!
Were you be able to generate the report?
Remember first to check this pref to see if firefox has any restriction using processes: dom.ipc.processCoun

Comment 28

[Marcela (Please NI request to Brindusa Tot)]

Hi,
I'm marking this as Resolved-Incomplete due to lack of response. If the issue is still reproducible with the latest Firefox version, feel free to reopen the bug with more information.
Thanks,