1735345 - Huge memory usage on macos Monterey in [NSCursor set]; macOS popup warning saying machine is maxed out. 70+ GBs of memory shown in warning

Status: RESOLVED WORKSFORME

(Core :: Widget: Cocoa, defect, P2)

Description

[williamyoung]

Attachment: Firefox-sample-10122021.txt

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:95.0) Gecko/20100101 Firefox/95.0

Steps to reproduce:

Standard browsing, nothing heavier than normal

Actual results:

Mac popped up warning of max memory usage and prompted me to quit firefox ((95.0a1 (2021-10-12) (64-bit))

Expected results:

never had this happen before about 2 weeks ago

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Widget: Cocoa' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 2

[Timothy Nikkel (:tnikkel)]

Could you save a copy of about:memory (anonymized) and upload here when you are experiencing the high memory? (Before it gets to the point of force quitting)

Comment 3

[williamyoung]

Attachment: memory-report.json.gz

Memory report attached. MacOS popup showed 78.96GB memory usage, soon as it popped up I grabbed this report.

Comment 4

[Timothy Nikkel (:tnikkel)]

Thanks. 70 GB of heap unclassified in the parent process.

You seem to have a decent number of addons, is it possible to disable some of those to see if they are causing the problem?

Comment 5

[williamyoung]

Attachment: extensions-reduced_memory-report.json.gz

Disabled all extensions but for 8 shown in pic linked below

Enabled Extensions

Comment 6

[williamyoung]

Disabled all but those mentioned in previous post, restarted browser, then generated that report. I'll monitor this and update you if it happens again with extensions disabled or if i do not encounter the problem over next 1-2 days I will also let you know.

Comment 7

[Jeff Muizelaar [:jrmuizel]]

https://www.reddit.com/r/firefox/comments/q7yeyv/insane_memory_leak/ is another similar report.

williamyoung, do you use fullscreen very much?

Comment 8

[Vic]

I can confirm I am seeing this same behaviour - uncapped memory consumption using FF in the latest macOS Monterey beta - often being used full-screen.

Comment 9

[Jeff Muizelaar [:jrmuizel]]

mstange is going to try updating to Monterey to see if he can reproduce.

Vic, would you be able to try running Nightly Firefox and getting a DMD report? You should be able to do this by starting Firefox from the terminal:

DMD=1 /Applications/Firefox\ Nightly.app/Contents/MacOS/firefox-bin

Then go to about:memory and click "Save" under "Save DMD output". It should output a bunch of extra stuff in the terminal like:

DMD[836] $DMD = '1'
DMD[837] $DMD = '1'
DMD[839] $DMD = '1'
DMD[840] $DMD = '1'
DMD[841] $DMD = '1'
DMD[842] $DMD = '1'
DMD[844] $DMD = '1'
DMD[845] $DMD = '1'
DMD[846] $DMD = '1'
DMD[847] $DMD = '1'
DMD[849] $DMD = '1'
DMD[850] $DMD = '1'
DMD[851] $DMD = '1'
Unable to read VR Path Registry from /Users/jrmuizel/Library/Application Support/OpenVR/.openvr/openvrpaths.vrpath
DMD[855] $DMD = '1'
DMD[856] $DMD = '1'
DMD[857] $DMD = '1'
DMD[836] opened /var/folders/z2/52nrrnjx24z2fd60wd6glw1w0000gn/T/dmd-1634226999-836.json.gz for writing

Then you can run dmd on the first file that it writes (e.g. /var/folders/z2/52nrrnjx24z2fd60wd6glw1w0000gn/T/dmd-1634226999-836.json.gz)

$ /Applications/Firefox\ Nightly.app/Contents/Resources/dmd.py /var/folders/z2/52nrrnjx24z2fd60wd6glw1w0000gn/T/dmd-1634226999-836.json.gz  > /tmp/report.txt

You can then attach /tmp/report.txt to the bug.

More details here: https://firefox-source-docs.mozilla.org/xpcom/dmd.html and I'm happy to help if any steps are unclear.

Comment 10

[Vic]

I am unable to run the nightly build - I get a dialog pop up that says "Your Firefox profile cannot be loaded. It may be missing or inaccessible".

In the console. I see:

DMD=1 /Applications/Firefox\ Nightly.app/Contents/MacOS/firefox-bin 
DMD[23944] $DMD = '1'
DMD[23951] $DMD = '1'
2021-10-14 13:09:55.912 plugin-container[23951:635616] nil host used in call to allowsSpecificHTTPSCertificateForHost
2021-10-14 13:09:55.912 plugin-container[23951:635616] nil host used in call to allowsAnyHTTPSCertificateForHost:
2021-10-14 13:09:55.922 plugin-container[23951:635616] nil host used in call to allowsSpecificHTTPSCertificateForHost
2021-10-14 13:09:55.922 plugin-container[23951:635616] nil host used in call to allowsAnyHTTPSCertificateForHost:
2021-10-14 13:09:55.922 plugin-container[23951:635623] nil host used in call to allowsSpecificHTTPSCertificateForHost
2021-10-14 13:09:55.922 plugin-container[23951:635623] nil host used in call to allowsAnyHTTPSCertificateForHost:
UNSUPPORTED (log once): POSSIBLE ISSUE: unit 1 GLD_TEXTURE_INDEX_2D is unloadable and bound to sampler type (Float) - using zero texture because texture unloadable

Comment 11

[Vic]

I was able to run it by removing my Firefox/ folder and letting it make a new one

I was able to dump DMD files, however I get this when trying to analyze it:

Traceback (most recent call last):
  File "/Applications/Firefox Nightly.app/Contents/Resources/./dmd.py", line 1010, in <module>
    main()
  File "/Applications/Firefox Nightly.app/Contents/Resources/./dmd.py", line 1002, in main
    digest = getDigestFromFile(args, args.input_file)
  File "/Applications/Firefox Nightly.app/Contents/Resources/./dmd.py", line 307, in getDigestFromFile
    fixStackTraces(inputFile, isZipped, opener)
  File "/Applications/Firefox Nightly.app/Contents/Resources/./dmd.py", line 293, in fixStackTraces
    tmpFile.write(fix(line))
  File "/Applications/Firefox Nightly.app/Contents/Resources/./dmd.py", line 268, in fix
    return fixModule.fixSymbols(line, jsonMode=True)
  File "/Applications/Firefox Nightly.app/Contents/Resources/fix_stacks.py", line 82, in fixSymbols
    initFixStacks(jsonMode, slowWarning, breakpadSymsDir, hide_errors)
  File "/Applications/Firefox Nightly.app/Contents/Resources/fix_stacks.py", line 38, in initFixStacks
    raise Exception("cannot find `fix-stacks`; please run `./mach bootstrap`")
Exception: cannot find `fix-stacks`; please run `./mach bootstrap`

Comment 12

[Jeff Muizelaar [:jrmuizel]]

You can try again with the --no-fix-stacks option.

Comment 13

[Vic]

Got it - download the report at https://mvgrafx.net/~vmark/ff_n_DMD_report.txt

Comment 14

[Vic]

Attachment: report.txt

DMD report

Comment 15

[Vic]

FYI - still confirm seeing unbound memory use with the nightly build with a completely fresh profile and no extensions.

Comment 16

[Markus Stange [:mstange]]

This is super useful, thank you Vic! It seems to point at mouse cursor images being leaked.

Could you run dmd.py again, and also specify --max-frames 16? At the moment we only get 8 stack frames (the default), which might hide some interesting details.

Comment 17

[Vic]

Attachment: report-16.txt

DMD report with --max-frames = 16

Comment 18

[Markus Stange [:mstange]]

I've filed FB9704004 about this.

-[NSCursor set] appears to be leaking images, causes 70GB leak in Firefox

This appears to be a regression in one of the recent macOS Monterey Betas.

Two Firefox users reported a very large memory usage. We’re tracking this in https://bugzilla.mozilla.org/show_bug.cgi?id=1735345 .
We have a memory report which blames the allocations on this stack:

#02: _malloc_zone_malloc[/usr/lib/system/libsystem_malloc.dylib +0x1e770]
#03: createImageWithSizeRenderInstructions[/System/Library/PrivateFrameworks/AccessibilitySupport.framework/Versions/A/Frameworks/AccessibilityFoundation.framework/Versions/A/AccessibilityFoundation +0x7430]
#04: -[_AXFMouseCursorGeneratorLayer _createImageForScale:color:][/System/Library/PrivateFrameworks/AccessibilitySupport.framework/Versions/A/Frameworks/AccessibilityFoundation.framework/Versions/A/AccessibilityFoundation +0x66e0]
#05: -[_AXFMouseCursorGenerator createImageForScale:][/System/Library/PrivateFrameworks/AccessibilitySupport.framework/Versions/A/Frameworks/AccessibilityFoundation.framework/Versions/A/AccessibilityFoundation +0x5064]
#06: -[_AXFMouseCursorGenerator _registerWithName:connection:isGlobal:setActive:seed:][/System/Library/PrivateFrameworks/AccessibilitySupport.framework/Versions/A/Frameworks/AccessibilityFoundation.framework/Versions/A/AccessibilityFoundation +0x5418]
#07: _AXFCursorSetAndReturnSeed[/System/Library/PrivateFrameworks/AccessibilitySupport.framework/Versions/A/Frameworks/AccessibilityFoundation.framework/Versions/A/AccessibilityFoundation +0x70b8]
#08: _AXInterfaceCursorSetAndReturnSeed[/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices +0x2d1d8]
#09: -[NSCursor _reallySet][/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit +0x1e01d0]
#10: -[NSCursor set][/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit +0x1e0050]

Firefox calls -[NSCursor set] very frequently during mouse motions, so many leaked images might accumulate.

Comment 19

[Vic]

Wanted to add to this if it helps: I realized that this issue probably corresponded to when I used the new accessibility features to make the mouse cursor larger and inverted the colours. To test the theory I went back to accessibility and set the cursor size back to normal and the colours to default.

The issue is either gone or significantly mitigated. More time is needed to see if it grows out of control - but as I write this I've been to a few different sites with multiple tabs and memory usage sits at a comfortable 608MB.

So this is definitely related to the new cursor accessibility features.

Comment 20

[Markus Stange [:mstange]]

That makes a lot of sense.

Comment 21

[Eric Schlegel]

Thanks for filing this issue with Apple; one of my coworkers is looking into it now.

Comment 22

[Markus Stange [:mstange]]

Thanks Eric!

Comment 23

[Markus Stange [:mstange]]

I'm also debugging why Firefox sets the cursor so frequently. This is some ancient code.

Comment 24

[Emilio Cobos Álvarez (:emilio)]

(In reply to Markus Stange [:mstange] from comment #23)

I'm also debugging why Firefox sets the cursor so frequently. This is some ancient code.

We do this unconditionally, but it seems we could avoid it?

Comment 25

[Emilio Cobos Álvarez (:emilio)]

I guess this is the problematic situation, and we can probably avoid it nowadays that binary plugins are not a thing?

Comment 26

[Markus Stange [:mstange]]

(mid-aired with emilio here)

It's from this code, which I added 12 years ago to work around a problem with plug-ins in bug 496601, and which gets called on every mouse move:

  [[NSCursor currentCursor] set];

I've filed bug 1736049 to try to remove it.

Comment 27

[Jeff Muizelaar [:jrmuizel]]

This is reproducible on x64 as well.

Comment 28

[Vic]

It seems like the underlying issue in macOS has been fixed as of Monterey RC2 (21A559)

Comment 29

[Jeff Muizelaar [:jrmuizel]]

Vic, are you sure? Are you testing Firefox 93? Haik can still reproduce it on 21A559

Comment 30

[Vic]

I re-enabled a large cursor and didn’t observe any rampant memory growth. I didn’t test extensively though. I will give it a more thorough test this evening

Comment 31

[Haik Aftandilian [:haik]]

Attachment: Repro Settings

I'm using the attached settings to reproduce the problem. Moving the mouse around a lot seems to accelerate the memory usage and it is attributed to the Firefox process in Activity Monitor. Closing all tabs doesn't reduce the Firefox memory use. In my case, it is still reproducible on macOS 21A559 with Firefox 93.0 on x64, MacBook Pro 2018.

Comment 32

[Vic]

Yes, sorry - I retract my claim. Though I will say it appears setting the colours has more of an impact than setting the size.

Comment 34

[dick fickling]

I've been encountering this bug as well. I'm on the official MacOS Monterey release and had my cursor set to a larger-than-normal size as well as custom colors. Having returned those settings to their default, the memory leak seems to have stopped. Happy to share a memory dump or anything if y'all need it, but it looks like Markus has fixed the issue in #1736049.

Comment 36

[Kelimutu [:kiki]]

We received similar report on Twitter recently: https://twitter.com/y80481629/status/1453305283317354496

macOS 12.0.1 (21A559)
Firefox 93.0 (64-bit)
MacBook Pro (16-inch, 2021)

Comment 37

[Markus Stange [:mstange]]

Firefox 94 just came out, and it contains a change which greatly reduces the severity of the leak. But some leak still remains, and it needs to be fixed by Apple in a macOS update.

Comment 38

[Kelimutu [:kiki]]

Thanks for the info, Markus! I'll ask folks to update in the meantime.

Comment 39

[L]

Seems this may have fixed it for some users:
https://news.ycombinator.com/item?id=29143270

Comment 40

[Eric Schlegel]

Apple engineering expects that the underlying issue in macOS is fixed in Monterey 12.1 Beta 3 (21C5039b), released today.

Comment 41

[Markus Stange [:mstange]]

I just tested 12.1 Beta (21C5039b) and it does not appear to be fixed. In Firefox 93 I can still get memory usage to climb up to 5GB within two minutes of moving the mouse.

Comment 42

[BugBot [:suhaib / :marco/ :calixte]]

Redirect a needinfo that is pending on an inactive user to the triage owner.
:spohl, since the bug has high priority, high severity and recent activity, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 43

[Stephen A Pohl [:spohl]]

I'm not able to reproduce this in macOS Monterey 12.5 Beta (21G5027d), but I didn't originally experience this issue. Haik or Markus, could one of you confirm that this is indeed fixed now?

Comment 44

[Markus Stange [:mstange]]

Looking at my comment history on the Apple bug report FB9704004, it looks like this was fixed a while ago in macOS 12.1 beta 4 (21C5045a). I confirmed the fix in macOS at the time.