937878 - Extremely high layer memory usage on Mac when browser windows are minimized

Status: VERIFIED FIXED

(Core :: Graphics, defect)

Description

[(no longer active)]

Attachment: DMD log (gzipped)

I have noticed Nightly sometimes using a ton of memory in the past few weeks (I caught it using 14GB on one occasion.)  Nearly all of this memory is in heap-unclassified.  I've been running my own DMD builds and I finally managed to capture a DMD log which I'm attaching here.

about:memory:

2,737.31 MB (100.0%) -- explicit
├──1,207.06 MB (44.10%) ── heap-unclassified
├──1,083.57 MB (39.59%) -- window-objects
│  ├────660.59 MB (24.13%) -- top(https://bugzilla.mozilla.org/show_bug.cgi?id=915735, id=144)/active
│  │    ├──420.79 MB (15.37%) ++ (19 tiny)
│  │    ├───47.98 MB (01.75%) -- window(https://tbpl.mozilla.org/?tree=Try&rev=f10ab7dbe376&embed=true)
│  │    │   ├──40.52 MB (01.48%) -- js-compartment(https://tbpl.mozilla.org/?tree=Try&rev=f10ab7dbe376&embed=true)
│  │    │   │  ├──38.85 MB (01.42%) -- objects
│  │    │   │  │  ├──33.75 MB (01.23%) -- gc-heap
│  │    │   │  │  │  ├──29.49 MB (01.08%) ── ordinary [2]
│  │    │   │  │  │  └───4.26 MB (00.16%) ++ (2 tiny)
│  │    │   │  │  └───5.10 MB (00.19%) ++ (2 tiny)
│  │    │   │  └───1.67 MB (00.06%) ++ (4 tiny)
│  │    │   └───7.46 MB (00.27%) ++ (3 tiny)
│  │    ├───47.97 MB (01.75%) -- window(https://tbpl.mozilla.org/?tree=Try&rev=7b028bad1246&embed=true)
│  │    │   ├──40.53 MB (01.48%) -- js-compartment(https://tbpl.mozilla.org/?tree=Try&rev=7b028bad1246&embed=true)
│  │    │   │  ├──38.86 MB (01.42%) -- objects
│  │    │   │  │  ├──33.76 MB (01.23%) -- gc-heap
│  │    │   │  │  │  ├──29.50 MB (01.08%) ── ordinary [2]
│  │    │   │  │  │  └───4.26 MB (00.16%) ++ (2 tiny)
│  │    │   │  │  └───5.10 MB (00.19%) ++ (2 tiny)
│  │    │   │  └───1.67 MB (00.06%) ++ (4 tiny)
│  │    │   └───7.44 MB (00.27%) ++ (3 tiny)
│  │    ├───47.96 MB (01.75%) -- window(https://tbpl.mozilla.org/?tree=Try&rev=ff699478f972&embed=true)
│  │    │   ├──40.53 MB (01.48%) -- js-compartment(https://tbpl.mozilla.org/?tree=Try&rev=ff699478f972&embed=true)
│  │    │   │  ├──38.86 MB (01.42%) -- objects
│  │    │   │  │  ├──33.76 MB (01.23%) -- gc-heap
│  │    │   │  │  │  ├──29.50 MB (01.08%) ── ordinary [2]
│  │    │   │  │  │  └───4.26 MB (00.16%) ++ (2 tiny)
│  │    │   │  │  └───5.10 MB (00.19%) ++ (2 tiny)
│  │    │   │  └───1.67 MB (00.06%) ++ (4 tiny)
│  │    │   └───7.43 MB (00.27%) ++ (3 tiny)
│  │    ├───47.95 MB (01.75%) -- window(https://tbpl.mozilla.org/?tree=Try&rev=6b1d8b2d7b7d&embed=true)
│  │    │   ├──40.53 MB (01.48%) -- js-compartment(https://tbpl.mozilla.org/?tree=Try&rev=6b1d8b2d7b7d&embed=true)
│  │    │   │  ├──38.86 MB (01.42%) -- objects
│  │    │   │  │  ├──33.76 MB (01.23%) -- gc-heap
│  │    │   │  │  │  ├──29.50 MB (01.08%) ── ordinary [2]
│  │    │   │  │  │  └───4.26 MB (00.16%) ++ (2 tiny)
│  │    │   │  │  └───5.10 MB (00.19%) ++ (2 tiny)
│  │    │   │  └───1.67 MB (00.06%) ++ (4 tiny)
│  │    │   └───7.42 MB (00.27%) ++ (3 tiny)
│  │    └───47.94 MB (01.75%) -- window(https://tbpl.mozilla.org/?tree=Try&rev=3a79c1775830&embed=true)
│  │        ├──40.52 MB (01.48%) -- js-compartment(https://tbpl.mozilla.org/?tree=Try&rev=3a79c1775830&embed=true)
│  │        │  ├──38.85 MB (01.42%) -- objects
│  │        │  │  ├──33.75 MB (01.23%) -- gc-heap
│  │        │  │  │  ├──29.49 MB (01.08%) ── ordinary [2]
│  │        │  │  │  └───4.26 MB (00.16%) ++ (2 tiny)
│  │        │  │  └───5.10 MB (00.19%) ++ (2 tiny)
│  │        │  └───1.67 MB (00.06%) ++ (4 tiny)
│  │        └───7.42 MB (00.27%) ++ (3 tiny)
│  ├────144.35 MB (05.27%) -- top(https://www.facebook.com/, id=15)/active
│  │    ├──138.12 MB (05.05%) -- window(https://www.facebook.com/)
│  │    │  ├───79.68 MB (02.91%) -- js-compartment(https://www.facebook.com/)
│  │    │  │   ├──53.36 MB (01.95%) -- objects
│  │    │  │   │  ├──35.99 MB (01.31%) ++ gc-heap
│  │    │  │   │  └──17.36 MB (00.63%) ++ (2 tiny)
│  │    │  │   └──26.32 MB (00.96%) ++ (7 tiny)
│  │    │  ├───42.70 MB (01.56%) -- dom
│  │    │  │   ├──37.65 MB (01.38%) ── orphan-nodes
│  │    │  │   └───5.05 MB (00.18%) ++ (5 tiny)
│  │    │  └───15.74 MB (00.57%) ++ (3 tiny)
│  │    └────6.23 MB (00.23%) ++ (3 tiny)
│  ├────125.01 MB (04.57%) ++ (96 tiny)
│  ├────115.08 MB (04.20%) -- top(https://mail.google.com/mail/u/0/?ui=2&shva=1#inbox, id=13)/active
│  │    ├───46.64 MB (01.70%) -- window(https://mail.google.com/mail/u/0/?ui=2&shva=1#inbox)
│  │    │   ├──28.53 MB (01.04%) ++ dom
│  │    │   └──18.11 MB (00.66%) ++ (4 tiny)
│  │    ├───44.73 MB (01.63%) -- window(https://mail.google.com/_/scs/mail-static/_/js/k=gmail.main.en.IrCl4h8mIb4.O/m=m_i,t,it/am=vyBOkvLv2oERh2EH9pNO__3ffRa78X9dvb0AoQIQ2Kv-f__C6gc-qD_O/rt=h/d=1/rs=AItRSTOaTLA3ULYJ1TYuLl_zkZ0RZSVM8Q)
│  │    │   ├──39.74 MB (01.45%) ++ js-compartment(https://mail.google.com/_/scs/mail-static/_/js/k=gmail.main.en.IrCl4h8mIb4.O/m=m_i,t,it/am=vyBOkvLv2oERh2EH9pNO__3ffRa78X9dvb0AoQIQ2Kv-f__C6gc-qD_O/rt=h/d=1/rs=AItRSTOaTLA3ULYJ1TYuLl_zkZ0RZSVM8Q)
│  │    │   └───4.99 MB (00.18%) ++ (5 tiny)
│  │    └───23.71 MB (00.87%) ++ (5 tiny)
│  └─────38.55 MB (01.41%) ++ top(none)/detached
├────219.21 MB (08.01%) -- js-non-window
│    ├──161.43 MB (05.90%) -- zones
│    │  ├──120.79 MB (04.41%) -- zone(0x10c728800)
│    │  │  ├───85.75 MB (03.13%) ++ (362 tiny)
│    │  │  └───35.04 MB (01.28%) ++ strings
│    │  ├───36.20 MB (01.32%) -- zone(0x1087ee800)
│    │  │   ├──35.77 MB (01.31%) ++ strings
│    │  │   └───0.42 MB (00.02%) ++ (4 tiny)
│    │  └────4.44 MB (00.16%) ++ (13 tiny)
│    ├───41.10 MB (01.50%) ++ runtime
│    └───16.67 MB (00.61%) ++ gc-heap
├─────91.84 MB (03.36%) ++ (17 tiny)
├─────54.61 MB (02.00%) -- heap-overhead
│     ├──50.87 MB (01.86%) ── waste
│     └───3.74 MB (00.14%) ++ (2 tiny)
├─────48.24 MB (01.76%) ++ add-ons
└─────32.78 MB (01.20%) ++ dmd

The most relevant bits of the DMD log:

 4402 Unreported: 40 blocks in stack trace record 1 of 7,923
 4403  162,930,688 bytes (162,865,264 requested / 65,424 slop)
 4404  6.86% of the heap (6.86% cumulative);  30.04% of unreported (30.04% cumulative)
 4405  Allocated at
 4406    replace_malloc (DMD.cpp:1227, in libdmd.dylib) 0x100008e2d
 4407    malloc (in libsystem_c.dylib) + 41 0x7fff93171183
 4408    malloc_zone_malloc (in libsystem_c.dylib) + 71 0x7fff93171bd7
 4409    js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) (jscntxtinlines.h:220, in XUL) 0x1029187e1
 4410    mozilla::layers::ISurfaceAllocator::AllocSurfaceDescriptorWithCaps(nsIntSize const&, gfxContentType, unsigned int, mozilla::layers::SurfaceDescriptor*) (mozalloc.h:219, in
 4411    mozilla::layers::ContentClientIncremental::GetUpdateSurface(mozilla::layers::ContentClientIncremental::BufferType, nsIntRegion&) (ContentClient.cpp:950, in XUL) 0x1028ec70
 4412    mozilla::layers::ContentClientIncremental::BeginPaintBuffer(mozilla::layers::ThebesLayer*, gfxContentType, unsigned int) (ContentClient.cpp:876, in XUL) 0x1028e6d85
 4413    mozilla::layers::ClientThebesLayer::PaintThebes() (nsRegion.h:368, in XUL) 0x1028e73e4
 4414    mozilla::layers::ClientThebesLayer::RenderLayer() (RefPtr.h:182, in XUL) 0x1028e5227
 4415    mozilla::layers::ClientContainerLayer::RenderLayer() (nsTArray.h:371, in XUL) 0x1028e5227
 4416    mozilla::layers::ClientContainerLayer::RenderLayer() (nsTArray.h:371, in XUL) 0x1028e63ec
 4417    mozilla::layers::ClientLayerManager::EndTransactionInternal(void (*)(mozilla::layers::ThebesLayer*, gfxContext*, nsIntRegion const&, mozilla::layers::DrawRegionClip, nsInt
 4418    mozilla::layers::ClientLayerManager::EndTransaction(void (*)(mozilla::layers::ThebesLayer*, gfxContext*, nsIntRegion const&, mozilla::layers::DrawRegionClip, nsIntRegion c
 4419    nsDisplayList::PaintForFrame(nsDisplayListBuilder*, nsRenderingContext*, nsIFrame*, unsigned int) const (nsDisplayList.h:302, in XUL) 0x10169acde
 4420    nsDisplayList::PaintRoot(nsDisplayListBuilder*, nsRenderingContext*, unsigned int) const (GeckoProfilerImpl.h:287, in XUL) 0x1016bc319
 4421    nsLayoutUtils::PaintFrame(nsRenderingContext*, nsIFrame*, nsRegion const&, unsigned int, unsigned int) (nsLayoutUtils.cpp:2283, in XUL) 0x1016d9269
 4422    PresShell::Paint(nsView*, nsRegion const&, unsigned int) (nsPresShell.cpp:5717, in XUL) 0x101b9b231
 4423    nsViewManager::ProcessPendingUpdatesForView(nsView*, bool) (nsRegion.h:82, in XUL) 0x1016e2c75
 4424    nsRefreshDriver::Tick(long long, mozilla::TimeStamp) (nsISupportsImpl.h:187, in XUL) 0x1016e3e28
 4425    mozilla::RefreshDriverTimer::Tick() (nsTArray.h:371, in XUL) 0x1028553f3
 4426    nsTimerImpl::Fire() (nsTimerImpl.cpp:561, in XUL) 0x1028554e8
 4427    nsTimerEvent::Run() (nsTimerImpl.cpp:630, in XUL) 0x102852bad
 4428    nsThread::ProcessNextEvent(bool, bool*) (nsThread.cpp:610, in XUL) 0x10280ee51
 4429    NS_ProcessPendingEvents(nsIThread*, unsigned int) (nsThreadUtils.cpp:201, in XUL) 0x1021a1fca

Comment 1

[(no longer active)]

I filed bug 937882 to add memory reporters for this stuff.

Comment 2

[Nathan Froyd [:froydnj]]

(In reply to :Ehsan Akhgari (needinfo? me!) from comment #0)
> 2,737.31 MB (100.0%) -- explicit
> ├──1,207.06 MB (44.10%) ── heap-unclassified
> ├──1,083.57 MB (39.59%) -- window-objects
> │  ├────660.59 MB (24.13%) --
> top(https://bugzilla.mozilla.org/show_bug.cgi?id=915735, id=144)/active
> │  │    ├──420.79 MB (15.37%) ++ (19 tiny)
> │  │    ├───47.98 MB (01.75%) --
> window(https://tbpl.mozilla.org/?tree=Try&rev=f10ab7dbe376&embed=true)

I'm really curious about this.  Do you really have 660MB for a bugzilla window?  (That seems extremely high.)  What's going on in those "tiny" 420MB bits?

Comment 3

[Jet Villegas (inactive)]

To Nick for a look. Can you review the DMD stack and Layer Manager code for clues here?

Comment 4

[Nicholas Nethercote [inactive]]

BTW, bug 915940 has patches implementing a memory reporter for these allocations, but it bounced due to problems with PGO builds.

Comment 5

[Nick Cameron [:nrc]]

I tried and failed to land memory reporters for this in bug 915940. I'll try again tomorrow. I had a similar bug like this but on Android and it was due to leaking the textures backing canvases. But from the stack, this looks like ContentClientIncremental. That is a bit odd because we should never have a texture larger than the screen there, so we would have to leak _a lot_ to get up to 14 GB. Also, it should affect everyone and be kind of noticeable. I'll look in more depth tomorrow.

Matt - any thoughts?

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Snippet from my DMD report

This is what DMD says is the culprit on my main profile, roughly 10 minutes after startup. Looks pretty similar to Ehsan's call stack, and mine will happily keep going until it's using > 10G of memory too.

Using new profile makes the problem go away, but I was able to reduce my main profile to these STR:
1, create new profile
2, open https://www.nelsontop10.co.nz/standard-cabin.php
3, minimize the window (to the dock)

I see the allocated memory go up by ~20MB every 15 seconds. The memory drops again when the window is brought back onto the display, but not all the way to what is was when first minimized. The memory stays constant if I leave the window on display. The problem goes away if layers.offmainthreadcomposition.enabled is set to false before starting Firefox up.

Comment 7

[(no longer active)]

(In reply to comment #5)
> I tried and failed to land memory reporters for this in bug 915940. I'll try
> again tomorrow. I had a similar bug like this but on Android and it was due to
> leaking the textures backing canvases. But from the stack, this looks like
> ContentClientIncremental. That is a bit odd because we should never have a
> texture larger than the screen there, so we would have to leak _a lot_ to get
> up to 14 GB. Also, it should affect everyone and be kind of noticeable. I'll
> look in more depth tomorrow.
> 
> Matt - any thoughts?

FWIW, I usually connect to an external monitor, and I was indeed connected to the said monitor  when this happened yesterday.  But most of the times that I've seen this I've been at home connected to my AppleTV through Air Play.  Not sure if any of this makes any difference, but I thought I'd mention it.

Comment 8

[(no longer active)]

(In reply to Nathan Froyd (:froydnj) from comment #2)
> (In reply to :Ehsan Akhgari (needinfo? me!) from comment #0)
> > 2,737.31 MB (100.0%) -- explicit
> > ├──1,207.06 MB (44.10%) ── heap-unclassified
> > ├──1,083.57 MB (39.59%) -- window-objects
> > │  ├────660.59 MB (24.13%) --
> > top(https://bugzilla.mozilla.org/show_bug.cgi?id=915735, id=144)/active
> > │  │    ├──420.79 MB (15.37%) ++ (19 tiny)
> > │  │    ├───47.98 MB (01.75%) --
> > window(https://tbpl.mozilla.org/?tree=Try&rev=f10ab7dbe376&embed=true)
> 
> I'm really curious about this.  Do you really have 660MB for a bugzilla
> window?  (That seems extremely high.)  What's going on in those "tiny" 420MB
> bits?

I have an add-on called BugzillaJS, and one of the things that it does is add an iframe for try server links in a bug's comments.  I don't have the same thing in my about:memory right now but that bug has a whole bunch of try runs in it, so I would expect most of that memory to be consumed by the TBPL iframes.

Comment 9

[(no longer active)]

(In reply to Nick Thomas [:nthomas] from comment #6)
> Created attachment 831395 [details]
> Snippet from my DMD report
> 
> This is what DMD says is the culprit on my main profile, roughly 10 minutes
> after startup. Looks pretty similar to Ehsan's call stack, and mine will
> happily keep going until it's using > 10G of memory too.
> 
> Using new profile makes the problem go away, but I was able to reduce my
> main profile to these STR:
> 1, create new profile
> 2, open https://www.nelsontop10.co.nz/standard-cabin.php
> 3, minimize the window (to the dock)
> 
> I see the allocated memory go up by ~20MB every 15 seconds. The memory drops
> again when the window is brought back onto the display, but not all the way
> to what is was when first minimized. The memory stays constant if I leave
> the window on display. The problem goes away if
> layers.offmainthreadcomposition.enabled is set to false before starting
> Firefox up.

Huh, interesting!  I do have a minimized window in my profile...

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Using a secondary monitor doesn't make a difference for me. FWIW, I have OSX 10.8.5, MacBook Pro Retina Mid 2012 (Intel HD Graphics 4000 and NVidia GeForce GT 650M).

Comment 11

[(no longer active)]

FWIW I just did an experiemnt.  My Firefox was consuming around 7.7GB of memory.  As soon as I restored the minimized window, the memory usage dropped to 2.5GB, so the window being minimized seems to be the cause of this leak.

Comment 12

[Matt Woodrow (:mattwoodrow)]

Attachment: Don't leak ContentHostIncremental queue up too many updates

We disable the compositor when the window is minimized, but this means that we never process the update queue for the compositables.

As long as something is animating and producing updates, then they just queue up until you bring the window back.

Fixing overproduction properly would be the better solution, but that's a lot more involved (and has its own bug already).

I think we should take this for now as its a fairly bad regression.

Comment 13

[Matt Woodrow (:mattwoodrow)]

This was a regression from bug 914437.

Comment 14

[Nick Cameron [:nrc]]

Comment on attachment 832611 [details] [diff] [review]
Don't leak ContentHostIncremental queue up too many updates

Review of attachment 832611 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks for the patch!

::: gfx/layers/composite/ContentHost.cpp
@@ +504,5 @@
>                                            const nsIntRegion& aUpdated,
>                                            const nsIntRect& aBufferRect,
>                                            const nsIntPoint& aBufferRotation)
>  {
> +  FlushUpdateQueue();

I would probably flush after appending to the queue (in both cases), but I don't feel strongly about that

@@ +520,5 @@
> +  // If we're not compositing for some reason (the window being minimized
> +  // is one example), then we never process these updates and it can consume
> +  // huge amounts of memory. Instead we forcibly process the updates (during the
> +  // transaction) if the list gets too long.
> +  static uint32_t kMaxUpdateCount = 4;

either use a define for this or make it const

Comment 15

[Nick Cameron [:nrc]]

Reassigning to mattwoodrow since he has a fix. Feel free to reassign back to me if it doesn't work and/or needs more investigation.

Comment 16

[Matt Woodrow (:mattwoodrow)]

(In reply to Nick Cameron [:nrc] from comment #14)
> Comment on attachment 832611 [details] [diff] [review]
> Don't leak ContentHostIncremental queue up too many updates
> 
> Review of attachment 832611 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Thanks for the patch!
> 
> ::: gfx/layers/composite/ContentHost.cpp
> @@ +504,5 @@
> >                                            const nsIntRegion& aUpdated,
> >                                            const nsIntRect& aBufferRect,
> >                                            const nsIntPoint& aBufferRotation)
> >  {
> > +  FlushUpdateQueue();
> 
> I would probably flush after appending to the queue (in both cases), but I
> don't feel strongly about that

Reasonable. I'll do that, but make the max value 6.

> 
> @@ +520,5 @@
> > +  // If we're not compositing for some reason (the window being minimized
> > +  // is one example), then we never process these updates and it can consume
> > +  // huge amounts of memory. Instead we forcibly process the updates (during the
> > +  // transaction) if the list gets too long.
> > +  static uint32_t kMaxUpdateCount = 4;
> 
> either use a define for this or make it const

Oops, that was definitely the intent.

Comment 17

[Matt Woodrow (:mattwoodrow)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/1f3975482a97

Comment 18

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

https://hg.mozilla.org/mozilla-central/rev/1f3975482a97

Comment 20

[Ioana (away)]

Reproduced with the 11/14 Nightly, verified as fixed on the 02/05 Firefox 28 beta (both Mac OS X 10.8.5).