Minimized windows seem to waste memory (unprocessed transactions?)

RESOLVED FIXED in Firefox 66

Status

()

defect
P3
normal
RESOLVED FIXED
10 months ago
7 months ago

People

(Reporter: mstange, Assigned: kats)

Tracking

(Depends on 1 bug, Blocks 1 bug)

Trunk
mozilla66
All
macOS
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox64 wontfix, firefox65 wontfix, firefox66 fixed)

Details

Attachments

(3 attachments)

I've observed Firefox using tons of memory on macOS, and I think it's caused by my minimized browser windows. Memory builds up over time until I get a dialog from the system that the system has run out of memory and that I should close Firefox. Before this dialog shows up, I've seen numbers as high as 90% for unclassified memory on about:memory.

We don't composite in minimized windows. I think we may be building up pending transactions, or pending frames?

Steps to reproduce (I think):
 1. Open a window and load https://webcompat.com/issues/18666 in it (this is just what I happened to have in one of my current minimized windows).
 2. Minimize that window.
 3. Go to about:memory and note the number of heap unclassified bytes.
 4. Use your browser for a while, or don't use it and just wait.
 5. After some time, trigger another measure on about:memory and notice that the heap unclassified number has risen by about 1 MB per minute.
 6. Unminimize the window.
 7. Watch Firefox freeze for a bit. (Profile is attached: the Renderer thread is busy deallocating memory under Renderer::update, mostly in Queue::pop.)
 8. After Firefox has unfrozen, measure memory again and see that it has dropped down to previous levels.
I'm going to try to reproduce this again once the new memory reporters are in Nightly and see if that gives more precise data.
This sounds pretty similar to bug 1486657 I think?

(In reply to Markus Stange [:mstange] from comment #1)
> I'm going to try to reproduce this again once the new memory reporters are
> in Nightly and see if that gives more precise data.

Keep in mind that the initial memory reporters only record the stuff I found on francine, because I figured it was more important to get them landed than to be thorough. If you find a testcase that still reports high heap-unclassified, do a DMD build and figure out what the unclassified stacks are (or give me STR and I'll do it).
Posted file dmd.txt
I took a DMD report. Here are the huge unreported stacks:

Unreported {
  103,839 blocks in heap block record 1 of 11,669
  850,649,088 bytes (664,569,600 requested / 186,079,488 slop)
  Individual block sizes: 8,192 x 103,839
  23.65% of the heap (23.65% cumulative)
  24.29% of unreported (24.29% cumulative)
  Allocated at {
    #01: replace_realloc(void*, unsigned long) (DMD.cpp:1307, in libmozglue.dylib)
    #02: Allocator<ReplaceMallocBase>::realloc(void*, unsigned long) (malloc_decls.h:39, in libmozglue.dylib)
    #03: realloc (malloc_decls.h:39, in libmozglue.dylib)
    #04: moz_xrealloc (mozalloc.cpp:93, in libmozglue.dylib)
    #05: realloc (MozGrMalloc.h:39, in XUL)
    #06: alloc::alloc::realloc::hcd268b892749fb0a (alloc.rs:98, in XUL)
    #07: _$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Alloc$GT$::realloc::ha8254c6b3d61e13f (alloc.rs:138, in XUL)
    #08: _$LT$alloc..raw_vec..RawVec$LT$T$C$$u20$A$GT$$GT$::reserve_internal::ha584806771f90e8c (raw_vec.rs:674, in XUL)
    #09: _$LT$alloc..raw_vec..RawVec$LT$T$C$$u20$A$GT$$GT$::reserve::h322d9ccdd99c6da5 (raw_vec.rs:499, in XUL)
    #10: _$LT$alloc..vec..Vec$LT$T$GT$$GT$::reserve::hfcd134d7485be9ba (vec.rs:465, in XUL)
    #11: _$LT$alloc..vec..Vec$LT$T$GT$$GT$::push::h5dda3739d51c7f5e (vec.rs:1060, in XUL)
    #12: webrender::render_task::RenderTaskTree::add::h84843b0910bedf4b (render_task.rs:82, in XUL)
    #13: webrender::prim_store::Primitive::update_clip_task_for_brush::h19fd911be37dafac (prim_store.rs:2277, in XUL)
    #14: webrender::prim_store::Primitive::update_clip_task::h6326df686cfbcc0d (prim_store.rs:2780, in XUL)
    #15: webrender::prim_store::PrimitiveStore::prepare_prim_for_render::h12307c6306a930ce (prim_store.rs:1795, in XUL)
    #16: webrender::prim_store::PrimitiveStore::prepare_prim_runs::hd3200684dd53eaca (prim_store.rs:1894, in XUL)
    #17: webrender::prim_store::PrimitiveStore::prepare_prim_for_render::h12307c6306a930ce (prim_store.rs:1640, in XUL)
    #18: webrender::prim_store::PrimitiveStore::prepare_prim_runs::hd3200684dd53eaca (prim_store.rs:1894, in XUL)
    #19: webrender::frame_builder::FrameBuilder::build_layer_screen_rects_and_cull_layers::heab53b0f424c8c83 (frame_builder.rs:253, in XUL)
    #20: webrender::frame_builder::FrameBuilder::build::h0cd713e9e0722b3c (frame_builder.rs:0, in XUL)
    #21: webrender::render_backend::Document::build_frame::h7ffc6a2ca5397e8f (render_backend.rs:0, in XUL)
    #22: webrender::render_backend::RenderBackend::update_document::hed24f04c0aeba4c6 (render_backend.rs:0, in XUL)
    #23: webrender::render_backend::RenderBackend::prepare_transaction::hbfc507548459089f (render_backend.rs:0, in XUL)
    #24: webrender::render_backend::RenderBackend::process_api_msg::h1e9cad3c26431462 (render_backend.rs:829, in XUL)
  }
}

Unreported {
  103,840 blocks in heap block record 2 of 11,669
  425,328,640 bytes (292,413,440 requested / 132,915,200 slop)
  Individual block sizes: 4,096 x 103,840
  11.83% of the heap (35.48% cumulative)
  12.14% of unreported (36.43% cumulative)
  Allocated at {
    #01: replace_realloc(void*, unsigned long) (DMD.cpp:1307, in libmozglue.dylib)
    #02: Allocator<ReplaceMallocBase>::realloc(void*, unsigned long) (malloc_decls.h:39, in libmozglue.dylib)
    #03: realloc (malloc_decls.h:39, in libmozglue.dylib)
    #04: moz_xrealloc (mozalloc.cpp:93, in libmozglue.dylib)
    #05: realloc (MozGrMalloc.h:39, in XUL)
    #06: alloc::alloc::realloc::hcd268b892749fb0a (alloc.rs:98, in XUL)
    #07: _$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Alloc$GT$::realloc::ha8254c6b3d61e13f (alloc.rs:138, in XUL)
    #08: _$LT$alloc..raw_vec..RawVec$LT$T$C$$u20$A$GT$$GT$::reserve_internal::h0946a6daf40db813 (raw_vec.rs:674, in XUL)
    #09: _$LT$alloc..raw_vec..RawVec$LT$T$C$$u20$A$GT$$GT$::reserve::h3ec8222f3447e3b3 (raw_vec.rs:499, in XUL)
    #10: _$LT$alloc..vec..Vec$LT$T$GT$$GT$::reserve::h1b5bda53761dafe4 (vec.rs:465, in XUL)
    #11: _$LT$alloc..vec..Vec$LT$T$GT$$GT$::push::ha5f150961d84785f (vec.rs:1060, in XUL)
    #12: webrender::batch::AlphaBatchContainer::merge::he4a21481a973ad75 (batch.rs:389, in XUL)
    #13: webrender::batch::AlphaBatchBuilder::build::h5d3e265fa63c87e3 (batch.rs:423, in XUL)
    #14: _$LT$webrender..tiling..ColorRenderTarget$u20$as$u20$webrender..tiling..RenderTarget$GT$::build::hee39e10c8048b5e0 (tiling.rs:0, in XUL)
    #15: webrender::tiling::RenderPass::build::he136cd463dfa7b88 (tiling.rs:829, in XUL)
    #16: webrender::frame_builder::FrameBuilder::build::h0cd713e9e0722b3c (frame_builder.rs:419, in XUL)
    #17: webrender::render_backend::Document::build_frame::h7ffc6a2ca5397e8f (render_backend.rs:0, in XUL)
    #18: webrender::render_backend::RenderBackend::update_document::hed24f04c0aeba4c6 (render_backend.rs:0, in XUL)
    #19: webrender::render_backend::RenderBackend::prepare_transaction::hbfc507548459089f (render_backend.rs:0, in XUL)
    #20: webrender::render_backend::RenderBackend::process_api_msg::h1e9cad3c26431462 (render_backend.rs:829, in XUL)
    #21: webrender::render_backend::RenderBackend::run::h3f85c9455635bbb3 (render_backend.rs:626, in XUL)
    #22: webrender::renderer::Renderer::new::_$u7b$$u7b$closure$u7d$$u7d$::h47e8009f07497ae8 (renderer.rs:1763, in XUL)
    #23: std::sys_common::backtrace::__rust_begin_short_backtrace::h432b94a009062336 (backtrace.rs:137, in XUL)
    #24: std::thread::Builder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::ha838ac5e2342d535 (mod.rs:410, in XUL)
  }
}

Unreported {
  297,018 blocks in heap block record 3 of 11,669
  304,146,432 bytes (268,504,272 requested / 35,642,160 slop)
  Individual block sizes: 1,024 x 297,018
  8.46% of the heap (43.94% cumulative)
  8.68% of unreported (45.11% cumulative)
  Allocated at {
    #01: replace_malloc(unsigned long) (DMD.cpp:1266, in libmozglue.dylib)
    #02: Allocator<ReplaceMallocBase>::malloc(unsigned long) (malloc_decls.h:37, in libmozglue.dylib)
    #03: malloc (malloc_decls.h:37, in libmozglue.dylib)
    #04: moz_xmalloc (mozalloc.cpp:70, in libmozglue.dylib)
    #05: malloc (MozGrMalloc.h:29, in XUL)
    #06: alloc::alloc::alloc::h6ea4fe9964925ed4 (alloc.rs:62, in XUL)
    #07: alloc::alloc::exchange_malloc::h40816d9af652b4e0 (alloc.rs:157, in XUL)
    #08: _$LT$std..sync..mpsc..mpsc_queue..Node$LT$T$GT$$GT$::new::h841665933829fe9c (mpsc_queue.rs:62, in XUL)
    #09: _$LT$std..sync..mpsc..mpsc_queue..Queue$LT$T$GT$$GT$::push::he8c1c7eeca4e83fd (mpsc_queue.rs:83, in XUL)
    #10: _$LT$std..sync..mpsc..shared..Packet$LT$T$GT$$GT$::send::h2381604805862c0a (shared.rs:171, in XUL)
    #11: _$LT$std..sync..mpsc..Sender$LT$T$GT$$GT$::send::hda8bebcd6d07dcca (mod.rs:850, in XUL)
    #12: webrender::render_backend::RenderBackend::update_document::hed24f04c0aeba4c6 (render_backend.rs:0, in XUL)
    #13: webrender::render_backend::RenderBackend::prepare_transaction::hbfc507548459089f (render_backend.rs:0, in XUL)
    #14: webrender::render_backend::RenderBackend::process_api_msg::h1e9cad3c26431462 (render_backend.rs:829, in XUL)
    #15: webrender::render_backend::RenderBackend::run::h3f85c9455635bbb3 (render_backend.rs:626, in XUL)
    #16: webrender::renderer::Renderer::new::_$u7b$$u7b$closure$u7d$$u7d$::h47e8009f07497ae8 (renderer.rs:1763, in XUL)
    #17: std::sys_common::backtrace::__rust_begin_short_backtrace::h432b94a009062336 (backtrace.rs:137, in XUL)
    #18: std::thread::Builder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::ha838ac5e2342d535 (mod.rs:410, in XUL)
    #19: _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h3b69476dd8845e34 (panic.rs:309, in XUL)
    #20: std::panicking::try::do_call::h13aa28bdcf51bc9d (panicking.rs:310, in XUL)
    #21: __rust_maybe_catch_panic (in XUL) + 12
    #22: std::panicking::try::h4cc727b0a6604e61 (panicking.rs:289, in XUL)
    #23: std::panic::catch_unwind::hed45ad6ec4c49b56 (panic.rs:392, in XUL)
    #24: std::thread::Builder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::h11546301cdc46ea8 (mod.rs:408, in XUL)
  }
}
So, all of these allocations are coming here:

https://searchfox.org/mozilla-central/rev/99cbc0aec3e1c0b65ff9052523fb5c181b248f57/gfx/webrender/src/render_backend.rs#883

When then gets shipped off in a transaction to the Render thread:

https://searchfox.org/mozilla-central/rev/99cbc0aec3e1c0b65ff9052523fb5c181b248f57/gfx/webrender/src/render_backend.rs#916

We actually do have memory reporters on the Renderer side for RenderTaskTree, and since these allocations don't seem to be getting reported via Renderer, I think it's reasonable to conclude that the messages are not getting received, and are thus all getting queued up in the channel.
Flags: needinfo?(nical.bugzilla)
Flags: needinfo?(mstange)
Duplicate of this bug: 1486657
(Note that this was with the spinning rectangle testcase from bug 1486657, so I think it's reasonable that we'd skip the scene builder thread like the stacks show).
(In reply to Bobby Holley (:bholley) from comment #2)
> This sounds pretty similar to bug 1486657 I think?

Hahaha, I'm surprised I forgot about that bug and didn't make the connection.

I'm going to defer to nical on how to fix this. It seems like we need to fix this on both ends of the queue: We shouldn't keep submitting new messages for windows that we're not going to paint, and we should service the queue frequently enough so that things don't accumulate without bounds.
Flags: needinfo?(mstange)
My work on avoiding the generation of frames that we are not going to render should help here. The scene would get updated but update_document would not produce a frame which would avoid piling stuff up in a renderer queue that don't get flushed because the window is not visible. Unfortunately I'm still wresting with some reftest failures.

On top of that work it sounds like we should also track whether the scene is valid and avoid building the scene when the window isn't shown to avoid wasting CPU cycles building scenes in the background.
Flags: needinfo?(nical.bugzilla)
Nical, Is this likely to affect MVP or is this Mac only?
Flags: needinfo?(nical.bugzilla)
Priority: -- → P3
I wasn't able to reproduce it on windows, but it's certainly plausible that it could reproduce in other ways. I don't have a good enough sense of what causes transaction queuing to understand if it could happen on other platforms.
I don't know enough to be 100% sure but I don't see a reason for this to be specific to mac. So I'd conservatively assume it can affect the MVP.
It's probably something that will get fixed with the general problem of generating work for no reason in webrender, and I think that it would be reasonable to fix at least a good portion of that for the MVP.
Flags: needinfo?(nical.bugzilla)
Priority: P3 → P4
Since this is actionable and could impact MVP, I'm making this a P3.
Priority: P4 → P3
Assignee: nobody → kats
I didn't repro on Windows, using either the URL in comment 0 or the infinite animation at [1]. However I am still seeing runaway heap-unclassified on Mac using [1] though it took a while. I'll go back and try Windows again giving it more time since I may not have waited long enough.

[1] https://mozilla.staktrace.com/anim.html
IIRC, on mac, mCompositor->GetWidget()->PreRender(&widgetContext)) in RendererOGL::UpdateAndRender() failed, when window is not shown. I wonder if it is related.
(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #13)
> I'll go back and try Windows again giving
> it more time since I may not have waited long enough.

I left it overnight and while I didn't see heap-unclassified go up, I did see ~4G in the clip-stores memory reporter in the GPU process. I'll attempt to repro and spin that out into a separate bug. We can keep this bug for the issue on Mac.

(In reply to Sotaro Ikeda [:sotaro] from comment #14)
> IIRC, on mac, mCompositor->GetWidget()->PreRender(&widgetContext)) in
> RendererOGL::UpdateAndRender() failed, when window is not shown. I wonder if
> it is related.

Ah, interesting! I verified that PreRender does return false for a minimized window, will check to see if that's related. Thanks for the tip!
Falling back to Update() if UpdateAndRender returns false at [1] clears the renderer queue but I still see heap-unclassified increasing. Will DMD it to dig more.

[1] https://searchfox.org/mozilla-central/rev/adcc169dcf58c2e45ba65c4ed5661d666fc3ac74/gfx/webrender_bindings/RenderThread.cpp#365
Some more investigation shows that my patch clears the renderer queue, but just moves the pileup a little further. Specifically the self.pending_texture_updates [1] and self.pending_gpu_cache_updates [2] structures keep accumulating stuff and don't process the entries, so we continue "leaking" memory until something allows us to render that window and process the updates.

[1] https://searchfox.org/mozilla-central/rev/fd32b3a6fa3eff1468311f6fcf32b45c117136df/gfx/wr/webrender/src/renderer.rs#2123
[2] https://searchfox.org/mozilla-central/rev/fd32b3a6fa3eff1468311f6fcf32b45c117136df/gfx/wr/webrender/src/renderer.rs#2131
In addition to the mac-specific bug, there's a cross-platform (at least mac and windows) bug for which I filed https://github.com/servo/webrender/issues/3416
In terms of fixing this bug, I think the best fix is to just allow rendering on minimized windows. That way PreRender won't return false and we just take the normal codepath. This will also bring mac in line with what we do on Windows, for example, but may adversely impact perf in that now we'll be doing more composition work on minimized windows. I still think this is reasonable because if we want to avoid doing that work on minimized windows, we should check for it further upstream and avoid scheduling the work in the first place. Like maybe turn off vsync ticking on minimized windows, or something along those lines.
Pushed by kgupta@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/99d2b63661e7
Allow rendering minimized windows on macOS. r=mstange
https://hg.mozilla.org/mozilla-central/rev/99d2b63661e7
Status: NEW → RESOLVED
Closed: 7 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla66
The memory leak here is specific to WR on Mac, so no need to uplift.
You need to log in before you can comment on or make changes to this bug.