1262736 - very high memory usage, with high heap-unclassified

Status: RESOLVED INCOMPLETE

(Firefox :: PDF Viewer, defect)

Description

[AM]

Attachment: memory-report.json.gz

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Firefox/45.0
Build ID: 20160315153207

Steps to reproduce:

I run firefox for several days on a macbook. using typically 20 tabs. Several add-ons
self-destructing cookies,tree-style-tab,ublock-origin. Latest MacOS. I often use bookmark manager to open "blocks" of up to 10 tabs. I see this on several machines- home and work.


Actual results:

Memory usage goes above 5Gb, firefox takes several seconds to react when typing.
heap-unclassified can go to 88%. about:memory output attached


Expected results:

firefox remains responsive,

Comment 1

[Kohei Yoshino [:kohei]]

Please review the following knowledge articles to solve the issue:

https://support.mozilla.org/kb/firefox-uses-too-much-memory-ram
https://support.mozilla.org/kb/troubleshoot-firefox-issues-using-safe-mode
https://support.mozilla.org/kb/profile-manager-create-and-remove-firefox-profiles

Usually extensions cause various issues (and such bug reports are not accepted).

Comment 2

[Timothy Nikkel (:tnikkel)]

4GB of heap-unclassified, that seems like something we want to investigate.

Comment 3

[Timothy Nikkel (:tnikkel)]

First thing is, do you see this problem with addons disabled? Maybe try disabling one at a time and see if it still happens?

Comment 4

[AM]

Oh, I am aware that you don't like addons. But it is so crazily bad that I thought it could be an interesting datapoint for memshrink. I know that memshrink did some work specifically to improve
behaviour of firefox with addons so it could be interesting.

The problem in terms of going individually through the addons is that it takes about a week to get
to this state. If it is a pairewise interaction between addons it will take time!

The output from about:memory is also clearly very sick - it has large negative values in it. Something is really smashing the memory management.

The work-around is simple - restart firefox once per day, but there is something wrong somewhere.

Comment 5

[Timothy Nikkel (:tnikkel)]

The negative gfx memory reports might be fixed by bug 1234343 which is in 46.

To diagnose the heap-unclassified we could make you a dmd build that you could run until heap unclassified grows (to say 40%, normal sessions have it about 20% of lower).

You could also try running beta (46) to see if the issue still appears.

Comment 6

[AM]

Ok, I'll pass at once to 46, is there a DMD build for download?

I write a fair amount of (C) code, so running in a non-standard environment/debugger would be fine for me too.

Comment 7

[Timothy Nikkel (:tnikkel)]

I believe DMD builds can be made for download, but I'm not familiar with that process. Nick?

The other option is building Firefox locally with DMD, it takes a while, but it's pretty straightforward to do. I can walk you through that.

Comment 8

[AM]

If you point me at some instructions I will manage. I have been developing on Unix for 30 years...

One other data point for (perhaps) unusual things I do - I have accounts on multiple machines on campus.
I have Sync on. When I get home the machine then has to pull in modifications from perhaps several sources -
bookmarks, tabs etc.

I am on 46b9 since yesterday evening, already at 45% heap-unclassified, total memory is reasonable however
at about 1Gb.

Comment 9

[Timothy Nikkel (:tnikkel)]

Instructions for building Firefox: https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Simple_Firefox_build

DMD instructions: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/DMD

Comment 10

[AM]

DMD build hangs -- summary

I have the build (beta branch) with debugging and dmd turned on.
Running in a fresh account, so no interaction with other instances of firefox
I started by disabling all extensions, but turn on firefox Sync and leave for an hour to repopulate bookmarks.

./mach run

seems to be healthy. However

./mach run --dmd

hangs and beachballs within a minute (when interacting with location bar)
  with the last message to the terminal always of the form:

DMD[32585] $DMD is undefined

[Parent 32500] WARNING: parent WaitForMessage() failed: 0x10004003 (ipc/rcv) timed out: file /Users/tony2/Mozilla/beta/ipc/glue/GeckoChildProcessHost.cpp, line 813

Comment 11

[Timothy Nikkel (:tnikkel)]

Hmm, I built beta with DMD and it worked fine for me. So your mozconfig has two lines, one for dmd and one for debug? Try removing the debug line, you shouldn't need that.

Comment 12

[Nicholas Nethercote [inactive]]

(In reply to Timothy Nikkel (:tnikkel) from comment #7)
> I believe DMD builds can be made for download, but I'm not familiar with
> that process. Nick?

You just force-enable DMD (via configuration code) and then do a try push :)

Comment 13

[Nicholas Nethercote [inactive]]

(In reply to AM from comment #10)
> DMD build hangs -- summary
> 
> I have the build (beta branch) with debugging and dmd turned on.
> Running in a fresh account, so no interaction with other instances of firefox
> I started by disabling all extensions, but turn on firefox Sync and leave
> for an hour to repopulate bookmarks.
> 
> ./mach run
> 
> seems to be healthy. However
> 
> ./mach run --dmd
> 
> hangs and beachballs within a minute (when interacting with location bar)

So it starts up ok but only hangs after a minute? Hmm. I've never seen that before.

>   with the last message to the terminal always of the form:
> 
> DMD[32585] $DMD is undefined

This is fine.

> [Parent 32500] WARNING: parent WaitForMessage() failed: 0x10004003 (ipc/rcv)
> timed out: file
> /Users/tony2/Mozilla/beta/ipc/glue/GeckoChildProcessHost.cpp, line 813

I don't know if this is significant or not.

Comment 14

[Timothy Nikkel (:tnikkel)]

If you have multiprocess enabled try disabling it.

Comment 15

[AM]

I have been traveling, so less work on this than I would have hoped.

1) I left one machine with a dmd (46) build running as another user (with fast user switching on the mac). Just three or four tabs open. I came back and examined the process - top/activity monitor claims process uses 2.4Gb, however about:memory gives only 600Mb, with low heap-unclassified. Machine (macbook air) unresponsive (4Gb memory) until firefox killed.

2) I have also a session which I ran on my main Imac. Here I have heap-unclassifed up to 40%, the biggest entry is below

3) My normal non-dmd build go bad much faster than the dmd build. I will try importing my full profile into the dmd directory to see if there is a preference that is strange (however I get high unclassified
on several different machines so something is going wrong on several sessions....)

-------------------------------


Unreported {
  9 blocks in heap block record 1 of 7,877
  193,241,088 bytes (193,208,256 requested / 32,832 slop)
  Individual block sizes: 21,471,232 x 9
  21.73% of the heap (21.73% cumulative)
  39.60% of unreported (39.60% cumulative)
  Allocated at {
    #01: replace_malloc (DMD.cpp:1258, in libdmd.dylib)
    #02: malloc_zone_malloc (in libsystem_malloc.dylib) + 71
    #03: malloc (in libsystem_malloc.dylib) + 42
    #04: SkMallocPixelRef::NewUsing(void* (*)(unsigned long), SkImageInfo const&, unsigned long, SkColorTable*) (SkMallocPixelRef.cpp:186, in XUL)
    #05: SkBitmap::HeapAllocator::allocPixelRef(SkBitmap*, SkColorTable*) (SkBitmap.cpp:430, in XUL)
    #06: SkBitmap::tryAllocPixels(SkBitmap::Allocator*, SkColorTable*) (SkBitmap.cpp:274, in XUL)
    #07: mozilla::gfx::DrawTargetSkia::Init(mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::SurfaceFormat) (DrawTargetSkia.cpp:855, in XUL)
    #08: mozilla::gfx::Factory::CreateDrawTarget(mozilla::gfx::BackendType, mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::SurfaceFormat) (Factory.cpp:353, in XUL)
    #09: gfxPlatform::CreateDrawTargetForBackend(mozilla::gfx::BackendType, mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::SurfaceFormat) (gfxPlatform.cpp:1253, in XUL)
    #10: mozilla::layers::PersistentBufferProviderBasic::PersistentBufferProviderBasic(mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits>, mozilla::gfx::SurfaceFormat, mozilla::gfx::BackendType) (AlreadyAddRefed.h:116, in XUL)
    #11: mozilla::layers::LayerManager::CreatePersistentBufferProvider(mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::SurfaceFormat) (RefPtr.h:88, in XUL)
    #12: mozilla::dom::CanvasRenderingContext2D::EnsureTarget(mozilla::dom::CanvasRenderingContext2D::RenderingMode) (AlreadyAddRefed.h:116, in XUL)
    #13: mozilla::dom::CanvasRenderingContext2D::SwitchRenderingMode(mozilla::dom::CanvasRenderingContext2D::RenderingMode) (CanvasRenderingContext2D.h:661, in XUL)
    #14: mozilla::dom::CanvasRenderingContext2D::EnsureTarget(mozilla::dom::CanvasRenderingContext2D::RenderingMode) (CanvasRenderingContext2D.cpp:1292, in XUL)
    #15: mozilla::dom::CanvasRenderingContext2D::Save() (CanvasRenderingContext2D.cpp:1733, in XUL)
    #16: mozilla::dom::CanvasRenderingContext2DBinding::save(JSContext*, JS::Handle<JSObject*>, mozilla::dom::CanvasRenderingContext2D*, JSJitMethodCallArgs const&) (CallArgs.h:197, in XUL)
    #17: mozilla::dom::GenericBindingMethod(JSContext*, unsigned int, JS::Value*) (BindingUtils.cpp:2717, in XUL)
    #18: js::Invoke(JSContext*, JS::CallArgs const&, js::MaybeConstruct) (jscntxtinlines.h:235, in XUL)
    #19: Interpret(JSContext*, js::RunState&) (Interpreter.cpp:2798, in XUL)
    #20: js::RunScript(JSContext*, js::RunState&) (Interpreter.cpp:425, in XUL)
    #21: js::Invoke(JSContext*, JS::CallArgs const&, js::MaybeConstruct) (Interpreter.cpp:493, in XUL)
    #22: js::Invoke(JSContext*, JS::Value const&, JS::Value const&, unsigned int, JS::Value const*, JS::MutableHandle<JS::Value>) (Interpreter.cpp:527, in XUL)
    #23: JS::Call(JSContext*, JS::Handle<JS::Value>, JS::Handle<JS::Value>, JS::HandleValueArray const&, JS::MutableHandle<JS::Value>) (jsapi.cpp:2882, in XUL)
    #24: mozilla::dom::AnyCallback::Call(JSContext*, JS::Handle<JS::Value>, JS::Handle<JS::Value>, JS::MutableHandle<JS::Value>, mozilla::ErrorResult&) (PromiseBinding.cpp:78, in XUL)
  }
}

Comment 16

[Timothy Nikkel (:tnikkel)]

Thank you, that's very helpful! Are there other entries taking up significant space?

Comment 17

[AM]

No, this is the only big one for the moment.

As I said, I will now import my full profile into the dmd build and report back
in a few days if I find more.

Comment 18

[Timothy Nikkel (:tnikkel)]

Milan, can we get someone to look at the dark matter from comment 15?

Comment 19

[Milan Sreckovic [:milan] (needinfo for best results)]

(In reply to Timothy Nikkel (:tnikkel) from comment #18)
> Milan, can we get someone to look at the dark matter from comment 15?

Looks like bug 1264161 would help if not resolve, and it just landed today.  Let's try it on a nightly after Apr 26th.

Comment 20

[AM]

that seems to be a e10s bug, I am running with it off.

Comment 21

[Milan Sreckovic [:milan] (needinfo for best results)]

Lee, can you take a look at the "unreported" part of comment 15, Skia draw targets?

AM: Do the problems go away if you restart Firefox after setting gfx.canvas.azure.backends to cg and gfx.canvas.azure.accelerated to false?  If they do, what about just when setting gfx.canvas.azure.accelerated to false?

Comment 22

[AM]

I have to say that dmd builds are behaving much much better than the normal releases. They remain usable much longer even with my full profile imported.

I propose going back to a release version to change the azure variables, to see if this makes any real difference.

My present session which has heap-unclassified at 40% (after two days) has the following top entries from the dmd report:

Summary {
  Total:          ~785,693,185 bytes (100.00%) in ~123,219 blocks (100.00%)
  Unreported:     ~323,622,222 bytes ( 41.19%) in ~37,324 blocks ( 30.29%)
  Once-reported:  ~462,070,963 bytes ( 58.81%) in ~85,895 blocks ( 69.71%)
  Twice-reported:           ~0 bytes (  0.00%) in      ~0 blocks (  0.00%)
}

   6.01% of unreported (6.01% cumulative)
  5.20% of unreported (11.22% cumulative)
  3.21% of unreported (14.43% cumulative)
  3.14% of unreported (17.58% cumulative)
  2.55% of unreported (20.12% cumulative)
  2.40% of unreported (22.52% cumulative)
  2.29% of unreported (24.82% cumulative)
  2.01% of unreported (26.83% cumulative)
Skia draw is not there this time, so there are no real top entries that dominate. For completeness let me give the top two:

Unreported {
  24 blocks in heap block record 1 of 7,868
  19,464,192 bytes (19,414,128 requested / 50,064 slop)
  Individual block sizes: 843,776 x 12; 778,240 x 12
  2.48% of the heap (2.48% cumulative)
  6.01% of unreported (6.01% cumulative)
  Allocated at {
    #01: replace_realloc (DMD.cpp:1306, in libdmd.dylib)
    #02: moz_xrealloc (mozalloc.cpp:106, in libmozglue.dylib)
    #03: ExpandingMemoryStream::WriteRaw(void const*, unsigned long) (gfxUserFontSet.cpp:76, in XUL)
    #04: ots::ots_post_serialise(ots::OTSStream*, ots::Font*) (post.cc:174, in XUL)
    #05: (anonymous namespace)::ProcessGeneric(ots::OpenTypeFile*, ots::Font*, unsigned int, ots::OTSStream*, unsigned char const*, unsigned long, std::vector<(anonymous namespace)::OpenTypeTable, std::allocator<(anonymous namespace)::OpenTypeTable> > const&, ots::Buffer&) (ots.cc:788, in XUL)
    #06: (anonymous namespace)::ProcessTTF(ots::OpenTypeFile*, ots::Font*, ots::OTSStream*, unsigned char const*, unsigned long, unsigned int) (ots.cc:220, in XUL)
    #07: ots::OTSContext::Process(ots::OTSStream*, unsigned char const*, unsigned long, unsigned int) (ots.cc:484, in XUL)
    #08: gfxUserFontEntry::SanitizeOpenTypeData(unsigned char const*, unsigned int, unsigned int&, gfxUserFontType) (gfxUserFontSet.cpp:254, in XUL)
    #09: gfxUserFontEntry::LoadPlatformFont(unsigned char const*, unsigned int&) (gfxUserFontSet.cpp:621, in XUL)
    #10: gfxUserFontEntry::FontDataDownloadComplete(unsigned char const*, unsigned int, nsresult) (gfxUserFontSet.cpp:746, in XUL)
    #11: nsFontFaceLoader::OnStreamComplete(nsIStreamLoader*, nsISupports*, nsresult, unsigned int, unsigned char const*) (nsFontFaceLoader.cpp:258, in XUL)
    #12: nsStreamLoader::OnStopRequest(nsIRequest*, nsISupports*, nsresult) (nsStreamLoader.cpp:104, in XUL)
    #13: nsCORSListenerProxy::OnStopRequest(nsIRequest*, nsISupports*, nsresult) (nsCORSListenerProxy.cpp:591, in XUL)
    #14: mozilla::net::nsHttpChannel::OnStopRequest(nsIRequest*, nsISupports*, nsresult) (nsHttpChannel.cpp:5954, in XUL)
    #15: nsInputStreamPump::OnStateStop() (ReentrantMonitor.h:74, in XUL)
    #16: nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*) (nsInputStreamPump.cpp:434, in XUL)
    #17: nsInputStreamReadyEvent::Run() (nsCOMPtr.h:578, in XUL)
    #18: nsThread::ProcessNextEvent(bool, bool*) (nsThread.cpp:995, in XUL)
    #19: NS_ProcessPendingEvents(nsIThread*, unsigned int) (nsThreadUtils.cpp:239, in XUL)
    #20: nsBaseAppShell::NativeEventCallback() (nsBaseAppShell.cpp:98, in XUL)
    #21: nsAppShell::ProcessGeckoEvents(void*) (nsAppShell.mm:388, in XUL)
    #22: __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ (in CoreFoundation) + 17
    #23: __CFRunLoopDoSources0 (in CoreFoundation) + 556
    #24: __CFRunLoopRun (in CoreFoundation) + 927
  }
}


Unreported {
  254 blocks in heap block record 2 of 7,868
  16,842,752 bytes (16,337,456 requested / 505,296 slop)
  Individual block sizes: 155,648 x 3; 143,360 x 28; 139,264 x 2; 135,168; 131,072 x 9; 126,976 x 5; 114,688 x 3; 94,208 x 6; 90,112 x 11; 81,920 x 3; 77,824 x 5; 73,728 x 3; 65,536 x 4; 61,440; 57,344 x 25; 53,248 x 18; 49,152 x 13; 45,056 x 27; 40,960 x 5; 36,864 x 4; 32,768 x 60; 28,672 x 16; 20,480; 8,192
  2.14% of the heap (4.62% cumulative)
  5.20% of unreported (11.22% cumulative)
  Allocated at {
    #01: replace_malloc (DMD.cpp:1258, in libdmd.dylib)
    #02: moz_xmalloc (mozalloc.cpp:84, in libmozglue.dylib)
    #03: gfxUserFontEntry::SanitizeOpenTypeData(unsigned char const*, unsigned int, unsigned int&, gfxUserFontType) (gfxUserFontSet.cpp:47, in XUL)
    #04: gfxUserFontEntry::LoadPlatformFont(unsigned char const*, unsigned int&) (gfxUserFontSet.cpp:621, in XUL)
    #05: gfxUserFontEntry::FontDataDownloadComplete(unsigned char const*, unsigned int, nsresult) (gfxUserFontSet.cpp:746, in XUL)
    #06: nsFontFaceLoader::OnStreamComplete(nsIStreamLoader*, nsISupports*, nsresult, unsigned int, unsigned char const*) (nsFontFaceLoader.cpp:258, in XUL)
    #07: nsStreamLoader::OnStopRequest(nsIRequest*, nsISupports*, nsresult) (nsStreamLoader.cpp:104, in XUL)
    #08: nsCORSListenerProxy::OnStopRequest(nsIRequest*, nsISupports*, nsresult) (nsCORSListenerProxy.cpp:591, in XUL)
    #09: mozilla::net::nsHttpChannel::OnStopRequest(nsIRequest*, nsISupports*, nsresult) (nsHttpChannel.cpp:5954, in XUL)
    #10: nsInputStreamPump::OnStateStop() (ReentrantMonitor.h:74, in XUL)
    #11: nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*) (nsInputStreamPump.cpp:434, in XUL)
    #12: nsInputStreamReadyEvent::Run() (nsCOMPtr.h:578, in XUL)
    #13: nsThread::ProcessNextEvent(bool, bool*) (nsThread.cpp:995, in XUL)
    #14: NS_ProcessPendingEvents(nsIThread*, unsigned int) (nsThreadUtils.cpp:239, in XUL)
    #15: nsBaseAppShell::NativeEventCallback() (nsBaseAppShell.cpp:98, in XUL)
    #16: nsAppShell::ProcessGeckoEvents(void*) (nsAppShell.mm:388, in XUL)
    #17: __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ (in CoreFoundation) + 17
    #18: __CFRunLoopDoSources0 (in CoreFoundation) + 556
    #19: __CFRunLoopRun (in CoreFoundation) + 927
    #20: CFRunLoopRunSpecific (in CoreFoundation) + 296
    #21: RunCurrentEventLoopInMode (in HIToolbox) + 235
    #22: ReceiveNextEventCommon (in HIToolbox) + 432
    #23: _BlockUntilNextEventMatchingListInModeWithFilter (in HIToolbox) + 71
    #24: _DPSNextEvent (in AppKit) + 1067
  }
}

Comment 23

[Milan Sreckovic [:milan] (needinfo for best results)]

Jonathan, thoughts on comment 22?

Comment 24

[Lee Salzman [:lsalzman]]

(In reply to Milan Sreckovic [:milan] from comment #21)
> Lee, can you take a look at the "unreported" part of comment 15, Skia draw
> targets?
> 
> AM: Do the problems go away if you restart Firefox after setting
> gfx.canvas.azure.backends to cg and gfx.canvas.azure.accelerated to false? 
> If they do, what about just when setting gfx.canvas.azure.accelerated to
> false?

We are doing very rough memory reporting for 2D canvases that should, while not ideal, at least just barely cover this case. It is accounting for the overhead of the bitmap allocated shown there. Though the canvas memory reporting only accounts for the pixel footprint (width * height * 4), and nothing else.

Comment 25

[Nicholas Nethercote [inactive]]

> We are doing very rough memory reporting for 2D canvases that should, while
> not ideal, at least just barely cover this case. It is accounting for the
> overhead of the bitmap allocated shown there. Though the canvas memory
> reporting only accounts for the pixel footprint (width * height * 4), and
> nothing else.

BTW, DMD only knows about the measurement of heap blocks that are measured via a MallocSizeOf function. There are some places where we instead compute what size of these blocks (theoretically) are. Such memory will be reported appropriately in about:memory but DMD will claim that it's unreported. This may be an example of such a case.

Comment 26

[Jonathan Kew [:jfkthame]]

If I'm understanding the reports in comment 22, it looks like we could use some memory-reporter support for downloadable fonts, so that whatever space they're taking up doesn't end up lumped in to "unclassified". Ideally, we'd record some kind of identification, too (URL of the font resource?) so that we can tell where they came from.

I'll have a look and see if there's something we can do there. (Might be slightly tricky because the allocation happens during the decode-and-sanitize process that OTS does for us?)

The "24 blocks in heap block record 1 of 7,868" sound a bit suspicious... given what it says about individual block sizes, I wonder whether we've downloaded and decoded multiple (12) copies each of the same two large font resources? That would be sad. Better reporting might help us figure out if this is the case.

Comment 27

[Jamaicanboy63]

here are my processes. I'm experiencing this bug too. I can barely use firefox for long without it using 2-3 gigs of ram.

http://pastebin.com/jp0miyfT

Comment 28

[Timothy Nikkel (:tnikkel)]

After bug 1268021 lands we should re-test to see if it helped.

Comment 29

[Jamaicanboy63]

(In reply to Timothy Nikkel (:tnikkel) from comment #28)
> After bug 1268021 lands we should re-test to see if it helped.

When will that solve our problem? Firefox is unusable for me so I have to use chrome.

Comment 30

[Jonathan Kew [:jfkthame]]

(In reply to Jamaicanboy63 from comment #29)
> (In reply to Timothy Nikkel (:tnikkel) from comment #28)
> > After bug 1268021 lands we should re-test to see if it helped.
> 
> When will that solve our problem? Firefox is unusable for me so I have to
> use chrome.

That bug won't "solve" things for you; it just aims to provide better reporting, to help us diagnose what's going on.

Comment 31

[Jamaicanboy63]

(In reply to Jonathan Kew (:jfkthame) from comment #30)
> (In reply to Jamaicanboy63 from comment #29)
> > (In reply to Timothy Nikkel (:tnikkel) from comment #28)
> > > After bug 1268021 lands we should re-test to see if it helped.
> > 
> > When will that solve our problem? Firefox is unusable for me so I have to
> > use chrome.
> 
> That bug won't "solve" things for you; it just aims to provide better
> reporting, to help us diagnose what's going on.

I see, guess I'll have to use chrome till that resolves or e10s will make it resolve itself.

Comment 32

[AM]

I have a 47b1, download  which is again at 75% heap-unclassified, after just a couple of days - it is
slow and unreactive.

Even after hammering on the dmd build I never get the same growth in the memory. Something seems very different in the behaviour of the dmd and the normal firefox builds.

Are there any other ways that you can suggest to work on this?

Again I note that dmd builds can use 4Gb of memory (in top) for quite small heap reports. However
this is perhaps a natural consequence of the dmd memory managment. Despite this large memory use they
remain more reactive than the download builds of firefox

Comment 33

[AM]

Attachment: dmd.h.gz

dmd output with 66% unrecorded,

Comment 34

[AM]

I have added a dmd run, up for about a week uptime. Gives 66% unreported. I will move onto using a recent
nightly to see if recent font reporting improves the behavior.

Summary {
  Total:          ~957,953,106 bytes (100.00%) in ~115,468 blocks (100.00%)
  Unreported:     ~641,509,341 bytes ( 66.97%) in ~64,861 blocks ( 56.17%)
  Once-reported:  ~316,443,765 bytes ( 33.03%) in ~50,607 blocks ( 43.83%)
  Twice-reported:           ~0 bytes (  0.00%) in      ~0 blocks (  0.00%)
}

Comment 35

[AM]

I have been running with a 10-may-2016 nightly (49.0a1), (post font-cache). After running a week  (electrolysis off) the top entries are below:
I guess I will stop reporting, unless you want any other specific information, or want results from other runs.

--------------

Summary {
  Total:           863,087,984 bytes (100.00%) in 3,235,242 blocks (100.00%)
  Unreported:      318,488,808 bytes ( 36.90%) in 1,505,581 blocks ( 46.54%)
  Once-reported:   544,599,176 bytes ( 63.10%) in 1,729,661 blocks ( 53.46%)
  Twice-reported:            0 bytes (  0.00%) in       0 blocks (  0.00%)
}
-----------------------

Unreported {
  1,162,454 blocks in heap block record 1 of 27,406
  80,009,992 bytes (75,633,764 requested / 4,376,228 slop)
  Individual block sizes: 2,048 x 21; 1,024 x 628; 512 x 584; 496 x 80; 480 x 1,060; 464 x 23; 448 x 267; 432 x 86; 416 x 48; 400 x 851; 384 x 243; 368 x 291; 352 x 275; 336 x 169; 320 x 335; 304 x 519; 288 x 2,233; 272 x 105,735; 256 x 1,791; 240 x 4,780; 224 x 1,127; 208 x 8,472; 192 x 8,249; 176 x 12,805; 160 x 6,130; 144 x 18,230; 128 x 18,387; 112 x 15,087; 96 x 16,181; 80 x 36,265; 64 x 69,175; 48 x 226,692; 32 x 239,012; 16 x 309,408; 8 x 57,215
  9.27% of the heap (9.27% cumulative)
  25.12% of unreported (25.12% cumulative)
  Allocated at {
    #01: (no stack trace recorded due to --stacks=partial)
  }
}

Unreported {
  2 blocks in heap block record 2 of 27,406
  35,774,464 bytes (35,769,600 requested / 4,864 slop)
  Individual block sizes: 21,544,960; 14,229,504
  4.14% of the heap (13.42% cumulative)
  11.23% of unreported (36.35% cumulative)
  Allocated at {
    #01: replace_malloc (DMD.cpp:1287, in libdmd.dylib)
    #02: malloc_zone_malloc (in libsystem_malloc.dylib) + 71
    #03: malloc (in libsystem_malloc.dylib) + 42
    #04: SkMallocPixelRef::NewUsing(void* (*)(unsigned long), SkImageInfo const&, unsigned long, SkColorTable*) (SkMallocPixelRef.cpp:186, in XUL)
    #05: SkBitmap::HeapAllocator::allocPixelRef(SkBitmap*, SkColorTable*) (SkBitmap.cpp:447, in XUL)
    #06: SkBitmap::tryAllocPixels(SkBitmap::Allocator*, SkColorTable*) (SkBitmap.cpp:285, in XUL)
    #07: mozilla::gfx::DrawTargetSkia::Init(mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::SurfaceFormat) (DrawTargetSkia.cpp:989, in XUL)
    #08: mozilla::gfx::Factory::CreateDrawTarget(mozilla::gfx::BackendType, mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::SurfaceFormat) (Factory.cpp:341, in XUL)
    #09: gfxPlatform::CreateDrawTargetForBackend(mozilla::gfx::BackendType, mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::SurfaceFormat) (gfxPlatform.cpp:1383, in XUL)
    #10: mozilla::layers::PersistentBufferProviderBasic::PersistentBufferProviderBasic(mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits>, mozilla::gfx::SurfaceFormat, mozilla::gfx::BackendType) (AlreadyAddRefed.h:116, in XUL)
    #11: mozilla::layers::LayerManager::CreatePersistentBufferProvider(mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::SurfaceFormat) (RefPtr.h:109, in XUL)
    #12: mozilla::dom::CanvasRenderingContext2D::EnsureTarget(mozilla::dom::CanvasRenderingContext2D::RenderingMode) (AlreadyAddRefed.h:116, in XUL)
    #13: mozilla::dom::CanvasRenderingContext2D::SwitchRenderingMode(mozilla::dom::CanvasRenderingContext2D::RenderingMode) (CanvasRenderingContext2D.h:655, in XUL)
    #14: mozilla::dom::CanvasRenderingContext2D::EnsureTarget(mozilla::dom::CanvasRenderingContext2D::RenderingMode) (CanvasRenderingContext2D.cpp:1332, in XUL)
    #15: mozilla::dom::CanvasRenderingContext2D::Scale(double, double, mozilla::ErrorResult&) (CanvasRenderingContext2D.cpp:3183, in XUL)
    #16: mozilla::dom::CanvasRenderingContext2DBinding::scale(JSContext*, JS::Handle<JSObject*>, mozilla::dom::CanvasRenderingContext2D*, JSJitMethodCallArgs const&) (ErrorResult.h:180, in XUL)
    #17: mozilla::dom::GenericBindingMethod(JSContext*, unsigned int, JS::Value*) (BindingUtils.cpp:2781, in XUL)
  }
}


Unreported {
  19 blocks in heap block record 3 of 27,406
  18,550,784 bytes (18,520,652 requested / 30,132 slop)
  Individual block sizes: 9,834,496; 2,666,496; 1,646,592; 1,376,256; 1,093,632; 425,984; 409,600; 102,400; 94,208 x 9; 77,824; 69,632
  2.15% of the heap (15.56% cumulative)
  5.82% of unreported (42.18% cumulative)
  Allocated at {
    #01: replace_malloc (DMD.cpp:1287, in libdmd.dylib)
    #02: malloc_zone_malloc (in libsystem_malloc.dylib) + 71
    #03: malloc (in libsystem_malloc.dylib) + 42
    #04: SkMallocPixelRef::NewUsing(void* (*)(unsigned long), SkImageInfo const&, unsigned long, SkColorTable*) (SkMallocPixelRef.cpp:186, in XUL)
    #05: SkBitmap::HeapAllocator::allocPixelRef(SkBitmap*, SkColorTable*) (SkBitmap.cpp:447, in XUL)
    #06: SkBitmap::tryAllocPixels(SkBitmap::Allocator*, SkColorTable*) (SkBitmap.cpp:285, in XUL)
    #07: mozilla::gfx::DrawTargetSkia::Init(mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::SurfaceFormat) (DrawTargetSkia.cpp:989, in XUL)
    #08: mozilla::gfx::Factory::CreateDrawTarget(mozilla::gfx::BackendType, mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::SurfaceFormat) (Factory.cpp:341, in XUL)
    #09: gfxPlatform::CreateOffscreenContentDrawTarget(mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::SurfaceFormat) (gfxPlatform.cpp:1383, in XUL)
    #10: non-virtual thunk to mozilla::layers::ContentClientBasic::CreateBuffer(gfxContentType, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&, unsigned int, RefPtr<mozilla::gfx::DrawTarget>*, RefPtr<mozilla::gfx::DrawTarget>*) (AlreadyAddRefed.h:116, in XUL)
    #11: mozilla::layers::RotatedContentBuffer::BeginPaint(mozilla::layers::PaintedLayer*, unsigned int) (RefPtr.h:290, in XUL)
    #12: mozilla::layers::BasicPaintedLayer::Validate(void (*)(mozilla::layers::PaintedLayer*, gfxContext*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::layers::DrawRegionClip, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, void*), void*, mozilla::layers::ReadbackProcessor*) (BasicPaintedLayer.cpp:163, in XUL)
    #13: non-virtual thunk to mozilla::layers::BasicContainerLayer::Validate(void (*)(mozilla::layers::PaintedLayer*, gfxContext*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::layers::DrawRegionClip, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, void*), void*, mozilla::layers::ReadbackProcessor*) (RefPtr.h:261, in XUL)
    #14: mozilla::layers::BasicLayerManager::EndTransactionInternal(void (*)(mozilla::layers::PaintedLayer*, gfxContext*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::layers::DrawRegionClip, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, void*), void*, mozilla::layers::LayerManager::EndTransactionFlags) (RefPtr.h:261, in XUL)
    #15: nsDisplayList::PaintRoot(nsDisplayListBuilder*, nsRenderingContext*, unsigned int) (nsDisplayList.h:486, in XUL)
    #16: nsLayoutUtils::PaintFrame(nsRenderingContext*, nsIFrame*, nsRegion const&, unsigned int, nsDisplayListBuilderMode, nsLayoutUtils::PaintFrameFlags) (AlreadyAddRefed.h:116, in XUL)
    #17: PresShell::Paint(nsView*, nsRegion const&, unsigned int) (nsPresShell.cpp:6366, in XUL)
    #18: nsViewManager::ProcessPendingUpdatesPaint(nsIWidget*) (nsRegion.h:75, in XUL)
    #19: nsViewManager::ProcessPendingUpdatesForView(nsView*, bool) (nsViewManager.cpp:409, in XUL)
    #20: nsViewManager::WillPaintWindow(nsIWidget*) (nsViewManager.cpp:719, in XUL)
    #21: nsView::WillPaintWindow(nsIWidget*) (nsISupportsImpl.h:321, in XUL)
    #22: -[ChildView viewWillDraw] (nsCOMPtr.h:295, in XUL)
    #23: -[NSView viewWillDraw] (in AppKit) + 1163
    #24: -[NSView _sendViewWillDrawInRect:clipRootView:] (in AppKit) + 1705
  }
}

Comment 36

[Lee Salzman [:lsalzman]]

Can you attach the full output? I would need more than just those top entries to diagnose.

Comment 37

[AM]

Attachment: dmd6.gz

Output from one week of dmd running on 49a, compiled after addition of font-cache
logging, electrolysis off. MacOS

Comment 38

[Lee Salzman [:lsalzman]]

(In reply to AM from comment #37)
> Created attachment 8753873 [details]
> dmd6.gz
> 
> Output from one week of dmd running on 49a, compiled after addition of
> font-cache
> logging, electrolysis off. MacOS

Hmm, it looks like it is just showing mostly canvas 2d, where we already do account for image data allocation, and the Skia glyph cache, which is bounded in size (to an amount of only some MBs) and would not otherwise contribute a huge unbounded memory usage.

So I can't really see where the bulk of it would be coming from... thus I am stumped.

Comment 39

[Timothy Nikkel (:tnikkel)]

AM, does about:memory still report a lot of heap-unclassified?

Comment 40

[AM]

no, this run went to 30% heap-unclassified, total heap was 1.5GB.
I find the program behaves  differently when running with dmd. It is much better behaved.

I don't see what I can add at this point.

Comment 41

[Nicholas Nethercote [inactive]]

(In reply to Lee Salzman [:lsalzman] from comment #38)
> 
> Hmm, it looks like it is just showing mostly canvas 2d, where we already do
> account for image data allocation, and the Skia glyph cache, which is
> bounded in size (to an amount of only some MBs) and would not otherwise
> contribute a huge unbounded memory usage.
> 
> So I can't really see where the bulk of it would be coming from... thus I am
> stumped.

One thing that may be relevant: DMD's view of what memory gets reported in about:memory (i.e. what memory doesn't end up in "heap-unclassified") is imperfect. DMD knows about the reporting of heap blocks that are measured with an appropriate MallocSizeOf function, but it doesn't know about the reporting of heap blocks whose size is computed. Most of our reporting is done via measurement, but some is done via computation, and that does include some gfx memory, which may include canvas memory.

So, DMD might be wrong when it says a block or blocks are unreported, alas.

Comment 42

[AM]

I really get the impression there is a leak, which goes away under dmd. Top/ActivityMonitor show very
high memory usage (I stop at 5GB) for release versions. heap-unclassified grows - first attachment is 88%.

The dmd builds Never have gigantic heap-unclassified. I imported a full copy of my firefox profile into the dmd run, so it is not even a strange/corrupted setting that could be the problem.

The release versions become  unresponsive - several seconds beach-ball to type a url, whereas even
with the the tracing overhead the dmd builds stay  reactive (after a week of use).

I will keep a dmd version running in the background, just to see if it does anything crazy. But I don't
think that I will be able to bring any more light to this.

Comment 43

[Nicholas Nethercote [inactive]]

I can't think of any reason why the DMD build would perform better. DMD basically just wraps memory allocations and deallocations and does some extra work on them. :(

Comment 44

[Timothy Nikkel (:tnikkel)]

Could you try DMD on the original profile (not the copy)? Or try a non-DMD build on the copy profile?

Comment 45

[AM]

Attachment: dmd_long.gz

result from dmd after running a couple of weeks

Comment 46

[AM]

Attachment: memory-report.json_long.gz

about:memory output from the long run

Comment 47

[AM]

I have been running various versions of firefox over the last weeks. Always from the home profile.

1) firefox 47 (downloaded), gives very high (75%) heap_unclassified within a day.

2) firefox 48beta (downloaded) is dramatically better, heap_unclassified stays at about 30%. I get the impression that there is a major difference between 47 and 48 in terms of memory behaviour.

3) I have also the results of a long run on 49 with dmd enabled. Memory usage after two weeks is rather
high 2.8GB; heap_unclassified remains at about 30% (800MB). I have included the memory-report and dmd data
from this version.

Comment 48

[ovidiu boca[:Ovidiu]]

Timothy the attached files by AM are helpful?

Comment 49

[Timothy Nikkel (:tnikkel)]

The latest dmd log looks a lot like the previous ones. There is some canvas draw targets which I think we determined were likely false positives. And then it's more font stuff. 30% heap-unclassified is a little high (closer to 20% might be more normal), but it's not crazy high like 80% you had before.

Comment 50

[ovidiu boca[:Ovidiu]]

Do you think that are enough information to set a component?

Comment 51

[AM]

I find 2.8GB rather high for memory usage, and 30% of it is quite a lot 
- but this was not my original complaint!

Yesterday I had a run-away firefox getting to 56GB, (48,b05) before being suspended by the OS.
I was unable to look at the memory profile due to the system performance, however I could see the process in
"activity monitor"/top

Comment 52

[Timothy Nikkel (:tnikkel)]

(In reply to AM from comment #51)
> I find 2.8GB rather high for memory usage, and 30% of it is quite a lot 
> - but this was not my original complaint!

Well, 800mb of it was for DMD, you wouldn't see that memory usage in a regular build. I agree it would be better to see both numbers be smaller.

Looking at your memory report again I see you have a detached window for chrome://browser/content/places/places.xul
Ben, do you know if this is covered by an existing bug?

> Yesterday I had a run-away firefox getting to 56GB, (48,b05) before being
> suspended by the OS.
> I was unable to look at the memory profile due to the system performance,
> however I could see the process in
> "activity monitor"/top

That is definitely something to investigate!

Comment 53

[Ben Kelly [:bkelly, not reviewing]]

Bug 1268898 describes a places.xul leak.  It just landed in mozilla-central.  Let's see if it fixes this.

Comment 54

[ovidiu boca[:Ovidiu]]

AM can you please retest this on your end with FF Nightly? The bug that Ben was talking about landed in FF Nightly(central). Here is the link that helps you to download Nightly: https://nightly.mozilla.org/

Comment 55

[ovidiu boca[:Ovidiu]]

AM, I will close this bug. Please fell free to reopen it if you encounter any issues. First please follow comment 54.

Comment 56

[AM]

I have been away from my main machines over the last weeks. I expect to get back to my lab in a week
and will build the newer nightly with the full dmd memory reporting to compare

Comment 57

[AM]

Attachment: dmd trace from recent nightly

I ran for some time with a dmd build: 51.0a1 (2016-08-17)
I have got up to 40% heap-unclassified.

Much growth seems to be associated with PDF files downloaded from arxiv
https://arxiv.org/list/cond-mat.stat-mech/new

I would normally read several documents per day 

these are math filled documents generated from latex.

Comment 58

[ovidiu boca[:Ovidiu]]

Timothy can you please take a look on what AM posted in comment 57? Can you make an idea of what component will fit this bug? Thanks

Comment 59

[Ben Kelly [:bkelly, not reviewing]]

Did you have any PDFs open when you captured this DMD log?

Looks like a ton of memory is being used by gfx to draw Canvas2D stuff.  If the PDF is open, this may be normal.  If the PDFs have all been closed then this could be a leak.

Also, do you tend to reuse the same tab or open new tabs most of the time?  I wonder if this memory could be for documents in the bfcache.

Unreported {
  17 blocks in heap block record 1 of 25,014
  332,378,112 bytes (332,343,552 requested / 34,560 slop)
  Individual block sizes: 22,286,336; 21,544,960 x 11; 14,716,928 x 4; 14,229,504
  31.46% of the heap (31.46% cumulative)
  59.90% of unreported (59.90% cumulative)
  Allocated at {
    #01: replace_malloc (DMD.cpp:1280, in libdmd.dylib)
    #02: malloc_zone_malloc (in libsystem_malloc.dylib) + 71
    #03: malloc (in libsystem_malloc.dylib) + 42
    #04: SkMallocPixelRef::NewUsing(void* (*)(unsigned long), SkImageInfo const&, unsigned long, SkColorTable*) (SkMallocPixelRef.cpp:186, in XUL)
    #05: SkBitmap::HeapAllocator::allocPixelRef(SkBitmap*, SkColorTable*) (SkBitmap.cpp:447, in XUL)
    #06: SkBitmap::tryAllocPixels(SkBitmap::Allocator*, SkColorTable*) (SkBitmap.cpp:285, in XUL)
    #07: SkGrPixelRef::onReadPixels(SkBitmap*, SkColorType, SkIRect const*) (SkGrPixelRef.cpp:191, in XUL)
    #08: SkROLockPixelsPixelRef::onNewLockPixels(SkPixelRef::LockRec*) (SkGrPixelRef.cpp:27, in XUL)
    #09: SkPixelRef::onRequestLock(SkPixelRef::LockRequest const&, SkPixelRef::LockResult*) (SkPixelRef.cpp:120, in XUL)
    #10: SkPixelRef::requestLock(SkPixelRef::LockRequest const&, SkPixelRef::LockResult*) (SkPixelRef.cpp:209, in XUL)
    #11: SkBitmap::requestLock(SkAutoPixmapUnlock*) const (SkBitmap.cpp:1324, in XUL)
    #12: SkDraw::drawBitmap(SkBitmap const&, SkMatrix const&, SkRect const*, SkPaint const&) const (SkDraw.cpp:1299, in XUL)
    #13: SkCanvas::onDrawBitmap(SkBitmap const&, float, float, SkPaint const*) (SkCanvas.cpp:2358, in XUL)
    #14: mozilla::gfx::DrawTargetSkia::CopySurface(mozilla::gfx::SourceSurface*, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntPointTyped<mozilla::gfx::UnknownUnits> const&) (RefPtrSkia.h:61, in XUL)
    #15: mozilla::layers::ClientCanvasLayer::UpdateTarget(mozilla::gfx::DrawTarget*) (ClientCanvasLayer.cpp:170, in XUL)
    #16: mozilla::layers::CanvasClient2D::Update(mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits>, mozilla::layers::ClientCanvasLayer*) (CanvasClient.cpp:138, in XUL)
    #17: mozilla::layers::ClientCanvasLayer::RenderLayer() (Layers.h:2485, in XUL)
    #18: mozilla::layers::ClientContainerLayer::RenderLayer() (ClientContainerLayer.h:139, in XUL)
    #19: mozilla::layers::ClientContainerLayer::RenderLayer() (ClientContainerLayer.h:139, in XUL)
    #20: mozilla::layers::ClientLayerManager::EndTransactionInternal(void (*)(mozilla::layers::PaintedLayer*, gfxContext*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::layers::DrawRegionClip, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, void*), void*, mozilla::layers::LayerManager::EndTransactionFlags) (ClientLayerManager.cpp:297, in XUL)
    #21: mozilla::layers::ClientLayerManager::EndTransaction(void (*)(mozilla::layers::PaintedLayer*, gfxContext*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::layers::DrawRegionClip, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, void*), void*, mozilla::layers::LayerManager::EndTransactionFlags) (ClientLayerManager.cpp:340, in XUL)
    #22: nsDisplayList::PaintRoot(nsDisplayListBuilder*, nsRenderingContext*, unsigned int) (nsDisplayList.h:492, in XUL)
    #23: nsLayoutUtils::PaintFrame(nsRenderingContext*, nsIFrame*, nsRegion const&, unsigned int, nsDisplayListBuilderMode, nsLayoutUtils::PaintFrameFlags) (AlreadyAddRefed.h:116, in XUL)
    #24: PresShell::Paint(nsView*, nsRegion const&, unsigned int) (nsPresShell.cpp:6047, in XUL)
  }
}

Comment 60

[AM]

I had two or three PDF's open. In fact I will often have pdf's open that will hang around for a week or
two - the time that I read and digest.

I will often have 10-20  tabs of which several pdf's , but if I have restarted they will typically not all be loaded.

My standard browse style is to "cmd-click" to open into a new tab, then switch/read/kill-tab

Most tabs are thus used just once

I also have groups of tabs that I like to read together - thus O(10) tabs opened from the bookmarks window
as a block. They are then read, and closed within a few minutes, and never re-used

Comment 61

[Ben Kelly [:bkelly, not reviewing]]

This is perhaps an issue with pdfjs opening too large of a canvas or something then.  Or maybe it should free things after a while if its not visible.

Brendan, what do you think?

Comment 62

[Timothy Nikkel (:tnikkel)]

(In reply to AM from comment #57)
> Created attachment 8784769 [details]
> dmd trace from recent nightly
> 
> I ran for some time with a dmd build: 51.0a1 (2016-08-17)
> I have got up to 40% heap-unclassified.
> 
> Much growth seems to be associated with PDF files downloaded from arxiv
> https://arxiv.org/list/cond-mat.stat-mech/new

I visited this page, opened the first 12 pdfs, scrolled through them, and I got 40% heap-unclassified. So looks like we have something that we can investigate on a developers machine now. This is good.

Thanks for the log. The actual dmd log looks about the same as the previous ones. Hopefully we can dig into this more now that we have a reproducible way to test.

Comment 63

[Timothy Nikkel (:tnikkel)]

I reproduced locally in a DMD build. I got the same results as AM. Specifically, the only significant allocation found by DMD was canvas. In the past we thought these were actually being reported properly in about:memory, but because they were calculated instead of measured DMD didn't think they were reported properly. But I think this may be incorrect because the amount of dark matter reported by DMD matches the heap-unclassified in about:memory, and the amount of memory reported about:memory for canvas does not match the amount of canvas allocations in the DMD report.

So my guess is that our canvas memory reporting is actually not reporting some allocations.

Comment 64

[Nicholas Nethercote [inactive]]

> So my guess is that our canvas memory reporting is actually not reporting
> some allocations.

The canvas-2d-pixels reporter appears to be totally busted at the moment, alas: bug 1241865.

Comment 65

[Marco Castelluccio [:marco]]

Canceling 6 years old needinfo.

Comment 66

[Brendan Dahl [:bdahl]]

Closing for now. This bug doesn't have much we can take action on. There several other memory use bugs with PDF that cover this case.