1298905 - Unused memory does not get released to OS

Status: RESOLVED WORKSFORME

(Core :: DOM: Workers, defect, P3)

Description

[The 8472]

The process has 4.9GB private bytes allocated and even more virtual memory. Explicit allocations only account for 2.3GB and overall only 3GB are resident. Which means 1.9GB are not used and unaccounted for and should be freed.

This is after several days of uptime and hitting "minimize memory usage" in about:memory.



Nightly 26-08-2016

    2,346.95 MB (100.0%) -- explicit
     
     
            891.76 MB ── canvas-2d-pixels
             75.83 MB ── d3d11-shared-textures
              0.00 MB ── gfx-d2d-vram-draw-target
              7.18 MB ── gfx-d2d-vram-source-surface
              1.99 MB ── gfx-surface-win32
              0.00 MB ── gfx-textures
              0.00 MB ── gfx-textures-peak
              0.00 MB ── gfx-tiles-waste
                    0 ── ghost-windows
              0.00 MB ── gpu-committed
            142.33 MB ── gpu-dedicated
             30.36 MB ── gpu-shared
          1,265.85 MB ── heap-allocated
              1.00 MB ── heap-chunksize
          1,946.00 MB ── heap-mapped
                   92 ── host-object-urls
             10.71 MB ── imagelib-surface-cache-estimated-locked
             10.71 MB ── imagelib-surface-cache-estimated-total
                    0 ── imagelib-surface-cache-overflow-count
              1.68 MB ── js-main-runtime-temporary-peak
          4,942.64 MB ── private
          3,024.63 MB ── resident
          2,922.34 MB ── resident-unique
              0.00 MB ── shmem-allocated
             43.17 MB ── shmem-mapped
             48.70 MB ── system-heap-allocated
          6,579.20 MB ── vsize

Comment 1

[Emil Pasca [:emilpasca]]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:51.0) Gecko/20100101 Firefox/51.0

I have tested this issue on Windows 10 x64 with the latest Firefox release (48.0.2) and the latest Nightly (51.0a1-20160831030224) and could not reproduce it.
I have tried to reproduce this by opening many different tabs with different content, however, I'm getting different results in the about:memory report.

Could you please provide us with more info, the OS and a full memory report?

Comment 2

[The 8472]

Attachment: memory-report unreleased, non-resident memory.json.gz

I don't have any particular STR, the memory just accumulates over time on a heavily used session with many tabs and various web apps doing refreshes in the background.

It's a 64bit build on windows 10.

And I don't really see a full memory report would provide any more insight considering that the leaked memory is not covered by explicit allocations (which you can already see in comment 0), but here you go.

Comment 3

[The 8472]

Attachment: memory-report private and virtual leak.json.gz

Here's another one with Build ID 	20160831030224

This one seems even more extreme, on top of the unaccounted-for private bytes there also is a large chunk of reserved virtual memory (a single 8GB chunk according to VMMap) and a high heap-unclassified figure too.

Comment 4

[The 8472]

Attachment: reserved.png

Comment 5

[Emil Pasca [:emilpasca]]

Due to the fact that the issue is reproducible for the reporter I will assign a component and perhaps there's someone with extensive knowledge on this area that might be able to help here.

Comment 6

[The 8472]

Attachment: memory-report 29.10 6GB+ overhead.json.gz

The problem seems to have gotten worse. With 2 days of uptime and 1 parent + 3 child processes i'm getting the following:

* two of the child process have a 1-2GB private bytes that are not tracked by private bytes or explicit allocations compared to 500-700MB explicit allocations. So each process has a ratio of about untracked allocations : explicit. The other child process only has a 500MB discrepancy.
* the parent process has 42% heap-unclassified (2GB) and another 3.6GB discrepancy between private bytes and working set or a 2.6GB discrepancy between private bytes and explicit allocations

After startup working set ≈ private bytes ≈ explicit (give or take maybe 50MB). So this discrepancy growing over time is a pretty big leak. Accumulated it amounts to about 8GB leaked memory.

At efaust's recommendation I tried capturing memory allocations with windows performance recorder as in bug 1300724 but they don't seem to contain symbols for the allocation stacks.

Do I need a special firefox build for that?

Comment 7

[Eric Rahm [:erahm]]

JS engine is probably not the right place for this, but I'm not sure where else to put it yet.

(In reply to The 8472 from comment #3)
> Created attachment 8787253 [details]
> memory-report private and virtual leak.json.gz
> 
> Here's another one with Build ID 	20160831030224
> 
> This one seems even more extreme, on top of the unaccounted-for private
> bytes there also is a large chunk of reserved virtual memory (a single 8GB
> chunk according to VMMap) and a high heap-unclassified figure too.

This was probably the WASM guard pages (which are now reported).

(In reply to The 8472 from comment #6)
> Created attachment 8805798 [details]
> memory-report 29.10 6GB+ overhead.json.gz
> 
> The problem seems to have gotten worse. With 2 days of uptime and 1 parent +
> 3 child processes i'm getting the following:
> 
> * two of the child process have a 1-2GB private bytes that are not tracked
> by private bytes or explicit allocations compared to 500-700MB explicit
> allocations. So each process has a ratio of about untracked allocations :
> explicit. The other child process only has a 500MB discrepancy.
> * the parent process has 42% heap-unclassified (2GB) and another 3.6GB
> discrepancy between private bytes and working set or a 2.6GB discrepancy
> between private bytes and explicit allocations
> 
> After startup working set ≈ private bytes ≈ explicit (give or take maybe
> 50MB). So this discrepancy growing over time is a pretty big leak.
> Accumulated it amounts to about 8GB leaked memory.
> 
> At efaust's recommendation I tried capturing memory allocations with windows
> performance recorder as in bug 1300724 but they don't seem to contain
> symbols for the allocation stacks.
> 
> Do I need a special firefox build for that?

Ting, do you know what we need to do to get a usable report from WPR?

Comment 8

[The 8472]

Attachment: 5GB discrepancy.png

Today's session deteriorated even further. 8GB footprint for the parent process alone, 5 of which are leaks.

Comment 9

[Ting-Yu Chou [:ting] (away)]

(In reply to Eric Rahm [:erahm] from comment #7)
> > At efaust's recommendation I tried capturing memory allocations with windows
> > performance recorder as in bug 1300724 but they don't seem to contain
> > symbols for the allocation stacks.
> > 
> > Do I need a special firefox build for that?
> 
> Ting, do you know what we need to do to get a usable report from WPR?

Bug 1300724 comment 43 should be sufficient to have allocation stacks in the record file.

For translating the symbols in WPA, since I was recording with my local debug build m-c, simply add the binary path to "Trace" > "Configure Symbol Paths" does work for me, and I assume it works for the other release builds if firefox symbol server url is added to the symbol paths. However, I just tried with Nightly and it didn't work. It complained xul.pdb was not found, the same symbol server url does work for VS2015 and WinDbg though. I am not sure why. Checked MDN [1] but didn't help.

dmajor, do you have any ideas about this?

[1] https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Profiling_with_Xperf

Comment 10

[Ting-Yu Chou [:ting] (away)]

My workaround for downloading xul.pdb is using symchk.exe, see https://developer.mozilla.org/en-US/docs/Mozilla/Using_the_Mozilla_symbol_server#Downloading_symbols_using_symchk.exe.

So for WPA, follow the steps to have symbol translated:

1. C:\path\to\symchk.exe /v C:\path\to\firefox\xul.dll /s SRV*C:\symcache*http://symbols.mozilla.org
2. Make sure the xul.pdb is downloaded
3. Open WPA, load the record file, add virtualalloc commit to the analysis view, and select "Trace" > "Configure Symbol Paths"
4. Add "SRV*C:\symcache*http://symbols.mozilla.org to the paths
5. I am not sure if this is needed, but close WPA and reload the record file so the new symbol path is picked up.
6. Again, add virtualalloc commit to the analysis view, select "Trace" > "Load Symbols".

The WPA I am using is version 10.0.14393.33, probably the workaround of symchk.exe is not needed for the other versions.

Comment 11

[(Away)]

Historically WPA has been pretty finicky about symbols. This page has some tips: https://randomascii.wordpress.com/2012/10/04/xperf-symbol-loading-pitfalls/

Comment 12

[Eric Rahm [:erahm]]

The 8472, are comment 9, comment 10, comment 11 enough for you to get working stacks? It sounds like you need configure a symbol server and/or download the symbols. Sorry it's such a hassle but it would be great to figure out what's going wrong for you.

Comment 13

[The 8472]

WPA's symbol downloading does not seem to work. Symchk seems to work in principle, but nightly got updated in the meantime, so it downloads the symbols for the wrong version of xul.dll.

Can I point it at the hash that wpa is looking for?

Comment 14

[Ting-Yu Chou [:ting] (away)]

symchk can use a manifest file to retrieve the symbols, see https://msdn.microsoft.com/en-us/library/windows/hardware/ff560061(v=vs.85).aspx.

A sample manifest is like:

  xul.pdb,D2E3DBCC63EC41D18FF9344A12AB63F82,1
  xul.dll,58174628325e000,1

Comment 15

[The 8472]

Getting connection errors when attempting to upload file, so i'm putting it on an external host: https://my.mixtape.moe/iybvvr.7z

WPA virtual alloc analysis output for firefox parent process as csv. Archive expands to 465MB.

Comment 16

[Ting-Yu Chou [:ting] (away)]

Could you put the WPR record file somewhere as well? Thanks.

Comment 17

[The 8472]

considering that this records my entire system and not a single process, no

Comment 18

[Ting-Yu Chou [:ting] (away)]

This stack contributes 783 MB (69%) to the impacting size:

[Root]/ntdll.dll!RtlUserThreadStart/kernel32.dll!BaseThreadInitThunk/ucrtbase.dll!_o__realloc_base/nss3.dll!<PDB not found>/nss3.dll!<PDB not found>/xul.dll!nsThread::ThreadFunc/xul.dll!MessageLoop::Run/xul.dll!MessageLoop::RunHandler/xul.dll!mozilla::ipc::MessagePumpForNonMainThreads::Run/xul.dll!NS_ProcessNextEvent/xul.dll!nsThread::ProcessNextEvent/xul.dll!`anonymous namespace'::WorkerThreadPrimaryRunnable::Run/xul.dll!`anonymous namespace'::WorkerJSContext::Initialize/xul.dll!mozilla::CycleCollectedJSContext::Initialize/xul.dll!JS_NewContext/xul.dll!JSRuntime::init/xul.dll!js::gc::GCRuntime::init/xul.dll!js::Nursery::init/xul.dll!js::Nursery::updateNumChunksLocked/xul.dll!js::gc::GCRuntime::getOrAllocChunk/xul.dll!js::gc::Chunk::allocate/xul.dll!js::gc::MapAlignedPages/KernelBase.dll!VirtualAlloc/ntdll.dll!NtAllocateVirtualMemory/ntoskrnl.exe!KiSystemServiceCopyEnd/ntoskrnl.exe!NtAllocateVirtualMemory/ntoskrnl.exe! ?? ::NNGAKEGL::`string'

Seems there're a load of worker threads?

Comment 19

[The 8472]

Load of workers? No, a handful maybe. But either way, workers seem to be covered by about:memory, so just the fact that there are workers doesn't explain the observed discrepancies in comment 6

Comment 20

[Ting-Yu Chou [:ting] (away)]

FWIW, the stack in comment 18 allocates 1 MB, so there're 783 worker threads. For the memory report in comment 6, I saw 1822 of window objects like this in the parent process:

  0.31 MB (00.01%) ++ top(<anonymized-6062>, id=6062)

which looks like a leak and is what I'm concerned.

Note the numbers of private and explicit allocation don't need to be the same, the difference can be from kernel space, external library, etc.

Comment 21

[The 8472]

> I saw 1822 of window objects like this in the parent process

that's just backgrounded tabs, not a leak

> FWIW, the stack in comment 18 allocates 1 MB, so there're 783 worker threads. 

those threads don't have to exist at the same though, do they? I think just create, leak, die, create, leak, die would do the job too.

Comment 22

[Ting-Yu Chou [:ting] (away)]

(In reply to The 8472 from comment #21)
> > I saw 1822 of window objects like this in the parent process
> 
> that's just backgrounded tabs, not a leak

So you have 1823 tabs opened, could you elaborate more about how you reproduced this? Except massive tabs, are there any specific web page or JS you're running?

> 
> > FWIW, the stack in comment 18 allocates 1 MB, so there're 783 worker threads. 
> 
> those threads don't have to exist at the same though, do they? I think just
> create, leak, die, create, leak, die would do the job too.

The definition of Impacting Size from MSDN [1]:

  The size of outstanding allocations or the size difference between the start and end of the selected time interval. This size adjusts based on the selected view port.

  The Impacting Size value will be zero if all memory allocated by a process is freed by the end of the visualized interval in WPA.

So contributing to impacting size in comment 18 means those worker were not dead yet from your attached analysis report in comment 15.

[1] https://msdn.microsoft.com/zh-tw/windows/hardware/commercialize/test/wpt/memory-footprint-optimization-exercise-2

Comment 23

[The 8472]

> So contributing to impacting size in comment 18 means those worker were not dead yet from your attached analysis report in comment 15.

Does it? Couldn't be that the workers are dead and just did not free the memory? Pretty much the definition of a leak.

I mean the workers don't show up in the about:memory reports after all.

> So you have 1823 tabs opened, could you elaborate more about how you reproduced this?

As already mentioned in comment 2 I don't really intentionally "reproduce" it. It just happens with session uptime. The discrepancy slowly grows over time.

> Except massive tabs, are there any specific web page or JS you're running?

Well, since you say it might be worker related, about:debugging says there's a bunch of chrome workers and one website using a service worker, but i don't know if they do anything special.

My guess would be that it could be a small leak that accumulates as workers are being torn down and started again over and over again.

Comment 24

[Eric Rahm [:erahm]]

This is sounding more like a worker leak, moving over to DOM :: Workers, cc bkelly who might be able to help out.

Comment 25

[Ben Kelly [:bkelly, not reviewing]]

Workers use our malloc heap like everything else.  You are saying that somehow workers are using memory that doesn't show in our heap or in heap-unclassified dark matter?

Can you post an about:support?

Comment 26

[The 8472]

> Can you post an about:support?

Which parts are relevant? It's fairly long.

> You are saying that somehow workers are using memory that doesn't show in our heap or in heap-unclassified dark matter?

I'm sometimes also seeing high heap-unclassified numbers, but that may be a separate problem.
Currently I'm again seeing a ~3GB discrepancy between explicit allocations and private bytes with heap-unclassified only hovering around 700MB.

Whether workers are responsible, that I do not know. I have to trust the analysis in comment 18

Comment 27

[Ben Kelly [:bkelly, not reviewing]]

(In reply to The 8472 from comment #26)
> > Can you post an about:support?
> 
> Which parts are relevant? It's fairly long.

All of it if possible.

I believe jemalloc has an algorithm for releasing memory back to the OS.  Mike, do you know what could be going on here?  (High private memory exceeding explicit allocations.)  Are there things gecko can do with malloc that makes this better or worse?

Comment 28

[Eric Rahm [:erahm]]

(wrong Mike)

I think doing a minimize in about:memory should clear up if jemalloc is involved, but at 3GiB it feels like that's probably not the issue.

Comment 29

[The 8472]

Attachment: about-support.json

(In reply to Ben Kelly [:bkelly] from comment #27)
> (In reply to The 8472 from comment #26)
> > > Can you post an about:support?
> > 
> > Which parts are relevant? It's fairly long.
> 
> All of it if possible.

Well, I'm a bit skeptical about providing information without you being able to say which parts are actually relevant to this particular issue because that kind of information has been used in the past to all but dismiss an issue. Like "oh, you have that addon? that's probably it then" when it wasn't.

Comment 30

[Eric Rahm [:erahm]]

(In reply to The 8472 from comment #29)
> Created attachment 8808316 [details]
> about-support.json
> 
> (In reply to Ben Kelly [:bkelly] from comment #27)
> > (In reply to The 8472 from comment #26)
> > > > Can you post an about:support?
> > > 
> > > Which parts are relevant? It's fairly long.
> > 
> > All of it if possible.
> 
> Well, I'm a bit skeptical about providing information without you being able
> to say which parts are actually relevant to this particular issue because
> that kind of information has been used in the past to all but dismiss an
> issue. Like "oh, you have that addon? that's probably it then" when it
> wasn't.

If you don't want to provide the output, then here's the standard spiel:
- Restart in safe mode, if the problems go away start re-enabling add-ons and see which one breaks things
- Identify any modified preferences, revert to defaults, if that fixes things start re-enabling
- Provide graphics info, we sometimes see memory issues with certain graphics setups (particularly on Windows)

Generally we can take a look at those and give pointers, but if you want to do that on your own that's fine.

Comment 31

[The 8472]

> If you don't want to provide the output

But I already did?

I merely explained why I am reluctant to do so. That standard spiel actually being part of the reason because it rarely is useful for me.

Comment 32

[Ben Kelly [:bkelly, not reviewing]]

This is nightly 45.0a1 built in Nov 2015?  Please upgrade to a current nightly or use one of our release channels like 45ESR.  Its very difficult to support old snapshots from a continuously changing branch like nightly.

Comment 33

[The 8472]

Attachment: about.support.json

Oops, sorry. I had two almost identically named files next to each other.

Comment 34

[Ben Kelly [:bkelly, not reviewing]]

I think a good place to start would be to revert some of your prefs back to their standard value.  For example:

    "dom.max_chrome_script_run_time": 0,
    "dom.ipc.processCount": 3,
    "javascript.options.mem.gc_incremental_slice_ms": 70,
    "javascript.options.mem.gc_refresh_frame_slices_enabled": false,

I think mucking with chrome script run time might have unexpected behavior.  I know I've seen increased fragmentation with multiple child processes.  Changing GC timing also seems suspect if you are running into memory issues that others do not see.

Comment 35

[The 8472]

> I think a good place to start 

What about the WPA recordings already done? Were they a false lead?

Comment 36

[Ben Kelly [:bkelly, not reviewing]]

TBH I have no idea what WPR is or how to interpret the results.  As I said, I don't see how allocations in workers using malloc could show up as memory outside of the explicit memory allocation in our reports.

I think trying to bring a non-standard config closer to standard is a reasonable approach to isolate the cause of behavior others are not seeing.

Anyway, I'll wait to see what Mike says about malloc releasing memory to the OS.

Comment 37

[Ting-Yu Chou [:ting] (away)]

(In reply to The 8472 from comment #23)
> > So contributing to impacting size in comment 18 means those worker were not dead yet from your attached analysis report in comment 15.
> 
> Does it? Couldn't be that the workers are dead and just did not free the
> memory? Pretty much the definition of a leak.
> 
> I mean the workers don't show up in the about:memory reports after all.

Once WorkerPrivate::DoRunLoop() calls DisableMemoryReporter(), the worker won't be listed in about:memory: 

  https://dxr.mozilla.org/mozilla-central/rev/8e8b146fcb8b268e3c09b646087c6b2ef9f0af6f/dom/workers/WorkerPrivate.cpp#4675.

Note the WorkerJSContext is destructed when the clause contains it is existed, which happens later than DisableMemoryReporter():

  https://dxr.mozilla.org/mozilla-central/rev/8e8b146fcb8b268e3c09b646087c6b2ef9f0af6f/dom/workers/RuntimeService.cpp#2852

Comment 38

[Ben Kelly [:bkelly, not reviewing]]

(In reply to Ting-Yu Chou [:ting] from comment #37)
> Once WorkerPrivate::DoRunLoop() calls DisableMemoryReporter(), the worker
> won't be listed in about:memory: 

Any allocations would still show up in explicit under heap-unclassified, though.

Comment 39

[Mike Hommey [:glandium]]

There are multiple fishy things in comment 0, when compared to each other:

    2,346.95 MB (100.0%) -- explicit
          1,265.85 MB ── heap-allocated
          1,946.00 MB ── heap-mapped
          4,942.64 MB ── private
          3,024.63 MB ── resident

- explicit is, essentially, all the things gecko knows has been allocated.
- heap-allocated is the amount of memory that was allocated by jemalloc.

That means there's a lot of memory that is known to gecko that is not allocated by jemalloc. Whatever that means.

- heap-mapped is the amount of memory that was mapped by jemalloc.

heap-mapped - heap-allocated can be seen as the amount of wasted memory, caused by fragmentation. The difference is unusually large, but doesn't account for all the memory sucked.

- private is memory private to the process, whatever the PrivateUsage field from GetProcessMemoryInfo actually means on Windows. That it's significantly larger than "explicit" is fishy.

- resident is everything that is mapped in the process and resides in physical memory. That it's significantly smaller than private is fishy. That it's significantly larger than heap-mapped is fishy.

Comment 40

[Eric Rahm [:erahm]]

(In reply to Mike Hommey [:glandium] from comment #39)
> There are multiple fishy things in comment 0, when compared to each other:
> 
>     2,346.95 MB (100.0%) -- explicit
>           1,265.85 MB ── heap-allocated
>           1,946.00 MB ── heap-mapped
>           4,942.64 MB ── private
>           3,024.63 MB ── resident
> 
> - explicit is, essentially, all the things gecko knows has been allocated.
> - heap-allocated is the amount of memory that was allocated by jemalloc.
> 
> That means there's a lot of memory that is known to gecko that is not
> allocated by jemalloc. Whatever that means.
> 
> - heap-mapped is the amount of memory that was mapped by jemalloc.
> 
> heap-mapped - heap-allocated can be seen as the amount of wasted memory,
> caused by fragmentation. The difference is unusually large, but doesn't
> account for all the memory sucked.
> 
> - private is memory private to the process, whatever the PrivateUsage field
> from GetProcessMemoryInfo actually means on Windows. That it's significantly
> larger than "explicit" is fishy.
> 
> - resident is everything that is mapped in the process and resides in
> physical memory. That it's significantly smaller than private is fishy. That
> it's significantly larger than heap-mapped is fishy.

So is there something in jemalloc that could contribute to the excessively large private measurement [1]? I'm thinking along the lines of not decomitting / not doing an MADV_FREE etc.

[1] https://blogs.msdn.microsoft.com/ricom/2005/08/01/private-bytes-performance-counter-beware/

Comment 41

[The 8472]

Would the VMMap output be helpful?

Comment 42

[Mike Hommey [:glandium]]

(In reply to Eric Rahm [:erahm] from comment #40)
> So is there something in jemalloc that could contribute to the excessively
> large private measurement [1]? I'm thinking along the lines of not
> decomitting / not doing an MADV_FREE etc.

Everything that has been allocated by jemalloc and is not part of the current heap-mapped is supposed to be VirtualFree'ed by jemalloc, so nothing jemalloc does can explain the difference between heap-mapped and private/resident... except if for some reason the VirtualFree's are not happening.

Comment 43

[The 8472]

Attachment: vmmap report

This is a VMMap[0] snapshot of a session with 5 days uptime with a 9GB discrepancy between private and working set.

[0] https://technet.microsoft.com/en-us/sysinternals/vmmap.aspx

Comment 44

[The 8472]

Attachment: about:memory report corresponding to vmmap report

And this is the corresponding about:memory report

Comment 45

[Ben Kelly [:bkelly, not reviewing]]

FWIW, I am also seeing high private bytes and RSS on my nightly on windows 10.  My main process has an explicit usage of 238MB, but private bytes and RSS of ~800MB.

Are we somehow writing to executable pages preventing them from being paged out?

Mike, how fishy is my memory report here?

Main Process
Explicit Allocations

238.82 MB (100.0%) ++ explicit

Other Measurements

134,217,727.94 MB (100.0%) ++ address-space

21.43 MB (100.0%) ++ decommitted

3,110 (100.0%) ++ event-counts

30 (100.0%) ++ file-blob-urls

182.78 MB (100.0%) ++ heap-committed

5.60 MB (100.0%) ++ images

89.47 MB (100.0%) ++ js-main-runtime

438 (100.0%) ++ js-main-runtime-compartments

42.76 MB (100.0%) ++ js-main-runtime-gc-heap-committed

734 (100.0%) ++ message-manager

1,825 (100.0%) ++ observer-service

522 (100.0%) ++ observer-service-suspect

1,298 (100.0%) ++ preference-service

0 (100.0%) ++ queued-ipc-messages

9.67 MB (100.0%) ++ window-objects

          0.00 MB ── gfx-d2d-vram-draw-target
          0.00 MB ── gfx-d2d-vram-source-surface
          0.95 MB ── gfx-surface-win32
          0.00 MB ── gfx-textures
          0.00 MB ── gfx-textures-peak
          0.00 MB ── gfx-tiles-waste
                0 ── ghost-windows
          0.00 MB ── gpu-committed
          0.00 MB ── gpu-dedicated
          0.00 MB ── gpu-shared
        144.51 MB ── heap-allocated
          1.00 MB ── heap-chunksize
        276.00 MB ── heap-mapped
               30 ── host-object-urls
          1.10 MB ── imagelib-surface-cache-estimated-locked
          1.10 MB ── imagelib-surface-cache-estimated-total
                0 ── imagelib-surface-cache-overflow-count
          1.73 MB ── js-main-runtime-temporary-peak
        783.32 MB ── private
        845.20 MB ── resident
        777.36 MB ── resident-unique
         34.07 MB ── shmem-allocated
         34.09 MB ── shmem-mapped
          2.94 MB ── system-heap-allocated
      1,563.34 MB ── vsize
130,525,468.25 MB ── vsize-max-contiguous

Comment 46

[Eric Rahm [:erahm]]

With Ben reproing this should probably be higher priority.

Comment 47

[Ben Kelly [:bkelly, not reviewing]]

I've been running FF50 since Wednesday on the same machine and I don't see RSS/Private Bytes elevated above the explicit memory there.  It seems something has changed between FF50 and Nightly.

Comparison with main process in comment 45:

                  50-Release        Nightly
Explicit:           320MB             238MB
Resident:           428MB             845MB
Resident-Unique:    387MB             777MB
Private Bytes:      568MB             783MB

So Resident-Unique is ~540MB more than explicit in nightly, but only ~67MB more than explicit in FF50.

Memory report for main process in FF50:

Main Process
Explicit Allocations

320.18 MB (100.0%) ++ explicit

Other Measurements

134,217,727.94 MB (100.0%) ++ address-space

53.43 MB (100.0%) ++ decommitted

3,555 (100.0%) ++ event-counts

2 (100.0%) ++ file-blob-urls

259.20 MB (100.0%) ++ heap-committed

6.59 MB (100.0%) ++ images

92.74 MB (100.0%) ++ js-main-runtime

407 (100.0%) ++ js-main-runtime-compartments

49.01 MB (100.0%) ++ js-main-runtime-gc-heap-committed

1,201 (100.0%) ++ message-manager

2,222 (100.0%) ++ observer-service

570 (100.0%) ++ observer-service-suspect

1,289 (100.0%) ++ preference-service

0 (100.0%) ++ queued-ipc-messages

56.99 MB (100.0%) ++ window-objects

        269.55 MB ── canvas-2d-pixels
         66.65 MB ── d3d11-shared-textures
          0.00 MB ── gfx-d2d-vram-draw-target
          0.76 MB ── gfx-d2d-vram-source-surface
          3.95 MB ── gfx-surface-win32
          0.00 MB ── gfx-textures
          0.00 MB ── gfx-textures-peak
          0.00 MB ── gfx-tiles-waste
                0 ── ghost-windows
          0.00 MB ── gpu-committed
        234.47 MB ── gpu-dedicated
         17.02 MB ── gpu-shared
        208.78 MB ── heap-allocated
          1.00 MB ── heap-chunksize
        382.00 MB ── heap-mapped
                2 ── host-object-urls
          3.88 MB ── imagelib-surface-cache-estimated-locked
          3.88 MB ── imagelib-surface-cache-estimated-total
                0 ── imagelib-surface-cache-overflow-count
          2.27 MB ── js-main-runtime-temporary-peak
        568.35 MB ── private
        428.01 MB ── resident
        387.65 MB ── resident-unique
          0.00 MB ── shmem-allocated
          1.36 MB ── shmem-mapped
         18.19 MB ── system-heap-allocated
      1,513.01 MB ── vsize
130,023,802.31 MB ── vsize-max-contiguous

Comment 48

[Ben Kelly [:bkelly, not reviewing]]

Once FF52 dev-edition updates are enabled we should see if this reproduces there or tracks nightly only.

Comment 49

[The 8472]

> I don't see RSS/Private Bytes elevated above the explicit memory there. 

That would appear to be a separate issue from what I'm seeing.

Comment 50

[The 8472]

New bit of information: The gfx process shows the same discrepancy as the parent process, so it might be something they have in common.

Comment 51

[The 8472]

Another bit: The problem also occurs with hw acceleration disabled.

Comment 52

[Danial Horton]

I believe I have the same issue in Nightly 55, I dropped memory reports in bug 1357225

Comment 53

[Danial Horton]

The comment on workers also implicates bug 1361016

Comment 54

[Danial Horton]

(In reply to The 8472 from comment #49)
> > I don't see RSS/Private Bytes elevated above the explicit memory there. 
> 
> That would appear to be a separate issue from what I'm seeing.



>>    2,346.95 MB (100.0%) -- explicit
>>          1,265.85 MB ── heap-allocated
>>          1,946.00 MB ── heap-mapped
>>          4,942.64 MB ── private
>>          3,024.63 MB ── resident

^I believe that is from the memory report you provided.

Ben was testing for it in Fx50, and he was simply commenting that the issue is not present in that version.

I am now attempting to reproduce it on 53 and Beta 54 to see if only nightly or nightly+beta are affected.

Comment 55

[Cervantes Yu [:cyu] [:cervantes]]

This symptom looks pretty similar to bug 1379250, and was recently fixed in bug 1377738.

Comment 56

[Danial Horton]

I can certainly confirm a difference, I will continue testing over the weekend and then let you know if its fully fixed.

Comment 57

[Wayne Mery (:wsmwk)]

(In reply to Danial Horton from comment #56)
> I can certainly confirm a difference, I will continue testing over the
> weekend and then let you know if its fully fixed.

Comment 58

[Danial Horton]

There is no longer a runaway increase in Commit Size when the browser is left idle.

VMMap also looks good.

Comment 59

[Wayne Mery (:wsmwk)]

> There is no longer a runaway increase in Commit Size when the browser is left idle.

the8472, do you agree problem is gone?

Comment 60

[The 8472]

Yes, memory use seems to be more stable over longer uptimes now.

Comment 61

[Wayne Mery (:wsmwk)]

Thanks for the updates