845430 - Memory consumption rises by 9 MiB during five minutes of inactivity in safe mode

Status: RESOLVED WORKSFORME

(Firefox :: Untriaged, defect)

Description

[jacques]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0
Build ID: 20130215130331

Steps to reproduce:

lauching FF, going to any webpage e.g google.com
- in normal mode
- in safe mode
- after disabling all modules & extensions (sorry , idid not removed them)



Actual results:

Task Mgr shows a continuous memory leak



Expected results:

no memory leak

Comment 1

[jacques]

just read the page on memory leak tools. Not a path I wish to take easily. I gave up C++ 10 years ago.

Comment 2

[Loic]

Try with a fresh profile: https://support.mozilla.org/en-US/kb/profile-manager-create-and-remove-firefox-profiles

In addition, type about:memory?verbose in location bar and attach the memory log to this current bug report when you're seeing the memory leak.

Comment 3

[Bogdan Maris, Desktop QA]

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0

Tried to reproduce but with no success, didn`t saw significant memory leak in task manager 70MB to 90MB.

Comment 4

[Nicholas Nethercote [inactive]]

> Task Mgr shows a continuous memory leak

Some measurements would be helpful.  Which numbers in the task manager were you looking at?  How fast were they changing?

And as per comment 2, the contents of about:memory?verbose after the leak has been active for a while (i.e. the memory consumption has gone up significantly) will be helpful.

Comment 5

[jacques]

Hi gents. glad to see you take interest
some tests I did :
I have *not* reintialized FF as I do not want to loose my config

- Launch In SAFE MODE
- call of about:memory?verbose  --> Data in Tab 1 of Xcel sheet
- wait 10 s and update page as processing of this page may have used memory --> Data in Tab 2 of Xcel
- WAIT 5 MN
- update of page --> Data in Tab 3 of xcel sheet

I join sample screen shot of task mgr. As taken in different time tahn emory in FF, data cannot be compared but show increase

Comment 6

[jacques]

Attachment: memore measure in safe mode showing memory increase

the data I measured showing memory increase while in safe mode

Comment 7

[jacques]

Attachment: TaskMgr screenshot

TaskMgr screenshot. This is how I saw I had a pb...

Comment 8

[Loic]

Paste the log from about:memory?verbose in plain text (.txt), Excel files are not really easy to open for people who don't have Office. Thanks.

Comment 9

[jacques]

Attachment: 1st load in TXT mode

for those who do not have Xcel nor Open office

Comment 10

[jacques]

Attachment: after page refresh

Comment 11

[jacques]

Attachment: after 5 mn

Comment 12

[Nicholas Nethercote [inactive]]

Jacques, thanks for the about:memory measurements -- that's very helpful!


First things first.  Before vs. after:

> 81,579,628 B (100.0%) -- explicit
> +--29,756,006 B (36.47%) -- js-non-window

> 90,708,108       (100.0%) -- explicit   
> +--31,866,822    (35.13%) -- js-non-window      

So JS is ~2 MiB out of ~9.2 MiB increase.


Then we have this:

> ¦  ¦  ¦  +---1,188,248 B (01.46%) -- compartment([System Principal], resource://services-common/utils.js)

> ¦  ¦  ¦  +---1,271,560   (01.40%) -- compartment([System Principal], resource://services-common/utils.js)       

the difference isn't large, but the size of the compartment is -- in my current session the same compartment is only 177 KiB.


Here's part of the increase:

¦  ¦  ¦  +-----511,216 B (00.63%) -- compartment([System Principal], resource://gre/modules/XPIProvider.jsm)

¦  ¦  ¦  +---1,118,928   (01.23%) -- compartment([System Principal], resource://gre/modules/XPIProvider.jsm)    


I also see this:

¦  ¦  ¦  +-----156,120 B (00.19%) -- compartment([System Principal], resource://services-sync/resource.js)

¦  ¦  ¦  +-----301,256   (00.33%) -- compartment([System Principal], resource://services-sync/resource.js)      


But here's the lion's share -- 6.5 MiB out of the total 9.2 MiB increase.

> +--20,209,608 B (24.77%) -- storage
> ¦  +--18,628,424 B (22.83%) -- sqlite
> ¦  ¦  +---8,258,924 B (10.12%) -- other
> ¦  ¦  +---7,944,904 B (09.74%) -- places.sqlite
> ¦  ¦  ¦   +--7,702,512 B (09.44%) -- cache-used [4]
> ¦  ¦  ¦   +----176,544 B (00.22%) -- stmt-used [4]
> ¦  ¦  ¦   +-----65,848 B (00.08%) -- schema-used [4]

> +--26,778,176    (29.52%) -- storage    
> ¦  +--25,196,992         (27.78%) -- sqlite     
> ¦  ¦  +--11,395,868      (12.56%) -- other      
> ¦  ¦  +--11,376,528      (12.54%) -- places.sqlite      
> ¦  ¦  ¦  +--11,074,368   (12.21%) -- cache-used [4]     
> ¦  ¦  ¦  +-----236,288   (00.26%) -- stmt-used [4]      
> ¦  ¦  ¦  +------65,872   (00.07%) -- schema-used [4]


It sounds to me like this is just Firefox Sync doing its job... does Sync run in safe mode?

Comment 13

[Richard Newman [:rnewman]]

Sync runs in Safe Mode. Sync uses memory. It inits about ten seconds after launch, and will have another spike on first sync (five minutes being one such interval).

Comment 14

[Gregory Szorc [:gps]]

Sync will also touch a lot of subsystems and possibly cause significant portions of databases like Places to get paged in. An increase of 9MiB for Sync seems in the territory I'd expect if the profile is literally idling.

A modern SQLite interface layer (like SQLite.jsm - used by FHR only right now) does automatic statement management and has APIs to force memory freeing - something FHR makes use of to reduce its footprint after heavy database activity. I suspect moving existing SQLite-touching JS components to SQLite.jsm and using its memory-compassionate features could help reduce the memory usage of storage.

Comment 15

[Nicholas Nethercote [inactive]]

Sounds like there's nothing unexpected going on here, then.  Thanks.


> I suspect moving existing SQLite-touching JS components
> to SQLite.jsm and using its memory-compassionate features could help reduce
> the memory usage of storage.

So Sync is one such JS component -- can you file a bug on that?  Are there any other such components?

Comment 16

[Richard Newman [:rnewman]]

(In reply to Nicholas Nethercote [:njn] from comment #15)

> So Sync is one such JS component -- can you file a bug on that?  Are there
> any other such components?

Sync doesn't do any direct DB access. What it does do is talk to form history, places, password manager, prefs, the add-ons manager, Necko, and other systems that access databases (sqlite or otherwise).

Sync's access might well be the first, and cause the initialization of DB connections, file handles, or what have you. At the very least it might result in filling of caches. (I'm pretty sure Places caches a bunch of stuff.)

I'm not convinced that this is a bug — memory usage is traded against future speed all the time.

(And we already know that optimizing for minimal memory usage when there's free RAM is a bad tradeoff on modern machines; it's better to fill that page cache, cache statements, keep connections alive, etc.)

I wouldn't want to categorically assert that Places should drop all of its caches and close the DB when Sync's done, for example — that seems foolish when so much of the operation of the browser will touch it.


But in this case much of the Sync-related space that I can see in the report is just "and now Sync is initialized" -- it's got the user's keys in memory, it's computed the URLs to which it will talk, it's tracking the current server IDs, caching the current set of add-ons, it has set up logging, started some timers, etc.

The utils module is swollen, but then it's the source for all of the timer instances, amongst other things.

And of course there's per-compartment overhead for each of Sync's multiple JavaScript modules.

So I'd write this up as "yeah, Sync is initialized".


With regards to "file a bug": we're not taking on any non-critical Sync development work; that product is on ice, and there are no heads assigned to it.

I've mentioned the importance of early perf review and involvement to the PICL team, and I will bring it up again in my next chat with Ben Adida.

Comment 17

[jacques]

sorry to jump into your technical discussion.
I do understand all the reasons given around Sync. Indded, i have a heavy user as I use it for all my passwords. (except my bank ;-))

1 )
Wouldn't it mean that after some time the memory usage will stabilize ? Shall I try along a longer period (few hours)

2 ) BTW, memory increase was low in safe mode compared to what I ddi experieced in 'normal' navigation, but I'd expect syn,c to behave the same way... Is it the unique root cause ? (open question)

3 ) I'll check but I am not sure I saw the issue on my laptop which of course I use Sync on (to synchronize with my desktop, of course). If no feedback, it is the same...


thxs for your time & support and good luck

Comment 18

[Richard Newman [:rnewman]]

(In reply to jacques from comment #17)

> Wouldn't it mean that after some time the memory usage will stabilize ?
> Shall I try along a longer period (few hours)

It should, assuming the garbage collector runs. (That is, if you click "Minimize memory usage" in about:memory, Sync's memory usage should stay roughly constant assuming you don't catch it mid-sync.)

> 2 ) BTW, memory increase was low in safe mode compared to what I ddi
> experieced in 'normal' navigation, but I'd expect syn,c to behave the same
> way... Is it the unique root cause ? (open question)

Sync would behave the same way. There are other components that would not, I imagine.

> 3 ) I'll check but I am not sure I saw the issue on my laptop which of
> course I use Sync on (to synchronize with my desktop, of course). If no
> feedback, it is the same...

It could be that there's something behaving differently (e.g., you have a sync that's never finishing, so there's a bunch of mid-sync state, such as HTTP request objects hanging around).

Comment 19

[Nicholas Nethercote [inactive]]

I think we can close this.