998753 - Firefox uses too much memory

Status: RESOLVED INCOMPLETE

(Core :: General, defect)

Description

[Craig Arno]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:28.0) Gecko/20100101 Firefox/28.0 (Beta/Release)
Build ID: 2014031500

Steps to reproduce:

Firefox 28.0 Windows and Linux

I'm using a web based genealogy program which has me performing frequent page loads from numerous "new" pages used for data entry and modification.  What is happening is Firefox quickly starts to use almost 3GB of RAM as reported in about:memory after a few hours of use:
1,390.12 MB ── resident
1,334.34 MB ── resident-unique
2,750.83 MB ── vsize

Most of the memory seems to be in "Explicit Allocations" from all these page loads:
1,286.39 MB (100.0%) -- explicit


Actual results:

What is happening is Firefox self terminates when it runs out of memory to use.  I can't use Firefox on anything except a well stocked state of the art computer with about 16GB of RAM.  I noticed in my laptop with 3GB RAM running Vista in about 1/2 hour of editing I get a system error telling me memory is low and I need to close Firefox.  If I ignore the warning, Firefox goes "poof"  and is gone, presumably when system memory is completely exhausted.  I added another 1GB to this computer to max it out at 4GB.  Now I can edit pages for several hours before hitting the limits and experiencing the same issue.

I retried this same activity on a Linux workstation with 16GB of RAM.  I'm able to edit all day long, without issue.  Memory usage is high, only about 1.3g according to "top", and the application doesn't crash.  In Windows I'm seeing 2.5GB numbers right before Firefox crashes.

I am unable to use Firefox for any length of time with this application on any platform which doesn't have almost unlimited resources available to Firefox.


Expected results:

I would like Firefox to monitor available system memory and when things get tight, start flushing some of the page caches which appear to be using up large quantities of memory.  Slightly longer page loads is preferable to having Firefox crash.  Since server and client are on a local network, interaction is almost "instant", and I may be loading a new page every 2-3 seconds.

From my point of view Firefox caching is too aggressive resulting in quick consumption of all available system resources on machines with limited resources, like a laptop, or a slightly older computer.  Firefox needs a way to monitor and return some of these resources when resources become scarce.

Comment 1

[Wayne Mery (:wsmwk)]

What you really want is relief from high memory usage, which may be one or more bugs.  

But there is nothing we can do with this bug report without a URL for your application, and the output from about:memory (which you can attach to this bug report). And also an assurance that this also happens with Firefox in safe mode.

Comment 2

[Craig Arno]

Attachment: FF28 Linux memory usage about:memory

Ok, I agree, as a better problem statement, "I need relief from high memory usage".

My previous statements are based on keeping track of my own usage pattern while watching about:memory until Firefox self terminates.  This was occurring on -both- Windows Vista x64 and OpenSUSE Linux12.3 x64 OS's while using Firefox 28.0 on both platforms.  The Linux system stopped having problems when I added another 8GB of memory (16GB total).  My Windows based Laptop is maxed out at 4GB and can't be expanded further.  I run both Firefox and Thunderbird (Bug 998755), and both Mozilla applications are heavy memory consumers.  I'll leave it to developers to determine why?  Both systems use 256GB SSD's for fast disk performance (implying gargantuan swap spaces are also not feasible).

The URL I used when I made the report is https://arno.com/tng loading new database admin pages at about a rate of 1 every 2-3 seconds for hours of editing activity.  For me, this site is on a local gigabit network connection, meaning its response to my activity is almost instant.

If it's important, I can also try to recreate the about:memory map on the Windows machine.  It isn't used as often as the Linux workstation.

Comment 3

[Craig Arno]

I'd like to add, I don't mind FF using lots of memory to maintain performance when memory is available.  What I need is for Firefox/Mozilla applications to recognize when memory as a resource is becoming limited/constrained, and reduce it's own consumption of this resource when limits are reached.  This might also need to occur when I launch a CAD or Office application, which also uses/needs RAM.

Comment 4

[Craig Arno]

Attachment: FF29 about:memory 15GB

This is Firefox v29 after running for about a day with ½ day editing https://arno.com/tng, showing 15GB of memory usage. Unfortunately I didn't disable Add-Ons for this one.  After posting this I'll restart in SAFE mode with Add-Ons disabled.  The update from v28 to v29 turned off SAFE mode.

% ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
craig    19523 31.4  8.9 15509104 1467304 ?    Sl   May02 462:39 /usr/lib64/firefox/firefox

This is on a 6-core Phenom T1100 processor with 16GB RAM installed, 256GB SSD.

Comment 5

[Craig Arno]

Attachment: FF29Lmemory-report.json20140507.gz

Since my last posting, this is Firefox v29 running in SAFE mode with light browser usage (only light editing of a web based database [https://arno.com/tng]).  Like my Thunderbird report, I need to take this application out of SAFE mode.  So unless more information is needed, this is my last report.

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
craig    19523 13.6  4.9 2503252 805064 ?      Sl   May02 1004:34 /usr/lib64/firefox/firefox
craig    21322  1.9  0.7 682728 127556 ?       Sl   May03 112:17 /usr/lib64/firefox/plugin-container /usr/lib64/browser-plugins/libflashplayer.so -greomni /usr/lib64/firefox/omni.ja -appomni /usr/lib64/firefox/browser/omni.ja -appdir /usr/lib64/firefox/browser 19523 true plugin

It looks like v29 may be performing better with memory usage than v28.  How much, I can't be sure.  At 2.5GB, I still need better memory behavior (need additional relief).

Comment 6

[Craig Arno]

Hmm, when I quit Firefox v29 SAFEmode and all Firefox processes stopped, memory usage went from 6.8GB to 5.9GB.  This leads me to believe FF29Linux is using ~1GB memory.  Not great, but not terribly bad either.  Again, that was Safe-mode.

Comment 7

[Wayne Mery (:wsmwk)]

Firefox better now?

Comment 8

[Craig Arno]

Attachment: Firefox 34.0.5 MS-Windows memory usage

Current Firefox memory usage without being used hard under Windows Vista SP2 shows ~450MB usage in Task Manager after being run off/on for light web browsing for several hours.  Attached is a memory report from this activity.

I'm not sure what to say about Firefox 34.0.5 memory usage.  It does appear to be improved, however I'm not spending the whole day using FF to update web based genealogy information as I was when I submitted this report.  I maxed out my Laptop memory at 4GB to reduce my pain, and increased my desktop to 16GB for the same reason.

If you've done something to address this issue between FF28 and FF34, then I'd say there is improvement in the form of memory relief.  I'll report more when I have more.

Thunderbird also appears to be better behaved at this time.  As I understand it, FF and TB share the same core Mozilla code for HTML rendering and perhaps more.  My sense of it is this situation for both applications is improved.

Comment 9

[Wayne Mery (:wsmwk)]

Thanks for the update. Let's call this WFM. If problem returns please update the bug

Comment 10

[Craig Arno]

Attachment: Typical first clue about memory shortage (4GB Vista x64 laptop)

When I exited Firefox 35.0.1, Windows Task Manager reported almost 1GB freed.  I admit I had a lot of other applications running consuming ~2.97GB of memory.  This is a time when I wish Firefox would detect low memory and yield some of the memory it is using, even if performance will suffer.  i.e. I believe Firefox should not insist on locking up 25% of system memory for whatever it is doing, particularly if it doesn't have current application focus.  It seems reasonable to me to start by flushing/dumping/releasing caches which can be re-established when the current memory log jam is relieved/reduced.  This will prevent Firefox being targeted by the OS as the greatest memory hog [read message at bottom of dialog].

Comment 11

[Craig Arno]

Attachment: FF-Taskmgr-Memory.gif

I received the memory warning again today.  Here is an image of the Taskmgr Process display sorted by program memory usage.  This will be followed by a Firefox about:memory dump.

I believe this is an important issue for Mozilla users since many users are using older machines, and the Firefox "upgrade" strategy forces all Firefox users into a "one size fits all" type strategy/solution.  Many are happy with their older machines and will find an alternative to Firefox if Firefox is identified as a "problem" rather than dumping their older systems.  I believe this memory consumption issue needs to be addressed for older machines, like mine where 4GB isn't enough, and Firefox / Thunderbird consistently hit the top of the memory usage charts.  Chrome doesn't experience these memory usage issues.  Don't get me wrong, I -really- like Firefox+sync and Mozilla user privacy protection policies.  We need to work this issue.

Comment 12

[Craig Arno]

Attachment: FFmemory-report20150128b.json.gz

about:memory measure+save from the time of Vista x64 memory warning.

Comment 13

[Craig Arno]

Hopefully additional data will allow this issue to continue to be addressed,

Comment 14

[Craig Arno]

Attachment: FF + Chrome + IE process memory usage

Since I said something about Chrome, I thought I'd start Firefox, Chrome, and Internet Explorer, set all to be looking at the same collection of pages, then use Taskmgr to see how much memory can be attributed to each browser implementation.  Here is what I found using MS Windows Vista SP2 x64, with all MS Vista Updates to date applied:

Chrome	524860K
Firefox	438276K
IE	295260K

It looks like Firefox is better about memory usage than Chrome, but not as good as Internet Explorer.  I still believe Firefox could perform better with regard to memory when it needs to do so as this is still a problem in a 4GB machine running a typical set of productivity applications performing real work.

In more detail:

	Total		
Firefox	438276		
		431432	Firefox *32
		6844	Plugin Container *32
IE	295260		
		29384	iexplore.exe *32
		65952	iexplore.exe *32
		30632	iexplore.exe *32
		19324	iexplore.exe *32
		39060	iexplore.exe *32
		11612	iexplore.exe *32
		43144	iexplore.exe *32
		56152	iexplore.exe *32
Chrome	524860		
		29160	chrome.exe *32
		9376	chrome.exe *32
		28592	chrome.exe *32
		32380	chrome.exe *32
		49072	chrome.exe *32
		34180	chrome.exe *32
		33148	chrome.exe *32
		12352	chrome.exe *32
		17716	chrome.exe *32
		10488	chrome.exe *32
		34456	chrome.exe *32
		51988	chrome.exe *32
		31896	chrome.exe *32
		70176	chrome.exe *32
		51380	chrome.exe *32
		28500	chrome.exe *32

Any thoughts from a Mozilla Firefox developer?  Another memory dialog appeared while I was doing this experiment, and it still wanted to kill Firefox.  I suspect this is because FF *32 is the largest single running process, where Chrome and IE memory usage is distributed among a greater collection of smaller processes.

Comment 15

[Craig Arno]

While I was shutting everything down, I noticed I overlooked Flash Player Plugin is part of Firefox memory usage.  So Firefox corrected memory usage at 490384K is more closely associated with Chrome's 524860K above.

Firefox	490384		
		431432	Firefox *32
		6844	Plugin Container *32
		48100	Flash Player Plugin *32
		4008	Flash Player Plugin *32

I really hope there is a way to address ½GB of memory usage for machines using Mozilla's "latest and greatest" forced update.  A dynamic scalable solution so Firefox can decide at run-time, and hopefully real-time on memory usage; "If you've got it, use it, and if you don't, lose it", based on available memory, not physical memory.  I'm also thinking of my 2GB 1.2GHz quad-core Tablet computer, today's "jumbo phones", and latest MS Windows 10 "common API" across platform efforts while typing this.  BTW, I usually use Linux myself.  I'd hate to see Firefox relegated to only being used on the biggest, newest, and most capable high end desktop and laptop systems because of forced upgrades.  Hopefully these issues/goals make it onto Mozilla's project roadmaps, for all Mozilla products, or in this case Firefox.

Comment 16

[Craig Arno]

Attachment: Firefox 35.0 memory report

Today I notice 14GB of my 16GB RAM is in use, and I'm not doing anything particularly strenuous with my OpenSUSE 12.3 AMD x64 Linux Workstation.  Firefox 35.0 and Thunderbird 31.3.0 have been running for about 3 weeks.  Here is what I see from a system level for these two:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
craig    11698  8.2  8.3 3174248 1363560 ?     Sl   Jan21 2600:52 /usr/lib64/firefox/firefox
craig    11815  5.7  0.5 1299940 96452 ?       Sl   Jan21 1818:30 /usr/lib64/firefox/plugin-container /usr/lib64/browser-plugins/libflashplayer.so -greomni /usr/lib64/firefox/omni.ja -appomni /usr/lib64/firefox/browser/omni.ja -appdir /usr/lib64/firefox/browser 11698 true plugin
craig    29117  1.3 12.2 3212088 2007888 ?     Sl   Jan17 500:41 /usr/lib64/thunderbird/thunderbird-bin

From the 'ps' man page:
VSZ (virtual size in kilobytes), RSS (real memory size or resident set size in 1024 byte units)

Between Firefox and Thunderbird; 3.3GB of real memory with a virtual size of 6.3GB?  Firefox memory dump attached.

I know this behavior would have choked out my smaller 4GB laptop long ago.  After 3 weeks, I would think there is some memory these two applications could release to provide system memory relief.

Comment 17

[Wayne Mery (:wsmwk)]

Except for comment 2, what is described so far does not have enough detail to allow someone to reproduce your problem(s). To get to that place, I think you need to simplfy your environment

1. install flashblock addon

2. focus this bug report only on comment 2, https://arno.com/tng, no thunderbird or other firefox activity
- provide detailed numbered steps to reproduce on short term usage, i.e. memory increase over a very short period of time like a few minutes
- document how much memory increase per repetition/reload
- describe the guts of the application
- if it's only reproducible as admin, hopefully provide admin access to someone so they can reproduce the problem

3. if you have memory issues that go beyond comment 2, then it deserves a different bug report, which I suggest you again simplify your evironment by structuring as follows:
- in firefox (still with flashblock) avoid using sites that use flash and instead use chrome for usch sites
- don't use https://arno.com/tng in Firefox, use chrome
- what you have left then in Firefox is more generic activity, and perhaps you can determine whic

Comment 18

[Craig Arno]

I believe I see what you are trying to achieve and don't think we can reproduce this issue this easily.  I use TNH infrequently since it has nothing to do with my income. And the issue usually doesn't appear for hours of serving up dynamic pages at a rate of a new one every 2-3 seconds while editing data on the site.  This will be very hard to reproduce since the effects change with updated data.

I re-enabled swap in Windows and set it to use as much of the SSD as it needs.  The laptop has only been running Firefox and other apps for a few houts, but is already consuming 5.2GB of memory+swap.  However with swap on and unbound, no system memory warnings are issued.  This will allow me to keep working using my Windows laptop and Linux wotkstation as needed.  I notice my Linux machine is consuming 15.5GB memory so I may need to perform a reboot soon to flush everything and start with a clean slate.

I had hoped memory dumps would provide you with enough information to narrow down what code is generating most of the memory consumption.  When I looked at the "measure" output I was being warned about certain numbers being very large or negative in the memory listing.  Reloading the same memory save now does not provide these same warnings.  At the time I also tried all 3 "Memory Free" buttons with little effect.

I think we're both stuck.  So I'll return this ticket to "Resolved->WorksForMe" until something changes again.  Thank you for your help.

Comment 19

[Wayne Mery (:wsmwk)]

Thaks for the update. 

WORSKFORME for bugzilla does not mean "I can can live with the problem".  We will use  Incomplete, i.e. we do not know how to describe it such that others can reproduce it (And it is also neither working as designed, nor FIXED in code, nor INVALID as in being caused by something other than fireefox.)

Also if there is a problem, at some point we may need URLs.  But all your recent memory reports are anonymized.

(In reply to Craig Arno from comment #16)
> Created attachment 8563487 [details]
> Firefox 35.0 memory report
> 
> Today I notice 14GB of my 16GB RAM is in use, and I'm not doing anything
> particularly strenuous with my OpenSUSE 12.3 AMD x64 Linux Workstation. 
> Firefox 35.0 and Thunderbird 31.3.0 have been running for about 3 weeks. 
> Here is what I see from a system level for these two:
> 
> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> craig    11698  8.2  8.3 3174248 1363560 ?     Sl   Jan21 2600:52
> /usr/lib64/firefox/firefox
> craig    11815  5.7  0.5 1299940 96452 ?       Sl   Jan21 1818:30
> /usr/lib64/firefox/plugin-container
> /usr/lib64/browser-plugins/libflashplayer.so -greomni
> /usr/lib64/firefox/omni.ja -appomni /usr/lib64/firefox/browser/omni.ja
> -appdir /usr/lib64/firefox/browser 11698 true plugin
> craig    29117  1.3 12.2 3212088 2007888 ?     Sl   Jan17 500:41
> /usr/lib64/thunderbird/thunderbird-bin
> 
> From the 'ps' man page:
> VSZ (virtual size in kilobytes), RSS (real memory size or resident set size
> in 1024 byte units)
> 
> Between Firefox and Thunderbird; 3.3GB of real memory with a virtual size of
> 6.3GB?  Firefox memory dump attached.

We really only concern ourselves with Firefox in this report.  The THunderbird issue we can handle in bug 998755. (And note - Thunderbird's biggest memory usage areas are nothing like Firefox's, so we won't be trying to draw comparisons or similarities)

So it should be noted that Firefox+plugin process is about 4GB.  For a 64bit build, open 3 weeks, and a few dozen tabs open, then 4GB usage does not surprise me. Especially if it includes Facebook activity.  I'd frankly be more concerned about Thunderbird's usage, and the applications that are consuming the other 7GB.

> I know this behavior would have choked out my smaller 4GB laptop long ago. 
> After 3 weeks, I would think there is some memory these two applications
> could release to provide system memory relief.

If they don't release it after 1 day, they are not going to release it after 3 weeks :)

Comment 20

[Boris Zbarsky [:bzbarsky]]

VSZ includes things that aren't memory usage at all (memory mapped files, reserved address space ranges, etc).  On 64-bit, you are likely to get all sorts of non-memory address space mappings, since the address space is large and there's no reason not to use it...