974811 - Rotating between Firefox pages and refreshing causes Firefox to Crash runs out of Memory

Status: RESOLVED INCOMPLETE

(Firefox :: Untriaged, defect)

Description

[Nigel]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0 (Beta/Release)
Build ID: 20140212131424

Steps to reproduce:

Have a number of tabs open with graphical data thats changed on each reload, even marginally.

Use an addon (you can do this manually and see a change but automation helps) to rotate between tabs and reload them constantly 


Actual results:

After a few hours the memory used by Firefox increased from around 200MB to over 800MB and after a day it was over 1.2GB.

Carrying on to several days it reaches over 2GB and crashes Firefox


Expected results:

Firefox should have limited the number of back pages allowed on each page thats refreshed and not continue until all resources are exausted

Comment 1

[Nigel]

This was not happening around a year ago but has progressively become worse with newer releases.

I know it was happening with Firefox 25 and think it was earlier as well.

Comment 2

[Nigel]

Note we tend to have around 8-10 tabs open

Comment 3

[Nigel]

Attachment: Please find attached the verbose memory reports showing the memory at startup and after a day (then Firefox hung today

Please find attached the two verbose memory reports showing the memory at startup and after a day (then Firefox hung today

Comment 4

[Nigel]

Have put this bug number into the Crash report when this occurred last time

Comment 5

[Gabriele Svelto [:gsvelto]]

The memory report contains 2+ GiB of raw image data, that sounds like we might be leaking those images somehow. Can you provide a link to an actual page - or even better a test page - exhibiting this behavior?

Comment 6

[Nigel]

The pages being used come from a tool called Zabbix.  Unfortunately where we use it is behind a DMZ.

I have on an error report that the browser sends, referenced this defect.  The images are a set of graphical data showing how systems are performing.  Typically we have 8 pages with up to 9 graphs on each

Comment 7

[Nigel]

I should of course say that we reload each page around once every 3 minutes I would guess.  Typical reload times 2-4 seconds

Comment 8

[Nigel]

Screen shots have images - but with more like this http://www.zabbix.com/img/screenshots/2.2/monitoring/Custom_screens1.png

Comment 9

[Nigel]

There are crashes at least once a day from 2 of our 4 browsers. All with the same settings and each opening similar number of screens.

No idea when Mozilla will get this fixed!  Not a good sign for the stability of the platform

Comment 10

[Gabriele Svelto [:gsvelto]]

I've found a workload similar to the one you have here but couldn't reproduce the issue on the latest nightly. Could you try this out using the latest nightly build? From what I could gather it's possible that this bug has already been fixed as part of some other issue. Alternatively it might be that I'm not reproducing exactly the scenario you're having.

Comment 11

[Gabriele Svelto [:gsvelto]]

I'm testing this with Zabbix's demo page and I think I was able to spot the problem so I wanted to ask if you could confirm that you're experiencing this problem specifically using Zabbix. The issue might be specific to how Zabbix works and I'd like to confirm that.

Comment 12

[Gabriele Svelto [:gsvelto]]

I'm confirming this bug as I seem to be able to reproduce the issue using Zabbix's demo server, the STR is the following:

1. Point the current nightly at this page
   https://www.zabbix.org/zabbix/charts.php?sid=76ab4ef3523cd403&form_refresh=1&fullscreen=0&groupid=0&hostid=0&graphid=73

2. Make sure the scrollbar at the middle of the screen is dragged all the way to the right so the graph refreshes itself automatically

3. Wait patiently (say one hour)

4. In about:memory check the 'explicit > images > content > used > raw', it should have grown from 0 when opening the page to a few MiB, invoking a memory minimization doesn't seem to free the associated memory

As I didn't have much time to investigate this I haven't yet figured out if we're leaking the images ourselves or if there's a bug in Zabbix's JS code that's leaking them accidentally.

BTW this was reported for Firefox 25 on Windows 7 but I could reproduce it with nightly on Linux.

Comment 13

[Nigel]

Confirm that we are using Zabbix and have only seen it with there software.

We have 4 instances running with a variety of graphs on them.  Two regularly fail and the remainder appear not to have an issue.

We have nightly on one.  I can on Monday (providing they have not crashed before) supply the memory stats for each.  For all recent failures we have had firefox report the issue and have given this ticket on each report

Comment 14

[Nigel]

Attachment: cc-edges.1028.1397214163.log.7z

Please find attached the verbose memory dump with nightly of a couple of days ago.  Based upon performance - it looks like the issue is continuing with Nightly and the crash is imminent

Comment 15

[Nigel]

I asked the team at Zabbix for their views.

The comments were

I think a memory leak in Firefox. I opened 15 tabs (news, wiki, youtube, etc), but didn't open zabbix frontend. After ~6 hours Firefox uses 599M instead 534M (please look attached screenshot).

Mozilla Firefox 28.0 for Ubuntu.

Comment 16

[Nigel]

Attachment: memory.png

Memory after a few hours without Zabbix

Comment 17

[Nicholas Nethercote [inactive]]

> I think a memory leak in Firefox. I opened 15 tabs (news, wiki, youtube,
> etc), but didn't open zabbix frontend. After ~6 hours Firefox uses 599M
> instead 534M (please look attached screenshot).

That tells us almost nothing. That increase could be entirely legitimate. It depends entirely on what those other 15 tabs were doing.

Comment 18

[Nicholas Nethercote [inactive]]

> I'm confirming this bug as I seem to be able to reproduce the issue using
> Zabbix's demo server, the STR is the following:

Does this reproduce in other browsers? That would indicate whether it's Firefox's fault or Zabbix's.

Comment 19

[Gabriele Svelto [:gsvelto]]

(In reply to Nicholas Nethercote [:njn] from comment #18)
> Does this reproduce in other browsers? That would indicate whether it's
> Firefox's fault or Zabbix's.

I gave this a spin in Chrome and it seems that copies of the main chart image (chart.php) keep piling up there too. The leak must be caused by Zabbix's code failing to drop references to the old images when refreshing the view. Closing this as it's definitely a Zabbix bug.

Comment 20

[Nigel]

We used to use Chrome around 18 months ago with an older version of Zabbix, and it crashed, however Firefox did not crash.  Over the last few months with a newer version of Zabbix and newer version of Firefox its started to crash.  One of the things to note is that what'd displayed has only recently been established and stabilised.

Thinking the issue through as I write this, we have up to 14 tabs set, and each tab has between 1 and 6 items on it (an item can be a list or graph), and we rotate the tabs every 15-20 seconds.  Zabbix has a reload every 30 seconds set.  Could the number of tabs open with the refresh frequencies mean that Firefox has no defined time to remove the old data as housekeeping is a background activity?  If thats the case then maybe at a certain point where resources are becoming exhausted the cleanup routine should have priority. Note as resources get used up (memory and disk available for caching) the performance slows and there is a catch 22 situation going on.

Comment 21

[Gabriele Svelto [:gsvelto]]

(In reply to Nigel from comment #20)
> Thinking the issue through as I write this, we have up to 14 tabs set, and
> each tab has between 1 and 6 items on it (an item can be a list or graph),
> and we rotate the tabs every 15-20 seconds.  Zabbix has a reload every 30
> seconds set.  Could the number of tabs open with the refresh frequencies
> mean that Firefox has no defined time to remove the old data as housekeeping
> is a background activity?

No, the issue here is that Zabbix's code is holding to all of the old chart images so Firefox can't free them. After reproducing your scenario I could find multiple copies of the chart image (the filename is 'chart.php') being held in both Firefox and Chrome. You should file a bug against Zabbix as this problem resides somewhere in their code.

Comment 22

[krists.krigers]

Attachment: spamtest.html

Comment 23

[krists.krigers]

Attachment: memory-report.json-29.0.1.gz

Comment 24

[krists.krigers]

I was able to reproduce memory leak without any client side code specific to zabbix - see attachement spamtest.phtml.

When this was run, after some time (~20 minutes) memory used for images in about:memory grew to 48 MB. Pausing spamtest and waiting/forcing GC/CC did not make Firefox to release this memory. 

I don't think anything in spamtest.phtml is holding on to previous versions of loaded images (if I happen to be wrong about this please give some hints!). If spamtest is run on Chrome, GC does frees the memory.

I have also uploaded memory report from about:memory which was made after spamtest was paused and GC/CC buttons pushed - see attachment memory-report.json-29.0.1.gz.

Used Firefox 24.4.0, Firefox 29.0.1. Memory report is from 29.0.1.

Comment 25

[Gabriele Svelto [:gsvelto]]

Thanks for posting a reduced test, I'll give this a spin tomorrow and see if I can reproduce the issue.

Comment 26

[Gabriele Svelto [:gsvelto]]

I did another couple of tests on this bug, the first one using a nightly build from ~2 weeks ago and then again with the current nightly as well as stable chrome. Note that I've modified the test to load the image from a local file and sped up the timer; the results I got were the same. This is what I got:

- old nightly: raw versions of the image kept piling up, pausing the timer did not free them nor invoking a minimization

- chrome: raw versions of the image keep piling up but only until the timer is paused, whenever the timer is paused the images are dropped

- today's nightly: only a certain amount of raw images pile up while the timer runs, after a while unused ones are flushed and memory consumption shrinks down, pausing the timer doesn't seem to affect this behavior, the unused images are regularly dropped anyway

So, I was about to reopen this but it seems that we've fixed it in the meantime. Can you confirm that the problem went away on your side too using the latest nightly?

Comment 27

[Nigel]

Downloaded nightly and based upon 6 hours of testing the main issue appears to have been resolved, the loss so far is around 8MB compared to around 40MB+ per hour.  Will leave for 24 hours to get a better view on the changes.

Comment 28

[Nigel]

I loaded the nightly as mentioned above and its been stable for several days.  Lets hope that the fix that has done this will maintain its way through to release.

I think that the status - Resolved Invalid is wrong - If anything its Resolved Duplicate - but I do not know which fix resolved it.

Thanks

Comment 29

[Gabriele Svelto [:gsvelto]]

(In reply to Nigel from comment #28)
> I think that the status - Resolved Invalid is wrong - If anything its
> Resolved Duplicate - but I do not know which fix resolved it.

Agreed, if I get the time I'll try to do a bisection around this issue and find which bug fixed this so we can properly dup it.

Comment 30

[Nigel]

This issue has reoccurred (and is in a worse state) between Firefox 33.1 and Firefox 34 releases

Comment 31

[Nigel]

Attachment: memory.zip

The attached file shows the issue without Zabbix and a 50Mb leak 

Note Adblock Plus Addon is installed on this machine (file) however on another PC we have had memory go from 500MB to over 3GB in 48 hours since Firefox 34 was released

Comment 32

[Nigel]

significantly worse with Firefox 34 and Firefox 35.  Issue appears to have gone away with the ESR release.  Looks like a regression occurred during a release

Comment 33

[Nickolay_Ponomarev]

Nigel, it would be best if you opened a new bug (if this wasn't fixed *again* during the 6-month pause). Reading through this bug is quite a barrier to entry.

Two things would be very much helpful in advancing this forward:
- a testcase and the clear steps to reproduce. spamtest.html from comment 22 doesn't actually load any images (it gets 404s from bugzilla), so I'm not sure if it can be used as is or requires any modifications. I'm also not sure if you're referring to the testcase or the original zabbix issue when you say it got significantly worse in subsequent releases (comment 30 and comment 32).
- a specific regression window, obtained via http://mozilla.github.io/mozregression/

Comment 34

[Vlad Bacia-Mociran [:VladB]]

Due to the fact that the reporter didn't provided the requested information until now, I will mark this issue as RESOLVED - INCOMPLETE.

Please feel free to reopen it or file a new bug if you are still experiencing this issue.