1476032 - 2GB structured-clone-holder memory usage with 9 extensions enabled

Status: RESOLVED FIXED

(WebExtensions :: Storage, defect, P1)

Description

[Sam Bull]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0
Build ID: 20180704194937

Steps to reproduce:

Since upgrading to Firefox Quantum I have frequently encountered the case where Firefox uses up several GBs of RAM, until my system runs out and then freezes up. At that point it typically takes about 10-30 mins for my system to recover, generally by killing the worker process, while I am unable to use my machine at all. This happens to me atleast a couple of times a week, and is still happening on FF 61.

Running on Ubuntu GNOME 16.04 without swap space.


Actual results:

It seems to me that each time I load a page, Firefox uses up more memory. A few seconds (5-30) after loading the page, Firefox cleans up the old memory, and memory usage goes back to normal.

The problem occurs when you flick through pages quickly, for example looking at a load of items on a shopping site where you click through to a product, then go back to the previous page, click to the next product etc.
  In this circumstance, if you are flicking through too quickly, then that idle delay is never reached, meaning each page load increases the RAM usage without freeing anything up until the system freezes.

I also encountered this at https://www.ssllabs.com/ssltest/index.html where the page refreshes every few seconds to display progress. Watching the memory usage, it appeared that half the time the refresh was happening just fast enough to stop the memory getting cleared (but not enough to be 100% reproducible).


Expected results:

Firefox should be smarter about it's memory management, perhaps ensuring that page loads of the same site (or even the same page) can share memory such that loading the same page again and again doesn't increase memory usage, and/or ensuring that memory is reclaimed a little more aggressively when it is starting to spiral out of control during frequent page loads.

Comment 1

[Abe - QA (:Abe_LV)]

Hi Sam,

I tested this on Ubuntu 16.04 with latest Nightly, but could not manage to reproduce.  Memory usage looks correct to me.

Could you please provide the memory report? Thanks

Here are the steps to generate a memory report:

1. Wait until memory usage is high enough
2. Open a new tab and type "about:memory" into the address bar and hit enter
3. Click on the "Measure and save..." button
4. Attach the file to this bug.

Detailed information is available here: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/about:memory


My test environment:
Version 	63.0a1
Build ID 	20180718220144
Update Channel 	nightly
User Agent 	Mozilla/5.0 (X11; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0

Comment 2

[Sam Bull]

Attachment: Large memory usage

Comment 3

[Sam Bull]

Attachment: After cleanup

Comment 4

[Sam Bull]

OK, so I managed to reproduce it by just clicking some links and going back several times. I switched to the memory tab and saved the report, I then saved another memory report 30 seconds later after Firefox had cleaned up the memory.

The clearly suspicious thing here is in the Web Content (5095) process, which is using 2GB of memory in the first log, mostly on explicit/dom/structured-clone-holder. In the second log after cleaning up, it has dropped to 150MB of memory.

Comment 5

[Abe - QA (:Abe_LV)]

erahm, comment or suggestion on this report? Thanks

Comment 6

[Eric Rahm [:erahm]]

Yes, the structured clone holder number is pretty high in the content process.

> 2,197.38 MB (100.0%) -- explicit
> ├──1,962.25 MB (89.30%) -- dom
> │  ├──1,962.20 MB (89.30%) ── structured-clone-holder [2415]

Can you test in safe mode [1]? My guess is this is extension related, if that helps can you try selectively disabling extensions to figure out which one is causing the issue?

[1] https://support.mozilla.org/kb/troubleshoot-firefox-issues-using-safe-mode

Comment 7

[Kris Maglione [:kmag]]

Hm. Well, the numbers after CC look pretty reasonable, so I don't think there's a leak. I'm surprised that it ever gets anywhere near as high as 2GB across all content processes, but my guess is that some extension is just doing *tons* of messaging.

It would be nice to find out which one.

I'm not sure there's much we can do about it, other than forcing the CC to run more often when the numbers get that high, though.

Comment 8

[Sam Bull]

Fortunately, I'm not using many extensions, so it was not difficult to find.

Seems the culprit is this extension: https://addons.mozilla.org/en-GB/firefox/addon/uautopagerize/

Annoyingly, there's no download link on the page and I can't figure out where the code is installed locally, so I'm unable to have a peek at the source code and see what it's doing.

Comment 9

[Kris Maglione [:kmag]]

(In reply to Sam Bull from comment #8)
> Fortunately, I'm not using many extensions, so it was not difficult to find.
> 
> Seems the culprit is this extension:
> https://addons.mozilla.org/en-GB/firefox/addon/uautopagerize/

Um. Yeah, so basically this add-on is crazy.

It stores about a MB of JSON data in its local storage, and loads that data for every page it's loaded into. That's what creates the StructuredCloneHolders, since we use them to transfer storage data to child processes.

It then iterates over a top-level list and checks the URL against each of 60-some regular expressions, and if it matches one, tries matching the URL against dozens of other URL patterns beneath it... All of which is crazy, though probably not related to the memory issues. It's just awful for performance.

I'm not sure how big that MB of JSON data winds up being when it's structured clone serialized, or how we wind up with enough of them to add up to 2GB, though.

In any case, those StructuredCloneHolder objects should go away with the indexedDB migration in bug 1406181. However, the content script will still keep an entire deserialized copy alive for every page it's loaded into, regardless of whether it's active on that page.

Comment 13

[Eric Rahm [:erahm]]

P1 for now, we're seeing quite a few dup's.

Comment 14

[User Dderss]

Can somebody please check profile recording attachment to #bug 1488051 (it is marked as duplicate).

Comment 15

[zheltonozhskiy]

Attachment: large memory usage

Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0
Ubuntu 18.04.1 LTS

Comment 16

[zheltonozhskiy]

Attachment: after cleanup

Comment 17

[zheltonozhskiy]

Attachment: a couple of minutes after cleanup

Comment 18

[Kris Maglione [:kmag]]

I'm going to try to work around this by neutering single-use StructuredCloneHolder objects, and dropping their storage, when we deserialize them. The objects themselves will still stick around until the GC cleans them up, but they should use much less memory.

Comment 19

[Kris Maglione [:kmag]]

Attachment: Bug 1476032: Neuter StructuredCloneHolder objects after deserializing. r=aswan

Comment 20

[Kris Maglione [:kmag]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/005a8fce14e71875a1e784d7adf3ec73fe5e694b
Bug 1476032: Neuter StructuredCloneHolder objects after deserializing. r=aswan

Comment 21

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/005a8fce14e7

Comment 23

[Victor Carciu]

I tested this on Ubuntu 16.04 x64 with Latest FF Nightly and I was not able to reproduce the issue. I will attach the memory usage for my test and https://addons.mozilla.org/en-GB/firefox/addon/uautopagerize/ addon installed.

Please let me know if the fix solved the problem for you.

Comment 24

[Victor Carciu]

Attachment: Post fix memory usage report

Comment 25

[Dominik]

This fix possibly causes a regression: bug 1552112