1619604 - High blob memory usage and lags

Status: RESOLVED WORKSFORME

(Core :: DOM: File, defect, P3)

Description

[Julien Wajsberg [:julienw]]

Attachment: memory-report.json.gz

Basic information

After a few days of running, my Firefox instance takes a lot of memory and feels laggy.

Steps to Reproduce:
This is difficult to come up with steps to reproduce. What I know is that this is common that my Firefox instance runs during a few days. Because I use nightly, generally I restart Firefox just to update it.
For example the current build I use is 20200223214228, so likely last time I started this instance of Firefox is Feb 23 (we are Mar 3).
Not only this is kept running but also it survived a lot of suspend/resume cycles.

Here are the main websites I use:

• github
• slack
• riot
• gmail
• other google docs
• deezer
• lemonde
• duckduckgo
• twitter
• bugzilla

I also use some extensions, most notably ublock origin and nocoin.

I have 1600 tabs but obviously they're not all loaded.

It's common I start devtools on one of these tabs just to test stuff.

---

More information

I attached the anonymized result of about:memory.
What will strike you is the amount of heap-unclassified in the main process. I think it would be nice to understand where this could come from so that we could classify this better.
I also took a memory snapshot from the devtools' memory tool, but it's 80MB and probably contains PII so I didn't want to attach it here directly. But I can share it with some individuals if you want.

Profile URL: https://perfht.ml/2uPiAo5
I took this when typing some text in this text box in bugzilla.
I kept only the interesting tracks. Especially we see:

• the main process is very busy doing GC all the time, which is kind of expected.
• one content process shows the interaction with this page
• another content process shows a big GC and could free up some memory
• the webextension process shows a big GC too (I just disabled an addon right before, maybe that's the reason)

Basic systems configuration:

OS version: Debian stable

GPU model: Mesa DRI Intel(R) HD Graphics 520 (Skylake GT2)

Number of cores: 2 cores, 4 if we count HT.

Amount of memory (RAM): 16 GB

Comment 1

[Julien Wajsberg [:julienw]]

Attachment: memory-report-after-restart.json.gz

For reference, here is memory usage measurements right after restarting with latest nightly.

Comment 2

[Andrew Overholt [:overholt]]

Eric, what should we do with this report? While I'm interested in Julien's issue and any root cause we can find, I'm particularly interested in this bug getting out of Core: General or having a priority assigned to it :)

Comment 3

[Eric Rahm [:erahm]]

The parent process has a ton of heap-unclassified:

7,253.26 MB (100.0%) -- explicit
├──5,732.10 MB (79.03%) ── heap-unclassified

There's a pretty crazy amount of blob storage going on in the content process:

1,838.48 MB (100.0%) -- explicit
├──1,418.91 MB (77.18%) -- dom
│  ├──1,418.75 MB (77.17%) ── memory-file-data/string [11350]
....
11,350 (100.0%) -- file-blob-urls
└──11,350 (100.0%) ── owner(<anonymized>)/<anonymized-stack><anonymized-url> [11350]

My best guess is we're leaking/retaining blobs, so vaguely this is dom related, but devtools has been known to cause leaks like this so that could be related. On the other hand, it could just be a website going crazy with createObjectURL and not performing the corresponding revokeObjectURL

Comment 4

[Julien Wajsberg [:julienw]]

I 'm sorry I didn't keep an unanonymized dump, but I'll make sure to look at the file-blob-urls line next time it happens (which is quite frequent).

Comment 5

[Julien Wajsberg [:julienw]]

Ah the Blob thing might be Deezer; I actually know that they create blob URLs in Firefox without revoking, because it was done as an "experimental" hack to make it work in Firefox. Without it we had to still use Flash. I just asked them if they still do it.

Comment 6

[Andrew Overholt [:overholt]]

OK I'm going to mark this as P3 to get the triage nag bot off my back but let's re-evaluate and/or move to a better component if/when we learn more.

Comment 7

[Julien Wajsberg [:julienw]]

Just to make it clear, from comment 3, do we have 2 separate problems, or is that linked?

Comment 8

[Julien Wajsberg [:julienw]]

Attachment: screenshot of about:memory

Yeah, the blob thing is probably deezer.com.

Eric, does that leak memory in the content or the parent process?

Comment 9

[Julien Wajsberg [:julienw]]

Closing the tab freed the blob URLs.

Comment 10

[Eric Rahm [:erahm]]

(In reply to Julien Wajsberg [:julienw] from comment #8)

Created attachment 9134409 [details]
screenshot of about:memory

Yeah, the blob thing is probably deezer.com.

Eric, does that leak memory in the content or the parent process?

If I recall correctly the memory is shared, so both. Tearing down the tab that retains the blobs would allow them to be freed.

Comment 11

[Julien Wajsberg [:julienw]]

I had another look today, after listening to music on deezer.com a lot in the past days.
I definitely see all blob usage coming from there (about 2G in my current usage).
I don't see the heap-unclassified bits anymore, instead I see ~2G of memory-file-data/stream, which seems to relate to the blob objects. By chance could the way we report blobs here have changed? Otherwise this is a different issue.
Closing the deezer tab freed this memory, I'd say after about 1 min.

So I think the issue with Deezer is a site issue instead of a Firefox issue. Still our users could blame Firefox for the problem....

Comment 12

[Julien Wajsberg [:julienw]]

Attachment: memory-report-20200423-anonymized.json.gz

Here is a new one, this time without any blob from deezer, but still a huge line of unclassified heap.
This time I kept a non-anonymized version of it locally, in case I need to look at it.

Comment 13

[Andrew Overholt [:overholt]]

baku, does anything about the blobs here strike you as interesting?
mccr8, any info in the memory report that we can go on or do you think it's a site leak?

Comment 14

[Andrea Marchesini [:baku]]

(In reply to Julien Wajsberg [:julienw] from comment #7)

Just to make it clear, from comment 3, do we have 2 separate problems, or is that linked?

The 2 file-blob-urls and memory-file-data/string signatures are linked. memory-file-data/string is the memory signature for StringBlobImpl. These blobs can be created via: webSocket, PresentationConnection, DataChannel and new Blob([some value here]).
The signature file-blob-urls is from file-based Blobs, such as TemporaryBlobImpl (for more than 1mb XHR/fetch/webSocket blob responses) and FileBlobImpl (from a <input type=file> or other FilePicker operations).

Comment 15

[Andrew McCreight [:mccr8]]

Sorry I didn't look at this earlier. I don't really have anything to add, except to say that if you can reliably see the high heap-unclassified value, you can run Nightly with DMD=1 and possibly get some useful information about what the heap-unclassified is by running dmd.py on the DMD log you can get in about:memory.

Comment 16

[Andrew Sutherland [:asuth] (he/him)]

(In reply to Julien Wajsberg [:julienw] from comment #5)

Without it we had to still use Flash. I just asked them if they still do it.

Did they respond? There are mitigations possible, but other than spilling the Blobs to disk, they're visible to content and could result in breakage. (And a naive fix that makes deezer work right could break more complicate usages.)

So I think the issue with Deezer is a site issue instead of a Firefox issue. Still our users could blame Firefox for the problem....

Indeed. Blob URLs are a real footgun. This seems like a case where we'd want about:performance to surface the site's memory/disk usage and possibly display an infobar similar to how we'd tell the user if the site was making the page unresponsive through script usage.

Comment 17

[Julien Wajsberg [:julienw]]

hey, sorry for the slow reply, I was afk for a week :-)

(In reply to Andrew Sutherland [:asuth] (he/him) from comment #16)

(In reply to Julien Wajsberg [:julienw] from comment #5)

Without it we had to still use Flash. I just asked them if they still do it.

Did they respond? There are mitigations possible, but other than spilling the Blobs to disk, they're visible to content and could result in breakage. (And a naive fix that makes deezer work right could break more complicate usages.)

No I didn't get any answer from them. Maybe we should use the more official mailing list (I believe we have one with them).

Also to make it clear, the issue with Deezer was a red herring for my bigger "heap-unclassified" problem. The report in comment 12 has this without any blob from Deezer.
But now I think this is very similar to other bugs we have about the devtools so this might be it.

Comment 18

[Mason]

This bug seems to be related to bug 1591722
(I don't think I am able to edit the "See Also" section.)

Comment 19

[Jens Stutte [:jstutte]]

It seems clear that the Deezer part of this is just a dupe of bug 1591722.

(In reply to Andrew McCreight [:mccr8] from comment #15)

Sorry I didn't look at this earlier. I don't really have anything to add, except to say that if you can reliably see the high heap-unclassified value, you can run Nightly with DMD=1 and possibly get some useful information about what the heap-unclassified is by running dmd.py on the DMD log you can get in about:memory.

Julien, did you ever try Andrew's suggestion? Is this still happening (without Deezer) ?

Comment 20

[Julien Wajsberg [:julienw]]

I haven't seen this much recently, but I also now have a better computer with much more memory and much better CPU. Also a lot less tabs on this new computer now. My about:memory doesn't seem to show anything weird right now (this instance runs for ~3 days).

I'd be fine with closing the issue for now, I can reopen if I still see this later.