Some various PDF files are causing massive memory leak/usage

VERIFIED FIXED in Firefox 58

Status

()

Firefox
PDF Viewer
P1
major
VERIFIED FIXED
a month ago
16 days ago

People

(Reporter: Virtual, Unassigned)

Tracking

(7 keywords)

58 Branch
Firefox 58
x86_64
Windows 7
csectype-oom, dogfood, footprint, mlk, nightly-community, regression, topmlk
Points:
---

Firefox Tracking Flags

(firefox-esr52 unaffected, firefox56 unaffected, firefox57 unaffected, firefox58 verified)

Details

(Whiteboard: [MemShrink:P1] [fixed by patch from bug #1413218], URL)

Attachments

(1 attachment)

Created attachment 8920542 [details]
2013_05_09_Szulc.pdf

STR:
1. Open this hypelink - http://kdg.ue.poznan.pl/att/Wyklady_otwarte/2013_05_09_Szulc.pdf
or "2013_05_09_Szulc.pdf" file in Attachment
and enjoy massive memory leak
I forgot to add that it's always reproducible
when user change in about:preferences#general these settings:
> Use recommended performance settings = Disabled
> Content process limit = 1 or 2 (as any other kills performance in 2 threaded CPU, which still nearly 70% of Firefox users are still using it [ https://hardware.metrics.mozilla.com/ ])
or change in about:config this preference:
> dom.ipc.processCount = 1 or 2 (in Mozilla Firefox Nightly)
or
> dom.ipc.processCount.web = 1 or 2 (in Mozilla Firefox stable and Beta)


some regression hunt:
Firefox 45.9.0 = PDF file opens instantly
Firefox 52.4.1 = PDF file opens instantly
Firefox 56.0.1 = no memory leak, but very high CPU usage for some amount of time before showing PDF file
Firefox 57.0 Beta 3 = no memory leak, but very high CPU usage for some amount of time before showing PDF file
Firefox 58.0 Alpha 1 Pre = massive memory leak/usage and massive CPU usage, system will hang due to OOM, if user will not kill Firefox processes fast enough


[Tracking Requested - why for this release]: Regression
Has STR: --- → yes
status-firefox56: ? → unaffected
status-firefox57: ? → unaffected
status-firefox-esr52: ? → unaffected
tracking-firefox58: --- → ?
Keywords: regression, regressionwindow-wanted
More to add, that when file is cached on drive, it's not reproducible, so always clean Cache, before doing STR.
Group: firefox-core-security
Duplicate of this bug: 1411113
Duplicate of this bug: 1408812
Duplicate of this bug: 1411377
Hello Brendan: Can someone from your team take a look at this bug and triage appropriately? Thanks.
Flags: needinfo?(bdahl)
So far I'm unable to reproduce on Windows using Firefox 58. Yury, could you give it a try?
Flags: needinfo?(bdahl) → needinfo?(ydelendik)
I'm unable to reproduce on Windows 10 using Nighly.

Virtual, can you reproduce the issue on Windows 10? Can you refine STR so it will include changes in the preferences as well, assuming the tester has default profile? I tried to follow comment 1 and comment 2 and still cannot reproduce.
Flags: needinfo?(ydelendik) → needinfo?(Virtual)
Also, can some type of antivirus will cause this behavior?
(In reply to Yury Delendik (:yury) from comment #8)
> Virtual, can you reproduce the issue on Windows 10?
I'm using Windows 7,
for time being I don't have access to Windows 10 to test it also there,
but looking on duplicates (if they're really same issues),
it's should be also reproducible on Windows 10 and Linux.


(In reply to Yury Delendik (:yury) from comment #8)
> Can you refine STR so it will include changes in the preferences as well,
> assuming the tester has default profile?
Sure.

STR:
1. Create clean new fresh profile without any addons (extensions, plugins, themes, etc.)
2. Change in about:preferences#general these settings:
> Use recommended performance settings = Disabled
> Content process limit = 1 or 2 (as any other value kills performance in 2 threaded CPU, which still nearly 70% of Firefox users are still using it [ https://hardware.metrics.mozilla.com/ ]) or change in about:config this preference:
> dom.ipc.processCount = 1 or 2 (in Mozilla Firefox Nightly)
or
> dom.ipc.processCount.web = 1 or 2 (in Mozilla Firefox stable and Beta)
3. Open this hypelink - http://kdg.ue.poznan.pl/att/Wyklady_otwarte/2013_05_09_Szulc.pdf
or "2013_05_09_Szulc.pdf" file in Attachment


(In reply to Yury Delendik (:yury) from comment #9)
> Also, can some type of antivirus will cause this behavior?
In my case, I'm using Avira Antivirus 15.0.32.12 (Free),
but I can reproduce issue, even with antivirus disabled.
Flags: needinfo?(Virtual)
Keywords: nightly-community
Duplicate of this bug: 1412794
Keywords: csectype-oom
Virtual, it maybe a mistake to merge issues based just on "excessive memory use" attribute, e.g. in the past, we had a totally unrelated issues fixed that exhibited the same symptoms.

Comment 13

25 days ago
with 58a1 I can see memory consumption slowly rising when scrolling this PDF up&down, I stopped when the content-process reached 1.9GB of RAM (on Linux, basic composition).

I am however not sure this is related to the bug 1412794, as memory ocnsumption directly after opening the PDF seems sane (~650mb for the active content process), whereas in 1412794 the content process OOMs during initial loading of the PDF.

Comment 14

25 days ago
the massive memory consumption when scrolling this PDF seems to come from:

Web Content (pid 7421)
Explicit Allocations

1,290.30 MB (100.0%) -- explicit
├────891.33 MB (69.08%) -- images/content/raster
│    ├──720.59 MB (55.85%) ++ used
│    └──170.74 MB (13.23%) ++ unused
├────144.71 MB (11.22%) -- workers
│    ├──124.00 MB (09.61%) -- workers(pdf.js)/worker(../build/pdf.worker.js, 0x7f71b481b000)
│    │  ├──114.60 MB (08.88%) -- zone(0x7f71b2efb000)
│    │  │  ├───96.83 MB (07.50%) -- compartment(web-worker)
(In reply to Yury Delendik (:yury) from comment #12)
> Virtual, it maybe a mistake to merge issues based just on "excessive memory
> use" attribute, e.g. in the past, we had a totally unrelated issues fixed
> that exhibited the same symptoms.

Could be, but I suspect that it's probably the same issue.
I'm basing on this, that it's reported as reproducible on Mozilla Firefox Nightly 58.0a1 and if user can't reproduce it on Mozilla Firefox Beta 57.


(In reply to Clemens Eisserer from comment #13)
> with 58a1 I can see memory consumption slowly rising when scrolling this PDF
> up&down, I stopped when the content-process reached 1.9GB of RAM (on Linux,
> basic composition).
> 
> I am however not sure this is related to the bug 1412794, as memory
> ocnsumption directly after opening the PDF seems sane (~650mb for the active
> content process), whereas in 1412794 the content process OOMs during initial
> loading of the PDF.
In my case, on Windows 7, I'm getting nearly immediately all available RAM eaten by Firefox with these PDF files from this bug and duped one, so it could be just system thing. I will ask more question in duped bug.
(In reply to Virtual_ManPL [:Virtual] - (please needinfo? me - so I will see your comment/reply/question/etc.) from comment #15)
> In my case, on Windows 7, I'm getting nearly immediately all available RAM
> eaten by Firefox with these PDF files from this bug and duped one, so it
> could be just system thing. I will ask more question in duped bug.


Since you can reproduce it locally, it will be useful to find the exact regression window. Virtual, can you bisect (manually, if mozregression does not work) to find the exact m-c/incoming build? IMHO it will be a better time spend vs soliciting the feedback from similar bugs.
Flags: needinfo?(Virtual)
I was able to reproduce (sort of) on a Windows 7 machine. I see memory spike to ~1.6GB then settle back down to 214MB after a few seconds.

Updated

25 days ago
Severity: blocker → normal
Priority: -- → P1
@Virtual_ManPL Are you able to run mozregression (http://mozilla.github.io/mozregression/install.html)? I was able to reproduce for awhile, but it is not longer happening, so I can't come up with a regression range.
@ Yury Delendik (:yury) & Brendan Dahl [:bdahl] - Surely, I will do that when I be back from my rest days, so fastest in next week.
Flags: needinfo?(Virtual)
I was able to consistently reproduce by downloading the PDF http://de.free.aero/contents/light2017de.pdf and opening it from a local apache host. 

I ran mozregression on windows and linux and they both pointed to bug 1341093 as the regression. On Linux I checked about:memory's "resident-peak" after loading the PDF and on windows I used process explorer's "Peak Working Set" to monitor memory usage. Before this change, on windows memory is usually around 300MB, but after it balloons to ~980MB while loading the above PDF. On linux I see it go as high as 6GB.


@jono, can you look into this regression?
Blocks: 1341093
Flags: needinfo?(jcoppeard)
Whiteboard: [MemShrink]
(In reply to Brendan Dahl [:bdahl] from comment #20)
Interesting, that change should have had the opposite effect.  I'll look into it.
(In reply to Brendan Dahl [:bdahl] from comment #20)
Looking at a build of current inbound, I can see a high peak memory use of 1.5GB on linux when loading light2017de.pdf but I see memory use drop almost immediately.  I can't reproduce a leak or runaway high usage.  However I guess this is still a problem if you only have a limited amount of RAM.

From about:memory most of the memory in the worker process is ArrayBuffer elements (~900MB worth).

This is probably related to the malloc accounting changes in bug 1405274 that dynamically increase the threshold for triggering GCs due to malloc allocations.  I'll investigate making this less agressive.
Flags: needinfo?(jcoppeard)

Updated

24 days ago
Depends on: 1413218

Updated

24 days ago
No longer blocks: 1341093
Whiteboard: [MemShrink] → [MemShrink:P1]
I'm seeing weird activity when closing a page that leaked memory - I navigated away from https://www.scientificamerican.com/article/conservative-hunters-and-fishers-may-help-determine-the-fate-of-national-monuments1/ (to https://www.scientificamerican.com/, and then to another page on the site).  It had been sitting there for an hour or two, and had leaked to the point where the content process was ~3.8GB.  Navigating caused it to start chewing more memory, slowly at first and then quickly.  Over 37 seconds, it used an additional 1.1GB of memory(!!)  

It then dropped back to 3.8GB, and then 10 sec later dropped 3.1, then 10 sec later 2.9, then 20 sec later 2.8 -- all with no interaction.  I imagine the later drops are GC or CC running.  But using that much additional memory is really concerning
Does this still reproduce in the latest nightly now that bug 1413218 has landed?
Flags: needinfo?(Virtual)
I'll note that my issues in comment 23 were with Beta 57
(In reply to Jon Coppeard (:jonco) from comment #24)
> Does this still reproduce in the latest nightly now that bug 1413218 has
> landed?

Brendan, would you be able to test as well?
Flags: needinfo?(bdahl)
I'm confirming, same as Brendan Dahl [:bdahl], that this issue was caused by patch from bug #1341093, as mozregression-gui pointed.
I'm also confirming, that this bug is fixed by patch from bug #1413218.
So I'm marking this bug as RESOLVED.
Thank you very much! \o/
Severity: normal → major
Status: NEW → RESOLVED
Has Regression Range: --- → yes
Last Resolved: 16 days ago
status-firefox58: affected → fixed
tracking-firefox58: ? → ---
Flags: needinfo?(Virtual)
Keywords: regressionwindow-wanted
Resolution: --- → FIXED
Whiteboard: [MemShrink:P1] → [MemShrink:P1] [fixed by patch from bug #1413218]
Status: RESOLVED → VERIFIED
status-firefox58: fixed → verified
Target Milestone: --- → Firefox 58
Thanks Virtual.
Flags: needinfo?(bdahl)
For posterity, I see the fixed bug reduce the memory usage but it still was getting pretty high. I also noticed the plain-web version of PDF.js loaded this PDF much faster, so I looked further into it. In summary, when a large PDF downloads very fast, PDF.js was doing something pretty horrible in the Firefox extension code where we were expanding and copying a typed array for each chunk of the PDF that was streamed in. In the case of the above 27MB PDF, PDF.js was expanding and copying the buffer nearly 200 times until it grew to the 27MB size (allocating roughly 2.7GB in the process!).

For more info see the PDF.js pull request https://github.com/mozilla/pdf.js/pull/9110
(In reply to Virtual_ManPL [:Virtual] - (please needinfo? me - so I will see your comment/reply/question/etc.) from comment #27)
Great to hear that.  Thanks for reporting this.
You need to log in before you can comment on or make changes to this bug.