1128999 - possible memory issue in fx/plugin-container.exe

Status: RESOLVED INVALID

(Core Graveyard :: Plug-ins, defect)

Description

[Kamil Jozwiak [:kjozwiak]]

Attachment: beforeCrash.png

I had the plugin-container.exe crash on me several times over the weekend with a windows crash appearing rather than the fx crash. I only managed to get the fx crash reporter once but when I attempted to send the report, everything hung and it never finished uploading. I ended up leaving it in the "sending" state for about 2 hours but it never managed to send.

I added some screenshots to illustrate the state of fx before the crash occurred as well as the crashes:

* beforeCrash.png - Before the crash all the flash components on the page started blinking and appearing black making fx unusable. As soon as I resized fx, it crashed. (I'm guessing it crashed during the reflow of the page when I re-sized)

* firstCrash.png - This is the first crash that I received. As mentioned above, it never ended up sending the report.

* secondCrash.png & thirdCrash.png - Same crashes as before using the same STR

STR: (note: this takes a LONG time to reproduce but I ended up getting the crashes on three different occasion using the same STR)

* download the latest nightly
* opened the following websites:
** www.premierleague.com/en-gb.html
** https://www.youtube.com/watch?v=K7SYvgB6SZ4 (pressed play and paused about 2-3 minutes into the clip)
** http://www1.skysports.com/watch/video/sports/football/9683332/the-art-of-scouting-episode-1 (let it play and paused it)

Once I had the above websites opened, I left "www.premierleague.com/en-gb.html" opened on the visible tab and let the transitions on the website run over and over for about two hours. It basically changes the front page every few seconds. If you leave this running long enough, the RAM of fx and the plugin-plugin container slowly starts increasing. Once you leave it going for about 2-3 hours everything will start appearing as black squares and you'll get something along the lines of "beforeCrash.png". Once that happens, your bound to crash. At this point, fx was using around 1.5GB 2GB worth of memory. When you get it into the state, resizing the browser definitely makes thins a lot worse.

Comment 1

[Kamil Jozwiak [:kjozwiak]]

Forgot to mention that I was using the latest m-c on a Windows 8.1 x64 machine.

Comment 2

[Kamil Jozwiak [:kjozwiak]]

Attachment: firstCrash.png

Comment 3

[Kamil Jozwiak [:kjozwiak]]

Attachment: secondCrash.png

Comment 4

[Kamil Jozwiak [:kjozwiak]]

Attachment: thirdCrash.png

Comment 5

[Benjamin Smedberg]

The youtube video doesn't use Flash. skysports.com does.

Kamil, do you think that it is an interaction between the three tabs that causes the problem, or a single tab? Could you maybe try this with just one tab at a time and watch memory usage both in about:memory and in the Windows task manager.

Comment 6

[Kamil Jozwiak [:kjozwiak]]

Ya sure, I'll narrow it down and see if it's a single tab that's the culprit. The machine is my home desktop so I'll go through it once I get home! This time around I'll watch about:memory as well.

Comment 7

[Kamil Jozwiak [:kjozwiak]]

Attachment: Mem Leak Crash.7z

It's definitely "www.premierleague.com/en-gb.html" that's causing the problem. If you open that website in a single tab and leave the browser running for a long period of time (6 hours or so), the memory will slowly increase over time and hit around 1.6GB at which point my browser ends up crashing. (reproduced this twice today) 

I ended up getting two crash reports but for some reason they don't want to upload. I tried several times without any luck so I've attached the two .dmp files from the crash folder and 3 snapshots of my memory using about:memory when they'are at 1GB, 1.3GB and 1.4GB. I can't take a snapshot when the browser is the above mentioned state (comment #0 screenshot) because any interaction at that point always results in an instant crash. I added the files into a single 7zip file so I didn't spam everyone)

Not sure if this is related to Flash but definitely seems like an issue. I think if you have a flash video opened at the same time and this happens, you'll get the plugin-container.exe crash as well (comment #2)

Let me know if there's anything else that I can do!

Comment 8

[Benjamin Smedberg]

about:memory doesn't show anything to explain the memory usage, so this isn't just a web page using lots of memory.

mccr8, as part of memshrink can somebody look into the memory usage here and see if there's something we're not measuring in about:memory or what else might be going on?

Comment 9

[Andrew McCreight [:mccr8]]

Sure, I can take a look.  I'll have to figure out how to open a 7zip file first.

Comment 10

[Kamil Jozwiak [:kjozwiak]]

It's an open source file archiver (http://www.7-zip.org). I'll use zip next time around to make things easier :)

Comment 11

[Andrew McCreight [:mccr8]]

This is the only significant change from the first to the last memory report:
 306.96 MB ── private
 309.86 MB ── resident
 608.09 MB ── vsize
-594.70 MB ── vsize-max-contiguous

Because explicit isn't changing, this isn't anything being allocated by the browser itself, so this is kind of out of the area of expertise of MemShrink people per se.  I assume Flash is allocating... something in the main process.  Maybe dmajor can look at the crash dump and get some kind of something out of it?

Sorry it took me so long for a not very useful answer.

Comment 12

[(Away)]

Wow. You've got bazillions of little 4k MEM_RESERVE regions eating up 1.6 gigs of address space in each dump. I haven't seen this before.

Could you grab an xperf trace of the memory growth? https://wiki.mozilla.org/Tracing_VirtualAlloc_With_Xperf

Comment 13

[Kamil Jozwiak [:kjozwiak]]

David, I followed the instructions in the link and uploaded the compressed .etl file. I also included the main folder that had all the *.ni.pdb folders as well. As this was my first time using the tool, let me know if I missed something!

Just a heads up, uncompressed it's almost 1.86GB (about 100MB compressed using 7zip) 

https://s3.amazonaws.com/mozrelated/WPR+Crash+Trace.7z

Comment 14

[(Away)]

Kamil: My 7-zip says the archive is invalid. Maybe can we try a plain .zip file?

Comment 15

[Kamil Jozwiak [:kjozwiak]]

Apologies David, not sure why that didn't work :/ Try this one (quickly checked and made sure extracting was working):

https://s3.amazonaws.com/mozrelated/WPRCrashTrace.zip

Comment 16

[(Away)]

The .zip doesn't work for me either :( I wonder if AWS is messing with the file or serving it wrong. Has this kind of thing worked in the past?

Comment 17

[Kamil Jozwiak [:kjozwiak]]

Ya not sure what's going.. I didn't bother compressing and just added it into my dropbox, let me know if this works?

* https://www.dropbox.com/sh/mt264nn5r3bzxx1/AAAOyWLlIZeushTkX-GvDZSwa?dl=0

Comment 18

[(Away)]

Thanks, that file worked.

And, aha: You appear to be running with pageheap. That would definitely explain the huge memory usage and the abundance of single-page regions. (It wouldn't explain memory growth over time, but it causes so much interference that I'm not convinced there's a real problem)

Could you try again with pageheap (and/or appverifier) turned off? I can help if you're not sure how to do that.

Comment 19

[(Away)]

(For anyone following along, the pageheap'ed allocations were primarily from the graphics driver. Pageheap doesn't do a lot for Firefox code since we use jemalloc.)

Comment 20

[Kamil Jozwiak [:kjozwiak]]

David, that was definitely it. I remember turning it on a long time ago to help debug another issue and figured it would clear itself once I restarted the machine etc.. Guess not :/

* "gflags /p" listed "firefox.exe: page heap enabled with flags (full trace)"
* Disabled using "gflags /p /disable firefox.exe"
* left the browser running over night and memory didn't increase substantially like before (only a few MB)

Anyways thanks again for the help! Apologies if I wasted your time.