Last Comment Bug 774281 - (F1398665248) Generic bug: Flash plugin-container crash in F1398665248
(F1398665248)
: Generic bug: Flash plugin-container crash in F1398665248
Status: NEW
[Flash-11.3] generic signature bug, p...
: crash, crashreportid, flashplayer, thirdparty, topcrash, topcrash-plugin, topcrash-win
Product: Core
Classification: Components
Component: Plug-ins (show other bugs)
: unspecified
: x86 Windows 7
: -- critical with 8 votes (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
:
: Benjamin Smedberg [:bsmedberg]
Mentors:
: 761597 774092 790153 935493 1078000 1099944 1133407 1188631 1188682 1188683 1221838 1249958 1249959 1250116 1299582 1312750 (view as bug list)
Depends on: 1137664 713138 763237 767867 782395 787014 790368 812047 814003 846126 891813 1082670 1120011 1122646 1157078 1275682
Blocks: flash-crashes
  Show dependency treegraph
 
Reported: 2012-07-16 07:55 PDT by Benjamin Smedberg [:bsmedberg]
Modified: 2016-10-31 07:59 PDT (History)
47 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
affected
affected
affected
affected
affected
affected
affected


Attachments
dmp/extra files & dxdiag (40.59 KB, application/octet-stream)
2013-05-12 00:04 PDT, MrX1980
no flags Details
Plugin crash rate graph on Aurora (49.64 KB, image/png)
2016-10-27 09:00 PDT, Mike Conley (:mconley)
no flags Details

Description Benjamin Smedberg [:bsmedberg] 2012-07-16 07:55:05 PDT
This is a generic bug covering the Flash 11.3 signature F1398665248_____________________________. If you have specific steps to reproduce a Flash crash, please do not comment in this bug! Instead, please file a *new* bug and mark it blocking this one.

The F1398665248_____________________________ signature in Flash crashes is a generic signature. It happens when Flash protected mode for Firefox is enabled and one of the other FlashPlayerPlugin processes crashes or is killed. Currently in Firefox 13 and 14, we only track crashes in the plugin-container process, so almost all Flash 11.3 crashes show up as this signature.

In Firefox 15 and 16 we have implemented crash reporting for the FlashPlayerPlugin processes as well, so this signature will usually not be reported; we'll see the more interesting crash signature in the Flash process instead.

It appears that this crash is uncontrolled behavior, and Adobe is working with us to make the behavior more controlled when the Flash process crashes, something like MOZ_CRASH, so this signature will very likely change in future versions of Flash 11.3.
Comment 1 Benjamin Smedberg [:bsmedberg] 2012-07-20 14:43:59 PDT
I can confirm that with the 11.3.300.267 build killing a FlashPlayerPlugin process while hulu is running produces a different (controlled) EXCEPTION_BREAKPOINT report.
Comment 2 Arthur K. 2012-07-26 10:06:13 PDT
Has it been tried with Flash 11.3.300.268?
Comment 3 Scoobidiver (away) 2012-07-26 10:37:02 PDT
(In reply to Arthur K. from comment #2)
> Has it been tried with Flash 11.3.300.268?
It's a generic bug, so there are no STR to test
Comment 4 Benjamin Smedberg [:bsmedberg] 2012-07-26 10:39:58 PDT
Well actually, you can usually reproduce the generic bug by playing a hulu.com movie (vimeo would probably work as well) and then killing one of the FlashPlayerPlugin.exe processes in the task manager.
Comment 5 Arthur K. 2012-08-21 17:51:02 PDT
Any improvement with 11.4.402.265?
Comment 7 mtalistu 2012-09-11 11:12:39 PDT
We implemented the suggested fix, similar to what is in MOZ_CRASH, for 11.3 and 11.4.402.265.

When we detect an error that requires us to shut down plugin-container, we do the following to exit the process:

__debugbreak();
::TerminateProcess(::GetCurrentProcess(), 3);

This is based on the MOZ_CRASH macro defined in http://mxr.mozilla.org/mozilla-central/source/mfbt/Assertions.h#153

Is there something else we should be doing to more cleanly exit the plugin-container process when we detect a fatal error so that this is not classified as a crash for this process?
Comment 8 Benjamin Smedberg [:bsmedberg] 2012-09-11 11:22:12 PDT
I'm not sure what you mean... we now see that the crash reason is correct (EXCEPTION_BREAKPOINT) and we can track the volume in a useful way, so I believe that the current behavior is as expected.
Comment 9 mtalistu 2012-09-11 11:29:23 PDT
Ah, perhaps there was a misunderstanding then. This bug/signature was in one of the top crashers lists we were looking at, so we thought that we still needed to do something to address it.

However, if this is behaving as expected, then I suppose we will just continue to see this crash as long as we still have other crashes that cause the sandbox process to go away unexpectedly.
Comment 10 Benjamin Smedberg [:bsmedberg] 2012-09-11 11:42:26 PDT
Yes. To the user it still represents "Flash player crashing", but to us it indicates that there was a problem in one of the other processes which was not caught usefully by the crash reporter and so what's left is a bucket of crashes which are good for volume measurements but often aren't useful in terms of engineering.

If we have new bugs with STR to reproduce this signature, I'd like to see them because I'd like to figure out why the crash reporter didn't catch the other process dying.
Comment 11 Daniel Veditz [:dveditz] 2012-09-12 10:36:42 PDT
*** Bug 790153 has been marked as a duplicate of this bug. ***
Comment 12 Arthur K. 2012-09-28 08:49:46 PDT
Any improvement with 11.5.500.80?
Comment 13 nemo 2012-10-12 12:51:15 PDT
FWIW, friend who'd encountered crasher (11.3 and 11.4 Flash in Windows 7) reports that the fix for him was disabling acceleration in Flash.

Well, maybe more of a workaround, but given the increasingly limited utility of Flash, really more of a fix for him.

Perhaps Adobe could push out a hotfix 11.4.1 w/ acceleration disabled by default? :)
Comment 14 nemo 2012-10-26 07:11:56 PDT
Have a second report of disabling acceleration being a workaround, also a Windows user.
Comment 15 Jeromie Clark 2012-11-19 14:36:55 PST
@Nemo: Please attach dxdiag output for affected machines, and ideally, URLs to sites where the crash is occurring consistently.  We'll attempt to build out a similar configuration and reproduce the issue.  Once we're sure that the crash is related to the HW/Driver combination, we can disable GPU acceleration for that particular profile.
Comment 16 nemo 2012-11-19 15:25:49 PST
Welp. I asked one of 'em, but he didn't answer me, and I'm betting he isn't too interested in doing debugging for Adobe when he already has a fix.  I might be able to convince him to pastebin about:support or otherwise give card info.

The other fellow I can't contact.
Comment 18 Scoobidiver (away) 2013-05-12 00:24:48 PDT
(In reply to MrX1980 from comment #17)
I doesn't crash for me with Flash 11.7

> https://crash-stats.mozilla.com/report/index/bp-12c89852-ea69-4ac2-95bf-
> cb1672130512
It's bug 772097 that spiked with Flash 11.8.

> https://crash-stats.mozilla.com/report/index/bp-8e450e15-61a7-4ef6-ad8f-
> 9a9092130512
It's not this bug which a generic one but will likely be fixed along with bug 772097.
Comment 19 Tracy Walker [:tracy] 2013-08-26 07:47:46 PDT
with Flash 11.8.800.94 is number one on https://crash-analysis.mozilla.com/bsmedberg/flash-topcrash-release.html
Comment 20 Robert Kaiser 2013-11-07 09:48:30 PST
*** Bug 935493 has been marked as a duplicate of this bug. ***
Comment 21 juan becerra [:juanb] 2014-02-03 07:29:04 PST
Tracy noticed a crash spike across branches over the weekend.

Looking at the explosive reports list for Fx29, the number of crashes with this signature doubled in the last few days, starting on 1/31.

The last Flash update was a couple of weeks ago. Also, the version of Flash reported in the reports I looked at varied from 11.9x to the latest 12.0.0.43, so it doesn't seem to be version specific.

Perhaps a change elsewhere, like a major website?
Comment 22 MrX1980 2014-02-03 09:11:31 PST
I think one reason could be "watching Super Bowl".
https://crash-stats.mozilla.com/report/index/a0d22cb0-887c-4ec7-9fea-035522140203
Comment 23 Ronny Perinke 2014-02-26 12:46:13 PST
Flash keeps on crashing with this error in my VS2013 builds of Fx29 on Win 8.1 x64.

Here is my latest crash report https://crash-stats.mozilla.com/report/index/9935723b-25b2-4282-819f-6c1232140226

In earlier builds around 1/11, Flash did not crash. It started somewhen between 1/12 and 1/23 (with Fx28).
Comment 25 alex_mayorga 2014-04-13 10:47:32 PDT
*** Bug 713138 has been marked as a duplicate of this bug. ***
Comment 26 banakon 2014-04-13 12:50:23 PDT Comment hidden (off-topic)
Comment 27 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2014-05-05 09:30:32 PDT
FYI, this signature is now the #1 explosive crash in Firefox 31:
https://crash-analysis.mozilla.com/rkaiser/2014-05-04/2014-05-04.firefox.31.explosiveness.html

I'm not sure if that indicates a worse situation or not, and I'm not sure what the next step should be. I just thought I'd document it here in case anyone has an idea about what to do.
Comment 28 Robert Kaiser 2014-05-05 10:42:51 PDT
Unless we see this as a permanent trend that probably relates to a patch on our side, I wouldn't care about those generic Flash issues on Aurora or Nightly. And we did not see any significant increase on the other channels at this point.
Comment 29 banakon 2014-05-05 10:46:37 PDT Comment hidden (me-too)
Comment 30 banakon 2014-05-05 10:50:38 PDT Comment hidden (me-too)
Comment 31 banakon 2014-05-05 13:15:57 PDT Comment hidden (me-too)
Comment 32 Benjamin Smedberg [:bsmedberg] 2015-02-19 06:35:32 PST
*** Bug 1133407 has been marked as a duplicate of this bug. ***
Comment 33 Benjamin Smedberg [:bsmedberg] 2015-02-28 08:59:22 PST
*** Bug 1099944 has been marked as a duplicate of this bug. ***
Comment 35 Virtual_ManPL [:Virtual] - (ni? me) 2015-05-03 04:49:45 PDT
*** Bug 761597 has been marked as a duplicate of this bug. ***
Comment 36 Virtual_ManPL [:Virtual] - (ni? me) 2015-05-03 04:51:46 PDT
*** Bug 774092 has been marked as a duplicate of this bug. ***
Comment 37 Virtual_ManPL [:Virtual] - (ni? me) 2015-05-03 05:44:58 PDT
*** Bug 1078000 has been marked as a duplicate of this bug. ***
Comment 38 Jeromie Clark 2015-06-15 12:15:00 PDT
Flash Player has shipped a number of changes to mitigate this as part of a recent collaborative engineering effort with Mozilla.  Those changes are dependent on additional changes in Firefox, and my understanding is that we're waiting on the corresponding Firefox changes to make their way through the various stability channels at this point in time.  

I can't get a clear read on whether the situation is improved looking at the data in crash-stats, but we would naturally welcome additional engineering engagements to work on this as necessary moving forward.
Comment 39 Robert Kaiser 2015-06-25 10:41:41 PDT
Aaron, in which Firefox version do or did the changes ship that Jeromie is talking about?

Jeromie, can you tell us at what time or in which Flash version your side of the improvements shipped?
Comment 40 Aaron Klotz [:aklotz] 2015-06-25 12:00:12 PDT
(In reply to Robert Kaiser (:kairo@mozilla.com) - on vacation or slow to reply until the end of June from comment #39)
> Aaron, in which Firefox version do or did the changes ship that Jeromie is
> talking about?

39.
Comment 41 Robert Kaiser 2015-06-30 08:35:31 PDT
This signature dropped significantly in volume with the Flash release that was shipped 2015-06-24.
Comment 42 Robert Kaiser 2015-06-30 08:39:07 PDT
(In reply to Robert Kaiser (:kairo@mozilla.com) - on vacation or slow to reply until the end of June from comment #41)
> This signature dropped significantly in volume with the Flash release that
> was shipped 2015-06-24.

Oops, sorry, probably not true, forgot for a moment that we are missing Flash symbols for the new release, so of course Flash crashes with a symbolized signature seem to have dropped.
Comment 43 Robert Kaiser 2015-07-29 07:32:20 PDT
*** Bug 1188631 has been marked as a duplicate of this bug. ***
Comment 44 Robert Kaiser 2015-07-29 07:33:12 PDT
*** Bug 1188682 has been marked as a duplicate of this bug. ***
Comment 45 Robert Kaiser 2015-07-29 07:33:47 PDT
*** Bug 1188683 has been marked as a duplicate of this bug. ***
Comment 48 Virtual_ManPL [:Virtual] - (ni? me) 2015-12-05 10:29:29 PST
*** Bug 1221838 has been marked as a duplicate of this bug. ***
Comment 49 Virtual_ManPL [:Virtual] - (ni? me) 2016-02-21 02:31:40 PST
*** Bug 1249959 has been marked as a duplicate of this bug. ***
Comment 50 YF (Yang) 2016-02-22 04:29:59 PST
*** Bug 1250116 has been marked as a duplicate of this bug. ***
Comment 52 Arthur K. 2016-06-01 14:11:07 PDT
(In reply to Yorgos from comment #51)
> https://crash-stats.mozilla.com/report/index/b04d4faf-edd8-4beb-a730-
> 26b5a2160601

AdapterVendorID: 0x10de, AdapterDeviceID: 0x0fc6, AdapterSubsysID: 00000000, AdapterDriverVersion: 10.18.13.6191

Could possibly be fixed by updating to (v365.19) 10.18.13.6519 drivers.
Comment 53 Yorgos 2016-06-01 14:13:39 PDT
there is something in the changelog from nvidia?
because this issue arised a lot of years ago.
Comment 54 Yorgos 2016-06-04 14:27:06 PDT
Now I have 10.18.13.6519 drivers.
Comment 55 Calixte Denizet (:calixte) 2016-07-19 09:23:24 PDT
Crash volume for signature 'F1398665248_____________________________':
  - nightly (50): 11899
  - aurora (49): 31684
  - beta (48): 16372
  - release (47): 92596
  - esr (45): 16611

Affected platform: Windows
Comment 56 Arthur K. 2016-07-19 09:34:46 PDT
(In reply to Yorgos from comment #54)
> Now I have 10.18.13.6519 drivers.

Did upgrading drivers reduce or eliminate crashes? You're on Flash 22.0.0.209 as well?
Comment 57 Yorgos 2016-07-19 09:36:34 PDT
A bit reduced, not fully resolved but very well reduced.
I am on 209.
Comment 58 Arthur K. 2016-07-19 09:38:12 PDT
(In reply to Yorgos from comment #57)
> A bit reduced, not fully resolved but very well reduced.
> I am on 209.

This is good to know as it implies the driver on Vista still has room for improvement. Thanks for reporting back.
Comment 59 Yorgos 2016-07-19 09:39:12 PDT
this is the final driver for Vista.
Comment 60 Release Management Account Bot 2016-08-25 11:14:06 PDT
Crash volume for signature 'F1398665248_____________________________':
 - nightly (version 51): 5697 crashes from 2016-08-01.
 - aurora  (version 50): 13400 crashes from 2016-08-01.
 - beta    (version 49): 47525 crashes from 2016-08-02.
 - release (version 48): 18140 crashes from 2016-07-25.
 - esr     (version 45): 22863 crashes from 2016-05-02.

Crash volume on the last weeks (Week N is from 08-22 to 08-28):
            W. N-1  W. N-2  W. N-3
 - nightly    2023    1497     412
 - aurora     4369    3907    2449
 - beta       4194   25129   15981
 - release    6004    5246    2874
 - esr        1464    1588    1814

Affected platform: Windows

Crash rank on the last 7 days:
           Browser     Content Plugin
 - nightly #14                 #1
 - aurora  #1                  #1
 - beta    #181                #1
 - release #1083               #1
 - esr     #1094               #1
Comment 61 Virtual_ManPL [:Virtual] - (ni? me) 2016-09-01 02:04:42 PDT
*** Bug 1299582 has been marked as a duplicate of this bug. ***
Comment 62 Virtual_ManPL [:Virtual] - (ni? me) 2016-09-01 02:06:13 PDT
*** Bug 1249958 has been marked as a duplicate of this bug. ***
Comment 63 Release Management Account Bot 2016-10-03 03:28:00 PDT
Crash volume for signature 'F1398665248_____________________________':
 - nightly (version 52): 2645 crashes from 2016-09-19.
 - aurora  (version 51): 8019 crashes from 2016-09-19.
 - beta    (version 50): 3383 crashes from 2016-09-20.
 - release (version 49): 16033 crashes from 2016-09-05.
 - esr     (version 45): 26082 crashes from 2016-06-01.

Crash volume on the last weeks (Week N is from 10-03 to 10-09):
            W. N-1  W. N-2
 - nightly    2223     422
 - aurora     7564     455
 - beta       2739     644
 - release   12016    4016
 - esr        1853    1731

Affected platform: Windows

Crash rank on the last 7 days:
           Browser     Content Plugin
 - nightly #13                 #1
 - aurora  #3                  #1
 - beta    #1868               #1
 - release #2096               #1
 - esr     #3692               #1
Comment 64 Gerry Chang [:gchang] 2016-10-25 09:19:43 PDT
*** Bug 1312750 has been marked as a duplicate of this bug. ***
Comment 65 Gerry Chang [:gchang] 2016-10-25 09:21:34 PDT
This crash is getting worse in 51 aurora and it's #2 in top crash. 
One of the crashes is caused by Flash Version: 23.0.0.185 Filename: NPSWF32_23_0_0_185.dll

We can also see the volume grows from 604 to 1109 on 2016-10-23 to 2016-10-24.
Comment 66 Benjamin Smedberg [:bsmedberg] 2016-10-25 11:51:59 PDT
Gerry, there are many possibilities here.
* A Firefox change caused a problem
* A Flash change caused a problem (either Flash release or Flash beta)
* Some common website changed their code and caused a problem
* We changed something about crash submission

Can you help with digging in and figuring out which of these is most likely? I don't know of any plugin changes that landed to aurora recently, and I don't think any of the auto-submission bits changed recently either (NI?mconley for confirmation). If this was a Flash or website change, it's likely that we'd see similar volume changes in beta/release.
Comment 67 Mike Conley (:mconley) 2016-10-27 09:00:00 PDT
Created attachment 8805166 [details]
Plugin crash rate graph on Aurora

Here's the plugin crash rate on Aurora according to Telemetry: https://chutten.github.io/telemetry_crashes/

The spike you mention around September 23rd / 24th, I believe, makes an appearance on this chart. It's the big spike in the middle - but it drops off dramatically.

Interestingly, this was very shortly after 51 uplifted to Aurora (2016-09-20), according to https://wiki.mozilla.org/RapidRelease/Calendar.

So it looks like the crashes were real, but have tapered off. Either something got patched in Gecko (I'm not familiar with what that might be), or something got patched in Flash, or something got patched on some website somewhere.
Comment 68 Mike Conley (:mconley) 2016-10-27 09:01:09 PDT
The only change to auto-submission in the last release is that we started showing users the "Submit Unsubmitted Crash Reports" notification, for which users can opt-in to auto-submission.
Comment 69 Marco Castelluccio [:marco] 2016-10-28 02:35:52 PDT
(In reply to Mike Conley (:mconley) - (digging out of review / needinfo backlog) from comment #68)
> The only change to auto-submission in the last release is that we started
> showing users the "Submit Unsubmitted Crash Reports" notification, for which
> users can opt-in to auto-submission.

Did we enable auto-submission? Which bug? I see bug 1270553 is still open.
Comment 70 Mike Conley (:mconley) 2016-10-31 07:59:20 PDT
(In reply to Marco Castelluccio [:marco] from comment #69)
> (In reply to Mike Conley (:mconley) - (digging out of review / needinfo
> backlog) from comment #68)
> > The only change to auto-submission in the last release is that we started
> > showing users the "Submit Unsubmitted Crash Reports" notification, for which
> > users can opt-in to auto-submission.
> 
> Did we enable auto-submission? Which bug? I see bug 1270553 is still open.

Bug 1287178 is where we exposed the preference and the button in the notification bar to allow users to auto-submit backlogged crash reports.

Note You need to log in before you can comment on or make changes to this bug.