1205979 - Java plugin crash in mozilla::plugins::AssertPluginThread()

Status: RESOLVED WONTFIX

(Core Graveyard :: Plug-ins, defect)

Description

[Ekanan Ketunuti]

This bug was filed from the Socorro interface and is 
report bp-580ba68d-13d8-42c1-855f-0bdc72150918.
=============================================================

STR:

1. Run java https://www.java.com/verify
2. Closed tab
3. Open about:crashes. you'll see 3 reports in there.

Plugin works fine, fake reports maybe?

Cannot reproduce in Beta.

Comment 1

[Benjamin Smedberg]

This is a real plugin-process crash which happens when tearing down a plugin instance. It's caused by a threading error in the Java plugin, and we recently became more strict about crashing on these threading bugs instead of ignoring them and hoping everything works out.

If you had multiple Java tabs open, you'd see the Java in the second tab crash when you closed the first tab.

Mozilla is not going to "fix" this bug: you need to report it to Oracle and they need to fix it within the Java plugin.

Comment 2

[Jim Mathies [:jimm]]

These are substantially higher under e10s. 6.78% of total crash volume compared to 0.99%. All are under Java.

https://crash-stats.mozilla.com/search/?product=Firefox&version=47.0b3&version=47.0b4&version=47.0b5&dom_ipc_enabled=!__null__&process_type=plugin&signature=~AssertPluginThread&_facets=signature&_facets=plugin_name&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-plugin_name

Comment 3

[Chris Peterson [:cpeterson]]

bsmedberg, these AssertPluginThread() Java crashes seem to have recently spiked in Nightly 50 (rank #3) and Aurora 49 (rank #5), but don't show up in the top 50 crashes for Beta 48 or Firefox 47.

Bug 1222036 comment 15 says Oracle has fixed this _releaseobject thread bug. Should we blocklist older unfixed Java versions?

https://bugs.openjdk.java.net/browse/JDK-8133523

e.g. bp-eea8c897-2523-4717-89b8-067882160624

Comment 4

[Benjamin Smedberg]

In general we should blocklist all known-insecure versions of Java. Do you have a proposed minimum Java version we should target? My understanding is that this was fixed in Java 9, but there are still supported versions of Java 8 that we typically would not blocklist. Perhaps though we should encourage upgrade for affected users using Shield.

Comment 5

[Chris Peterson [:cpeterson]]

Oracle fixed this bug in Java 8 Update 66 according to bug 1140616 comment 84 and OpenJDK bug JDK-8133523 [1]. According to AMO [2] and bug 1259458, we block (click-to-play) Java 8 Update 76 and below. The latest version of Java 8 is Update 92.

Do we already have a mechanism in Shield to encourage people to update their plugins? How do we do that?

[1] https://bugs.openjdk.java.net/browse/JDK-8133523
[2] https://addons.mozilla.org/en-US/firefox/blocked/p1144

Comment 6

[Benjamin Smedberg]

No, but if we've already issued a plugin block there is very little we can do. And we also care not-at-all.

Comment 8

[Sylvestre Ledru [:Sylvestre]]

Benjamin, with the last beta or 48 and 49.0b1, this is causing a huge spike in term of startup crashes:
https://crash-analysis.mozilla.com/rkaiser/crash-report-tools/longtermgraph/?fxbeta-bcat

Is there anything we can do here?
It seems that it is impacting VLC & the JVM

Comment 9

[Chris Peterson [:cpeterson]]

All of the crashes seem to point the same release assert in NPN_ReleaseObject. We could just change that one release assert to a debug assert and return from the function early. Leaking an object is preferable to crashing.

https://hg.mozilla.org/integration/mozilla-inbound/annotate/51377a641589/dom/plugins/ipc/PluginModuleChild.cpp#l2300

Comment 10

[Liz Henry (:lizzard) (relman/hg->git project)]

Tracking this for 49 at least. Is this really a startup crash ? How can we tell?

Comment 11

[Liz Henry (:lizzard) (relman/hg->git project)]

From discussion on IRC it sounds like the uptime is process uptime, so this may be the plugin process. 
Is there some change in how we are counting plugin crashes in crash-stats? i.e. we catch these in crash stats now when we failed to in the past?

Comment 12

[Benjamin Smedberg]

This is not a "startup crash" according to our standard definition. It's a plugin-process crash.

I do not believe that leaking is preferable to crashing in this case: leaking means gradual slowdown while crashes are immediate and provide feedback.

I strongly believe we should not make any Firefox changes here. If this is happening in current versions of Oracle Java, we should reach out to them: old versions are already blocklisted so there's not a lot we can do if users keep choosing to use those versions.

As for VLC, you could reach out to the developers but we'd reach the point of diminishing returns very quickly.

Comment 13

[Liz Henry (:lizzard) (relman/hg->git project)]

Are we seeing the spike in crash-stats move from nightly to aurora to beta to release, because of some change in how plugin crashes are reported, then?

Comment 14

[Chris Peterson [:cpeterson]]

Benjamin, can we completely disallow (not just "This plugin is vulnerable" soft block that still allows click-to-play [1]) Java versions older than the known-fixed version? Or can we reset everyone's Java "Allow and Remember" setting?

According to bug 1140616 comment 84, Oracle says fixed this bug in Java 8 Update 66. I looked at a dozen of these Java crash reports and the most recent crashing version appears to be 8.0.650.17, which looks like a version right before the fixed Java 8 Update 66.

We already soft block (click-to-play) Java 8 Update 76 and below (bug 1259458), so users are manually activating and crashing old plugins or they set Java's click-to-play setting as "Allow and Remember".

[1] https://support.mozilla.org/en-US/kb/why-do-i-have-click-activate-plugins#w_how-click-to-activate-works

Comment 15

[Benjamin Smedberg]

(In reply to Chris Peterson [:cpeterson] from comment #14)
> Benjamin, can we completely disallow (not just "This plugin is vulnerable"
> soft block that still allows click-to-play [1]) Java versions older than the
> known-fixed version?

We can but we shouldn't. Enterprise Java deployments are very slow to update, and only a minority of users/Java applets are affected by this bug.

Comment 16

[Chris Peterson [:cpeterson]]

(In reply to Benjamin Smedberg [:bsmedberg] from comment #15)
> We can but we shouldn't. Enterprise Java deployments are very slow to
> update, and only a minority of users/Java applets are affected by this bug.

This is our #1 topcrash in Beta 49. I think some external change has triggered this spike.

When I search for the number of AssertPluginThread crashes in Beta versions *restricted to their six-week release cycle dates*:

Beta 46 =  3110
Beta 47 =  1232
Beta 48 =  1458 https://is.gd/pl7KGZ
Beta 49 = 25557 in just one week! https://is.gd/hyABIl

And laggards still running Beta 48 over the last week = 139163 crashes! https://is.gd/ChqUC6

Comment 17

[[:philipp]]

i think bug 1269998 landing in 49.0b triggered the spike, since 99.8% of these reports are currently coming from the infobar submission mechanism.

Comment 18

[Sylvestre Ledru [:Sylvestre]]

Philipp, you are suggesting that we had this bug for a long time and the implementation of bug 1269998 increased the submission rate?

Comment 19

[[:philipp]]

yes, if you just look at the reports submitted through the normal crash reporting process (by excluding crashes submitted from the infobar), the numbers for 49.0b are not out of the ordinary and in line with what we have seen in older versions as well: http://bit.ly/2aK1eZA

Comment 20

[Sylvestre Ledru [:Sylvestre]]

Off topic but maybe, if we can, we should annotate these crashes to know that we didn't get them much in the past and that they are now pushed thanks to  bug 1269998

Comment 21

[Marco Castelluccio [:marco]]

A relatively small number of users (e.g. 1560 installations on 49.0b1) is generating a huge number of crash reports (e.g. 29620 reports on 49.0b1)

Comment 22

[Liz Henry (:lizzard) (relman/hg->git project)]

Looks like now we won't get those (for beta 2) because we turned off the submission interface for beta and release channels, post beta 1 in bug 1291277.

Comment 23

[Liz Henry (:lizzard) (relman/hg->git project)]

Note also Marco is checking these reports to see the versions of Java used.

Comment 24

[Marco Castelluccio [:marco]]

Attachment: Java versions.txt

Comment 25

[Liz Henry (:lizzard) (relman/hg->git project)]

Jorge, how can I tell which versions of Java we  have blocklisted already? Marco's list above shows the versions that are currently showing up as crashing most often. So if we haven't blocklisted the worst offenders there, maybe we should do that now.

Comment 26

[Andrew McCreight [:mccr8]]

Are these even real crashes a user might notice? The other crashes that bug 1269998 has tended to get reports generated for are slow shutdown issues that don't even look like crashes. It seems bad to block somebody from using Java for an issue they aren't even noticing.

Comment 27

[Jorge Villalobos [:jorgev] (he/him)]

The full list of blocked add-ons and plugins can be found here: https://addons.mozilla.org/en-US/firefox/blocked/

For Java, we should be blocking Java Plugin 8 update 76 and lower, and Java Plugin 7 update 97 and lower. I don't know how those map to the extended version numbers in comment #24.

Comment 28

[Marco Castelluccio [:marco]]

(In reply to Jorge Villalobos [:jorgev] from comment #27)
> The full list of blocked add-ons and plugins can be found here:
> https://addons.mozilla.org/en-US/firefox/blocked/
> 
> For Java, we should be blocking Java Plugin 8 update 76 and lower, and Java
> Plugin 7 update 97 and lower. I don't know how those map to the extended
> version numbers in comment #24.

There do seem to be versions in the list that are supposedly blocked.
For example, 7.0.510.13 (7U51) is the top one with ~4000 crashes.
There are also versions starting with 6.

If it works like I think it works (MAJOR.0.UPDATE0.??? or MAJOR.0.UPDATE.???),
then all those versions should be blocked.

Comment 29

[Benjamin Smedberg]

That's not surprising. Users can override the plugin blocks, and it's amazingly common for enterprises to never update their Java.

Comment 30

[Marco Castelluccio [:marco]]

(In reply to Benjamin Smedberg [:bsmedberg] from comment #29)
> That's not surprising. Users can override the plugin blocks, and it's
> amazingly common for enterprises to never update their Java.

It is interesting though that most crashes with this signature are coming from Aurora (from a
small number of installations).

Comment 31

[Chris Peterson [:cpeterson]]

These are definitely a lot of enterprise apps. Many of the non-empty crash URLs are intranet IP addresses, file:/// URLs, and various countries' .gov sites.

Benjamin said in comment 12 that crashing immediately is better than a leak leading to a gradual slowdown. But the crash stops the user from completing their enterprise task, while the leak would be limited to the plugin process, which is reaped when the plugin is no longer in use. That seems like a reasonable trade-off to me. To minimize the slowdown issue, we could also reap unused Java plugin processes more frequently than other plugin processes.

Comment 32

[Marco Castelluccio [:marco]]

(In reply to Chris Peterson [:cpeterson] from comment #31)
> These are definitely a lot of enterprise apps. Many of the non-empty crash
> URLs are intranet IP addresses, file:/// URLs, and various countries' .gov
> sites.
> 
> Benjamin said in comment 12 that crashing immediately is better than a leak
> leading to a gradual slowdown. But the crash stops the user from completing
> their enterprise task, while the leak would be limited to the plugin
> process, which is reaped when the plugin is no longer in use. That seems
> like a reasonable trade-off to me. To minimize the slowdown issue, we could
> also reap unused Java plugin processes more frequently than other plugin
> processes.

~98% of the crashes have an empty URL (in Aurora ~100% of the crashes).

It still seems very strange to me that this crash has been #1 in the top
crashers list for Aurora for a while, since I wouldn't expect enterprise
users to use Aurora.

I've noticed this crash was not a top-crasher before we had the infobar
to submit crashes, and indeed basically all crashes with this signature
on Aurora have submitted_from_infobar=true.

Comment 33

[Marco Castelluccio [:marco]]

(In reply to Marco Castelluccio [:marco] from comment #32)
> I've noticed this crash was not a top-crasher before we had the infobar
> to submit crashes, and indeed basically all crashes with this signature
> on Aurora have submitted_from_infobar=true.

Oh, philipp already noticed this before (comment 19).

Comment 34

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

[Tracking Requested - why for this release]: This was mentioned as a top crasher in the channel meeting today.

Comment 35

[Gerry Chang [:gchang]]

Track for 51+ as the volume is getting larger.

Comment 36

[Gerry Chang [:gchang]]

Mark 51 fix-optional as the volume of installation of the crash is low.