Closed Bug 1140616 (CVE-2015-7196) Opened 5 years ago Closed 5 years ago
crashes in GC with Java applet
The first for crashes are all [@ js::Nursery::MinorGCCallback(JSTracer*, void**, JSGCTraceKind) ], which may make this related to bug 991746, but perhaps not a duplicate, given that 991746 exists in version 31 but you say version 34 and 31.5.0 work for you
All crash signatures were produced with same setup. Firefox 34 was installed on the same machine and there were no crashes. Firefox ESR 31.5 also has no problems, although it is run on different host, but same OS - Windows 7. Firefox 35 and 36 crash all the time with same application/web page containing applets, on different Windows OSes on different hosts. Not sure if it is the same issue, could be, but other crashes of same thing have different signatures. Is there anything I could help trace issue? Unfortunately, there is no public access to the page that causes issue, but maybe something could be arranged privately, as I am developer from team that produces this Web application that causes crash in Firefox.
(In reply to vcoolas from comment #2) > Is there anything I could help trace issue? Would you mind trying this debug build? http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-release-win32-debug/1426976301/firefox-36.0.4.en-US.win32.zip Please let us know if it also crashes and attach the console output to this bug if it does (it should open a new console window when you start it, I think).
We appear to have introduced a bug in NPAPI in 35 that causes us to somehow embed a broken pointer into the store buffer. Unfortunately, there's not much in the existing stacks that helps much as the error happens long before the GC trips over it. Hopefully the debug build will show something up; otherwise someone with a debugger will need to backtrack the problem.
I am attaching console output logs from four runs. Except for one case, crashes with this debug version seem to happen much faster. Firefox crashed in every test run.
Assertion failure: CurrentThreadCanAccessRuntime(rt), at c:\builds\moz2_slave\m-rel-w32-d-000000000000000000\build\js\src\gc/Heap.h:1190 Nice! We need the backtrace from that line. Do you have link to the crash-stats upload?
Unfortunately, when debug version crashes, window to submit crash information does not even show up. Windows only show dialog that application needs to be closed. Maybe crash information gets saved somewhere locally on disk? If yes, where? Or maybe Firefox could be started with some additional options to produce this information? I have just unzipped indicated archive with debug version and ran. Should I install/register something to get crash reporter application to pop up?
Hello all, Any progression on this bug ? We also have a web application loading a Java Applet, and since Firefox 35 the browser crashes when the applet is unloaded. This issue is really critical... I'm sorry but I can't share with you any public link of our web application because the product is not public. Crash reports: https://crash-stats.mozilla.com/report/index/9b310cd6-6ea4-4614-adfd-cc0992150402 https://crash-stats.mozilla.com/report/index/b8166c59-deda-4433-93ed-e52722150323 https://crash-stats.mozilla.com/report/index/753fa1e3-e2f8-4c9f-b982-71bfe2150323 https://crash-stats.mozilla.com/report/index/4fdc8917-343b-4050-b5dc-6b5c82150323
Hi Since a couple of weeks we are reproducing similar behaviour with our applet using FF 37.0.2 and different versions of Java (7u79, 8u45). Not happening with 31.6 (havent tried other versions). This is more or less the scenario we are currently analyzing: -Browse foo.htm which loads an applet (OK) -Invoke function foo that loads and some libraries (NSS) for signature (OK) -Invoke function foo that loads and some libraries (NSS) for signature (OK) - ...you can do this as many times as you want... -Invoke function foo that loads and some libraries (NSS) for signature (OK) -Browse (as fast as you can after using the applet) to bar.htm -Result: Firefox crash Java console is showing a java.lang.InterruptedException, and seems Java plugin is crashing on Applet unload, causing a Firefox Crash. I could agree the applet is doing something wrong, but a plugin crash should never crash the browser! Can you give us some feedback?
Not happening on Ubuntu 14.04 amd64 neither with OpenJDK (7u79) or Oracle (8u45). Also happening on Windows 7 x86. Current working test: -Go to foo.htm (applet load) -invoke/use applet (our applet is currently doing a lot of things, so we have still to analize if this is an issue that depends on applet code) -CTRL+F5 = firefox crash
Maybe I'm missing something, but where is foo.htm?
LOL...thats was just an example!. I'll try to have a simple testcase ASAP ;) Are you going to be the one dealing with this problem?
It depends where the bug is. If we're crashing in GC it's generally because the heap got hosed at some point in the distant past -- I can't even tell you where the bug is without being able to reproduce it locally and walk backwards from the crash.
Hi all, there's any progression on this bug? Or any workaround? Thanks.
We would need to be able to reproduce the bug. Victor, are you seeing this crash? Can you give us more information? A crash signature from about:crashes maybe? Thanks.
Right now, we have some crashes with this signature on 38.0.5 but none for 39+.
Hi Liz, I don't know if you are asking this, but this is the crash ID that I got yesterday: bp-ab3c08fb-f52e-4165-a117-d2fe92150622 I am using Firefox ESR 38.0.1 Tell me if you need something more. Thanks.
[Tracking Requested - why for this release]: Regression, crash, is appearing in ESR.
Hi Liz, it seems that in Firefox 39, this error is not appearing. Do you know if the next release of Firefox ESR will have the fix apparently introduced in Firefox 39? I mean, Firefox ESR 38.2 will have this fix? Thanks.
(In reply to Victor from comment #20) > Hi Liz, > > it seems that in Firefox 39, this error is not appearing. Do you know if the > next release of Firefox ESR will have the fix apparently introduced in > Firefox 39? > I mean, Firefox ESR 38.2 will have this fix? > > Thanks. Hi Victor, Actually, it does affect version 39, and even beta 40.0. I don't know why someone told that it was not appearing, because after just 1 minute of testing I could crash FF. Cheers
Yes, it does affect version 39. This is the signature: bp-0ea63e95-addd-48e2-ab4b-7471b2150720
Let's try to see if we can get this fixed for ESR38 release. Naveed, can you please help find an owner for this issue? We are hoping to get a fix soon. Thanks!
Victor, iannakin, anyone: it'd be incredibly useful if one of us at Mozilla could get access to a page that crashes reliably so we can debug this. A remote desktop session might also work, but we'd probably have to install some developer stuff so it'd be more time consuming.
Victor gave me access to a crashing website and I was able to reproduce this on Windows \o/ After spending some quality time with Visual Studio, it looks like the Java plugin is deallocating nsJSObjWrapper on a thread other than the main thread. nsJSObjWrapper's Heap<JSObject*> member is not happy with this scheme: we don't remove the store buffer entry if the Heap<JSObject*> is destructed on another thread, so the next GC will crash because it has a bogus CellPtrEdge :( While testing this, I also sometimes got it to fail the assert in isOkayToUseBuffer, when the main thread is doing a GC while we're destructing our Heap member. The plugin code right now breaks the single-threaded-JSRuntime invariant and I'm not sure how to fix it. It seems we could either (1) MOZ_CRASH if we're on a different thread, and let Oracle et al deal with it (can we get away with this?) or (2) avoid deallocating the nsJSObjWrappers immediately but flag them and deallocate them later on the main thread (the plugin code is really complicated, no idea if this is feasible..).
Hmm. I don't really know that much about this code (ccing some people who might), but I assume that it's nsJSObjWrapper::NP_Deallocate that's being called on the wrong thread? I see nothing in principle impossible about handing that delete call to the main thread; the question is whether anything relies on the NP_Invalidate in its destructor happening eagerly, I guess. I assume the OnWrapperDestroyed bit is ok to be called a bit later when the wrapper is really destroyed.
(In reply to Boris Zbarsky [:bz] from comment #26) > but I assume that it's nsJSObjWrapper::NP_Deallocate that's being > called on the wrong thread? Yes, the plugin calls _releaseObject and that calls NP_Deallocate. _releaseObject has a log message when it's called on the wrong thread, but that's easy to ignore: https://hg.mozilla.org/mozilla-central/file/888e8026ed60/dom/plugins/base/nsNPAPIPlugin.cpp#l1394 If _releaseObject could set a flag indicating the nsJSObjWrapper can be destroyed, then presumably DelayedReleaseGCCallback could do the actual deallocation, it already has some similar code.. We'd have to be careful to ignore any flagged nsJSObjWrapper entries when we do hash table lookups though, there are at least two such tables. It would be great if someone familiar with the plugin code could take a look.
Benjamin, do you have any ideas? You've looked at some "plugins doing stuff on the wrong thread" stuff recently. Thanks.
Flags: needinfo?(nihsanullah) → needinfo?(benjamin)
Hi All, Are there any news on this bug? Will be solved in ESR 38.2 or 38.3? We have recommended to our community of hundreds of organizations and many thousands of users to use Firefox ESR. It was good for us because the community avoid this error from version 35 to 40 because we have firefox ESR 31 working perfectly. Now with the last version ESR31.8 Now we have a complicated situation because I think that tomorrow the version Firefox ESR 38.2 will be released and the users with ESR 31.8 will be updated to ESR 38.2. If this bug still exists in ESR38.2 we have to recommend to our community to disabled updates and stay in ESR 31.8 with the security risk associated (only a temporary solution) or migrate to Internet Explorer 11 as the final solution. We have critical use cases of our business affected by this bug, so any information or workaround to avoid them will be Thanks in advance Luis
(In reply to Jan de Mooij [:jandem] from comment #25) > It seems we could either (1) MOZ_CRASH if we're > on a different thread bsmedberg suggested we do this. I'll work on a patch.
We should have a main-thread check in _releaseObject (NPN_ReleaseObject) and crash if it's called on the wrong thread. Also this is clearly a bug in the Java plugin and if you can reproduce it you should report it to Oracle support.
This patch adds a MOZ_CRASH to _releaseObject. I confirmed we hit this crash on the page where I can reproduce the problem. I kept the NPN_PLUGIN_LOG call, because MOZ_CRASH doesn't print anything in opt builds and the logging might be useful to someone.
Assignee: nobody → jdemooij
Status: NEW → ASSIGNED
Attachment #8646867 - Flags: review?(benjamin)
(In reply to Benjamin Smedberg [:bsmedberg] from comment #31) > Also this is clearly a bug in the > Java plugin and if you can reproduce it you should report it to Oracle > support. I reported it to Oracle (http://bugreport.java.com) Let's hope they take it seriously and get back to us soon.
Hi! Are you planning to give to the community any workarround before the answer of Oracle? or are you waiting for Oracle? Do you have any ideas of the planning of the solution? Thanks in advance, Luis
Comment on attachment 8646867 [details] [diff] [review] Patch This is going to cause user regressions :-( I wonder if we should move Java back out of process at the same time.
Attachment #8646867 - Flags: review?(benjamin) → review+
(In reply to Benjamin Smedberg [:bsmedberg] from comment #35) > I wonder if we should move Java back out of process at the same time. I've no idea how that works so someone else would have to do that. Let me know if we should wait for that, else I'll land this on Monday.
Attached a patch to move Java OOPP in bug 874176. I propose that they land together.
(In reply to Benjamin Smedberg [:bsmedberg] from comment #37) > Attached a patch to move Java OOPP in bug 874176. I propose that they land > together. Bug 874167 seems to have stuck so I pushed this: https://hg.mozilla.org/integration/mozilla-inbound/rev/51f188b7abd8
OK, so as I understand this, now Firefox 43 users (and ESR versions built from 43, if we take this patch and the one in bug 874167 for the next ESR) will still crash sometimes when using Java applets, but the crash will be safer or potentially less exploitable. Benjamin do you want to request uplift for this to 42 or further? If we do that, would it need both patches? It's hard to tell here what the tradeoff would be for ESR, though I'd lean toward crashing more often but more safely.
On the non-ESR branches, we should take both patches at least to 42. I don't know what to do about ESR: both patches will have some compatibility issues with Java hangs or crashes. Moving Java out-of-process will keep Firefox alive but Java's going to keep crashing regardless of what we do here. That fix has to come from Oracle.
Donald, I think you should be aware of this as well as bug 874167.
Jan can you nominate for uplift?
Comment on attachment 8646867 [details] [diff] [review] Patch Nominating for Aurora to match bug 874167. Approval Request Comment [Feature/regressing bug #]: Java plugin issue. [User impact if declined]: Plugin crashes, maybe security related. This patch makes us crash immediately before things go off the rails. [Describe test coverage new/current, TreeHerder]: On m-c for a few days. [Risks and why]: More Java plugin crashes. [String/UUID change made/needed]: None.
Attachment #8646867 - Flags: approval-mozilla-aurora?
Comment on attachment 8646867 [details] [diff] [review] Patch Make sense to take it in 42. Thanks
Attachment #8646867 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
Let's look at uplifting this again for ESR 38.4.0, since it's not in beta 41.
Jan, do you think we should consider fixing this in ESR38.4.0? Based on comment 41, it seems moving the java plug-in out of proc (even though it keeps crashing) is better than Firefox crashing. Thoughts?
(In reply to Ritu Kothari (:ritu) from comment #49) > Jan, do you think we should consider fixing this in ESR38.4.0? Based on > comment 41, it seems moving the java plug-in out of proc (even though it > keeps crashing) is better than Firefox crashing. Thoughts? The patch here is trivial to backport, but we'd also have to backport bug 874167, so I'll forward this to bsmedberg.
Flags: needinfo?(jdemooij) → needinfo?(benjamin)
Moving Java OOP will cause compatibility issues for enterprise users. We should probably not mess with ESR unless there is no other way to fix the security issue.
I looked at the crash reports for this one and the product breakdown indicates this is showing up on Thurderbird 38.3.0 with a total of 19 crashes. Benjamin, given the complexity of fixing this in esr38 (no straight forward fix), do you suggested we wontfix this or leave it open and wait for more data from ESR38.4.0?
What more data? The question from comment 51 was about security evaluation, not about data.
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #48) > Let's look at uplifting this again for ESR 38.4.0, since it's not in beta 41. Here we are and it didn't make it...
This never got a security rating (partly my fault as I didn't notice this lacked a rating or sec-approval). Breaking compatibility in some unknown way sounds worse to me than this crash. How bad is it that we fixed this without fixing it in ESR? (thus potentially revealing the vulnerability)? Is there another way we could address the security issue without moving Java OOP? I don't want to land this on ESR and cause possibly worse regressions for the people already crashing and potentially everyone else using ESR. Is this worth doing more investigation or trying to figure out what this and the patch from bug 874167 would break? With all these questions still up in the air, I am leaning towards the thought we need to bother Oracle more to fix this on their side, and keep wontfixing this for ESR, though that is depressing.
Hi, does it mean that this issue won't be fixed in ESR 38.4.0?
Victor, it seems unlikely as we're making final ESR builds real soon now.
Jan, should we still be backporting this to ESR if we don't also take bug 874167 (Which Benjamin says we should not take in ESR)? Will it at least fix the security issue (even if we still crash)?
Comment on attachment 8646867 [details] [diff] [review] Patch Nominating for ESR38 as discussed. [Approval Request Comment] User impact if declined: Security issues. Fix Landed on Version: Backported to 42+ Risk to taking this patch (and alternatives if risky): More (intentional) browser crashes when plugins misbehave. String or UUID changes made by this patch: None.
Attachment #8646867 - Flags: approval-mozilla-esr38?
Comment on attachment 8646867 [details] [diff] [review] Patch May be crashier but more secure crashes anyway. Approved for uplift to esr38.
Attachment #8646867 - Flags: approval-mozilla-esr38? → approval-mozilla-esr38+
Hi All, As I explained we recommended to our community of hundreds of organizations and many thousands of users to use Firefox ESR. It was good for us because the community avoid this error from version 35 to 40, because we have firefox ESR 31 working perfectly until ESR31.8 Now we have a complicated situation after Firefox ESR 38.2 was published and Firefox is asking for updates and there are no security updates for ESR31. Administrators are working to achieve users continue with ESR31.8 but this situation is bad because of security. Our solutions are: -- recommend to our community to disabled updates and stay in ESR 31.8 with the security risk associated (only a temporary solution). We are in this situation Now -- or migrate to Internet Explorer 11 or Edge as the final solution for clients that doesn’t accept the risk We have critical use cases of our business affected by this bug, so any information or workaround to avoid them will be grateful. We were wating for a possible solution in 38.4 but if you won’t fix in 38.4 our problem is bigger. For us: -- there is an important regression between 31 ESR and 38 ESR -- Is it possible to collaborate in anyway? ---- Maybe you can build a beta version with this changes and our team tester can verify if with this fix our regression disappears. Maybe we are wating for nothing. -- Maybe you can create a parameter to control this issue. You can introduce this patch but disabled by default and we can enabled with a parameter in [about:config] I think is better for security and disponibility of firefox to execute JAVA OOP. Thanks in advance, Luis
Luis, the fix will be in ESR 38.4.0. I haven't built it yet. It should release next Tuesday.
Thanks it's good news for us We are going to test the 38.4 next Tuesday
Can you please verify that the issue is fixed on Firefox 42 and on ESR 38.4.0? I cannot reproduce the initial issue and thus I can't verify it.
Luis, if you want to give it a try now, here is the current release candidate for ESR 38.4.0, hope that helps. https://ftp.mozilla.org/pub/firefox/candidates/38.4.0esr-candidates/build2/
I have tested the release candidate and still crashes, but with a different signature (mozilla::plugins::parent::_releaseobject) This is the ID: ba750b9c-8688-4e95-b3f3-994292151029
Sadly, It's the same as Victor, Firefox still crash. Can we help you in anyway?
Jan had access to our application where he was able to reproduce the issue. Do you still have access Jan? Were you able to reproduce the problem with this new version? Liz, if you need access too I can send you the instructions to connect and reproduce the problem. If there's something more that we can do for you, please tell us. Thanks.
(In reply to Victor from comment #67) > I have tested the release candidate and still crashes, but with a different > signature (mozilla::plugins::parent::_releaseobject) The patch in this bug just makes the browser crash when the Java plugin misbehaves, before it can corrupt memory and cause crashes or security issues later on. So unfortunately that crash is expected. It's up to Oracle to fix the Java plugin. Bug 874167 moved the Java plugin out-of-process, so that only the plugin crashes and not the whole browser. That patch wasn't backported to ESR 38 though.
Thanks Jan. Do you know if Bug 874167 can be included in ESR 38.4.0?
No, we will not include bug 874167 into the ESR because it has more serious compatibility concerns.
What was the bug ID for the bug submitted to java.com? Also, can someone provide a link to an example that causes the crash? If one is provided in the bug report then that should be fine, but I need the JI number.
I found the JBS issue: https://bugs.openjdk.java.net/browse/JDK-8133523 A sample would be very helpful in evaluating this.
David, Would help you to connect remotely to our systems to evaluate directly this issue?
(In reply to Luis from comment #75) > Would help you to connect remotely to our systems to evaluate directly this > issue? Victor, Luis: it'd be great if you could help David reproduce this crash so they can confirm their Java fix works :)
I have just sent an email to David with the instructions to reproduce the issue. David, did you receive the email? If you need something more, please tell me. Thanks.
Flags: needinfo?(victor.bail) → needinfo?(david.dehaven)
(In reply to Victor from comment #77) > David, did you receive the email? If you need something more, please tell me. I did, thank you.
(In reply to Luis from comment #75) > Would help you to connect remotely to our systems to evaluate directly this > issue? Remote access through our WAN is usually met with frustration and anger. I'll try Victor's link first and let you know.
As per my comment on Bug 1221448 Oracle is aware of the issue specific to 38.4, and deciding the best way to make that available. The timeframe is likely 'days', not 'week(s)'.
/s/that/a fix to the JRE/
FWIW, the JRE fix for this should now be available, once we have more feedback that the build we're making available does indeed fully address the issue, we'll push more broadly. Details below: You can now get version 1.8.0_66-b18, of the JRE or JDK, here: http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html Note that you want b18, and you want 8u66, *not* 8u55. Also this is Windows only. Build 1.8.0_66-b18 contains the fix for this issue. Be sure to check double check after install with "java -version", or after accepting the license you'll notice the url path for the download includes -b18 instead of -b17. It is possible you may hit a cache with -b17 still out there. We'd greatly appreciate your help testing this out. Once we've had more public validation of this fix, we'll push more aggressively, likely very early next week. If you experience any issues, please report at java.com/report FWIW, release notes are here: http://www.oracle.com/technetwork/java/javase/8u66-relnotes-2692847.html
Hey, how could I write something like that and not have at least one mistake. When I say you "want 8u66, *not* 8u55", I really meant you do not want "8u65"... To be more precise, this fix is in 8u66-b18 Windows only on OTN for now...
I have tested our web application with java 8u66-b18 and now it's working fine. Thank You!
Our first test has been positive too with 8u66-b18, thanks !!
Hi Donald, do you know when will be available the fix for all users? Thanks.
Oracle development has a fix that is under code review and test currently. Assuming the fix passes regression testing and security reviews we will slot this into the earliest release possible. We're concerned with compounding issues here, so want to do more extensive testing. In the meantime we would recommend the FF ESR or other versions where this is known to work.
You need to log in before you can comment on or make changes to this bug.