Last Comment Bug 492492 - 3.5b4 crash [@ libawt.jnilib@0x11e1 ]
: 3.5b4 crash [@ libawt.jnilib@0x11e1 ]
Status: RESOLVED FIXED
: crash, fixed1.9.1, topcrash+
Product: Plugins Graveyard
Classification: Graveyard
Component: Java (Java Embedding Plugin) (show other bugs)
: unspecified
: All Mac OS X
: -- critical
: ---
Assigned To: Steven Michaud [:smichaud] (Retired)
:
Mentors:
: 493017 493898 (view as bug list)
Depends on: 493629
Blocks:
  Show dependency treegraph
 
Reported: 2009-05-11 23:01 PDT by chris hofmann
Modified: 2016-04-28 09:08 PDT (History)
9 users (show)
mbeltzner: blocking1.9.1+
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
Screenshot of JavaFX install dialog (157.26 KB, image/png)
2009-05-12 15:16 PDT, Steven Michaud [:smichaud] (Retired)
no flags Details
SystemExit Java applet source (652 bytes, text/plain)
2009-05-13 16:34 PDT, Steven Michaud [:smichaud] (Retired)
no flags Details
Good stack for this bug's crashes (12.88 KB, text/plain)
2009-05-14 13:03 PDT, Steven Michaud [:smichaud] (Retired)
no flags Details

Description chris hofmann 2009-05-11 23:01:01 PDT
bouncing around #7-#14 top crash for fx3.5b4

associated with a variety of download sites.
14	http://downloads.garmin.com/Garmin_RMU_CNNANT2010C.dmg
7	http://downloads.garmin.com/Garmin_rmu_cneunt2010.dmg
2	http://www.downloadsquad.com/timewasters/

and mac users trying to download windows 7

http://downloads.technet.microsoft.com/download/release/msdn/en_windows_7_ultimate_rc_x64_dvd_347803.iso?lcid=1033&MTBPD=bf1558ca-830f-492a-9bea-b4b225a8e1bb&RURL=http%3a%2f%2ftechnet.microsoft.com%2fen-us%2fsubscriptions%2fdefault.aspx&__gda__=1243862420 

I couldn't reproduce any of these or several other urls from crash reports

next step might be to dig out version info for the library out of the module list

stack trace looks like:

0  	libawt.jnilib  	libawt.jnilib@0x11e1  	
1 	libawt.jnilib 	libawt.jnilib@0x40ab 	
2 	libawt.jnilib 	libawt.jnilib@0x3fff 	
3 	libobjc.A.dylib 	libobjc.A.dylib@0x68b7 	
4 	libobjc.A.dylib 	libobjc.A.dylib@0x678b 	
5 	libobjc.A.dylib 	libobjc.A.dylib@0x678b 	
6 	libobjc.A.dylib 	libobjc.A.dylib@0x678b 	
7 	libobjc.A.dylib 	libobjc.A.dylib@0x678b 	
8 	libobjc.A.dylib 	libobjc.A.dylib@0x5238 	
9 	libobjc.A.dylib 	libobjc.A.dylib@0x156d5 	
10 	XUL 	-[ToolbarWindow sendEvent:] 	widget/src/cocoa/nsCocoaWindow.mm:1959
11 	AppKit 	AppKit@0xdb9fc 	
12 	JavaEmbeddingPlugin 	JavaEmbeddingPlugin@0x12f00 	
13 	XUL 	nsAppShell::ProcessNextNativeEvent 	widget/src/cocoa/nsAppShell.mm:636
14 	XUL 	nsBaseAppShell::DoProcessNextNativeEvent 	widget/src/xpwidgets/nsBaseAppShell.cpp:151
15 	XUL 	nsBaseAppShell::OnProcessNextEvent 	widget/src/xpwidgets/nsBaseAppShell.cpp:278
16 	XUL 	nsAppShell::OnProcessNextEvent 	widget/src/cocoa/nsAppShell.mm:789
17 	XUL 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:497
18 	XUL 	NS_ProcessPendingEvents_P 	nsThreadUtils.cpp:180
19 	XUL 	nsBaseAppShell::NativeEventCallback 	widget/src/xpwidgets/nsBaseAppShell.cpp:121
20 	XUL 	nsAppShell::ProcessGeckoEvents 	widget/src/cocoa/nsAppShell.mm:405
21 	CoreFoundation 	CoreFoundation@0x72614 	
22 	CoreFoundation 	CoreFoundation@0x72cf7 	
23 	HIToolbox 	HIToolbox@0x2f47f 	
24 	HIToolbox 	HIToolbox@0x2f1d1 	
25 	HIToolbox 	HIToolbox@0x2f10c 	
26 	AppKit 	AppKit@0x403ec 	
27 	AppKit 	AppKit@0x3fc9f 	
28 	JavaEmbeddingPlugin 	JavaEmbeddingPlugin@0x12fc2 	
29 	JavaEmbeddingPlugin 	JavaEmbeddingPlugin@0x6f29 	
30 	JavaEmbeddingPlugin 	JavaEmbeddingPlugin@0x25dc3 	
31 	MRJPlugin 	MRJPlugin@0x3de4 	
32 	MRJPlugin 	MRJPlugin@0xbb4a 	
33 	MRJPlugin 	MRJPlugin@0x24dc 	
34 	MRJPlugin 	MRJPlugin@0x268f 	
35 	XUL 	CreateProxyJNI 	modules/oji/src/ProxyJNI.cpp:1744
36 	XUL 	JVM_GetJNIEnv 	modules/oji/src/jvmmgr.cpp:289
37 	XUL 	create_java_vm_impl 	modules/oji/src/lcglue.cpp:342
38 	XUL 	jsj_ConnectToJavaVM 	js/src/liveconnect/jsj.c:474
39 	XUL 	JSJ_AttachCurrentThreadToJava 	js/src/liveconnect/jsj.c:698
40 	XUL 	map_js_context_to_jsj_thread_impl 	modules/oji/src/lcglue.cpp:180
41 	XUL 	jsj_EnterJava 	js/src/liveconnect/jsj_utils.c:465
42 	XUL 	JavaPackage_resolve 	js/src/liveconnect/jsj_JavaPackage.c:199
43 	libmozjs.dylib 	js_LookupPropertyWithFlags 	js/src/jsobj.cpp:3974
44 	libmozjs.dylib 	js_GetPropertyHelper 	js/src/jsobj.cpp:4332
45 	libmozjs.dylib 	js_Interpret 	js/src/jsinterp.cpp:4409
46 	libmozjs.dylib 	js_Execute 	js/src/jsinterp.cpp:1599
47 	libmozjs.dylib 	JS_EvaluateUCScriptForPrincipals 	js/src/jsapi.cpp:5145
48 	XUL 	nsJSContext::EvaluateString 	dom/src/base/nsJSEnvironment.cpp:1603
49 	XUL 	nsGlobalWindow::RunTimeout 	dom/src/base/nsGlobalWindow.cpp:7763
50 	XUL 	nsGlobalWindow::TimerCallback 	dom/src/base/nsGlobalWindow.cpp:8116
51 	XUL 	nsTimerImpl::Fire 	xpcom/threads/nsTimerImpl.cpp:420
52 	XUL 	nsTimerEvent::Run 	xpcom/threads/nsTimerImpl.cpp:512
53 	XUL 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:510
54 	XUL 	NS_ProcessPendingEvents_P 	nsThreadUtils.cpp:180
55 	XUL 	nsBaseAppShell::NativeEventCallback 	widget/src/xpwidgets/nsBaseAppShell.cpp:121
56 	XUL 	nsAppShell::ProcessGeckoEvents 	widget/src/cocoa/nsAppShell.mm:405
57 	CoreFoundation 	CoreFoundation@0x72614 	
58 	CoreFoundation 	CoreFoundation@0x72cf7 	
59 	HIToolbox 	HIToolbox@0x2f47f 	
60 	HIToolbox 	HIToolbox@0x2f1d1 	
61 	HIToolbox 	HIToolbox@0x2f10c 	
62 	AppKit 	AppKit@0x403ec 	
63 	AppKit 	AppKit@0x3fc9f 	
64 	AppKit 	AppKit@0x38cda 	
65 	XUL 	nsAppShell::Run 	widget/src/cocoa/nsAppShell.mm:716
66 	XUL 	nsAppStartup::Run 	toolkit/components/startup/src/nsAppStartup.cpp:193
67 	XUL 	XRE_main 	toolkit/xre/nsAppRunner.cpp:3298
68 	firefox-bin 	main 	browser/app/nsBrowserApp.cpp:156
69 	firefox-bin 	firefox-bin@0x1541 	
70 	firefox-bin 	firefox-bin@0x1468 	
71 		@0x1
Comment 1 chris hofmann 2009-05-11 23:19:51 PDT
one of the reported crash urls is http://javafx.com/samples/

I hit a crash by going there, then clicking on calendar app, then back, and hit the "interesting photos" in the upper left corner -> boom!

for some reason I didn't get the crash reporter.
Comment 2 chris hofmann 2009-05-11 23:23:01 PDT
on the second attempt to try the same steps in comment #1 I wasn't able to reproduce
Comment 3 chris hofmann 2009-05-11 23:33:22 PDT
there is another large download file (300MB) that one user was downloading when they hit the crash.

http://downloadns.citrix.com.edgesuite.net/3982/FREE_XenServer-5.0.0-Update3-install.iso

also seen while visiting a few banks
irefox	3.5b4	8	http://danskebank.dk/da-dk/Pages/default.aspx	libawt.jnilib@0x11e1
Firefox	3.5b4	2	https://www.usaa.com/inet/bank_deposit/BkHomeDeposit	libawt.jnilib@0x11e1
Firefox	3.5b4	1	http://www.danskebank.dk/da-dk/Pages/default.aspx	libawt.jnilib@0x11e1
Comment 4 Samuel Sidler (old account; do not CC) 2009-05-12 00:37:53 PDT
It's very useful to have a URL that includes a list of recent crashes at that stack. Please do so in the future.

http://crash-stats.mozilla.com/report/list?product=Firefox&version=Firefox%3A3.5b4&query_search=signature&query_type=exact&query=&date=&range_value=1&range_unit=weeks&do_query=1&signature=libawt.jnilib%400x11e1

Note: This seems to only happen on x86 (note PPC) and only one report was using 10.5.5, the rest were using 10.5.6 (with a lone 10.5.7 in there too).

Before you ask, no, this doesn't appear to be the "Java crashes" bug, in that no reports I looked at have exception notes, which bug 492251 has. Steven can feel free to correct me though.
Comment 5 Steven Michaud [:smichaud] (Retired) 2009-05-12 09:12:01 PDT
Chris:  When you followed your STR from comment #1, were you prompted
to install something called JavaFX?  And if so, did you allow it to be
installed?

I'd also like to know if it has an uninstaller.  I haven't yet dared
to install it on my main partition, and am currently debating creating
a "disposable" partition on which to test with it.
Comment 6 chris hofmann 2009-05-12 09:24:49 PDT
> Chris:  When you followed your STR from comment #1, were you prompted
> to install something called JavaFX?

no prompt to install javafx when I visited http://javafx.com/samples/

it just showed me a 5x4 matrix of images/apps and I started clicking around on them.
Comment 7 Steven Michaud [:smichaud] (Retired) 2009-05-12 09:32:49 PDT
You might already have JavaFX installed.

But I don't yet know how to tell this.
Comment 8 Steven Michaud [:smichaud] (Retired) 2009-05-12 12:00:48 PDT
The plot thickens ...

So far I've only tested without installing JavaFX, and only at the URL
from comment #1 (http://javafx.com/samples/).  But I've found a way to
make the browser quit abnormally *without* a crash, in multiple
browsers and on multiple platforms.

1) Visit http://javafx.com/samples/.

   In my tests, I always waited for the page to finish loading.

2) Click on "InterestingPhotos" in the upper left.

   After a while you will be asked if you want to trust a signed Java
   applet.  (On Windows and Linux a Java dialog will ask you if you
   want to run a signed application.)

   (In IE on Windows XP, the first time you take these steps the above
   dialog is preceded by one asking if you want to trust/run an
   ActiveX control.)

   If you're asked to run a signed application, first uncheck the
   "Always trust content from this publisher" box.

   Choose to trust the signed applet (or run the signed application).

3) After a while (perhaps 30 seconds or longer) another dialog will
   appear, asking if you want to install the JavaFX Desktop Runtime.

   Click the Cancel button (meaning you don't want to install JavaFX).

4) Press the back button.

   After a few seconds the browser should quit.

   When I did this with a custom build under gdb (a current
   Shiretoko build running on OS X 10.5.6), gdb didn't catch any crash
   (it said "Program exited normally").

   But if the next copy of FF I ran was a downloaded distro, I got the
   "Restore Previous Session" dialog.

I've reproduced these steps with FF 3.0.X on OS X (10.5.6), Windows
and Linux, with FF 3.5 on OS X 10.5.6, with Safari on OS X (10.5.6)
and with IE 7 on Windows XP.
Comment 9 Steven Michaud [:smichaud] (Retired) 2009-05-12 12:05:06 PDT
Forgot to mention that I can also reproduce comment #8's STR with Firefox 2.0.0.X on both OS X and Windows.
Comment 10 Steven Michaud [:smichaud] (Retired) 2009-05-12 12:30:01 PDT
None of the other URLs listed above in this bug are at all helpful.
They're either inaccessible, inscruitable, or don't launch Java when
invoked
(http://downloadns.citrix.com.edgesuite.net/3982/FREE_XenServer-5.0.0-Update3-install.iso
belongs to the last category).
Comment 11 Steven Michaud [:smichaud] (Retired) 2009-05-12 13:47:16 PDT
I've now installed JavaFX (on my disposable partition) and have been
playing around with it for a while.

To my considerable surprise, I can't find any problems with it.  For
example I'm not able to reproduce the crashes reported in comment #1.

JavaFX doesn't have an uninstaller (that I can find).  And I haven't
been able to figure out where it installed its files.  That's a bit
disturbing ... but I'm afraid fairly typical of Java on OS X
(including Apple's own JVMs).

So the only way I've found of telling whether or not you have it
installed is as follows:

1) Visit a page containing Java applets that require JavaFX
   (e.g. http://javafx.com/samples/).

2) Click on a representation of one of the applets
   (e.g. InterestingPhotos in the upper left).

   You'll be prompted once or twice to trust a signed Java applet.  If
   you do, you should eventually see a Java applet displayed on the
   page.  (One way to tell it's a Java applet is to scroll or resize
   the page -- it will temporarily disappear while you do this (this
   is a limitation of the JEP).)

So Chris, do you have JavaFX installed?
Comment 12 Steven Michaud [:smichaud] (Retired) 2009-05-12 13:51:35 PDT
It appears that JavaFX requires the use of signed applets.  If true, I
think this is a severe design flaw -- signed applets can do anything
to your computer (if you "trust" them), including deleting all the
files belonging to the user you've signed in as.

The whole point of Java applets is the sandbox they run in.  Break
down the sandbox and you might as well be using ActiveX controls :-)
Comment 13 Steven Michaud [:smichaud] (Retired) 2009-05-12 14:01:14 PDT
(Following up comment #11)

I should add that, once I installed JavaFX, I could no longer
reproduce the "abnormal quits" I describe in comment #8.
Comment 14 chris hofmann 2009-05-12 14:06:44 PDT
yes, I've seen the trust dialogs and clicked ok to them.  I also see the applet and can watch it go away on resizing the window.
Comment 15 Steven Michaud [:smichaud] (Retired) 2009-05-12 14:12:05 PDT
I'm afraid we're at a dead end here ... until we can get more information.
Comment 16 Steven Michaud [:smichaud] (Retired) 2009-05-12 14:17:24 PDT
(From comment #1)

> for some reason I didn't get the crash reporter.

This sounds like one of my "abnormal quits without a crash".

Could the "crashes" you describe in comment #1 have happened before you
installed JavaFX?

Can you still reproduce them?
Comment 17 chris hofmann 2009-05-12 14:50:06 PDT
I can't reproduce now, but I don't recall anything that looked like a javafx install.  

I'm going through a longer list of urls now, but still no luck hitting a crash.
Comment 18 Steven Michaud [:smichaud] (Retired) 2009-05-12 15:16:35 PDT
Created attachment 377014 [details]
Screenshot of JavaFX install dialog
Comment 19 chris hofmann 2009-05-12 15:35:43 PDT
ok, maybe it's possible I saw the dialog in the comment 18 attachment late last night.

If I did, then we might be getting a bit closer.

It would seem you need to not have javafx installed, 
run the steps in comment 8, 
then try to somehow get a crash instead of a "quit" like Steven and I saw

we either need to find a way for Steven and/or I to uninstall javafx or recruit others to join in the fun here to reproduce.
Comment 20 Steven Michaud [:smichaud] (Retired) 2009-05-12 15:38:03 PDT
I've finally figured out how to "uninstall" JavaFX on OS X ... sort
of:

Delete the following two files from ~/Library/Preferences/

  .javafx_eula_accepted
  .javafx_ping_sent

Once you've done this, you'll once more be able to reproduce all the
steps of comment #8's STR (including the "abnormal quits without a
crash").
Comment 21 Steven Michaud [:smichaud] (Retired) 2009-05-12 19:27:16 PDT
I've found that if you run an opt build of Shiretoko (with debug
symbols) in gdb and set a breakpoint on 'exit', you'll hit that
breakpoint at step 4 of my comment #8 STR.

exit() is called (indirectly) from JVM_RaiseSignal() on a secondary
thread (presumably a Java thread).  It appears that JVM_RaiseSignal()
sends a SIGTERM signal to Java's (and the browser's) own process.
(For some reason you can't break on 'JVM_RaiseSignal' itself, or I'd
know for sure what signal it sends.)

Here are two examples of stacks.  From their differences you can see
that the symbols between 'exit' and 'JVM_RaiseSignal' are spurious (I
don't know why):

#0  0x94159bd5 in exit ()
#1  0x1e2087f3 in dyld_stub_vImageLookupTable_Planar8toPlanarF ()
#2  0x1e208752 in dyld_stub_vImageLookupTable_Planar8toPlanarF ()
#3  0x1e208618 in dyld_stub_vImageLookupTable_Planar8toPlanarF ()
#4  0x1e1aac1d in dyld_stub_vImageLookupTable_Planar8toPlanarF ()
#5  0x1e1aa937 in dyld_stub_vImageLookupTable_Planar8toPlanarF ()
#6  0x1e412935 in JVM_RaiseSignal ()
#7  0x941ad204 in _pthread_body ()

#0  0x94159bd5 in exit ()
#1  0x1dfe97f3 in dyld_stub_strlen ()
#2  0x1dfe9752 in dyld_stub_strlen ()
#3  0x1dfe9618 in dyld_stub_strlen ()
#4  0x1df8bc1d in dyld_stub_strlen ()
#5  0x1df8b937 in dyld_stub_strlen ()
#6  0x1e1f3935 in JVM_RaiseSignal ()
#7  0x941ad204 in _pthread_body ()

I think this shows that one of the JavaFX applets (which as you'll
remember are signed, and can do anything) is killing Java's (and the
browser's) process.
Comment 22 Steven Michaud [:smichaud] (Retired) 2009-05-12 19:35:57 PDT
Actually I now think the 'JVM_RaiseSignal' symbol is also spurious --
this'd be why you can't break on it.  The same symbol appears near the
bottom of the stacks of many Java threads when you break and do
'thread apply all bt' while Java is running.

But the 'exit' symbol clearly isn't spurious.  And I still think my
stacks from comment #21 show that one of the JavaFX applets is killing
Java's (and the browser's) process.
Comment 23 Steven Michaud [:smichaud] (Retired) 2009-05-12 19:38:33 PDT
Presumably one of the JavaFX applets calls System.exit().
Comment 24 Steven Michaud [:smichaud] (Retired) 2009-05-13 10:04:46 PDT
I've decompiled the main JavaFX applet (javafx-rt__V1.1.1.jar) and
found that it does call System.exit().

Here's the exit() method of its
com.sun.javafx.runtime.provider.AWT_EDT_RuntimeProvider class:

    public void exit()
    {
        try {
            if(WindowImpl.getApplet() == null)
                System.exit(0);
        }
        catch(Throwable throwable) { }
        WindowImpl.disposeAll();
    }

My next step will be to write and post a signed applet that does
nothing but call System.exit().  We can use it first to try to
reproduce the "abnormal quits without a crash" that I describe in
comment #8, then to try to reproduce the libawt.jnilib crashes.
Comment 25 Steven Michaud [:smichaud] (Retired) 2009-05-13 16:34:29 PDT
Created attachment 377286 [details]
SystemExit Java applet source

As promised, here's a signed Java applet that does nothing but call
System.exit() (from its start() method).

http://people.mozilla.com/~stmichaud/bmo/SystemExit.html

If you visit this URL, you'll be asked to "trust" the applet.  If you
choose not to, you'll get an access denied error (since ordinary
applets aren't allowed to call System.exit()).  If you do "trust" the
applet, the browser will terminate immediately afterwards.

You'll need to turn on the Java Console to see the "access denied"
error, using Applications : Utilities : Java : Java Preferences.
Comment 26 timeless 2009-05-14 02:01:58 PDT
steven: if you make a LD_PRELOAD_LIBRARY which provides <exit>, what happens when the exit() function is called? does jvm mind it not exiting?
Comment 27 Steven Michaud [:smichaud] (Retired) 2009-05-14 08:31:41 PDT
(In reply to comment #26)

I didn't know about LD_PRELOAD.  I'll add it to my bag of tricks.

But I don't think it will work in this case, because the OS X
equivalent (DYLD_INSERT_LIBRARIES) seems to require a flat namespace
-- which is rarely used these days.

Here's a code snippet I found that contains a lot of information about
LD_PRELOAD and friends:

http://www.opensource.apple.com/source/sudo/sudo-28/sudo/env.c

    /*
     * Preload a noexec file?  For a list of LD_PRELOAD-alikes, see
     * http://www.fortran-2000.com/ArnaudRecipes/sharedlib.html
     * XXX - should prepend to original value, if any
     */
    if (noexec && def_noexec_file != NULL) {
#if defined(__darwin__) || defined(__APPLE__)
    insert_env(format_env("DYLD_INSERT_LIBRARIES", def_noexec_file, VNULL), 1);
    insert_env(format_env("DYLD_FORCE_FLAT_NAMESPACE", VNULL), 1);
#else
# if defined(__osf__) || defined(__sgi)
  insert_env(format_env("_RLD_LIST", def_noexec_file, ":DEFAULT", VNULL), 1);
# else
  insert_env(format_env("LD_PRELOAD", def_noexec_file, VNULL), 1);
# endif
#endif
    }
Comment 28 Steven Michaud [:smichaud] (Retired) 2009-05-14 08:37:48 PDT
(Following up comment #27)

In any case, though, I've now hit the libawt crash myself, in
circumstances that indicate it's not related to the JavaFX problem.
It hit it just once, and it isn't reproducible.  But I got a good
stack, it seems to be an Apple bug, and I have hopes that I'll be able
to work around it in the JEP.

I won't say any more until I've had a chance to investigate further.

Once I'm not quite so busy, I'll open a new bug on the JavaFX problem,
and report it to Sun.
Comment 29 Mike Beltzner [:beltzner, not reading bugmail] 2009-05-14 12:14:35 PDT
*** Bug 493017 has been marked as a duplicate of this bug. ***
Comment 30 Mike Beltzner [:beltzner, not reading bugmail] 2009-05-14 12:20:02 PDT
FWIW, this is hitting some people who are trying to download the Windows 7 RC. It was nominated for blocking there, so I'm carrying it over. It worked for me, fwiw.
Comment 31 Steven Michaud [:smichaud] (Retired) 2009-05-14 13:03:30 PDT
Created attachment 377501 [details]
Good stack for this bug's crashes

Here's the "good stack" I talked about in comment #28.  Notice that
the addresses in system libraries (like libawt.jnilib) are
(accurately) translated into symbols.

Unlike the Breakpad stacks, it also tells me what kind of crash it was
-- in this case a NULL dereference.

This crash happened at http://javafx.com/samples/, just after I'd
clicked on InterestingPhotos (in the upper left) and let go the mouse
button (while I was running a custom build of Shiretoko in gdb).  I
didn't have JavaFX installed when this happened, but I think these
crashes can happen with or without JavaFX installed.

The crash happens as the JVM is being loaded (another reason to think
JavaFX isn't involved).  The call to [ThreadUtilities getJNIEnv]
triggers a NULL dereference.  Seeing this, I guessed [ThreadUtilities
getJNIEnv] was being called too early, before Apple's JVM's AWT
libraries were fully loaded and initialized.  I confirmed my guess by
deliberately calling [ThreadUtilities getJNIEnv] before the call to
[AppletView loadAWTLibraries], but after the JVM itself had been
loaded.  Doing this caused this bug's crash to happen every time I
tried to load a Java applet, on OS X 10.5.7 and 10.4.11.

So this is an Apple bug:  [ThreadUtilities getJNIEnv] doesn't take
account of the possibility that it will be called before the AWT
subsystem is fully initialized, and crashes on dereferencing an
internal pointer that's unexpectedly NULL.

Deliberately calling [ControlModel initialize] before the call to
[AppletView loadAWTLibraries] only crashes on OS X 10.5.X, and then
only when using J2SE 5.0.  It doesn't happen on OS X 10.4.11 or when
using Java 1.4.2 on OS X 10.5.X.  In these cases a call to
[ControlModel initialize] doesn't (for some reason) trigger a call to
[ThreadUtilities getJNIEnv].  This is why none of these crashes have
been reported on OS X 10.4.11.

I don't know why these crashes haven't been reported in Firefox 3.0.X,
even on OS X 10.5.X.  I assume JEP 0.9.7 (currently only bundled on
the 1.9.1 branch) made subtle timing changes, which resulted in
[ControlModel initialize] sometimes being called before the AWT
subsystem had been initialized (or at least to this happening more
often).

Note that a small number of these crashes *have* been reported on PPC
Macs.  They just take place at a different address
(libawt.jnilib@0x18d4).

http://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=libawt&date=&range_value=1&range_unit=weeks&do_query=1&signature=libawt.jnilib%400x18d4
Comment 32 Steven Michaud [:smichaud] (Retired) 2009-05-14 13:18:13 PDT
It's very easy for the JEP to work around the Apple bug that causes
these crashes:  I just hook [ThreadUtilities getJNIEnv] and only let
it call super when awtLibrariesLoaded is TRUE.  When
awtLibrariesLoaded is FALSE I use other (safer) ways to get the
current thread's JNIEnv.

Adding this code made the crashes stop happening even when I
deliberately called [ThreadUtilities getJNIEnv] or [ControlModel
initialize] before the call to [AppletView loadAWTLibraries].

I can't reproduce this bug's crashes without fiddling with the JEP's
code.  And I doubt anyone else will be able to, either.  So I can't be
100% sure my workaround will fix this bug's crashes.  But what I said
in comment #31 shows I have good reason to be 95%-99% sure.

Sounds like I should release a new version of the JEP (0.9.7.1)
sometime in the next week or two -- in time for inclusion in the first
RC of FF 3.5.

There are a couple of other problems I also need to resolve in JEP
0.9.7.1 (bug 492251 and bug 491600).  Both have very simple
workarounds.  And bug 491600 is actually quite a serious problem.
Comment 33 Samuel Sidler (old account; do not CC) 2009-05-14 15:15:54 PDT
(In reply to comment #32)
> Sounds like I should release a new version of the JEP (0.9.7.1)
> sometime in the next week or two -- in time for inclusion in the first
> RC of FF 3.5.

You should aim for a week since code freeze is set for then. Two weeks will miss the release, which would be a shame given the amount of users experiencing this bug. (When a Mac-specific topcrash is in the top 10, it's really bad since most of our users are on Windows.)

I think this bug should block the release of Firefox 3.5.
Comment 34 Mike Beltzner [:beltzner, not reading bugmail] 2009-05-14 15:20:46 PDT
I thought this was only in JavaFX? has that morphed from earlier comments? Do we know why it's only affecting some users?

Tentatitvely blocking here. Steven: what would an ETA be for another JEP? Any chance we could get it for next Wednesday?
Comment 35 Samuel Sidler (old account; do not CC) 2009-05-14 15:23:14 PDT
Steven: What changed in the latest JEP that made this Apple bug visible? We previously weren't seeing any crashes like this, so it's something new on our end that's making it visible.
Comment 36 Steven Michaud [:smichaud] (Retired) 2009-05-14 15:25:50 PDT
> You should aim for a week since code freeze is set for then.

Thanks for the info.  I'll aim for a week.

I've actually already put in the fixes for this bug (bug 492492) and
bug 492251.  I'm waiting for advice from Apple on how to fix bug
419600, but have a very simple solution I can fall back on.

Aside from these, what remains to be done is the testing.  That can
(with luck) be finished in 2-3 days.

> Steven: what would an ETA be for another JEP? Any chance we could
> get it for next Wednesday?

That's doable, I think.

> I thought this was only in JavaFX? has that morphed from earlier
> comments? Do we know why it's only affecting some users?

See comment #28, comment #31 and comment #32.
Comment 37 Steven Michaud [:smichaud] (Retired) 2009-05-14 15:27:46 PDT
> What changed in the latest JEP that made this Apple bug visible?

As I said in comment #31, I don't really know.  My best guess is that
JEP 0.9.7 introduced some subtle timing changes.  I'd like to think
it's faster :-)
Comment 38 Steven Michaud [:smichaud] (Retired) 2009-05-18 14:14:44 PDT
I've just released JEP 0.9.7.1, which should fix these crashes.

See bug 493629.
Comment 39 Steven Michaud [:smichaud] (Retired) 2009-05-19 10:21:54 PDT
JEP 0.9.7.1 has been landed on the 1.9.1 branch (bug 493629), so I'm
tentatively marking this bug fixed (and fixed1.9.1).

But, because this bug isn't reproducible (and possibly also because of
difficulties with Socorro), we probably can't know for sure that this
bug has been fixed until some time after the RC is released.

Here's Socorro's current list of all this bug's crashes:

http://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=libawt&date=&range_value=1&range_unit=weeks&do_query=1&signature=libawt.jnilib%400x11e1

All but 4 of them belong to version 3.5b4 and have build date
20090423191946.  (Three belong to version 3.5b5pre and have a couple
of different build dates.  One actually belongs to version 3.0.10.)

I know that nightlies get less exposure than betas.  But it still
puzzles me that there are so many more 3.5b4 crashes than 3.5b5pre (or
3.5b4pre) crashes.  I wonder if Socorro undercounts crashes in
nightlies.
Comment 40 chris hofmann 2009-05-19 12:37:51 PDT
> I know that nightlies get less exposure than betas. 

b4 now has about 600k daily users.  b5pre might have 5-10k.
Comment 41 Steven Michaud [:smichaud] (Retired) 2009-05-19 12:51:40 PDT
5K is 1/120th of 600K (about .83%)

But there are currently 1144 instances of this crash in Socorro for FF
3.5 Beta 4, and only 4 for other builds (a ratio of 1/286, or about
.35%).

I know it's difficult to be sure of the '5-10K', or even of the
'600K'.  But it seems to me that, over the last few months, it's been
more difficult to find crashing nightlies (as opposed to crashing
"releases") in Socorro than it used to be.  This is very subjective,
but this bug might provide an opportunity to begin to quantify it.
Comment 42 Steven Michaud [:smichaud] (Retired) 2009-05-19 13:15:45 PDT
Over two weeks (instead of the default one week) the ratio is even worse:

2971 crashes in FF 3.5 Beta 4, 4 in other builds.
Comment 43 James Rome 2009-05-19 13:26:35 PDT
The reason that I can't easily use nightlies is that for some awful reason, you rename it to Shiretoko or something else. That breaks everything. This policy should be reexamined. I used to run the daily nightlies and had to give that up because of this.
Comment 44 Steven Michaud [:smichaud] (Retired) 2009-05-19 13:41:16 PDT
> That breaks everything.

I don't understand.

And is there a bug on this?
Comment 45 Samuel Sidler (old account; do not CC) 2009-05-19 14:21:04 PDT
(In reply to comment #42)
> Over two weeks (instead of the default one week) the ratio is even worse:
> 
> 2971 crashes in FF 3.5 Beta 4, 4 in other builds.

Err... huh? In the last four weeks, there have been 23 crashes (#2 on Mac) with this stack.

http://crash-stats.mozilla.com/query/query?do_query=1&product=Firefox&version=Firefox%3A3.5b5pre&platform=mac&date=&range_value=2&range_unit=weeks&query_search=signature&query_type=exact&query=
Comment 46 Steven Michaud [:smichaud] (Retired) 2009-05-19 14:42:39 PDT
(In reply to comment #45)

Your search link doesn't work.

And in any case I searched a different way.

First I searched for all crashes whose "signature" contains libawt on
Mac OS X, over two weeks:

http://crash-stats.mozilla.com/query/query?do_query=1&platform=mac&date=&range_value=2&range_unit=weeks&query_search=signature&query_type=contains&query=libawt

Then I right-clicked on the first one in the resulting list, opened it
in a new tab, and clicked twice on the "Version" tab (so that I first
ordered all the crashes by version twice, first in "normal" order and
then in "reverse" order).

This gave (and still gives) me only four crashes whose "version" isn't
'3.5b4'.  I get the same results if I use the "Build" tab instead of
the "Version" tab.

The total number of crashes (over 2 weeks) now is 2905 ... which is a
bit screwy (since the number has dropped).
Comment 48 Steven Michaud [:smichaud] (Retired) 2009-05-19 15:04:54 PDT
So maybe I just need to change how I've been doing Socorro searches :-)

The business with the tabs used to work fine (several months ago).  But now it apparently doesn't.
Comment 49 Samuel Sidler (old account; do not CC) 2009-05-19 15:17:45 PDT
(In reply to comment #46)
> Your search link doesn't work.

Yes it does if you actually go to the full URL (including the trailing '=').

Searching the way you did is wrong because we limit the amount of given reports to 500. It's not surprising you only saw 4 in that case.
Comment 50 Samuel Sidler (old account; do not CC) 2009-05-20 08:29:24 PDT
*** Bug 493898 has been marked as a duplicate of this bug. ***
Comment 51 doug redecopp 2009-08-18 18:49:54 PDT
When I go to Mygarmin and plug in my nuvi 255, windows pops up the little window. "Firefox is causing a problem and will closed"
mygarmin/dashboard
garmin nuvi 255
Latest garmin communicator
firefox 3.5.2
Also commented in https://bugzilla.mozilla.org/show_bug.cgi?id=494417

Note You need to log in before you can comment on or make changes to this bug.