Environment: - Ubuntu 10.04 b1, up-to-date as of earlier today - mozilla-central nightly build, with fresh profile Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.3a4pre) Gecko/20100326 Minefield/3.7a4pre - Java plugin, according to about:plugins, is IcedTea NPR Web Browser Plugin (using IcedTea6 1.8pre (6b18~pre3-0ubuntu1)) STEPS TO REPRODUCE: Load URL, http://zxspectrum.net/ EXPECTED RESULTS: The emulator should load. That is -- the gray box under "ZX Spectrum" title) should flash black, and then go to a screen saying "(c) 1982 Sinclair Research Ltd" ACTUAL RESULTS: The gray box stays gray, and I get a few "*** glibc detected *** /home/dholbert/programs/firefox-upToDate/mozilla-runtime: free(): invalid next size (fast): 0x08b9d1a8 ***"
Regression range: WORKS FINE: http://hg.mozilla.org/mozilla-central/rev/bcd9709de08a Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.3a1pre) Gecko/20100127 Minefield/3.7a1pre DOESN'T WORK: (emulator doesn't load, glibc messages are spammed, "a plugin crashed" dialog pops up -- which doesn't happen in current nightly, FWIW) http://hg.mozilla.org/mozilla-central/rev/6712bed154ed Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.3a1pre) Gecko/20100128 Minefield/3.7a1pre REGRESSION PUSHLOG: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=bcd9709de08a&tochange=6712bed154ed I'm guessing this is at least in part due to a bug in IcedTea java plugin. Still, it used to work (recently, w/in the last 2 months on trunk), and it doesn't anymore. --> 'regression' keyword
Did you manually set dom.ipc.plugins.enabled or use the default when checking the regression? The range includes Benjamin Smedberg — Bug 531142 - Turn on multi-process plugins by default which seems highly suspect.
Ah, that's probably part of it. I tried that in my up-to-date build, but not in older builds. (I'd thought we'd defaulted-to-IPC-being-on longer ago than that. :)) I'll try toggling that and report back... NOTE: this is WFM on a different Ubuntu 9.04 machine, using sun-java6-plugin. NOTE: Loading the URL in the 20100128 m-c nightly, I get a multiple "a plugin crashed" dialogs, as noted in comment 1. Just now, it gave me 3 dialogs (for a single load of the page). I pressed 'submit' on all of them, and here are the resulting reports: http://crash-stats.mozilla.com/report/index/6dc07239-e14b-47cf-ac6b-ab21d2100326 http://crash-stats.mozilla.com/report/index/08c64732-e7ac-4341-9cb3-a43762100326 http://crash-stats.mozilla.com/report/index/e1a50854-d48d-49a9-8293-840692100326
bsmedberg is right in comment 2 -- this is only an issue with "dom.ipc.plugins.enabled" turned on. Problem goes away if I disable that pref. (double-checked that w/ the 20100128 nightly and also today's nightly)
Created attachment 435323 [details] std[out/err], IPC disabled Here's the combined stdout + stderr, with IPC disabled, with page loading just fine. (Contains a few "java.*.*Exception" backtraces, and nothing scarier than that)
Created attachment 435324 [details] std[out/err], IPC enabled and here's the combined stdout + stderr with IPC enabled, with all the "glibc detected" memory issues.
I tried building from source but gave up after IcedTea required me to install nspr, xulrunner, libxul-unstable, and mozjs (very very bad sign!). I tried the system Ubuntu 9.10 IcedTea and confirmed it's broken; it apparently doesn't report a MIME type LoadPlugin() /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/IcedTeaPlugin.so returned 30b5a90 GetMIMEDescription() returned "" I'll try hard coding application/x-java-applet for IcedTea.so and see if I can repro. Otherwise I'll spin up an Ubuntu 10.4 VM.
So, the Ubuntu 9.10 version of IcedTea (6b16-1.6.1-3ubuntu1) is an XPCOM plugin. _ZN8nsCOMPtrI9nsIThreadED1Ev _ZN8nsCOMPtrI11nsIRunnableEC1EPS0_ NS_GetComponentManager NS_GetServiceManager NS_Alloc NS_Free NS_Realloc WONTFIX for that. I'll spin up a 10.4 VM.
(In reply to comment #8) > So, the Ubuntu 9.10 version of IcedTea (6b16-1.6.1-3ubuntu1) is an XPCOM > plugin. > (It doesn't define any of the NPAPI entry functions.)
dholbert, you try running "strings IcedTeaPlugin.so | grep NP_" and see if you get anything?
Yup: [dholbert@orange:~]$ locate IcedTeaPlugin.so /usr/lib/jvm/java-6-openjdk/jre/lib/i386/IcedTeaPlugin.so [dholbert@orange:~]$ strings /usr/lib/jvm/java-6-openjdk/jre/lib/i386/IcedTeaPlugin.so | grep NP_ NP_GetMIMEDescription NP_Shutdown NP_GetValue NP_Initialize NP_GetMIMEDescription NP_GetMIMEDescription return NP_Shutdown NP_Shutdown return NP_GetValue NP_GetValue return NP_Initialize NP_Initialize: using %s NP_Initialize return NP_Shutdown: deleting output fifo: %s NP_Shutdown: deleted output fifo: %s NP_Shutdown: deleting input fifo: %s NP_Shutdown: deleted input fifo: %s NP_GetValue: returning plugin name. NP_GetValue: returning plugin description.
IcedTea dies in my 64-bit VM because IcedTea wants SSE2 and it's either absent or IcedTea is failing to detect it. Need to give a 32-bit VM a shot. On the bright side, I found that VMWare linux 64-bit will now record/replay! I've got a fun project for next week.
(In reply to comment #12) > IcedTea dies in my 64-bit VM because IcedTea wants SSE2 and it's either absent > or IcedTea is failing to detect it. Need to give a 32-bit VM a shot. > To clarify, this was when running under valgrind. I repro'd running normally.
fwiw, bug 552891 was my last encounter with icedtea...
Had some VM trouble, finally got icedtea to compile from source (needed to use mercurial latest). Here's one problem $ nm -g --defined-only ~/.mozilla/plugins/libicedteadplugin.so | grep NP_ 000000000000bc90 T _Z11NP_GetValuePv11NPPVariableS_ 000000000000b4f0 T _Z11NP_Shutdownv 000000000000be90 T _Z13NP_InitializeP16_NPNetscapeFuncsP14_NPPluginFuncs 000000000000a970 T _Z21NP_GetMIMEDescriptionv We're not resolving any npapi entry functions because they're not |extern "C"|. Need to file this against IcedTea. Whipping up a bandaid patch atm.
Nm, this was my fault; I bypassed pkg-config incorrectly. The fix was to -DXP_UNIX in MOZILLA_CFLAGS.
Here's one fun icedtea bug right off the bat command_line = (gchar**) malloc(sizeof(gchar)*5); command_line = g_strdup(appletviewer_executable); command_line = g_strdup("sun.applet.PluginMain"); command_line = g_strdup(out_pipe_name); command_line = g_strdup(in_pipe_name); command_line = NULL; Oopsie! Not sure yet if this is what's causing the crash, but it sure upsets valgrind.
Yep, this is the cause of the bug. #8 0x00007fde9709ddd6 in malloc_printerr (action=3, str=0x7fde9715f720 "free(): invalid next size (normal)", ptr=<value optimized out>) at malloc.c:6217 #9 0x00007fde970a274c in *__GI___libc_free (mem=<value optimized out>) at malloc.c:3716 #10 0x00007fde970c7b9d in __closedir (dirp=0x315d) at ../sysdeps/unix/closedir.c:52 #11 0x00007fde949cee78 in fdwalk (child_err_report_fd=15, stdin_fd=-1, stdout_fd=-1, stderr_fd=-1, working_directory=<value optimized out>, argv=0x18b9a40, envp=0x0, close_descriptors=1, search_path=0, stdout_to_null=0, stderr_to_null=0, child_inherits_stdin=0, file_and_argv_zero=0, child_setup=0, user_data=0x0) at /build/buildd/glib2.0-2.22.3/glib/gspawn.c:952 #12 do_exec (child_err_report_fd=15, stdin_fd=-1, stdout_fd=-1, stderr_fd=-1, working_directory=<value optimized out>, argv=0x18b9a40, envp=0x0, close_descriptors=1, search_path=0, stdout_to_null=0, stderr_to_null=0, child_inherits_stdin=0, file_and_argv_zero=0, child_setup=0, user_data=0x0) at /build/buildd/glib2.0-2.22.3/glib/gspawn.c:1026 #13 0x00007fde949cf51b in fork_exec_with_pipes (intermediate_child=<value optimized out>, working_directory=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>, close_descriptors=<value optimized out>, search_path=<value optimized out>, stdout_to_null=0, stderr_to_null=0, child_inherits_stdin=0, file_and_argv_zero=0, child_setup=0, user_data=0x0, child_pid=0x7fde8cda47f4, standard_input=0x0, standard_output=0x0, standard_error=0x0, error=0x7fde8cda4ae0) at /build/buildd/glib2.0-2.22.3/glib/gspawn.c:1283 #14 0x00007fde949cfb88 in IA__g_spawn_async_with_pipes (working_directory=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>, flags=<value optimized out>, child_setup=<value optimized out>, user_data=<value optimized out>, child_pid=0x7fde8cda47f4, standard_input=0x0, standard_output=0x202, standard_error=0x0, error=0x7fde8cda4ae0) at /build/buildd/glib2.0-2.22.3/glib/gspawn.c:631 #15 0x00007fde949cfc8c in IA__g_spawn_async (working_directory=0x315d <Address 0x315d out of bounds>, argv=0x315d, envp=0x6, flags=4294967295, child_setup=0, user_data=0x7fde970b1340, child_pid=0x7fde8cda47f4, error=0x7fde8cda4ae0) at /build/buildd/glib2.0-2.22.3/glib/gspawn.c:122 #16 0x00007fde8cb81d1d in plugin_start_appletviewer () at /home/cjones/src/icedtea6/plugin/icedteanp/IcedTeaNPPlugin.cc:1483 #17 start_jvm_if_needed () at /home/cjones/src/icedtea6/plugin/icedteanp/IcedTeaNPPlugin.cc:491 #18 0x00007fde8cb82d59 in GCJ_New (pluginType=<value optimized out>, instance=0x18b7e00, mode=<value optimized out>, argc=<value optimized out>, argn=0x18b7c30, argv=<value optimized out>, saved=0x0) at /home/cjones/src/icedtea6/plugin/icedteanp/IcedTeaNPPlugin.cc:310 #19 0x00007fde99d7c1ee in mozilla::plugins::PluginModuleChild::AnswerPPluginInstanceConstructor (this=0x1829d48, aActor=0x18b7db0, aMimeType=..., aMode=@0x7fde8f0c17a4, aNames=..., aValues=..., rv=0x7fde8f0c179c) at /home/cjones/mozilla/mozilla-central/dom/plugins/PluginModuleChild.cpp:1600 #20 0x00007fde99e06778 in mozilla::plugins::PPluginModuleChild::OnCallReceived (this=0x1829d48, msg=..., reply=@0x7fde8f0c1840) at PPluginModuleChild.cpp:483 #21 0x00007fde99d99b85 in mozilla::ipc::RPCChannel::DispatchIncall (this=0x1829d58, call=...) at /home/cjones/mozilla/mozilla-central/ipc/glue/RPCChannel.cpp:485 #22 0x00007fde99d99a9a in mozilla::ipc::RPCChannel::Incall (this=0x1829d58, call=..., stackDepth=0) at /home/cjones/mozilla/mozilla-central/ipc/glue/RPCChannel.cpp:471 #23 0x00007fde99d997e7 in mozilla::ipc::RPCChannel::OnMaybeDequeueOne (this=0x1829d58) at /home/cjones/mozilla/mozilla-central/ipc/glue/RPCChannel.cpp:413 #24 0x00007fde99d9f424 in DispatchToMethod<mozilla::ipc::RPCChannel, void (mozilla::ipc::RPCChannel::*)()> (obj=0x1829d58, method=0x7fde99d995c0 <mozilla::ipc::RPCChannel::OnMaybeDequeueOne()>, arg=...) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/tuple.h:383 #25 0x00007fde99d9f2cc in RunnableMethod<mozilla::ipc::RPCChannel, void (mozilla::ipc::RPCChannel::*)(), Tuple0>::Run (this=0x182ab30) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/task.h:307 #26 0x00007fde99d9b00b in mozilla::ipc::RPCChannel::RefCountedTask::Run (this=0x182ab70) at ../../dist/include/mozilla/ipc/RPCChannel.h:421 #27 0x00007fde99d9b10e in mozilla::ipc::RPCChannel::DequeueTask::Run (this=0x18b5af0) at ../../dist/include/mozilla/ipc/RPCChannel.h:446 #28 0x00007fde99f566ee in MessageLoop::RunTask (this=0x7fde8f0c1e20, task=0x18b5af0) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_loop.cc:336 #29 0x00007fde99f5675e in MessageLoop::DeferOrRunPendingTask (this=0x7fde8f0c1e20, pending_task=...) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_loop.cc:344 #30 0x00007fde99f56b5c in MessageLoop::DoWork (this=0x7fde8f0c1e20) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_loop.cc:444 #31 0x00007fde99fca101 in base::MessagePumpForUI::HandleDispatch (this=0x182c0a0) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_pump_glib.cc:264 #32 0x00007fde99fc977b in WorkSourceDispatch (source=0x182c1f0, unused_func=0, unused_data=0x0) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_pump_glib.cc:109 #33 0x00007fde94998bce in g_main_dispatch (context=0x182c110) at /build/buildd/glib2.0-2.22.3/glib/gmain.c:1960 #34 IA__g_main_context_dispatch (context=0x182c110) at /build/buildd/glib2.0-2.22.3/glib/gmain.c:2513 #35 0x00007fde9499c598 in g_main_context_iterate (context=0x182c110, block=<value optimized out>, dispatch=<value optimized out>, self=<value optimized out>) at /build/buildd/glib2.0-2.22.3/glib/gmain.c:2591 #36 0x00007fde9499c6c0 in IA__g_main_context_iteration (context=0x182c110, may_block=1) at /build/buildd/glib2.0-2.22.3/glib/gmain.c:2654 #37 0x00007fde99fc9e31 in base::MessagePumpForUI::RunWithDispatcher (this=0x182c0a0, delegate=0x7fde8f0c1e20, dispatcher=0x0) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_pump_glib.cc:195 #38 0x00007fde99fca4c6 in base::MessagePumpForUI::Run(base::MessagePump::Delegate*) () from ./libxul.so #39 0x00007fde99f561f9 in MessageLoop::RunInternal (this=0x7fde8f0c1e20) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_loop.cc:216 #40 0x00007fde99f5617e in MessageLoop::RunHandler (this=0x7fde8f0c1e20) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_loop.cc:199 #41 0x00007fde99f5610f in MessageLoop::Run (this=0x7fde8f0c1e20) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_loop.cc:173 #42 0x00007fde99f7d406 in base::Thread::ThreadMain (this=0x1829ca0) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/thread.cc:165 #43 0x00007fde99fb0d6b in ThreadFunc (closure=0x1829ca0) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/platform_thread_posix.cc:26 #44 0x00007fde9ba0da04 in start_thread (arg=<value optimized out>) at pthread_create.c:300 #45 0x00007fde9710780d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #46 0x0000000000000000 in ?? () I'll try to see why we don't hit this in the in-process case ...
Huh, I get the same crash running IcedTea in-process. Shrug. Definitely an IcedTea bug. I'll patch it up, verify the fix, and file against IcedTea.
Created attachment 436098 [details] [diff] [review] IcedTea fix (for archival purposes) Wfm OOP and IP.
Created attachment 436103 [details] [diff] [review] Hack around IcedTea bug by forcing it into debug mode Still waiting for my account info for the IcedTea bugzilla. I *really* hope IcedTea doesn't ship in the official Ubuntu 10.4 release with such an egregious bug, but we may be too short on time. Here's a hacky "fix" to hedge our bets. If IcedTea is fixed in time we can rip it out.
Karl, the code this fix applies to is http://icedtea.classpath.org/hg/icedtea6/file/1188b1a313b9/plugin/icedteanp/IcedTeaNPPlugin.cc line 1462.
Comment on attachment 436103 [details] [diff] [review] Hack around IcedTea bug by forcing it into debug mode In debug mode, the plugin opens a Java debug socket on localhost:8787 $ socklist type port inode uid pid fd name tcp 8787 1639542 1000 5951 5 java This means other users on the machine can attach to the plugin and read possibly sensitive information (verified this with a program that seteuid's/setegid's to nobody/nogroup and connected to the debug socket). So we're just going to have to blacklist this plugin until the bug is fixed.
Created attachment 436125 [details] [diff] [review] Attempt to blocklist IcedTea (though not working locally) I don't know if this is the right way to blocklist a plugin or from whom I should request review. Will resume tomorrow. We basically have three options for dealing with IcedTea (1) "OOP blacklist" it. This exposes firefox-bin to definite crash cases and a possible security vulnerability (2) start linking mozilla-runtime with jemalloc so that IcedTea will sort of work in mozilla-runtime. This pushes the browser problems into the plugin process. (Karl rightly points out that this might cause plugin perf degradation if they're tuned to the libc allocator. I'm going to spin this investigation off in its own bug.) (3) blocklist until the IcedTea bug is fixed I personally lean towards (3), but my understanding is that Fedora Core 12 already uses IcedTeaPlugin, and I see little chance of this bug being fixed before Ubuntu 10.4 ships.
Comment on attachment 436125 [details] [diff] [review] Attempt to blocklist IcedTea (though not working locally) See https://bugs.launchpad.net/ubuntu/+source/openjdk-6/+bug/552287. Matthias Klose wrote 7 hours ago: #5 Re: https://bugzilla.mozilla.org/show_bug.cgi?id=555342#c24 the fix is in IcedTea and uploaded to lucid. (3) afaik FC12 does use the IcedTeaPlugin.cc, not the IcedTeaNPPlugin.cc. The former is not affected. No release enables the IcedTeaNPPlugin.cc by default.
(AFAICT the IcedTea fix should make it into Ubuntu 10.4 beta 2.)
Thanks for the fix, Chris! Great work. :)
FWIW bug 556198 will serve as a band-aid for this problem, hope to land it today.
FWIW, this is fixed for me, with previously-broken versions of Firefox -- hence, it looks like Chris's IcedTea patch (from comment 25) has made it into the Ubuntu repositories. My current icedtea6-plugin version is "6b18~pre4-1ubuntu1", and it's WFM with today's nightly and with the previously-"doesn't work" nightly from comment 1. (I verified that "mozilla-runtime" is running in both instances, indicating that the plugin is indeed being run OOP without crashing.)