555342 - [OOPP] zxspectrum.net Java applet doesn't load, with glibc memory-freeing errors spammed to stderr (IcedTea java plugin on Ubuntu 10.04b1)

Status: RESOLVED FIXED

(Plugins Graveyard :: Java (IcedTea), defect)

Description

[Daniel Holbert [:dholbert]]

Environment:
 - Ubuntu 10.04 b1, up-to-date as of earlier today
 - mozilla-central nightly build, with fresh profile
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.3a4pre) Gecko/20100326 Minefield/3.7a4pre
 - Java plugin, according to about:plugins, is IcedTea NPR Web Browser Plugin (using IcedTea6 1.8pre (6b18~pre3-0ubuntu1))

STEPS TO REPRODUCE:
 Load URL, http://zxspectrum.net/

EXPECTED RESULTS:  The emulator should load.  That is -- the gray box under "ZX Spectrum" title) should flash black, and then go to a screen saying "(c) 1982 Sinclair Research Ltd"

ACTUAL RESULTS:
The gray box stays gray, and I get a few "*** glibc detected *** /home/dholbert/programs/firefox-upToDate/mozilla-runtime: free(): invalid next size (fast): 0x08b9d1a8 ***"

Comment 1

[Daniel Holbert [:dholbert]]

Regression range:
WORKS FINE:
http://hg.mozilla.org/mozilla-central/rev/bcd9709de08a
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.3a1pre) Gecko/20100127 Minefield/3.7a1pre

DOESN'T WORK:
(emulator doesn't load, glibc messages are spammed, "a plugin crashed" dialog pops up -- which doesn't happen in current nightly, FWIW)
http://hg.mozilla.org/mozilla-central/rev/6712bed154ed
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.3a1pre) Gecko/20100128 Minefield/3.7a1pre

REGRESSION PUSHLOG:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=bcd9709de08a&tochange=6712bed154ed

I'm guessing this is at least in part due to a bug in IcedTea java plugin.  Still, it used to work (recently, w/in the last 2 months on trunk), and it doesn't anymore. --> 'regression' keyword

Comment 2

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Did you manually set dom.ipc.plugins.enabled or use the default when checking the regression?  The range includes

Benjamin Smedberg — Bug 531142 - Turn on multi-process plugins by default

which seems highly suspect.

Comment 3

[Daniel Holbert [:dholbert]]

Ah, that's probably part of it.  I tried that in my up-to-date build, but not in older builds.  (I'd thought we'd defaulted-to-IPC-being-on longer ago than that. :))  I'll try toggling that and report back...

NOTE: this is WFM on a different Ubuntu 9.04 machine, using sun-java6-plugin.

NOTE: Loading the URL in the 20100128 m-c nightly, I get a multiple "a plugin crashed" dialogs, as noted in comment 1.  Just now, it gave me 3 dialogs (for a single load of the page). I pressed 'submit' on all of them, and here are the resulting reports:
http://crash-stats.mozilla.com/report/index/6dc07239-e14b-47cf-ac6b-ab21d2100326
http://crash-stats.mozilla.com/report/index/08c64732-e7ac-4341-9cb3-a43762100326
http://crash-stats.mozilla.com/report/index/e1a50854-d48d-49a9-8293-840692100326

Comment 4

[Daniel Holbert [:dholbert]]

bsmedberg is right in comment 2 -- this is only an issue with "dom.ipc.plugins.enabled" turned on.  Problem goes away if I disable that pref. (double-checked that w/ the 20100128 nightly and also today's nightly)

Comment 5

[Daniel Holbert [:dholbert]]

Attachment: std[out/err], IPC disabled

Here's the combined stdout + stderr, with IPC disabled, with page loading just fine.  (Contains a few "java.*.*Exception" backtraces, and nothing scarier than that)

Comment 6

[Daniel Holbert [:dholbert]]

Attachment: std[out/err], IPC enabled

and here's the combined stdout + stderr with IPC enabled, with all the "glibc detected" memory issues.

Comment 7

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

I tried building from source but gave up after IcedTea required me to install nspr, xulrunner, libxul-unstable, and mozjs (very very bad sign!).  I tried the system Ubuntu 9.10 IcedTea and confirmed it's broken; it apparently doesn't report a MIME type

LoadPlugin() /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/IcedTeaPlugin.so returned 30b5a90
GetMIMEDescription() returned ""

I'll try hard coding application/x-java-applet for IcedTea.so and see if I can repro.  Otherwise I'll spin up an Ubuntu 10.4 VM.

Comment 8

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

So, the Ubuntu 9.10 version of IcedTea (6b16-1.6.1-3ubuntu1) is an XPCOM plugin.

_ZN8nsCOMPtrI9nsIThreadED1Ev
_ZN8nsCOMPtrI11nsIRunnableEC1EPS0_
NS_GetComponentManager
NS_GetServiceManager
NS_Alloc
NS_Free
NS_Realloc

WONTFIX for that.  I'll spin up a 10.4 VM.

Comment 9

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

(In reply to comment #8)
> So, the Ubuntu 9.10 version of IcedTea (6b16-1.6.1-3ubuntu1) is an XPCOM
> plugin.
> 

(It doesn't define any of the NPAPI entry functions.)

Comment 10

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

dholbert, you try running "strings IcedTeaPlugin.so | grep NP_" and see if you get anything?

Comment 11

[Daniel Holbert [:dholbert]]

Yup:

[dholbert@orange:~]$ locate IcedTeaPlugin.so
/usr/lib/jvm/java-6-openjdk/jre/lib/i386/IcedTeaPlugin.so
[dholbert@orange:~]$ strings /usr/lib/jvm/java-6-openjdk/jre/lib/i386/IcedTeaPlugin.so | grep NP_
NP_GetMIMEDescription
NP_Shutdown
NP_GetValue
NP_Initialize
NP_GetMIMEDescription
NP_GetMIMEDescription return
NP_Shutdown
NP_Shutdown return
NP_GetValue
NP_GetValue return
NP_Initialize
NP_Initialize: using %s
NP_Initialize return
NP_Shutdown: deleting output fifo: %s
NP_Shutdown: deleted output fifo: %s
NP_Shutdown: deleting input fifo: %s
NP_Shutdown: deleted input fifo: %s
NP_GetValue: returning plugin name.
NP_GetValue: returning plugin description.

Comment 12

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

IcedTea dies in my 64-bit VM because IcedTea wants SSE2 and it's either absent or IcedTea is failing to detect it.  Need to give a 32-bit VM a shot.

On the bright side, I found that VMWare linux 64-bit will now record/replay!  I've got a fun project for next week.

Comment 13

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

(In reply to comment #12)
> IcedTea dies in my 64-bit VM because IcedTea wants SSE2 and it's either absent
> or IcedTea is failing to detect it.  Need to give a 32-bit VM a shot.
>

To clarify, this was when running under valgrind.  I repro'd running normally.

Comment 14

[timeless]

fwiw, bug 552891 was my last encounter with icedtea...

Comment 15

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Had some VM trouble, finally got icedtea to compile from source (needed to use mercurial latest).  Here's one problem

$ nm -g --defined-only ~/.mozilla/plugins/libicedteadplugin.so | grep NP_
000000000000bc90 T _Z11NP_GetValuePv11NPPVariableS_
000000000000b4f0 T _Z11NP_Shutdownv
000000000000be90 T _Z13NP_InitializeP16_NPNetscapeFuncsP14_NPPluginFuncs
000000000000a970 T _Z21NP_GetMIMEDescriptionv

We're not resolving any npapi entry functions because they're not |extern "C"|.  Need to file this against IcedTea.  Whipping up a bandaid patch atm.

Comment 16

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Nm, this was my fault; I bypassed pkg-config incorrectly.  The fix was to -DXP_UNIX in MOZILLA_CFLAGS.

Comment 17

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Here's one fun icedtea bug right off the bat

       command_line = (gchar**) malloc(sizeof(gchar)*5);
       command_line[0] = g_strdup(appletviewer_executable);
       command_line[1] = g_strdup("sun.applet.PluginMain");
       command_line[2] = g_strdup(out_pipe_name);
       command_line[3] = g_strdup(in_pipe_name);
       command_line[4] = NULL;

Oopsie!  Not sure yet if this is what's causing the crash, but it sure upsets valgrind.

Comment 18

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Yep, this is the cause of the bug.

#8  0x00007fde9709ddd6 in malloc_printerr (action=3, str=0x7fde9715f720 "free(): invalid next size (normal)", ptr=<value optimized out>) at malloc.c:6217
#9  0x00007fde970a274c in *__GI___libc_free (mem=<value optimized out>) at malloc.c:3716
#10 0x00007fde970c7b9d in __closedir (dirp=0x315d) at ../sysdeps/unix/closedir.c:52
#11 0x00007fde949cee78 in fdwalk (child_err_report_fd=15, stdin_fd=-1, stdout_fd=-1, stderr_fd=-1, working_directory=<value optimized out>, argv=0x18b9a40, envp=0x0, close_descriptors=1, search_path=0, stdout_to_null=0, stderr_to_null=0, child_inherits_stdin=0, file_and_argv_zero=0, child_setup=0, user_data=0x0) at /build/buildd/glib2.0-2.22.3/glib/gspawn.c:952
#12 do_exec (child_err_report_fd=15, stdin_fd=-1, stdout_fd=-1, stderr_fd=-1, working_directory=<value optimized out>, argv=0x18b9a40, envp=0x0, close_descriptors=1, search_path=0, stdout_to_null=0, stderr_to_null=0, child_inherits_stdin=0, file_and_argv_zero=0, child_setup=0, user_data=0x0) at /build/buildd/glib2.0-2.22.3/glib/gspawn.c:1026
#13 0x00007fde949cf51b in fork_exec_with_pipes (intermediate_child=<value optimized out>, working_directory=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>, close_descriptors=<value optimized out>, search_path=<value optimized out>, stdout_to_null=0, stderr_to_null=0, child_inherits_stdin=0, file_and_argv_zero=0, child_setup=0, user_data=0x0, child_pid=0x7fde8cda47f4, standard_input=0x0, standard_output=0x0, standard_error=0x0, error=0x7fde8cda4ae0) at /build/buildd/glib2.0-2.22.3/glib/gspawn.c:1283
#14 0x00007fde949cfb88 in IA__g_spawn_async_with_pipes (working_directory=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>, flags=<value optimized out>, child_setup=<value optimized out>, user_data=<value optimized out>, child_pid=0x7fde8cda47f4, standard_input=0x0, standard_output=0x202, standard_error=0x0, error=0x7fde8cda4ae0) at /build/buildd/glib2.0-2.22.3/glib/gspawn.c:631
#15 0x00007fde949cfc8c in IA__g_spawn_async (working_directory=0x315d <Address 0x315d out of bounds>, argv=0x315d, envp=0x6, flags=4294967295, child_setup=0, user_data=0x7fde970b1340, child_pid=0x7fde8cda47f4, error=0x7fde8cda4ae0) at /build/buildd/glib2.0-2.22.3/glib/gspawn.c:122
#16 0x00007fde8cb81d1d in plugin_start_appletviewer () at /home/cjones/src/icedtea6/plugin/icedteanp/IcedTeaNPPlugin.cc:1483
#17 start_jvm_if_needed () at /home/cjones/src/icedtea6/plugin/icedteanp/IcedTeaNPPlugin.cc:491
#18 0x00007fde8cb82d59 in GCJ_New (pluginType=<value optimized out>, instance=0x18b7e00, mode=<value optimized out>, argc=<value optimized out>, argn=0x18b7c30, argv=<value optimized out>, saved=0x0) at /home/cjones/src/icedtea6/plugin/icedteanp/IcedTeaNPPlugin.cc:310
#19 0x00007fde99d7c1ee in mozilla::plugins::PluginModuleChild::AnswerPPluginInstanceConstructor (this=0x1829d48, aActor=0x18b7db0, aMimeType=..., aMode=@0x7fde8f0c17a4, aNames=..., aValues=..., rv=0x7fde8f0c179c) at /home/cjones/mozilla/mozilla-central/dom/plugins/PluginModuleChild.cpp:1600
#20 0x00007fde99e06778 in mozilla::plugins::PPluginModuleChild::OnCallReceived (this=0x1829d48, msg=..., reply=@0x7fde8f0c1840) at PPluginModuleChild.cpp:483
#21 0x00007fde99d99b85 in mozilla::ipc::RPCChannel::DispatchIncall (this=0x1829d58, call=...) at /home/cjones/mozilla/mozilla-central/ipc/glue/RPCChannel.cpp:485
#22 0x00007fde99d99a9a in mozilla::ipc::RPCChannel::Incall (this=0x1829d58, call=..., stackDepth=0) at /home/cjones/mozilla/mozilla-central/ipc/glue/RPCChannel.cpp:471
#23 0x00007fde99d997e7 in mozilla::ipc::RPCChannel::OnMaybeDequeueOne (this=0x1829d58) at /home/cjones/mozilla/mozilla-central/ipc/glue/RPCChannel.cpp:413
#24 0x00007fde99d9f424 in DispatchToMethod<mozilla::ipc::RPCChannel, void (mozilla::ipc::RPCChannel::*)()> (obj=0x1829d58, method=0x7fde99d995c0 <mozilla::ipc::RPCChannel::OnMaybeDequeueOne()>, arg=...) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/tuple.h:383
#25 0x00007fde99d9f2cc in RunnableMethod<mozilla::ipc::RPCChannel, void (mozilla::ipc::RPCChannel::*)(), Tuple0>::Run (this=0x182ab30) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/task.h:307
#26 0x00007fde99d9b00b in mozilla::ipc::RPCChannel::RefCountedTask::Run (this=0x182ab70) at ../../dist/include/mozilla/ipc/RPCChannel.h:421
#27 0x00007fde99d9b10e in mozilla::ipc::RPCChannel::DequeueTask::Run (this=0x18b5af0) at ../../dist/include/mozilla/ipc/RPCChannel.h:446
#28 0x00007fde99f566ee in MessageLoop::RunTask (this=0x7fde8f0c1e20, task=0x18b5af0) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_loop.cc:336
#29 0x00007fde99f5675e in MessageLoop::DeferOrRunPendingTask (this=0x7fde8f0c1e20, pending_task=...) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_loop.cc:344
#30 0x00007fde99f56b5c in MessageLoop::DoWork (this=0x7fde8f0c1e20) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_loop.cc:444
#31 0x00007fde99fca101 in base::MessagePumpForUI::HandleDispatch (this=0x182c0a0) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_pump_glib.cc:264
#32 0x00007fde99fc977b in WorkSourceDispatch (source=0x182c1f0, unused_func=0, unused_data=0x0) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_pump_glib.cc:109
#33 0x00007fde94998bce in g_main_dispatch (context=0x182c110) at /build/buildd/glib2.0-2.22.3/glib/gmain.c:1960
#34 IA__g_main_context_dispatch (context=0x182c110) at /build/buildd/glib2.0-2.22.3/glib/gmain.c:2513
#35 0x00007fde9499c598 in g_main_context_iterate (context=0x182c110, block=<value optimized out>, dispatch=<value optimized out>, self=<value optimized out>) at /build/buildd/glib2.0-2.22.3/glib/gmain.c:2591
#36 0x00007fde9499c6c0 in IA__g_main_context_iteration (context=0x182c110, may_block=1) at /build/buildd/glib2.0-2.22.3/glib/gmain.c:2654
#37 0x00007fde99fc9e31 in base::MessagePumpForUI::RunWithDispatcher (this=0x182c0a0, delegate=0x7fde8f0c1e20, dispatcher=0x0) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_pump_glib.cc:195
#38 0x00007fde99fca4c6 in base::MessagePumpForUI::Run(base::MessagePump::Delegate*) () from ./libxul.so
#39 0x00007fde99f561f9 in MessageLoop::RunInternal (this=0x7fde8f0c1e20) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_loop.cc:216
#40 0x00007fde99f5617e in MessageLoop::RunHandler (this=0x7fde8f0c1e20) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_loop.cc:199
#41 0x00007fde99f5610f in MessageLoop::Run (this=0x7fde8f0c1e20) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/message_loop.cc:173
#42 0x00007fde99f7d406 in base::Thread::ThreadMain (this=0x1829ca0) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/thread.cc:165
#43 0x00007fde99fb0d6b in ThreadFunc (closure=0x1829ca0) at /home/cjones/mozilla/mozilla-central/ipc/chromium/src/base/platform_thread_posix.cc:26
#44 0x00007fde9ba0da04 in start_thread (arg=<value optimized out>) at pthread_create.c:300
#45 0x00007fde9710780d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#46 0x0000000000000000 in ?? ()

I'll try to see why we don't hit this in the in-process case ...

Comment 19

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Huh, I get the same crash running IcedTea in-process.

Shrug.  Definitely an IcedTea bug.  I'll patch it up, verify the fix, and file against IcedTea.

Comment 20

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Attachment: IcedTea fix (for archival purposes)

Wfm OOP and IP.

Comment 21

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Attachment: Hack around IcedTea bug by forcing it into debug mode

Still waiting for my account info for the IcedTea bugzilla.

I *really* hope IcedTea doesn't ship in the official Ubuntu 10.4 release with such an egregious bug, but we may be too short on time.  Here's a hacky "fix" to hedge our bets.  If IcedTea is fixed in time we can rip it out.

Comment 22

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Karl, the code this fix applies to is http://icedtea.classpath.org/hg/icedtea6/file/1188b1a313b9/plugin/icedteanp/IcedTeaNPPlugin.cc line 1462.

Comment 23

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Comment on attachment 436103 [details] [diff] [review]
Hack around IcedTea bug by forcing it into debug mode

In debug mode, the plugin opens a Java debug socket on localhost:8787

$ socklist
type  port      inode     uid    pid   fd  name
tcp   8787    1639542    1000   5951    5  java

This means other users on the machine can attach to the plugin and read possibly sensitive information (verified this with a program that seteuid's/setegid's to nobody/nogroup and connected to the debug socket).

So we're just going to have to blacklist this plugin until the bug is fixed.

Comment 24

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Attachment: Attempt to blocklist IcedTea (though not working locally)

I don't know if this is the right way to blocklist a plugin or from whom I should request review.  Will resume tomorrow.

We basically have three options for dealing with IcedTea

 (1) "OOP blacklist" it.  This exposes firefox-bin to definite crash cases and a possible security vulnerability
 (2) start linking mozilla-runtime with jemalloc so that IcedTea will sort of work in mozilla-runtime.  This pushes the browser problems into the plugin process.  (Karl rightly points out that this might cause plugin perf degradation if they're tuned to the libc allocator.  I'm going to spin this investigation off in its own bug.)
 (3) blocklist until the IcedTea bug is fixed

I personally lean towards (3), but my understanding is that Fedora Core 12 already uses IcedTeaPlugin, and I see little chance of this bug being fixed before Ubuntu 10.4 ships.

Comment 25

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Comment on attachment 436125 [details] [diff] [review]
Attempt to blocklist IcedTea (though not working locally)

See https://bugs.launchpad.net/ubuntu/+source/openjdk-6/+bug/552287.

Matthias Klose  wrote 7 hours ago:  	  #5

Re: https://bugzilla.mozilla.org/show_bug.cgi?id=555342#c24

the fix is in IcedTea and uploaded to lucid.

(3) afaik FC12 does use the IcedTeaPlugin.cc, not the IcedTeaNPPlugin.cc. The former is not affected.

No release enables the IcedTeaNPPlugin.cc by default.

Comment 26

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

(AFAICT the IcedTea fix should make it into Ubuntu 10.4 beta 2.)

Comment 27

[Daniel Holbert [:dholbert]]

Thanks for the fix, Chris! Great work. :)

Comment 28

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

FWIW bug 556198 will serve as a band-aid for this problem, hope to land it today.

Comment 29

[Daniel Holbert [:dholbert]]

FWIW, this is fixed for me, with previously-broken versions of Firefox -- hence, it looks like Chris's IcedTea patch (from comment 25) has made it into the Ubuntu repositories.

My current icedtea6-plugin version is "6b18~pre4-1ubuntu1", and it's WFM with today's nightly and with the previously-"doesn't work" nightly from comment 1. (I verified that "mozilla-runtime" is running in both instances, indicating that the plugin is indeed being run OOP without crashing.)