577967 - Crash [@ NSAddImage ] on startup on 2.0 branch (FF 4.0b1)

Status: RESOLVED FIXED

(Core Graveyard :: Plug-ins, defect)

Description

[Steven Michaud [:smichaud] (Retired)]

There have been occasional crashes at NSAddImage (deep in OS code), on
OS X, for many years.  But there's been a sudden, very large increase
on 4.0b1.

There are many different kinds of crash stacks whose "signature" is
NSAddImage -- so it's likely that many of these crashes have different
causes.  But the four or five 4.0b1 crash stacks I looked at all look
like what follows.  That is, they all have
nsPluginFile::LoadPlugin(PRLibrary*&) on the stack.  So I'm calling
this a plugins bug.

0   @0x1b3ad4e7
1   @0x1b3acdda
2   @0x1b3d6906
3   @0x8fe0ed6c
4   @0x8fe0d31d
5   @0x8fe0d3d0
6   @0x8fe0200a
7   @0x8fe0a520
8   @0x8fe0a799
9   libSystem.B.dylib  NSAddImage
10  libnspr4.dylib     pr_LoadViaDyld
                         nsprpub/pr/src/linking/prlink.c:646
11  libnspr4.dylib     PR_LoadLibraryWithFlags
                         nsprpub/pr/src/linking/prlink.c:780
12  libnspr4.dylib     PR_LoadLibrary
                         nsprpub/pr/src/linking/prlink.c:475
13  XUL                nsPluginFile::LoadPlugin
                         modules/plugin/base/src/nsPluginsDirDarwin.cpp:254
14  XUL                nsPluginHost::ScanPluginsDirectory
                         modules/plugin/base/src/nsPluginHost.cpp:3308
15  XUL                nsPluginHost::ScanPluginsDirectoryList
                         modules/plugin/base/src/nsPluginHost.cpp:3416
16  XUL                nsPluginHost::FindPlugins
                         modules/plugin/base/src/nsPluginHost.cpp:3504
17  XUL                nsPluginHost::LoadPlugins
                         modules/plugin/base/src/nsPluginHost.cpp:3439
18  XUL                nsPluginHost::IsFullPagePluginEnabledForType
                         modules/plugin/base/src/nsPluginHost.cpp:2876
19  XUL                nsIContentUtils::FindInternalContentViewer
                         content/base/src/nsContentUtils.cpp:6191
20  XUL                nsDocShell::CreateAboutBlankContentViewer
                         docshell/base/nsDocShell.cpp:6195
21  XUL                nsDocShell::EnsureContentViewer
                         docshell/base/nsDocShell.cpp:6120
22  XUL                nsDocShell::GetInterface
                         docshell/base/nsDocShell.cpp:870
23  XUL                nsGetInterface::operator const
                         nsIInterfaceRequestorUtils.cpp:52
24  XUL                nsCOMPtr_base::assign_from_helper
                         nsCOMPtr.cpp:150
25  XUL                nsGlobalWindow::GetDocument
26  XUL                nsDataDocumentContentPolicy::ShouldLoad
                         content/base/src/nsDataDocumentContentPolicy.cpp:71
27  XUL                nsContentPolicy::ShouldLoad
                         content/base/src/nsContentPolicy.cpp:157
28  XUL                nsDocShell::InternalLoad
                         cyUtils.h:223
29  XUL                nsDocShell::LoadURI
                         docshell/base/nsDocShell.cpp:1381
30  XUL                nsWindowWatcher::OpenWindowJSInternal
                         embedding/components/windowwatcher/src/nsWindowWatcher.cpp:945
31  XUL                nsWindowWatcher::OpenWindow
                         embedding/components/windowwatcher/src/nsWindowWatcher.cpp:422
32  XUL                NS_InvokeByIndex_P
                         xpcom/reflect/xptcall/src/md/unix/xptcinvoke_unixish_x86.cpp:179
33  XUL                XPCWrappedNative::CallMethod
                         js/src/xpconnect/src/xpcwrappednative.cpp:3028
34  XUL                XPC_WN_CallMethod
                         js/src/xpconnect/src/xpcwrappednativejsops.cpp:1789
35  libmozjs.dylib     js_Invoke
                         js/src/jsinterp.cpp:654
36  libmozjs.dylib     js_Interpret
                         js/src/jsops.cpp:2158
37  libmozjs.dylib     js_Invoke
                         js/src/jsinterp.cpp:664
38  XUL                nsXPCWrappedJSClass::CallMethod
                         js/src/xpconnect/src/xpcwrappedjsclass.cpp:1689
39  XUL                nsXPCWrappedJS::CallMethod
                         js/src/xpconnect/src/xpcwrappedjs.cpp:570
40  XUL                PrepareAndDispatch
                         xpcom/reflect/xptcall/src/md/unix/xptcstubs_unixish_x86.cpp:93
41  XUL                nsXPTCStubBase::Stub3
42  XUL                nsCommandLine::EnumerateHandlers
                         toolkit/components/commandlines/src/nsCommandLine.cpp:603
43  XUL                nsCommandLine::Run
                         toolkit/components/commandlines/src/nsCommandLine.cpp:677
44  XUL                XRE_main
                         toolkit/xre/nsAppRunner.cpp:3595
45  firefox-bin        main
                         browser/app/nsBrowserApp.cpp:158
46  firefox-bin        firefox-bin@0xbf5

Comment 1

[Steven Michaud [:smichaud] (Retired)]

Apparently all these crashes (those on 4.0b1) happen on startup:

http://hg.mozilla.org/mozilla-central/annotate/65c30e4ee631/toolkit/xre/nsAppRunner.cpp#l3595

Comment 2

[Steven Michaud [:smichaud] (Retired)]

(I can't find a keyword for startup crashes.)

Comment 3

[Steven Michaud [:smichaud] (Retired)]

A couple of comments indicate that this crash happens after installing
the "VMP Plugin"
(http://www.viewpoint.com/technologies/viewpoint-media-player.shtml)
and (presumably) restarting.

This plugin doesn't appear in the correlation data (at
http://people.mozilla.com/crash_analysis/).  But fairly strong
correlations with the WebKit and WebCore frameworks do appear there.

We need to find out if the "VMP plugin" uses WebKit/WebCore.

Comment 4

[Steven Michaud [:smichaud] (Retired)]

I'm able to reproduce this bug by installing the VMP plugin and then restarting Firefox.  The crash keeps happening on subsequent restarts.

My crash stacks are slightly different, but I'll bet they have the same cause.

bp-2434d951-492c-44c2-b181-ef5752100711
bp-91e7995b-35d7-4d3f-9192-246962100711

Hooray!  I'll take this bug.

(I tested on OS X 10.5.8, with a mozilla-central nightly from last week.)

Comment 5

[Steven Michaud [:smichaud] (Retired)]

> We need to find out if the "VMP plugin" uses WebKit/WebCore.

It does appear to, though only indirectly.

Comment 6

[Steven Michaud [:smichaud] (Retired)]

It turns out that, once the VMP Plugin has been successfully loaded
(without triggering a crash) in any Firefox version, it no longer
crashes any version of Firefox on startup.  So if (for example) you
run FF 3.6.4 once with the VMP Plugin installed, you'll no longer
crash on startup even in the most recent Minefield nightlies.

So here's revised STR:

1) If the VMP Plugin is installed, first uninstall it.  To do this
   remove the following directories and their contents:

   a) /Library/Receipts/VMP.pkg/
   b) /Library/Application Support/Viewpoint/

   Also remove the following soft link:

   c) /Library/Internet Plug-Ins/npViewpoint.plugin

2) Install the VMP Plugin.

3) Run a recent Minefield nightly -- you should crash on startup.

Comment 7

[Steven Michaud [:smichaud] (Retired)]

Oops, this still isn't right.  To make the crash happen (in
susceptible versions of Minefield) you just need to delete
pluginreg.dat before starting the browser.

So here's yet another revised STR:

1) Make sure the VMP Plugin is installed.

2) Delete the pluginreg.dat file if present.

   This file is in ~/Library/Application Support/Profiles/[profile-dir]/.

3) Run a recent Minefield nightly -- you should crash on startup.

Comment 8

[Steven Michaud [:smichaud] (Retired)]

Here's the regression range for my STR from comment #7:

firefox-2010-02-08-03-mozilla-central
firefox-2010-02-09-03-mozilla-central

http://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2010-02-08+03%3A00%3A00&enddate=2010-02-09+03%3A00%3A00

This probably implicates the patch for bug 544579 (http://hg.mozilla.org/mozilla-central/rev/9ab5faed990f).

Comment 9

[Steven Michaud [:smichaud] (Retired)]

> This probably implicates the patch for bug 544579
> (http://hg.mozilla.org/mozilla-central/rev/9ab5faed990f).

I've confirmed that the patch for bug 544579 does trigger this bug.  My STR from comment #7 "works" (i.e. crashes) on a build "updated" to http://hg.mozilla.org/mozilla-central/rev/9ab5faed990f, and doesn't work on a build "updated" to the previous patch (http://hg.mozilla.org/mozilla-central/rev/4760079804e4).

Comment 10

[Steven Michaud [:smichaud] (Retired)]

I've figured out why this bug happens.

The patch for bug 544579 changed when the initialization code in a
bundle's executable runs:  On the trunk (as of the patch for bug
544579) the initialization code runs as the bundle is "loaded" (during
the call to PR_LoadLibrary()).  On the 1.9.2 branch and earlier, the
initialization code only runs the first time you try to resolve a
symbol in the bundle (the first time you call PR_FindSymbol() on it).

For most bundles this makes no difference.  But some have problems if
their initialization code runs too early -- the VMP Plugin is an
example.

I haven't been able to figure out exactly why the VMP Plugin's
initialization code crashes when run too early.  But it's clear that
timing is a crucial factor -- backing out the patch for bug 544579 in
32-bit builds makes my STR from comment #7 no longer "work" (it makes
the crashes go away).  And it's likely the VMP Plugin isn't doing
anything wrong -- since there are clearly many other plugin bundles
that have the same problem.

I suspect this problem also exists on 64-bit builds, so ultimately we
should completely back out the patch for bug 544579.  Currently NSPR
can't use the "normal" method for loading bundles (CFBundleCreate())
on 64-bit builds.  But (as the patch for bug 544579 shows)
CFBundleCreate() works just fine on 64-bit builds, and at some point
we will need to change NSPR to work the same way on 64-bit builds as
it does on 32-bit builds (on OS X).  In the meantime there are very
few 64-bit plugin bundles, which (presumably) makes this bug much less
likely to happen with them.

Comment 11

[Steven Michaud [:smichaud] (Retired)]

Attachment: Gdb stack trace of crash on OS X 10.5.8

Here's a stack trace of this bug's crash (following the STR from
comment #7) on OS X 10.5.8.

The stack was taken just before the crash, at a call to printf() that
the VMP Plugin happens to make when errors start happening that lead
to the crash.  Gdb itself crashes if I try to use "bt" after the crash
has occurred!

Comment 12

[Steven Michaud [:smichaud] (Retired)]

Attachment: Stack trace of plugin init code on the trunk

Here and in the next comment are stack traces which show when a plugin
bundle gets loaded on the trunk and the 1.9.2 branch.  They were made
using my DebugEventsPlugin from bug 441880 (I set a breakpoint at
"loadHandler::loadHandler").

Comment 13

[Steven Michaud [:smichaud] (Retired)]

Attachment: Stack trace of plugin init code on the 1.9.2 branch

Comment 14

[Steven Michaud [:smichaud] (Retired)]

Attachment: Fix

Comment 15

[Steven Michaud [:smichaud] (Retired)]

(Unfortunately, I'm going to be gone (and largely away from the net) for the rest of this week and part of next week.)

Comment 16

[Steven Michaud [:smichaud] (Retired)]

Comment on attachment 457120 [details] [diff] [review]
Fix

Landed on trunk:
http://hg.mozilla.org/mozilla-central/rev/a6e8d20f950e