Crash [@ NSAddImage ] on startup on 2.0 branch (FF 4.0b1)

RESOLVED FIXED

Status

()

--
critical
RESOLVED FIXED
8 years ago
7 years ago

People

(Reporter: smichaud, Assigned: smichaud)

Tracking

({crash, topcrash})

Trunk
All
Mac OS X
crash, topcrash
Points:
---

Firefox Tracking Flags

(blocking2.0 beta3+)

Details

(Whiteboard: [4b1], crash signature)

Attachments

(4 attachments)

There have been occasional crashes at NSAddImage (deep in OS code), on
OS X, for many years.  But there's been a sudden, very large increase
on 4.0b1.

There are many different kinds of crash stacks whose "signature" is
NSAddImage -- so it's likely that many of these crashes have different
causes.  But the four or five 4.0b1 crash stacks I looked at all look
like what follows.  That is, they all have
nsPluginFile::LoadPlugin(PRLibrary*&) on the stack.  So I'm calling
this a plugins bug.

0   @0x1b3ad4e7
1   @0x1b3acdda
2   @0x1b3d6906
3   @0x8fe0ed6c
4   @0x8fe0d31d
5   @0x8fe0d3d0
6   @0x8fe0200a
7   @0x8fe0a520
8   @0x8fe0a799
9   libSystem.B.dylib  NSAddImage
10  libnspr4.dylib     pr_LoadViaDyld
                         nsprpub/pr/src/linking/prlink.c:646
11  libnspr4.dylib     PR_LoadLibraryWithFlags
                         nsprpub/pr/src/linking/prlink.c:780
12  libnspr4.dylib     PR_LoadLibrary
                         nsprpub/pr/src/linking/prlink.c:475
13  XUL                nsPluginFile::LoadPlugin
                         modules/plugin/base/src/nsPluginsDirDarwin.cpp:254
14  XUL                nsPluginHost::ScanPluginsDirectory
                         modules/plugin/base/src/nsPluginHost.cpp:3308
15  XUL                nsPluginHost::ScanPluginsDirectoryList
                         modules/plugin/base/src/nsPluginHost.cpp:3416
16  XUL                nsPluginHost::FindPlugins
                         modules/plugin/base/src/nsPluginHost.cpp:3504
17  XUL                nsPluginHost::LoadPlugins
                         modules/plugin/base/src/nsPluginHost.cpp:3439
18  XUL                nsPluginHost::IsFullPagePluginEnabledForType
                         modules/plugin/base/src/nsPluginHost.cpp:2876
19  XUL                nsIContentUtils::FindInternalContentViewer
                         content/base/src/nsContentUtils.cpp:6191
20  XUL                nsDocShell::CreateAboutBlankContentViewer
                         docshell/base/nsDocShell.cpp:6195
21  XUL                nsDocShell::EnsureContentViewer
                         docshell/base/nsDocShell.cpp:6120
22  XUL                nsDocShell::GetInterface
                         docshell/base/nsDocShell.cpp:870
23  XUL                nsGetInterface::operator const
                         nsIInterfaceRequestorUtils.cpp:52
24  XUL                nsCOMPtr_base::assign_from_helper
                         nsCOMPtr.cpp:150
25  XUL                nsGlobalWindow::GetDocument
26  XUL                nsDataDocumentContentPolicy::ShouldLoad
                         content/base/src/nsDataDocumentContentPolicy.cpp:71
27  XUL                nsContentPolicy::ShouldLoad
                         content/base/src/nsContentPolicy.cpp:157
28  XUL                nsDocShell::InternalLoad
                         cyUtils.h:223
29  XUL                nsDocShell::LoadURI
                         docshell/base/nsDocShell.cpp:1381
30  XUL                nsWindowWatcher::OpenWindowJSInternal
                         embedding/components/windowwatcher/src/nsWindowWatcher.cpp:945
31  XUL                nsWindowWatcher::OpenWindow
                         embedding/components/windowwatcher/src/nsWindowWatcher.cpp:422
32  XUL                NS_InvokeByIndex_P
                         xpcom/reflect/xptcall/src/md/unix/xptcinvoke_unixish_x86.cpp:179
33  XUL                XPCWrappedNative::CallMethod
                         js/src/xpconnect/src/xpcwrappednative.cpp:3028
34  XUL                XPC_WN_CallMethod
                         js/src/xpconnect/src/xpcwrappednativejsops.cpp:1789
35  libmozjs.dylib     js_Invoke
                         js/src/jsinterp.cpp:654
36  libmozjs.dylib     js_Interpret
                         js/src/jsops.cpp:2158
37  libmozjs.dylib     js_Invoke
                         js/src/jsinterp.cpp:664
38  XUL                nsXPCWrappedJSClass::CallMethod
                         js/src/xpconnect/src/xpcwrappedjsclass.cpp:1689
39  XUL                nsXPCWrappedJS::CallMethod
                         js/src/xpconnect/src/xpcwrappedjs.cpp:570
40  XUL                PrepareAndDispatch
                         xpcom/reflect/xptcall/src/md/unix/xptcstubs_unixish_x86.cpp:93
41  XUL                nsXPTCStubBase::Stub3
42  XUL                nsCommandLine::EnumerateHandlers
                         toolkit/components/commandlines/src/nsCommandLine.cpp:603
43  XUL                nsCommandLine::Run
                         toolkit/components/commandlines/src/nsCommandLine.cpp:677
44  XUL                XRE_main
                         toolkit/xre/nsAppRunner.cpp:3595
45  firefox-bin        main
                         browser/app/nsBrowserApp.cpp:158
46  firefox-bin        firefox-bin@0xbf5
(Assignee)

Updated

8 years ago
Keywords: crash, topcrash
Whiteboard: [4b1]
(Assignee)

Comment 1

8 years ago
Apparently all these crashes (those on 4.0b1) happen on startup:

http://hg.mozilla.org/mozilla-central/annotate/65c30e4ee631/toolkit/xre/nsAppRunner.cpp#l3595
(Assignee)

Comment 2

8 years ago
(I can't find a keyword for startup crashes.)
(Assignee)

Comment 3

8 years ago
A couple of comments indicate that this crash happens after installing
the "VMP Plugin"
(http://www.viewpoint.com/technologies/viewpoint-media-player.shtml)
and (presumably) restarting.

This plugin doesn't appear in the correlation data (at
http://people.mozilla.com/crash_analysis/).  But fairly strong
correlations with the WebKit and WebCore frameworks do appear there.

We need to find out if the "VMP plugin" uses WebKit/WebCore.
(Assignee)

Comment 4

8 years ago
I'm able to reproduce this bug by installing the VMP plugin and then restarting Firefox.  The crash keeps happening on subsequent restarts.

My crash stacks are slightly different, but I'll bet they have the same cause.

bp-2434d951-492c-44c2-b181-ef5752100711
bp-91e7995b-35d7-4d3f-9192-246962100711

Hooray!  I'll take this bug.

(I tested on OS X 10.5.8, with a mozilla-central nightly from last week.)
Assignee: nobody → smichaud
Status: NEW → ASSIGNED
(Assignee)

Comment 5

8 years ago
> We need to find out if the "VMP plugin" uses WebKit/WebCore.

It does appear to, though only indirectly.
(Assignee)

Updated

8 years ago
Summary: Crash [@ NSAddImage ] → Crash [@ NSAddImage ] on startup
(Assignee)

Updated

8 years ago
Summary: Crash [@ NSAddImage ] on startup → Crash [@ NSAddImage ] on startup on 2.0 branch (FF 4.0b1)
(Assignee)

Updated

8 years ago
blocking2.0: --- → ?
(Assignee)

Comment 6

8 years ago
It turns out that, once the VMP Plugin has been successfully loaded
(without triggering a crash) in any Firefox version, it no longer
crashes any version of Firefox on startup.  So if (for example) you
run FF 3.6.4 once with the VMP Plugin installed, you'll no longer
crash on startup even in the most recent Minefield nightlies.

So here's revised STR:

1) If the VMP Plugin is installed, first uninstall it.  To do this
   remove the following directories and their contents:

   a) /Library/Receipts/VMP.pkg/
   b) /Library/Application Support/Viewpoint/

   Also remove the following soft link:

   c) /Library/Internet Plug-Ins/npViewpoint.plugin

2) Install the VMP Plugin.

3) Run a recent Minefield nightly -- you should crash on startup.
(Assignee)

Comment 7

8 years ago
Oops, this still isn't right.  To make the crash happen (in
susceptible versions of Minefield) you just need to delete
pluginreg.dat before starting the browser.

So here's yet another revised STR:

1) Make sure the VMP Plugin is installed.

2) Delete the pluginreg.dat file if present.

   This file is in ~/Library/Application Support/Profiles/[profile-dir]/.

3) Run a recent Minefield nightly -- you should crash on startup.
(Assignee)

Comment 8

8 years ago
Here's the regression range for my STR from comment #7:

firefox-2010-02-08-03-mozilla-central
firefox-2010-02-09-03-mozilla-central

http://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2010-02-08+03%3A00%3A00&enddate=2010-02-09+03%3A00%3A00

This probably implicates the patch for bug 544579 (http://hg.mozilla.org/mozilla-central/rev/9ab5faed990f).
(Assignee)

Comment 9

8 years ago
> This probably implicates the patch for bug 544579
> (http://hg.mozilla.org/mozilla-central/rev/9ab5faed990f).

I've confirmed that the patch for bug 544579 does trigger this bug.  My STR from comment #7 "works" (i.e. crashes) on a build "updated" to http://hg.mozilla.org/mozilla-central/rev/9ab5faed990f, and doesn't work on a build "updated" to the previous patch (http://hg.mozilla.org/mozilla-central/rev/4760079804e4).
I've figured out why this bug happens.

The patch for bug 544579 changed when the initialization code in a
bundle's executable runs:  On the trunk (as of the patch for bug
544579) the initialization code runs as the bundle is "loaded" (during
the call to PR_LoadLibrary()).  On the 1.9.2 branch and earlier, the
initialization code only runs the first time you try to resolve a
symbol in the bundle (the first time you call PR_FindSymbol() on it).

For most bundles this makes no difference.  But some have problems if
their initialization code runs too early -- the VMP Plugin is an
example.

I haven't been able to figure out exactly why the VMP Plugin's
initialization code crashes when run too early.  But it's clear that
timing is a crucial factor -- backing out the patch for bug 544579 in
32-bit builds makes my STR from comment #7 no longer "work" (it makes
the crashes go away).  And it's likely the VMP Plugin isn't doing
anything wrong -- since there are clearly many other plugin bundles
that have the same problem.

I suspect this problem also exists on 64-bit builds, so ultimately we
should completely back out the patch for bug 544579.  Currently NSPR
can't use the "normal" method for loading bundles (CFBundleCreate())
on 64-bit builds.  But (as the patch for bug 544579 shows)
CFBundleCreate() works just fine on 64-bit builds, and at some point
we will need to change NSPR to work the same way on 64-bit builds as
it does on 32-bit builds (on OS X).  In the meantime there are very
few 64-bit plugin bundles, which (presumably) makes this bug much less
likely to happen with them.
Created attachment 457075 [details]
Gdb stack trace of crash on OS X 10.5.8

Here's a stack trace of this bug's crash (following the STR from
comment #7) on OS X 10.5.8.

The stack was taken just before the crash, at a call to printf() that
the VMP Plugin happens to make when errors start happening that lead
to the crash.  Gdb itself crashes if I try to use "bt" after the crash
has occurred!
Created attachment 457081 [details]
Stack trace of plugin init code on the trunk

Here and in the next comment are stack traces which show when a plugin
bundle gets loaded on the trunk and the 1.9.2 branch.  They were made
using my DebugEventsPlugin from bug 441880 (I set a breakpoint at
"loadHandler::loadHandler").
Created attachment 457082 [details]
Stack trace of plugin init code on the 1.9.2 branch
(Assignee)

Updated

8 years ago
Attachment #457081 - Attachment description: Stack trace of loading a plugin on the trunk → Stack trace of plugin init code on the trunk
(Assignee)

Updated

8 years ago
Attachment #457082 - Attachment description: Stack trace of loading a plugin on the 1.9.2 branch → Stack trace of plugin init code on the 1.9.2 branch
Created attachment 457120 [details] [diff] [review]
Fix
Attachment #457120 - Flags: review?(joshmoz)
(Unfortunately, I'm going to be gone (and largely away from the net) for the rest of this week and part of next week.)

Updated

8 years ago
blocking2.0: ? → beta3+

Updated

8 years ago
Attachment #457120 - Flags: review?(joshmoz) → review+
(Assignee)

Updated

8 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Crash Signature: [@ NSAddImage ]
You need to log in before you can comment on or make changes to this bug.