14.28 KB, text/plain
5.90 KB, text/plain
5.86 KB, text/plain
2.25 KB, patch
|Details | Diff | Splinter Review|
There have been occasional crashes at NSAddImage (deep in OS code), on OS X, for many years. But there's been a sudden, very large increase on 4.0b1. There are many different kinds of crash stacks whose "signature" is NSAddImage -- so it's likely that many of these crashes have different causes. But the four or five 4.0b1 crash stacks I looked at all look like what follows. That is, they all have nsPluginFile::LoadPlugin(PRLibrary*&) on the stack. So I'm calling this a plugins bug. 0 @0x1b3ad4e7 1 @0x1b3acdda 2 @0x1b3d6906 3 @0x8fe0ed6c 4 @0x8fe0d31d 5 @0x8fe0d3d0 6 @0x8fe0200a 7 @0x8fe0a520 8 @0x8fe0a799 9 libSystem.B.dylib NSAddImage 10 libnspr4.dylib pr_LoadViaDyld nsprpub/pr/src/linking/prlink.c:646 11 libnspr4.dylib PR_LoadLibraryWithFlags nsprpub/pr/src/linking/prlink.c:780 12 libnspr4.dylib PR_LoadLibrary nsprpub/pr/src/linking/prlink.c:475 13 XUL nsPluginFile::LoadPlugin modules/plugin/base/src/nsPluginsDirDarwin.cpp:254 14 XUL nsPluginHost::ScanPluginsDirectory modules/plugin/base/src/nsPluginHost.cpp:3308 15 XUL nsPluginHost::ScanPluginsDirectoryList modules/plugin/base/src/nsPluginHost.cpp:3416 16 XUL nsPluginHost::FindPlugins modules/plugin/base/src/nsPluginHost.cpp:3504 17 XUL nsPluginHost::LoadPlugins modules/plugin/base/src/nsPluginHost.cpp:3439 18 XUL nsPluginHost::IsFullPagePluginEnabledForType modules/plugin/base/src/nsPluginHost.cpp:2876 19 XUL nsIContentUtils::FindInternalContentViewer content/base/src/nsContentUtils.cpp:6191 20 XUL nsDocShell::CreateAboutBlankContentViewer docshell/base/nsDocShell.cpp:6195 21 XUL nsDocShell::EnsureContentViewer docshell/base/nsDocShell.cpp:6120 22 XUL nsDocShell::GetInterface docshell/base/nsDocShell.cpp:870 23 XUL nsGetInterface::operator const nsIInterfaceRequestorUtils.cpp:52 24 XUL nsCOMPtr_base::assign_from_helper nsCOMPtr.cpp:150 25 XUL nsGlobalWindow::GetDocument 26 XUL nsDataDocumentContentPolicy::ShouldLoad content/base/src/nsDataDocumentContentPolicy.cpp:71 27 XUL nsContentPolicy::ShouldLoad content/base/src/nsContentPolicy.cpp:157 28 XUL nsDocShell::InternalLoad cyUtils.h:223 29 XUL nsDocShell::LoadURI docshell/base/nsDocShell.cpp:1381 30 XUL nsWindowWatcher::OpenWindowJSInternal embedding/components/windowwatcher/src/nsWindowWatcher.cpp:945 31 XUL nsWindowWatcher::OpenWindow embedding/components/windowwatcher/src/nsWindowWatcher.cpp:422 32 XUL NS_InvokeByIndex_P xpcom/reflect/xptcall/src/md/unix/xptcinvoke_unixish_x86.cpp:179 33 XUL XPCWrappedNative::CallMethod js/src/xpconnect/src/xpcwrappednative.cpp:3028 34 XUL XPC_WN_CallMethod js/src/xpconnect/src/xpcwrappednativejsops.cpp:1789 35 libmozjs.dylib js_Invoke js/src/jsinterp.cpp:654 36 libmozjs.dylib js_Interpret js/src/jsops.cpp:2158 37 libmozjs.dylib js_Invoke js/src/jsinterp.cpp:664 38 XUL nsXPCWrappedJSClass::CallMethod js/src/xpconnect/src/xpcwrappedjsclass.cpp:1689 39 XUL nsXPCWrappedJS::CallMethod js/src/xpconnect/src/xpcwrappedjs.cpp:570 40 XUL PrepareAndDispatch xpcom/reflect/xptcall/src/md/unix/xptcstubs_unixish_x86.cpp:93 41 XUL nsXPTCStubBase::Stub3 42 XUL nsCommandLine::EnumerateHandlers toolkit/components/commandlines/src/nsCommandLine.cpp:603 43 XUL nsCommandLine::Run toolkit/components/commandlines/src/nsCommandLine.cpp:677 44 XUL XRE_main toolkit/xre/nsAppRunner.cpp:3595 45 firefox-bin main browser/app/nsBrowserApp.cpp:158 46 firefox-bin firefox-bin@0xbf5
Keywords: crash, topcrash
Apparently all these crashes (those on 4.0b1) happen on startup: http://hg.mozilla.org/mozilla-central/annotate/65c30e4ee631/toolkit/xre/nsAppRunner.cpp#l3595
(I can't find a keyword for startup crashes.)
A couple of comments indicate that this crash happens after installing the "VMP Plugin" (http://www.viewpoint.com/technologies/viewpoint-media-player.shtml) and (presumably) restarting. This plugin doesn't appear in the correlation data (at http://people.mozilla.com/crash_analysis/). But fairly strong correlations with the WebKit and WebCore frameworks do appear there. We need to find out if the "VMP plugin" uses WebKit/WebCore.
I'm able to reproduce this bug by installing the VMP plugin and then restarting Firefox. The crash keeps happening on subsequent restarts. My crash stacks are slightly different, but I'll bet they have the same cause. bp-2434d951-492c-44c2-b181-ef5752100711 bp-91e7995b-35d7-4d3f-9192-246962100711 Hooray! I'll take this bug. (I tested on OS X 10.5.8, with a mozilla-central nightly from last week.)
Assignee: nobody → smichaud
Status: NEW → ASSIGNED
> We need to find out if the "VMP plugin" uses WebKit/WebCore. It does appear to, though only indirectly.
Summary: Crash [@ NSAddImage ] → Crash [@ NSAddImage ] on startup
Summary: Crash [@ NSAddImage ] on startup → Crash [@ NSAddImage ] on startup on 2.0 branch (FF 4.0b1)
It turns out that, once the VMP Plugin has been successfully loaded (without triggering a crash) in any Firefox version, it no longer crashes any version of Firefox on startup. So if (for example) you run FF 3.6.4 once with the VMP Plugin installed, you'll no longer crash on startup even in the most recent Minefield nightlies. So here's revised STR: 1) If the VMP Plugin is installed, first uninstall it. To do this remove the following directories and their contents: a) /Library/Receipts/VMP.pkg/ b) /Library/Application Support/Viewpoint/ Also remove the following soft link: c) /Library/Internet Plug-Ins/npViewpoint.plugin 2) Install the VMP Plugin. 3) Run a recent Minefield nightly -- you should crash on startup.
Oops, this still isn't right. To make the crash happen (in susceptible versions of Minefield) you just need to delete pluginreg.dat before starting the browser. So here's yet another revised STR: 1) Make sure the VMP Plugin is installed. 2) Delete the pluginreg.dat file if present. This file is in ~/Library/Application Support/Profiles/[profile-dir]/. 3) Run a recent Minefield nightly -- you should crash on startup.
Here's the regression range for my STR from comment #7: firefox-2010-02-08-03-mozilla-central firefox-2010-02-09-03-mozilla-central http://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2010-02-08+03%3A00%3A00&enddate=2010-02-09+03%3A00%3A00 This probably implicates the patch for bug 544579 (http://hg.mozilla.org/mozilla-central/rev/9ab5faed990f).
> This probably implicates the patch for bug 544579 > (http://hg.mozilla.org/mozilla-central/rev/9ab5faed990f). I've confirmed that the patch for bug 544579 does trigger this bug. My STR from comment #7 "works" (i.e. crashes) on a build "updated" to http://hg.mozilla.org/mozilla-central/rev/9ab5faed990f, and doesn't work on a build "updated" to the previous patch (http://hg.mozilla.org/mozilla-central/rev/4760079804e4).
I've figured out why this bug happens. The patch for bug 544579 changed when the initialization code in a bundle's executable runs: On the trunk (as of the patch for bug 544579) the initialization code runs as the bundle is "loaded" (during the call to PR_LoadLibrary()). On the 1.9.2 branch and earlier, the initialization code only runs the first time you try to resolve a symbol in the bundle (the first time you call PR_FindSymbol() on it). For most bundles this makes no difference. But some have problems if their initialization code runs too early -- the VMP Plugin is an example. I haven't been able to figure out exactly why the VMP Plugin's initialization code crashes when run too early. But it's clear that timing is a crucial factor -- backing out the patch for bug 544579 in 32-bit builds makes my STR from comment #7 no longer "work" (it makes the crashes go away). And it's likely the VMP Plugin isn't doing anything wrong -- since there are clearly many other plugin bundles that have the same problem. I suspect this problem also exists on 64-bit builds, so ultimately we should completely back out the patch for bug 544579. Currently NSPR can't use the "normal" method for loading bundles (CFBundleCreate()) on 64-bit builds. But (as the patch for bug 544579 shows) CFBundleCreate() works just fine on 64-bit builds, and at some point we will need to change NSPR to work the same way on 64-bit builds as it does on 32-bit builds (on OS X). In the meantime there are very few 64-bit plugin bundles, which (presumably) makes this bug much less likely to happen with them.
Created attachment 457075 [details] Gdb stack trace of crash on OS X 10.5.8 Here's a stack trace of this bug's crash (following the STR from comment #7) on OS X 10.5.8. The stack was taken just before the crash, at a call to printf() that the VMP Plugin happens to make when errors start happening that lead to the crash. Gdb itself crashes if I try to use "bt" after the crash has occurred!
Created attachment 457081 [details] Stack trace of plugin init code on the trunk Here and in the next comment are stack traces which show when a plugin bundle gets loaded on the trunk and the 1.9.2 branch. They were made using my DebugEventsPlugin from bug 441880 (I set a breakpoint at "loadHandler::loadHandler").
Attachment #457081 - Attachment description: Stack trace of loading a plugin on the trunk → Stack trace of plugin init code on the trunk
Attachment #457082 - Attachment description: Stack trace of loading a plugin on the 1.9.2 branch → Stack trace of plugin init code on the 1.9.2 branch
Created attachment 457120 [details] [diff] [review] Fix
Attachment #457120 - Flags: review?(joshmoz)
(Unfortunately, I'm going to be gone (and largely away from the net) for the rest of this week and part of next week.)
Comment on attachment 457120 [details] [diff] [review] Fix Landed on trunk: http://hg.mozilla.org/mozilla-central/rev/a6e8d20f950e
Status: ASSIGNED → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.