746254 - crash in nsPluginInstanceOwner::FixUpPluginWindow caused/triggered by AdobePDFViewerNPAPI plugin in 32-bit mode

Status: RESOLVED WORKSFORME

(Core Graveyard :: Plug-ins, defect)

Description

[Scoobidiver (away)]

It's #1 top crasher in 13.0a2 on Mac OS X.

According to some comments, it's related to the PDF plugin.

Signature 	nsPluginInstanceOwner::FixUpPluginWindow More Reports Search
UUID	19de699b-e26b-434b-84fe-cce9f2120415
Date Processed	2012-04-15 19:11:51
Uptime	325
Last Crash	3.6 days before submission
Install Age	6.8 minutes since version was first installed.
Install Time	2012-04-15 19:04:14
Product	Firefox
Version	13.0a2
Build ID	20120415042011
Release Channel	aurora
OS	Mac OS X
OS Version	10.6.8 10K549
Build Architecture	x86
Build Architecture Info	family 6 model 14 stepping 8
Crash Reason	EXC_BAD_ACCESS / KERN_PROTECTION_FAILURE
Crash Address	0x4
User Comments	I was trying to open a pdf within Aurora. I guess the plugin did not work.
App Notes 	
AdapterVendorID: 0x8086, AdapterDeviceID: 0x27a2

Frame 	Module 	Signature 	Source
0 	XUL 	nsPluginInstanceOwner::FixUpPluginWindow 	dom/plugins/base/nsPluginInstanceOwner.cpp:3497
1 	XUL 	nsObjectFrame::CallSetWindow 	layout/generic/nsObjectFrame.cpp:728
2 	XUL 	nsPluginInstanceOwner::CallSetWindow 	dom/plugins/base/nsPluginInstanceOwner.cpp:3706
3 	XUL 	nsPluginHost::InstantiateFullPagePlugin 	dom/plugins/base/nsPluginHost.cpp:1168
4 	XUL 	nsObjectLoadingContent::InstantiatePluginInstance 	content/base/src/nsObjectLoadingContent.cpp:635
5 	XUL 	mozilla::dom::PluginStreamListener::SetupPlugin 	content/html/document/src/PluginDocument.cpp:156
6 	XUL 	mozilla::dom::PluginStreamListener::OnStartRequest 	content/html/document/src/PluginDocument.cpp:121
7 	XUL 	nsDocumentOpenInfo::OnStartRequest 	uriloader/base/nsURILoader.cpp:304
8 	XUL 	nsHttpChannel::CallOnStartRequest 	netwerk/protocol/http/nsHttpChannel.cpp:764
9 	XUL 	nsHttpChannel::ContinueOnStartRequest2 	netwerk/protocol/http/nsHttpChannel.cpp:4245
10 	XUL 	nsHttpChannel::OnStartRequest 	netwerk/protocol/http/nsHttpChannel.cpp:4209
11 	XUL 	nsInputStreamPump::OnStateStart 	netwerk/base/src/nsInputStreamPump.cpp:444
12 	XUL 	nsInputStreamPump::OnInputStreamReady 	netwerk/base/src/nsInputStreamPump.cpp:399
13 	XUL 	nsInputStreamReadyEvent::Run 	xpcom/io/nsStreamUtils.cpp:114
14 	XUL 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:657
15 	XUL 	NS_ProcessPendingEvents_P 	obj-firefox/i386/xpcom/build/nsThreadUtils.cpp:195
16 	XUL 	nsBaseAppShell::NativeEventCallback 	widget/xpwidgets/nsBaseAppShell.cpp:130
17 	XUL 	nsAppShell::ProcessGeckoEvents 	widget/cocoa/nsAppShell.mm:441
18 	CoreFoundation 	__CFRunLoopDoSources0

More reports at:
https://crash-stats.mozilla.com/report/list?signature=nsPluginInstanceOwner%3A%3AFixUpPluginWindow

Comment 1

[Steven Michaud [:smichaud] (Retired)]

> According to some comments, it's related to the PDF plugin.

In fact it's 100% correlated with Adobe's AdobePDFViewerNPAPI plugin.  A new version must have come out in the last few weeks, which is causing/triggering these crashes.

Comment 2

[Steven Michaud [:smichaud] (Retired)]

Also note that these crashes happen only in 32-bit mode.

Marcia, I'll ask you to CC the appropriate Adobe contact for these ... if you can.

Comment 3

[Steven Michaud [:smichaud] (Retired)]

These crashes jump suddenly in volume on 2012-04-10.  Adobe must have released a new version of the AdobePDFViewerNPAPI plugin on or very near that date.

Comment 4

[Steven Michaud [:smichaud] (Retired)]

These crashes happen at:
http://hg.mozilla.org/mozilla-central/annotate/3fa30b0edd15/dom/plugins/base/nsPluginInstanceOwner.cpp#l3505

I'd bet the proximate cause is that 'pluginPort' is unexpectedly null.

Comment 5

[Steven Michaud [:smichaud] (Retired)]

It'd be nice to get good STR for these :-)

Comment 6

[Steven Michaud [:smichaud] (Retired)]

When (at https://crash-stats.mozilla.com/) you do module correlations by versions, you find that these crashes are 100% correlated with the following version of AdobePDFViewerNPAPI:

AdobePDFViewerNPAPI (6075D23DB256536CF67C5367946FA0B20)

Comment 7

[Scoobidiver (away)]

(In reply to Steven Michaud from comment #1)
> In fact it's 100% correlated with Adobe's AdobePDFViewerNPAPI plugin.  A new
> version must have come out in the last few weeks, which is
> causing/triggering these crashes.
There's likely a code change in Firefox that makes this crash happen in 13.0 and above.

Comment 8

[Josh Aas]

Rudi - can you help us debug this?

Comment 9

[Steven Michaud [:smichaud] (Retired)]

> There's likely a code change in Firefox that makes this crash happen in 13.0 and
> above.

Possibly.  But it's also possible that the presumed AdobePDFViewerNPAPI bug just has a different signature in FF 12 and below.

For example there's a high volume crash at Acrobat@0xefeb63 in FF 12 and below, which appears to have started at the same time, and which is also 100% correlated with AdobePDFViewerNPAPI.

Comment 10

[Rudi Sherry]

Steven:

http://hg.mozilla.org/mozilla-central/annotate/3fa30b0edd15/dom/plugins/base/nsPluginInstanceOwner.cpp#l3505

...happens inside a clause 'if (drawingModel == NPDrawingModelQuickDraw)'. We return CoreGraphicsDrawingModel, not sure how the NPDrawingModelQuickDraw clause is being executed, I have a feeling the sources aren't the same.

The original bug crash has it on line 3497.

First order is try to reproduce it; we haven't seen it. I'll look at some of the crash logs to see if I can see some specific environment that might be involved.


...probably not the same sources


I'll continue to look a bit

Comment 11

[Steven Michaud [:smichaud] (Retired)]

Thanks, Rudi.

Do also look at the other crash I reported in comment #9 (Acrobat@0xefeb63).

Please also try to find STR for either/both of these crashes.

Getting the drawing model wrong might be our fault (there have been other bugs in the past where this was true).  But it could also be Adobe's.  It'll be a lot easier to dig into this once we have STR.

Comment 12

[Steven Michaud [:smichaud] (Retired)]

Attachment: Testcase

Here's a file (part of a test for our own PDF plugin) with which I can always reproduce the nsPluginInstanceOwner::FixUpPluginWindow crash on OS X 10.5.8, if the current version of Adobe Acrobat is installed and its plugin is enabled.

Comment 13

[Steven Michaud [:smichaud] (Retired)]

(Following up comment #12)

Not in FF 11, though.

I'll spend the next hour finding out exactly which FF versions crash with it (and the Adobe Acrobat plugin), on which OSes, and so forth.

Comment 14

[Steven Michaud [:smichaud] (Retired)]

I get the same results (using the testcase from comment #12) in 32-bit mode on OS X 10.6.8 as I do on OS X 10.5.8 (where FF always runs in 32-bit mode):  Current mozilla-central (14-branch), aurora (13-branch) *and beta* (12-branch) nightlies crash.  But the FF12 betas don't crash, and neither does FF 11.

Since the 12-branch nightlies crash while the 12-branch betas don't, the AdobePDFViewerNPAPI plugin must be changing behavior based on what kind of environment it thinks it's running in.  But changing the user-agent string doesn't seem to effect its behavior (doing that has no effect on the crashes).

Comment 15

[Steven Michaud [:smichaud] (Retired)]

The distros that crash in 32-bit mode with the plugin running in-process also crash in 32-bit mode with the plugin running out-of-process (to make this happen set dom.ipc.plugins.enabled.i386.adobepdfviewernpapi.plugin to true).

The testcase simply doesn't work in 64-bit mode -- nothing gets displayed.

Comment 16

[Rudi Sherry]

We can reproduce the problem here with any PDF.

Blank means we are loaded but decide (at runtime) that we support this version (yes I know that's lame).

We don't change behaviors on Firefox versions of any kind, but we do look at our process' bundle ID to decide our environment; we look for "org.mozilla.firefox".  Anything else is unsupported (it's not perfect, and I'll take suggestions for a better solution for subsequent releases).  That, combined with the fact that the 12betas don't crash but the 12 nightlies do... is it showing a PDF in the nightlies?  If no, then it's probably bundle ID; if yes, then that suggests that some new code in the 12 nightlies but not yet in the betas, is doing it?

As for 64-bit, we do not want to support a user-set preference to move in-process, and we cannot yet run out-of-process, so both are considered unsupported. It would be great if we could support the 32-bit version and have Firefox act like we didn't exist for the 64-bit, but there really isn't a way; both architectures of Firefox look in the same place in the same way for the plug-in.

Comment 17

[Scoobidiver (away)]

The patch that has caused those crashes has probably been backed out in 12.0a2.

Comment 18

[Steven Michaud [:smichaud] (Retired)]

This is almost certainly not a regression in our code.  But I haven't yet had a chance to confirm this.

> we look for "org.mozilla.firefox".  Anything else is unsupported

Thanks for letting us know, Rudy.

This is unfortunate (I'll expand on this in a later comment).  But it (probably) explains everything.

My hunch is that, if the bundle ID is "org.mozilla.firefox", the Acrobat plugin tells the browser that it wants to use the Core Graphics drawing model, and (in 32-bit mode at least) everything is fine.  But if the bundle id is something else, the Acrobat plugin doesn't specify a drawing model, we default to the QuickDraw drawing model, and we crash (because the Acrobat plugin still assumes we'll be using the Core Graphics drawing model).

As of some time ago (when the patch for bug 553073 was landed on the trunk on 2010-03-23) we only use the "org.mozilla.firefox" bundle id for "release" versions -- i.e. actual releases and betas.

Comment 19

[Steven Michaud [:smichaud] (Retired)]

Changing today's mozilla-central nightly's bundle ID from org.mozilla.nightly to org.mozilla.firefox stops the crashes from happening.  Changing FF 12.0b5's bundle ID from org.mozilla.firefox to org.mozilla.whatever makes it crash with this bug's STR.

These crashes weren't triggered by any change in our code.

Comment 20

[Rudi Sherry]

Another data point: I changed the shipping Firefox 11.0 bundle ID to "org.mozilla.whatever" and it crashes.

I found another place: in NPP_Initialize(), we just check for a bundle ID prefix of "org.mozilla"; if it doesn't have that (and some other prefixes for other companies), it returns NPERR_INCOMPATIBLE_VERSION_ERROR.  This was preparation to run either in-proc (org.mozilla.firefox) our out-of-proc (org.mozilla.plugincontainer), while not trying to limit the bundle ID.

In this scenario, we return no error from anything, but don't end up negotiating the drawing model (due to the internal disagreement over whether we're supported, since "org.mozilla" succeeds but "org.mozilla.whatever" fails).  NPP_New() returns NPERR_NO_ERR, then it crashes before anything else is called.

It sounds like we (reader) need to align our internal checks, and perhaps we also should come up with something other than bundleID for environment testing.

Comment 21

[Steven Michaud [:smichaud] (Retired)]

Thanks, Rudi, for the additional info.  I'll add it to the debugging
patch I'm writing.

> I found another place: in NPP_Initialize(), we just check for a
> bundle ID prefix of "org.mozilla"; if it doesn't have that (and some
> other prefixes for other companies), it returns
> NPERR_INCOMPATIBLE_VERSION_ERROR.

So the Reader plugin returns an error from NPP_Initialize() whenever
the bundle ID prefix (in Firefox) is something else than
"org.mozilla"?  In this case the browser should never call NPP_New(),
and I assume we aren't.

> In this scenario, we return no error from anything, but don't end up
> negotiating the drawing model (due to the internal disagreement over
> whether we're supported, since "org.mozilla" succeeds but
> "org.mozilla.whatever" fails).  NPP_New() returns NPERR_NO_ERR, then
> it crashes before anything else is called.

Sounds like you're confirming my hunch from comment #18.  Are you?

Comment 22

[Rudi Sherry]

> Sounds like you're confirming my hunch from comment #18.  Are you?

Yes.

Comment 23

[Steven Michaud [:smichaud] (Retired)]

I've finished at least a preliminary version of my debugging patch,
and have reconfirmed my hunch from comment #18 -- the
AdobePDFViewerNPAPI plugin only sets the drawing model (to
NPDrawingModelCoreGraphics) when FF's bundle ID is
"org.mozilla.firefox".

But the	Acrobat plugin always sets the event model to NPEventModelCocoa,
which (as I understand it) is incompatible with the QuickDraw drawing
model.  This is why 'pluginPort' gets set to NULL just before this
bug's crashes (at the following line in nsChildView.mm):

http://hg.mozilla.org/mozilla-central/annotate/da53be684794/widget/cocoa/nsChildView.mm#l520

So I'm wondering if, when a plugin sets the event model to
NPEventModelCocoa, we should also check if the drawing model is set to
NPDrawingModelQuickDraw, and if so change it to
NPDrawingModelCoreGraphics.  We wouldn't (of course) change the
drawing model if it *wasn't* set to NPDrawingModelQuickDraw.

What do you think, Josh?

There's also at least one other change I'd like to make to our current
code.  We shouldn't crash when the Acrobat plugin decides to throw up
its hands (as it also always does in 64-bit mode).  Instead we should
just leave the plugin frame blank, as we do in 64-bit mode.

Patch coming up later today or tomorrow.

Comment 24

[Josh Aas]

(In reply to Steven Michaud from comment #23)

> So I'm wondering if, when a plugin sets the event model to
> NPEventModelCocoa, we should also check if the drawing model is set to
> NPDrawingModelQuickDraw, and if so change it to
> NPDrawingModelCoreGraphics.  We wouldn't (of course) change the
> drawing model if it *wasn't* set to NPDrawingModelQuickDraw.
> 
> What do you think, Josh?

We can potentially do a lot of this sort of thing - guessing for better behavior - and our guess won't always be right. What if they assume CA? We should really only take fixes for plugins into the browser in critical situations. I'm inclined to have us spend our time on other things and let them fix their plugin.

> There's also at least one other change I'd like to make to our current
> code.  We shouldn't crash when the Acrobat plugin decides to throw up
> its hands (as it also always does in 64-bit mode).

I'm not sure what this means. If it is a separate issue can you file a separate bug with a better explanation?

Thanks for figuring this out! Hopefully Adobe can fix this quickly, it should be easy.

Comment 25

[Steven Michaud [:smichaud] (Retired)]

> We can potentially do a lot of this sort of thing - guessing for
> better behavior - and our guess won't always be right. What if they
> assume CA? We should really only take fixes for plugins into the
> browser in critical situations. I'm inclined to have us spend our
> time on other things and let them fix their plugin.

I can go either way on this.

Tomorrow I'll prepare a patch that follows my suggestion and post it
here.  We can keep it in reserve in case we decide we want it.

> I'm not sure what this means. If it is a separate issue can you file
> a separate bug with a better explanation?

It's not really a separate issue.  And in fact I've now changed my
mind about making any other changes than the one I specifically
suggested.

> Thanks for figuring this out! Hopefully Adobe can fix this quickly,
> it should be easy.

You're quite welcome.  And yes, it should be quite easy for Adobe to
fix this bug.  I'm currently more worried about bug 746713 (which is
in Acrobat code and isn't yet reproducible).

Comment 26

[Steven Michaud [:smichaud] (Retired)]

Attachment: Debugging patch (no fix)

Here's the debugging patch I've been using.

And here's a tryserver build made from it:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-22d73165f5f6/try-macosx64/firefox-14.0a1.en-US.mac.dmg

It doesn't contain the fix I described in comment #23 (it's there, but commented out).

Comment 27

[Rudi Sherry]

I looked at at the patch; AFAICT it adds lots of printf statements but has no functional changes; is this for me?

Comment 28

[Steven Michaud [:smichaud] (Retired)]

(In reply to comment #27)

No, it doesn't have any functional changes.  It's for whoever wants to try debugging this problem on their own, and to show the basis for the conclusions I've drawn in previous comments.

Comment 29

[Alex Keybl [:akeybl]]

Tracking for FF13/14 since this is a top crasher, and sending over to Steven since he appears to be leading the investigation.

Comment 30

[Alex Keybl [:akeybl]]

Actually, since this crash only occurs on channels with a non-standard bundle ID, no need to track for release.

Comment 31

[Scoobidiver (away)]

There's one crash in 13.0.

Comment 32

[Steven Michaud [:smichaud] (Retired)]

bp-76d8afec-9156-4f67-8d63-4f79f2120613

This stack is quite different from the others reported here.  Among other things the crash doesn't happen while instantiating a plugin, but some time afterwards (while trying to draw the plugin).

Probably unrelated.

Comment 33

[Steven Michaud [:smichaud] (Retired)]

(Following up comment #32)

> bp-76d8afec-9156-4f67-8d63-4f79f2120613

Also AdobePDFViewerNPAPI isn't listed among the loaded modules.

Comment 34

[Virtual_ManPL [:Virtual] 🇵🇱 - (please needinfo? me - so I will see your comment/reply/question/etc.)]

I'm marking this bug as WORKSFORME as bug crashlog signature didn't appear from a long time (over half year) [except some obsolete <4 versions, no crashes starting since 4 version].