806461 - crashes in np_fastbid.dll since 13.x release

Status: RESOLVED INVALID

(Core Graveyard :: Plug-ins, defect)

Description

[helpdesk]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0
Build ID: 20121010144125

Steps to reproduce:

Install the plugin on this page, (which has worked well since v3.x):

   http://www.bxwa.com/fastbid/moz_check.html?about:blank

...then try the test link.


Actual results:

4.x - 12.x works great
13.x always crashed
14.x usually crashed, especially if upgraded from 13.x, sometimes worked if upgraded directly from 12.x
15.x usually crashed if upgraded from 13.x/14.x but usually worked if direct from 12.x
16.x always crashes (again)

Occasionally a consistently crashing version will actually work on the test page but then crash in actual use when viewing plans, usually fairly quickly.

This is under Windows but consistent regardless of OS version, (i.e. XP, Vista, Windows 7).  We have also experimented with various combinations of settings for video hardware acceleration, disabling other plugins, turning off anti-virus, even using a clean new machine with nothing else installed etc.  We have thousands of people using it but had not submitted a bug report sooner as we could see many of the more mainstream plugins were having similar issues and that things were slowly getting better through 14.x and 15.x.  With 16.x breaking it completely again though it appears that there could be other factors.  Please let us know if there's anything we can do to assist in debugging this issue.


Expected results:

Users should be able to view plan sheets.  E.g. anything linked from any of these projects:
   http://www.bxwa.com/bxwa_toc/pub/1519/toc.html

Comment 1

[Scoobidiver (away)]

Can you provide the crash ID from about:crashes?

Comment 2

[helpdesk]

My apologies, I had submitted this bug by clicking the link within a crash report so assumed it had been associated.  Here's a handful from one of our test machines:

bp-400a99c7-518d-464f-a406-20684212102210/22/201212:43 PM
bp-0c4d6721-2885-4c9d-8cd1-338a1212100410/4/201211:36 AM
74281eaf-bd14-4f54-a36b-4e2039c690ba9/17/20122:07 PM
ba31b00a-b3ca-421c-9f33-cce1d151d6379/17/20121:16 PM
88b52de4-ba0a-4add-ae41-656d8f209b2b9/17/20121:12 PM
59283470-2fb0-4f1c-8e68-348519f9f1c49/14/20124:46 PM
960027c4-98ad-4a85-a2d3-ec926e7c4aea9/14/20124:45 PM
800a0bc4-8030-4af6-a639-4466669ecd6d9/14/20123:01 PM

Comment 3

[Scoobidiver (away)]

The only exploitable crash IDs are bp-400a99c7-518d-464f-a406-206842121022 and bp-0c4d6721-2885-4c9d-8cd1-338a12121004, others haven't been submitted to the crash server (throttle of 10% in release versions).

It's a low volume crash with only 5 crashes in 16.0.1.

I am not sure the cause is in the Firefox code. You should see: https://developer.mozilla.org/docs/Plugins

Comment 4

[helpdesk]

Most of our users have either switched to other browsers or temporarily downgraded to v12.  Had not realized that the actual crash report submission is 10%, we can generate and manually submit as many as you wish!  :)

The reason we suspect FireFox code is that it coincides exactly with the new memory management and plugin architecture in v13 and plenty of other plugins had the exact same symptoms then.  Things have improved with subsequent releases, but our plugin probably stretches the limits a little further than the average media player etc.  Additionally, you'll notice that it is actually FireFox crashing, *not* our plugin, even though the new plugin architecture is supposed to keep FireFox from being affected even if a plugin does crash.  Either way it still comes back to a crashing bug in FireFox code.

For what it's worth we can also duplicate the issue in v17(beta), and we are also now seeing some consistent variation in which test machines crash, so it's quite possible we can narrow it down further.  (Could end up being a video acceleration bug or something, the plugin is quite graphically intensive.)

Also, if it helps any, there should be plenty more crash reports for prior versions as well, (we had hundreds of support calls when 13.x came out).  Might be some clues in whether they are different from 16.x reports or not, especially vis-a-vis the variances observed depending on upgrade path.

Comment 5

[helpdesk]

For what it's worth, following on to the possible cause being video related, we actually did do some rudimentary testing around that back in 13.x/14.x, and now again in 17.x, to see if disabling various of the layers and gfx options would help, but didn't find anything significant at the time.
   But just let us know what versions to focus on and we'll provide as much info as you wish, testers are standing by! :)

Comment 6

[John Schoenick [:johns]]

Looking at http://dxr.allizom.org/mozilla-central/dom/plugins/base/nsNPAPIPlugin.cpp#l247 it looks like we don't run plugins OOP by default, so this plugin crashing is taking down firefox as well.

If you go into about:config and add a new pref (Right Click -> New -> Boolean) named "dom.ipc.plugins.enabled.np_fastbid.dll" and set it to true, then restart Firefox, this plugin will be run out-of-process, similar to Flash.

@bsmedberg - is it intended that we don't OOP unknown plugins by default?

Comment 7

[Scoobidiver (away)]

(In reply to helpdesk from comment #0)
> we could see many of the more mainstream plugins were having similar issues
The spike in Flash crashes is caused by Flash 11.3 that introduced Firefox Protected Mode that interacts with some users' configurations (security software, other plugins, HW acceleration ON in Flash and OFF in Firefox...) and not by a change in Firefox code.

(In reply to helpdesk from comment #4)
> Most of our users have either switched to other browsers or temporarily
> downgraded to v12.
Firefox 12 is vulnerable so, if your client are organizations, recommend them to switch to Firefox 10 ESR (see http://www.mozilla.org/firefox/organizations/faq/) until the problem is figured out and at least workaroundable.

Comment 8

[Georg Fritzsche [:gfritzsche]]

(In reply to John Schoenick [:johns] from comment #6)
> it looks like we don't run plugins OOP by default

OOP is on per default via the stock prefs for Firefox. I think it's this file:
http://mxr.mozilla.org/mozilla-central/source/browser/app/profile/firefox.js#892

Comment 9

[Georg Fritzsche [:gfritzsche]]

(In reply to helpdesk from comment #4)
> Additionally, you'll notice that it is
> actually FireFox crashing, *not* our plugin, even though the new plugin
> architecture is supposed to keep FireFox from being affected even if a
> plugin does crash.

Do you possibly have the out-of-process-plugins feature disabled on your test-machines?
Also, as it is crashing in np_fastbid.dll - have you tried running it under a debugger to see what goes wrong?

Comment 10

[John Schoenick [:johns]]

(In reply to Georg Fritzsche [:gfritzsche] from comment #8)
> (In reply to John Schoenick [:johns] from comment #6)
> > it looks like we don't run plugins OOP by default
> 
> OOP is on per default via the stock prefs for Firefox. I think it's this
> file:
> http://mxr.mozilla.org/mozilla-central/source/browser/app/profile/firefox.
> js#892

Sorry, I meant "we don't run unknown plugins OOP by default", Flash does get OOP'd. We may have regressed this at some point -- This plugin does not get OOP on a fresh profile, which can be seen by using the link in the first comment. Flipping | dom.ipc.plugins.enabled.np_fastbid.dll | to true changes this from a crash to a plugin-container crash, so something is fishy.

Comment 11

[Georg Fritzsche [:gfritzsche]]

(In reply to John Schoenick [:johns] from comment #10)
> Sorry, I meant "we don't run unknown plugins OOP by default", Flash does get
> OOP'd. We may have regressed this at some point -- This plugin does not get
> OOP on a fresh profile, which can be seen by using the link in the first
> comment. Flipping | dom.ipc.plugins.enabled.np_fastbid.dll | to true changes
> this from a crash to a plugin-container crash, so something is fishy.

Oh, i see. Installing this on a fresh profile results in "dom.ipc.plugins.enabled.np_fastbid.dll = false" in the config, so it must be the "Fastbid Image Viewer" addon that is setting this.

Comment 12

[Georg Fritzsche [:gfritzsche]]

Or is there something that could set this up for plugins embedded in extensions?

Comment 13

[Georg Fritzsche [:gfritzsche]]

(In reply to Georg Fritzsche [:gfritzsche] from comment #12)
> Or is there something that could set this up for plugins embedded in
> extensions?

Never mind, the addon sets this in it's defaults/preferences/user.js

Comment 14

[helpdesk]

Thanks to all who have commented, it looks like we're on the trail!

In normal use the plugin can request ~100MB of memory space.

It appears that the new FireFox plugin architecture uses some sort of formula based on available system resources to decide how much memory can be given to a non-OOP plugin and has either a hard-coded limit, or a formula which consistently sets a much lower limit, for OOP plugins. (The plugin crashes are due to failing memory allocation).

Are there any about:config options we can use to test this hypothesis?

Comment 15

[Benjamin Smedberg]

We don't apply any limits, no. Are you sure that you're not corrupting the heap and it thinks it's out of memory prematurely? Especially are you using NPN_MemAlloc and NPN_MemFree appropriately? If you are calling free() on memory you shouldn't, it could easily corrupt the heap.

Comment 16

[helpdesk]

The plugin was written many years ago, we would need to do a code review to see what it actually is or isn't doing memory-wise.  But a quick text search shows no use of free() at all.  Alloc calls appear to all be mb_alloc or sb_malloc with corresponding mb_free and sb_mfree.
   Further, it's never had any crashing issues prior to v13.  
   However, it's possible that it could be making calls to free zero bytes of memory, (or possibly even to allocate?), and I did see a note somewhere recently about freeing zero bytes causing problems with more of the more recent versions of FireFox. 
   Could older versions have possibly been handling "odd" requests like that more gracefully?

Comment 17

[Benjamin Smedberg]

Are you linking the windows CRT statically or dynamically? If you run a debug build do you get any CRT heap assertions? Note that we did switch Windows CRT versions in Firefox 13, see bug 563318. Previously we used a modified version of the Visual Studio 2005 CRT. Now we're using the Microsoft-compiled CRT for Visual Studio 2010.

I'm 99% sure that this is a bug in memory handling within the plugin, so I'm going to mark it INVALID and let you examine the CRT linkage and memory handling yourself. You can email me questions directly if you need additional clues.

Comment 18

[helpdesk]

Good to know, we'll dig into it further on our end - thanks for the info!