Last Comment Bug 715744 - DLL block request: BExternal.dll
: DLL block request: BExternal.dll
Status: RESOLVED WONTFIX
[dll][3rd-party-bustage]
: qawanted
Product: Toolkit
Classification: Components
Component: Blocklisting (show other bugs)
: unspecified
: All All
: -- normal (vote)
: ---
Assigned To: :Ehsan Akhgari
:
:
Mentors:
Depends on:
Blocks: 715757 721645
  Show dependency treegraph
 
Reported: 2012-01-05 17:20 PST by Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)
Modified: 2016-03-07 15:30 PST (History)
31 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
+
+
wontfix


Attachments
Patch (821 bytes, patch)
2012-01-05 17:36 PST, Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary)
ehsan: review-
Details | Diff | Splinter Review
Patch (v1) (928 bytes, patch)
2012-03-02 08:58 PST, :Ehsan Akhgari
benjamin: review+
akeybl: approval‑mozilla‑aurora+
Details | Diff | Splinter Review

Description Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary) 2012-01-05 17:20:44 PST
DLL name: BExternal.dll
DLL versions to block: Unversioned
Applications, versions, and platforms affected: Firefox on Windows

Homepage and other references and contact info: Unknown

Reasons:
BExternal.dll appears to be injecting itself into the Firefox process and proceeding to manipulate our preferences off the main thread.

See crash reports like
https://crash-stats.mozilla.com/report/index/2a925d21-8e15-40c7-8cf5-5b9642120105
https://crash-stats.mozilla.com/report/index/73983a26-dafe-4384-967e-88f3a2120105
Comment 1 Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary) 2012-01-05 17:36:09 PST
Created attachment 586291 [details] [diff] [review]
Patch
Comment 2 Johnathan Nightingale [:johnath] 2012-01-06 07:54:18 PST
The internet thinks this is Babylon software, which is a translation app: http://www.babylon.com/windows

Kev - have we had to deal with them before?
Comment 3 Alex Keybl [:akeybl] 2012-01-09 13:01:20 PST
Prior to tracking for any release, it'd be good to better understand the affected user population and number of crashes highly correlated with the DLL (cc'ing Justin and Sheila).

Also looping Marcia in case we decide to move forward with this. We'd want to test a try build with the DLL blocklist entry included and also make sure that there's no related add-on (or regressions once blocked).
Comment 4 Justin Scott [:fligtar] 2012-01-13 12:02:12 PST
(In reply to Alex Keybl [:akeybl] from comment #3)
> Prior to tracking for any release, it'd be good to better understand the
> affected user population and number of crashes highly correlated with the
> DLL (cc'ing Justin and Sheila).

Doesn't seem like this is an add-on, so I can't really give a usage count.
Comment 5 Alex Keybl [:akeybl] 2012-01-17 12:41:38 PST
[Triage Comment]
I think we're too close to the end of the cycle to prepare and execute on a DLL blocklist addition for FF10. It's not clear that this needs to be tracked for FF11 either. We would, however, consider uplifting the entry once ready.
Comment 6 Luke Wagner [:luke] 2012-01-17 16:17:52 PST
(In reply to Alex Keybl [:akeybl] from comment #5)
Alex, were you aware that BExternal.dll is partially causing a topcrash: see bug 715757 comment 4.
Comment 7 Alex Keybl [:akeybl] 2012-01-17 16:41:38 PST
(In reply to Luke Wagner [:luke] from comment #6)
> (In reply to Alex Keybl [:akeybl] from comment #5)
> Alex, were you aware that BExternal.dll is partially causing a topcrash: see
> bug 715757 comment 4.

I was not. At this point, it's a difficult path to blocklisting in FF10 to resolve 44% of Windows crashes caused by 715757. We'd need to reach out to Babylon (Kev?), and then decide whether or not to blocklist based upon their response. Remember, if they plan on fixing we can't blocklist an unversioned DLL and then revoke the block before FF11's release. If we decide to move forward with the blocklist, we'd then need to test to make sure we're not leaving users in an even worse state once the DLL is blocklisted (for whatever reason).

I'll leave the door open for now and track this, but it's really a stretch.
Comment 8 Scoobidiver (away) 2012-01-18 00:21:54 PST
The crash in CrashInJS caused by BExternal.dll (41%) is #10 top crasher in 10.0b4
Comment 9 Luke Wagner [:luke] 2012-01-18 09:45:15 PST
Bug 715757 comment 7 says that blocking this and gemgecko10.dll remove 99% of the #8 topcrash.
Comment 10 Alex Keybl [:akeybl] 2012-01-20 13:30:06 PST
As mentioned in comment#7, blocklisting a legitimate unversioned DLL may cause user pain until FF11, and only then if the plugin author has resolved the issue in a later version and pushes it out to their users.
Comment 11 :Ehsan Akhgari 2012-01-21 10:21:22 PST
Comment on attachment 586291 [details] [diff] [review]
Patch

r-.  The DLL name must be in all lower case, otherwise the blocklist won't take effect at all.
Comment 12 Luke Wagner [:luke] 2012-01-21 17:59:57 PST
Comment on attachment 586291 [details] [diff] [review]
Patch

I may have miscommunicated on irc, but this patch wasn't requesting review due to comment 10.
Comment 13 Jorge Villalobos [:jorgev] 2012-02-06 09:18:03 PST
(In reply to Justin Scott [:fligtar] from comment #4)
> (In reply to Alex Keybl [:akeybl] from comment #3)
> > Prior to tracking for any release, it'd be good to better understand the
> > affected user population and number of crashes highly correlated with the
> > DLL (cc'ing Justin and Sheila).
> 
> Doesn't seem like this is an add-on, so I can't really give a usage count.

This is related to the Babylon Toolbar add-on, actually. The Babylon tool installs the toolbar in Firefox, and it is one of the most used add-ons around. The stats indicate roughly 6.7M ADU. I don't know if the DLL is part of the add-on, but there are other block requests for it, like bug 721264.
Comment 14 Kev Needham [:kev] 2012-02-13 14:31:01 PST
Working on getting a contact with Babylon Software. Apologies for the delay, this one was flying under my radar.
Comment 15 Alex Keybl [:akeybl] 2012-02-23 13:18:56 PST
(In reply to Kev [:kev] Needham from comment #14)
> Working on getting a contact with Babylon Software. Apologies for the delay,
> this one was flying under my radar.

How's outreach to Babylon going? Have we found a solid contact?
Comment 16 Igor Bukanov 2012-02-24 10:24:11 PST
If we cannot blacklist the DLL and cannot contact developers, what about adding a non-main thread check at the earliest possible point before we reach the assert and then reporting a failure? For example, XPCPerThreadData::GetDataImpl or XPCJSContextStack::GetSafeJSContext could be a good place for that as at that point we can check for non-main thread with a simple if.

If that fixes the crash, we can get some time before deciding with blacklisting.
Comment 17 Luke Wagner [:luke] 2012-02-24 10:49:57 PST
There is a risk that those new sources of dynamic errors would trigger some other problem that would be harder to track.  Blacklisting is much lower risk (assuming we cannot quickly get in contact with them).  If Babylon actually cares to release a new .dll that doesn't crash, can't they avoid the unversioned blocklist by changing their .dll name?
Comment 18 Alex Keybl [:akeybl] 2012-02-27 14:27:21 PST
(In reply to Luke Wagner [:luke] from comment #17)
> If Babylon
> actually cares to release a new .dll that doesn't crash, can't they avoid
> the unversioned blocklist by changing their .dll name?

We also don't want to pull the rug out from under our users who actually benefit from the software. That's why we're checking on the status of our outreach.
Comment 19 Luke Wagner [:luke] 2012-02-27 14:42:23 PST
Perhaps a more immediate question we can answer is: does the addon cause a crash for 100% of users?  If so, then the set of users benefiting from it right now might be empty.
Comment 20 Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary) 2012-02-27 17:08:49 PST
(In reply to Alex Keybl [:akeybl] from comment #18)
> (In reply to Luke Wagner [:luke] from comment #17)
> > If Babylon
> > actually cares to release a new .dll that doesn't crash, can't they avoid
> > the unversioned blocklist by changing their .dll name?
> 
> We also don't want to pull the rug out from under our users who actually
> benefit from the software. That's why we're checking on the status of our
> outreach.

I realize I'm opening a can of worms here, but do users actually benefit from the Babylon Toolbar?
Comment 21 Robert Kaiser 2012-02-28 05:44:33 PST
(In reply to Luke Wagner [:luke] from comment #17)
> If Babylon
> actually cares to release a new .dll that doesn't crash, can't they avoid
> the unversioned blocklist by changing their .dll name?

Yes, AFAIK. This possibility was one of the main reasons why we clearly determined that the blocklist is no good measure against malware.

(In reply to Luke Wagner [:luke] from comment #19)
> Perhaps a more immediate question we can answer is: does the addon cause a
> crash for 100% of users?

We only have data for loaded DLLs from crash reports, so we don't know for how many users this DLL might be working.

What we know is this:

Correlations for signature "CrashInJS" on Firefox 11.0, Windows, 2012-02-27:

Add-ons:
     15% (145/956) vs.   8% (1722/22655) ffxtlbr@babylon.com
          0% (2/956) vs.   0% (26/22655) 1.1.3
          0% (0/956) vs.   0% (5/22655) 1.1.8
         11% (101/956) vs.   4% (988/22655) 1.1.9
          4% (42/956) vs.   3% (703/22655) 1.2.0

Modules:
    100% (955/956) vs.   4% (956/22655) BExternal.dll

From that data, we can see that out of 22655 total crash reports yesterday on Firefox 11.0 (beta) builds, 1722 had the ffxtlbr@babylon.comm add-on installed (the version breakdown of the add-on is just FYI, to show that unfortunately not all crashes are on one add-on version), interestingly only 145 of the 956 crash reports with the affected signature have the add-on, so an add-on blocklist would not be effective.
ALL BUT ONE of the affected crashes with this signature have BExternal.dll (unfortunately unversioned) installed though, and that's only 1 difference from the number of total Firefox 11 crashes ports that have this library loaded.

That means that yesterday on Firefox 11.0 and Windows, every crash report but one that had this module loaded did crash with this same signature, which is pretty impressive.

Again, we don't know about Firefox runs that don't crash at all or don't send a report, but from those that crash in any place and sent us a report, basically everyone crashed this way, which dmandelin clearly indicates in bug 715757 comment #19 as "an add-on is doing bad stuff". This IMHO is a very strong argument for blocking this DLL ASAP on 11.0 beta.
Comment 22 Alex Keybl [:akeybl] 2012-02-29 12:49:13 PST
(In reply to Kev [:kev] Needham from comment #14)
> Working on getting a contact with Babylon Software. Apologies for the delay,
> this one was flying under my radar.

Kev - has there been response from Babylon? It'd be preferable to give them a heads up prior to blocklisting the DLL.
Comment 23 Alex Keybl [:akeybl] 2012-02-29 12:58:53 PST
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #20)
> I realize I'm opening a can of worms here, but do users actually benefit
> from the Babylon Toolbar?

It does appear to provide benefit:

"One click translation application offers over 75 languages to be translated from language to language, Babylon provides precise translations and updated content (e.g. Wikipedia in 21 languages) in a single click, without interrupting your workflow."

If we're interested in moving forward with this blocklist, my preference would be to first hear back from Babylon about updating their software. If that's not possible, let's prepare a blocklist patch and some try builds so that we can determine if we need to block the add-on toolbar at the same time (and verify that the blocklist addition works). We could push this into the FF11 beta 6 build (3/5) depending on our status on Monday.
Comment 24 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2012-03-01 13:49:34 PST
It's not clear to me what, if anything, QA can do -- what is the specific request for qawanted?
Comment 25 Alex Keybl [:akeybl] 2012-03-01 13:52:41 PST
(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #24)
> It's not clear to me what, if anything, QA can do -- what is the specific
> request for qawanted?

Sorry - errant keyword added.
Comment 26 :Ehsan Akhgari 2012-03-02 08:58:46 PST
Created attachment 602380 [details] [diff] [review]
Patch (v1)
Comment 27 :Ehsan Akhgari 2012-03-02 08:59:20 PST
Try build: https://tbpl.mozilla.org/?tree=Try&rev=a1bd00172de6
Comment 28 Alex Keybl [:akeybl] 2012-03-02 10:59:21 PST
(In reply to Ehsan Akhgari [:ehsan] from comment #27)
> Try build: https://tbpl.mozilla.org/?tree=Try&rev=a1bd00172de6

Thanks guys - much appreciated! Adding qawanted to do exploratory testing around Babylon's software (comment#2) with this try build. We want figure out whether we need to blocklist any associated add-ons and make sure we're not leaving users in a worse position than before the blocklist.
Comment 29 Kev Needham [:kev] 2012-03-02 11:34:12 PST
Howdy all, I've tried contacting babylon several times, and have not received a response. I believe we've satisfied the criteria in attempting to work things through with the vendor, and will inform them of that we'll be blocklisting the DLL (sometimes that generates a response, too).

Should not block this, though.

(In reply to Alex Keybl [:akeybl] from comment #22)
> (In reply to Kev [:kev] Needham from comment #14)
> > Working on getting a contact with Babylon Software. Apologies for the delay,
> > this one was flying under my radar.
> 
> Kev - has there been response from Babylon? It'd be preferable to give them
> a heads up prior to blocklisting the DLL.
Comment 30 :ibai 2012-03-02 11:37:57 PST
The SUMO team has found out that the Babylon toolbar (as other Montiera toolbars) are pretty much non desired for our users. Between 42% and 54% of the users who have it installed specifically don't want it.

You can find more info about the survey we run in here: https://etherpad.mozilla.org/babylon-toolbar-survey
Comment 31 Jorge Villalobos [:jorgev] 2012-03-02 11:57:35 PST
Bug 721264 is in file for blocking the Babylon Toolbar. Let me know if we want to do this, and when.

Also, should it be blocked for Firefox 10 and above only, or other versions as well? Do we know which versions of the toolbar are causing the crashes?
Comment 32 Alex Keybl [:akeybl] 2012-03-02 12:12:34 PST
(In reply to Jorge Villalobos [:jorgev] from comment #31)
> Bug 721264 is in file for blocking the Babylon Toolbar. Let me know if we
> want to do this, and when.
> 
> Also, should it be blocked for Firefox 10 and above only, or other versions
> as well? Do we know which versions of the toolbar are causing the crashes?

No decision has been made on the toolbar yet. We've not confirmed it causes crashes at all. The fact that many users don't want it should be a separate discussion, unless the add-on becomes nonfunctional after blocklisting the DLL.
Comment 33 Alex Keybl [:akeybl] 2012-03-02 12:14:13 PST
Sorry, that reads funny - let me try again. Blocklisting the toolbar should be a separate discussion unless we find that it doesn't function after blocklisting the DLL. We're trying to resolve a topcrasher here, not remove an unpopular add-on.
Comment 34 juan becerra [:juanb] 2012-03-02 15:12:39 PST
I haven't been able to get the BExternal.ddl to load with Fx11b5 with different types of interactions with the Babylon 9 toolbar. I launch it, take a look at the loaded dlls in firefox.exe, use the toolbar a bit more, but this particular dll doesn't show up in the list. A similar thing happens with the try-server builds, so I can't tell the effect of the block.
Comment 35 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2012-03-02 15:18:05 PST
Tried the TRY build on Windows 7 64-bit. Firefox asks if I want to allow the following add-ons to install:
 * Babylon Translation Activation 1.1
 * Babylon 1.2.0
 * Babylon Spelling and Proofreading 1.0.0.1

Disallowing the installation disables the add-ons but I still get a Babylon icon in my title bar. I can also manually enable the add-ons in about:addons. Allowing the installation enables the add-ons by default.

In Firefox 11.0b5, I get no such opportunity to allow/disallow.
Comment 36 Alex Keybl [:akeybl] 2012-03-02 16:41:55 PST
(In reply to juan becerra [:juanb] from comment #34)
> I haven't been able to get the BExternal.ddl to load with Fx11b5 with
> different types of interactions with the Babylon 9 toolbar. I launch it,
> take a look at the loaded dlls in firefox.exe, use the toolbar a bit more,
> but this particular dll doesn't show up in the list. A similar thing happens
> with the try-server builds, so I can't tell the effect of the block.

To clarify, you're testing with http://www.babylon.com/redirects/download.cgi?type=100 (Babylon9_setup.exe)? This bug specifically pertains to having the babylon DLL loaded when running Firefox, and then seeing it blocked with the try build. If we're not able to repro BExternal.dll loading at all, there's no point in further testing because we have the wrong installer.
Comment 37 juan becerra [:juanb] 2012-03-02 16:51:59 PST
(In reply to Alex Keybl [:akeybl] from comment #36)
> (In reply to juan becerra [:juanb] from comment #34)
> > I haven't been able to get the BExternal.ddl to load with Fx11b5 with
> > different types of interactions with the Babylon 9 toolbar. I launch it,
> > take a look at the loaded dlls in firefox.exe, use the toolbar a bit more,
> > but this particular dll doesn't show up in the list. A similar thing happens
> > with the try-server builds, so I can't tell the effect of the block.
> 
> To clarify, you're testing with
> http://www.babylon.com/redirects/download.cgi?type=100 (Babylon9_setup.exe)?
> This bug specifically pertains to having the babylon DLL loaded when running
> Firefox, and then seeing it blocked with the try build. If we're not able to
> repro BExternal.dll loading at all, there's no point in further testing
> because we have the wrong installer.

Correct. I used the Babylon9_setup.exe installer.
Comment 38 Alex Keybl [:akeybl] 2012-03-02 17:29:28 PST
(In reply to juan becerra [:juanb] from comment #37)
> Correct. I used the Babylon9_setup.exe installer.

They may have renamed their DLL at some point. After installing Babylon 9 I see captlib.dll loading in Firefox, but not bexternal.dll.

Can we try some of the older versions at http://forum.oldversion.com/showthread.php?108-Babylon&p=23831&viewfull=1#post23831 to see if bexternal.dll is loaded with any of them?
Comment 39 juan becerra [:juanb] 2012-03-05 13:02:03 PST
I tried versions 3.2, 4.0316, and 5.0 from the list in the link in comment #38. The other ones were dead links. From the ones I tried none loaded bexternal.dll

I tried looking in other places for older versions of this software, but they all point to some version of the latest.
Comment 40 Alex Keybl [:akeybl] 2012-03-05 13:13:08 PST
(In reply to juan becerra [:juanb] from comment #39)
> I tried versions 3.2, 4.0316, and 5.0 from the list in the link in comment
> #38. The other ones were dead links. From the ones I tried none loaded
> bexternal.dll
> 
> I tried looking in other places for older versions of this software, but
> they all point to some version of the latest.

Thanks Juan. In that case, I'm not comfortable with fixing in FF11 given our lack of understanding of the blocklist's effect. The resolution of this bug is on hold until we find the software that installs BExternal.dll.
Comment 41 Scoobidiver (away) 2012-03-05 13:27:29 PST
Babylon Toolbar uses 10 DLLs: BExternal.dll, IECookieLow.dll, sqlite3.dll, BabyFox.dll, BabylonIEPI.dll, BabylonOfficePI.dll, BContentServer.dll, BContentServerExt.dll, BContentServerLite.dll, captlib.dll (see http://www.threatexpert.com/report.aspx?md5=4d2965e27ebe75c22e0423f705b4e3e0).
So there should be certain conditions to have BExternal.dll loaded.
Comment 42 Alex Keybl [:akeybl] 2012-03-05 16:12:57 PST
(In reply to Scoobidiver from comment #41)
> Babylon Toolbar uses 10 DLLs: BExternal.dll, IECookieLow.dll, sqlite3.dll,
> BabyFox.dll, BabylonIEPI.dll, BabylonOfficePI.dll, BContentServer.dll,
> BContentServerExt.dll, BContentServerLite.dll, captlib.dll (see
> http://www.threatexpert.com/report.
> aspx?md5=4d2965e27ebe75c22e0423f705b4e3e0).
> So there should be certain conditions to have BExternal.dll loaded.

Perhaps the DLL is loaded post-startup? Let's continue to try to get BExternal.dll to load in QA. May be worth trying to translate selected text and other features of Babylon.
Comment 43 Robert Kaiser 2012-03-06 04:12:18 PST
(In reply to Alex Keybl [:akeybl] from comment #40)
> Thanks Juan. In that case, I'm not comfortable with fixing in FF11 given our
> lack of understanding of the blocklist's effect. The resolution of this bug
> is on hold until we find the software that installs BExternal.dll.


IMHO, that's the completely wrong way to go. We *know* that a huge number of real crashes for users are caused by this DLL, so we *know* that users have installed it and have problems. The absolute worst case this patch can have is that it has no effect, so the risk is zero in practice. We need to be hugely more aggressive in blocking libraries and add-ons that even *can* have bad side effects for users, we know that the top problems users have with Firefox are all issues with third-party software, and we need to become huge less forgiving for their errors, esp. when it's parties that don't even communicate with us as in this case.

Please reconsider this decision for 11.

In the position we are in without this, I cannot recommend shipping 11 at all, because it will cause a lot of grief for users (and for the CrashKill team, but users are the main concern).
Comment 44 Andrew Williamson [:eviljeff] 2012-03-06 06:03:52 PST
From a full code inspection of the 3 addons the Babylon9 installer creates, the dll isn't loaded directly from within the addons.  It would have to be loaded as a plugin, XPCOM component, with ctypes or executed via nsIProcess.  

It appears that communication with the binary part of the toolbar is via a local server component (ports 9876 and 22678), so I'd guess the offending dlls are loaded via that instead.
Comment 45 :Ehsan Akhgari 2012-03-06 08:46:02 PST
It's not clear to me whether I'm supposed to land this patch on mozilla-central or not...
Comment 46 Alex Keybl [:akeybl] 2012-03-06 11:33:05 PST
(In reply to Ehsan Akhgari [:ehsan] from comment #45)
> It's not clear to me whether I'm supposed to land this patch on
> mozilla-central or not...

We'll be discussing this bug in today's channel meeting at 2PM to discuss the risk of blocklisting the DLL without a testable case. We'll comment in this bug afterwards with a decision.
Comment 47 Alex Keybl [:akeybl] 2012-03-06 16:48:54 PST
Let's land this on m-c and Aurora 12 so that we can see if crash numbers drop on those branches before considering taking this for FF11 (drop dead date 3/9).
Comment 48 Axel Hecht 2012-03-07 04:02:25 PST
CCing the bugzilla accounts I found at babylon.com
Comment 49 Jorge Villalobos [:jorgev] 2012-03-07 07:46:12 PST
I received a response from Babylon claiming that the crashes are being caused by a previous version of their software. That could explain the difficulty in reproducing the crashes. I'm trying to get information about the crashy version so we can test it.
Comment 51 Jorge Villalobos [:jorgev] 2012-03-07 12:59:09 PST
Can someone post up to date crash stats? Are we seeing any change in the numbers? The Babylon people are saying they released a fixed version (I don't know when exactly) and would like to know if that has had any effect.
Comment 52 Scoobidiver (away) 2012-03-07 13:42:41 PST
Aurora is in version 12, not 11.
Comment 53 arky [:arky] 2012-03-08 03:49:16 PST
Testing Nightly,Aurora and Stable release of Firefox on Window 7 with Babylon 8 installed. Firefox blacklists the plugin. 

Unfortunately I didn't notice the BExternal.dll file anywhere. Let me know if you need further help.
Comment 54 Robert Kaiser 2012-03-08 06:53:05 PST
(In reply to Jorge Villalobos [:jorgev] from comment #51)
> Can someone post up to date crash stats? Are we seeing any change in the
> numbers? The Babylon people are saying they released a fixed version (I
> don't know when exactly) and would like to know if that has had any effect.

The CrashInJS signatures are seeing a downward trend since March 4, and have seen a huge drop in yesterday's data.
Comment 55 Ed Morley [:emorley] 2012-03-08 14:00:56 PST
https://hg.mozilla.org/mozilla-central/rev/6151695f9df4
Comment 56 Robert Kaiser 2012-03-09 10:03:12 PST
On Firefox 12 (Aurora), where the blocklist landed, we had ~200 (CrashInJS | GetContextFromObject) crashes daily from March 1 to 4, then a slight dropoff and a sharp decline on 7th to 47. Yesterday (8th) we had none at all, which seems to be the blocklist in effect.

So there, we seem to have a good positive trend due to the fix mentioned in comment #51 and then full success of the blocklist in yesterday's data.


Now, for those versions that don't have the blocklist in place (and where, for other reasons, the signature is CrashInJS | JS_AbortIfWrongThread instead):

On 11, we had 1600-1800 crashes daily from March 1 through 6, a steep decline to 488 on 7th and 47 yesterday. The signature is now ranked #100 (for all of 11) and far out of topcrashers.

On 10, we had ~500-600 crashes after throttling (to 10%, so it's ~5000-6000 in reality) from March 1 through 6, a bit of a decline to 299 on 7th and further significant decline on 7th to 180. The crash rank is now #66, also out of topcrashers.

Looking at those, it looks like the new Babylon version itself had a tremendous effect, even without blocklisting the DLL. On 10, we probably have people adopting to updates a bit more slowly so the decline is not that steep but it's still pretty significant.

Judging from that, I'm unsure if we need to blocklist on 11 as well, it looks to me that Babylon solved this well enough on their side.
Comment 57 Alex Keybl [:akeybl] 2012-03-09 10:57:43 PST
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #56)
> Judging from that, I'm unsure if we need to blocklist on 11 as well, it
> looks to me that Babylon solved this well enough on their side.

Thanks for looking into this KaiRo! Is there any reason to leave the blocklist on central/aurora in that case?
Comment 58 Robert Kaiser 2012-03-09 11:20:41 PST
(In reply to Alex Keybl [:akeybl] from comment #57)
> Thanks for looking into this KaiRo! Is there any reason to leave the
> blocklist on central/aurora in that case?

In the light of those developments, it might be reasonable to undo those again. Right now, I'm unsure if we should or not, knowing too little about the whole library and its use myself.
Comment 59 Alex Keybl [:akeybl] 2012-03-09 11:27:24 PST
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #58)
> (In reply to Alex Keybl [:akeybl] from comment #57)
> > Thanks for looking into this KaiRo! Is there any reason to leave the
> > blocklist on central/aurora in that case?
> 
> In the light of those developments, it might be reasonable to undo those
> again. Right now, I'm unsure if we should or not, knowing too little about
> the whole library and its use myself.

Let's continue to watch FF10 data and if it doesn't plateau with a high correlation to BExternal.dll, we can back out of 12/13.
Comment 60 Gen Kanai [:gen] 2012-03-12 17:41:08 PDT
Adding Tomer Cohen of Babylon.
Comment 61 :Ehsan Akhgari 2012-03-12 20:10:55 PDT
I'm assuming that if this needs to be backed out from central/Aurora, somebody will notify me.  :-)
Comment 62 Tomer Cohen :tomer 2012-03-13 04:12:01 PDT
(In reply to Gen Kanai [:gen] from comment #60)
> Adding Tomer Cohen of Babylon.
For the record, I am not the same person as tomer@babylon. ☺

Re: https://groups.google.com/forum/?fromgroups#!topic/mozilla.dev.l10n/0TgNk5NMDEQ

Some contact information aggregated from the web:

Technical support and users services: +972-3-5382111
Fax: +972-3-5334080
Email: support@babylon.com , helpdesk@babylon.com
Comment 63 Tomer Cohen :tomer 2012-03-13 04:38:04 PDT
I've contacted Babylon Support as well and made sure they are aware of this issue…
Comment 64 Jorge Villalobos [:jorgev] 2012-03-13 08:53:38 PDT
(In reply to Tomer Cohen :tomer from comment #63)
> I've contacted Babylon Support as well and made sure they are aware of this
> issue…

We're already talking with Babylon, so this is unnecessary. Thanks anyway :)

(In reply to Ehsan Akhgari [:ehsan] from comment #61)
> I'm assuming that if this needs to be backed out from central/Aurora,
> somebody will notify me.  :-)

I think we should back out the block. Alex?
Comment 66 :Ehsan Akhgari 2012-03-14 16:08:41 PDT
Also backed out from central: https://hg.mozilla.org/integration/mozilla-inbound/rev/9938ccc5f1e2
Comment 67 Justin Lebar (not reading bugmail) 2012-03-14 16:26:49 PDT
Marking with [3rd-party-bustage], a new whiteboard tag I'm using to track problems caused by non-AMO add-ons.
Comment 68 Alex Keybl [:akeybl] 2012-03-20 15:41:27 PDT
(In reply to Jorge Villalobos [:jorgev] from comment #64)
> I think we should back out the block. Alex?

I'm late to the party, but yes - I agree that this should be backed out from m-c.

Note You need to log in before you can comment on or make changes to this bug.