Closed Bug 715744 Opened 12 years ago Closed 12 years ago

DLL block request: BExternal.dll

Categories

(Toolkit :: Blocklist Policy Requests, defect)

Product:
Toolkit
Component:
Blocklist Policy Requests
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED WONTFIX
Tracking Flags:
Tracking Status
firefox10 + ---
firefox11 + wontfix

People

(Reporter: khuey, Assigned: ehsan.akhgari)

References

Details

(Keywords: qawanted, Whiteboard: [dll][3rd-party-bustage])

Attachments

(1 file, 1 obsolete file)

Patch (v1)
928 bytes, patch
benjamin
: review+
akeybl
: approval-mozilla-aurora+
Details | Diff | Splinter Review 
DLL name: BExternal.dll
DLL versions to block: Unversioned
Applications, versions, and platforms affected: Firefox on Windows

Homepage and other references and contact info: Unknown

Reasons:
BExternal.dll appears to be injecting itself into the Firefox process and proceeding to manipulate our preferences off the main thread.

See crash reports like
https://crash-stats.mozilla.com/report/index/2a925d21-8e15-40c7-8cf5-5b9642120105
https://crash-stats.mozilla.com/report/index/73983a26-dafe-4384-967e-88f3a2120105
The internet thinks this is Babylon software, which is a translation app: http://www.babylon.com/windows

Kev - have we had to deal with them before?
Blocks: 715757
Prior to tracking for any release, it'd be good to better understand the affected user population and number of crashes highly correlated with the DLL (cc'ing Justin and Sheila).

Also looping Marcia in case we decide to move forward with this. We'd want to test a try build with the DLL blocklist entry included and also make sure that there's no related add-on (or regressions once blocked).
tracking-firefox11: --- → ?
(In reply to Alex Keybl [:akeybl] from comment #3)
> Prior to tracking for any release, it'd be good to better understand the
> affected user population and number of crashes highly correlated with the
> DLL (cc'ing Justin and Sheila).

Doesn't seem like this is an add-on, so I can't really give a usage count.
[Triage Comment]
I think we're too close to the end of the cycle to prepare and execute on a DLL blocklist addition for FF10. It's not clear that this needs to be tracked for FF11 either. We would, however, consider uplifting the entry once ready.
tracking-firefox10: ? → -
tracking-firefox11: ? → -
(In reply to Alex Keybl [:akeybl] from comment #5)
Alex, were you aware that BExternal.dll is partially causing a topcrash: see bug 715757 comment 4.
tracking-firefox10: - → ?
tracking-firefox11: - → ?
(In reply to Luke Wagner [:luke] from comment #6)
> (In reply to Alex Keybl [:akeybl] from comment #5)
> Alex, were you aware that BExternal.dll is partially causing a topcrash: see
> bug 715757 comment 4.

I was not. At this point, it's a difficult path to blocklisting in FF10 to resolve 44% of Windows crashes caused by 715757. We'd need to reach out to Babylon (Kev?), and then decide whether or not to blocklist based upon their response. Remember, if they plan on fixing we can't blocklist an unversioned DLL and then revoke the block before FF11's release. If we decide to move forward with the blocklist, we'd then need to test to make sure we're not leaving users in an even worse state once the DLL is blocklisted (for whatever reason).

I'll leave the door open for now and track this, but it's really a stretch.
tracking-firefox10: ? → +
tracking-firefox11: ? → +
The crash in CrashInJS caused by BExternal.dll (41%) is #10 top crasher in 10.0b4
Bug 715757 comment 7 says that blocking this and gemgecko10.dll remove 99% of the #8 topcrash.
As mentioned in comment#7, blocklisting a legitimate unversioned DLL may cause user pain until FF11, and only then if the plugin author has resolved the issue in a later version and pushes it out to their users.
Comment on attachment 586291 [details] [diff] [review]
Patch

r-.  The DLL name must be in all lower case, otherwise the blocklist won't take effect at all.
Attachment #586291 - Flags: review-
Comment on attachment 586291 [details] [diff] [review]
Patch

I may have miscommunicated on irc, but this patch wasn't requesting review due to comment 10.
Attachment #586291 - Attachment is obsolete: true
(In reply to Justin Scott [:fligtar] from comment #4)
> (In reply to Alex Keybl [:akeybl] from comment #3)
> > Prior to tracking for any release, it'd be good to better understand the
> > affected user population and number of crashes highly correlated with the
> > DLL (cc'ing Justin and Sheila).
> 
> Doesn't seem like this is an add-on, so I can't really give a usage count.

This is related to the Babylon Toolbar add-on, actually. The Babylon tool installs the toolbar in Firefox, and it is one of the most used add-ons around. The stats indicate roughly 6.7M ADU. I don't know if the DLL is part of the add-on, but there are other block requests for it, like bug 721264.
Working on getting a contact with Babylon Software. Apologies for the delay, this one was flying under my radar.
(In reply to Kev [:kev] Needham from comment #14)
> Working on getting a contact with Babylon Software. Apologies for the delay,
> this one was flying under my radar.

How's outreach to Babylon going? Have we found a solid contact?
If we cannot blacklist the DLL and cannot contact developers, what about adding a non-main thread check at the earliest possible point before we reach the assert and then reporting a failure? For example, XPCPerThreadData::GetDataImpl or XPCJSContextStack::GetSafeJSContext could be a good place for that as at that point we can check for non-main thread with a simple if.

If that fixes the crash, we can get some time before deciding with blacklisting.
There is a risk that those new sources of dynamic errors would trigger some other problem that would be harder to track.  Blacklisting is much lower risk (assuming we cannot quickly get in contact with them).  If Babylon actually cares to release a new .dll that doesn't crash, can't they avoid the unversioned blocklist by changing their .dll name?
(In reply to Luke Wagner [:luke] from comment #17)
> If Babylon
> actually cares to release a new .dll that doesn't crash, can't they avoid
> the unversioned blocklist by changing their .dll name?

We also don't want to pull the rug out from under our users who actually benefit from the software. That's why we're checking on the status of our outreach.
Perhaps a more immediate question we can answer is: does the addon cause a crash for 100% of users?  If so, then the set of users benefiting from it right now might be empty.
(In reply to Alex Keybl [:akeybl] from comment #18)
> (In reply to Luke Wagner [:luke] from comment #17)
> > If Babylon
> > actually cares to release a new .dll that doesn't crash, can't they avoid
> > the unversioned blocklist by changing their .dll name?
> 
> We also don't want to pull the rug out from under our users who actually
> benefit from the software. That's why we're checking on the status of our
> outreach.

I realize I'm opening a can of worms here, but do users actually benefit from the Babylon Toolbar?
(In reply to Luke Wagner [:luke] from comment #17)
> If Babylon
> actually cares to release a new .dll that doesn't crash, can't they avoid
> the unversioned blocklist by changing their .dll name?

Yes, AFAIK. This possibility was one of the main reasons why we clearly determined that the blocklist is no good measure against malware.

(In reply to Luke Wagner [:luke] from comment #19)
> Perhaps a more immediate question we can answer is: does the addon cause a
> crash for 100% of users?

We only have data for loaded DLLs from crash reports, so we don't know for how many users this DLL might be working.

What we know is this:

Correlations for signature "CrashInJS" on Firefox 11.0, Windows, 2012-02-27:

Add-ons:
     15% (145/956) vs.   8% (1722/22655) ffxtlbr@babylon.com
          0% (2/956) vs.   0% (26/22655) 1.1.3
          0% (0/956) vs.   0% (5/22655) 1.1.8
         11% (101/956) vs.   4% (988/22655) 1.1.9
          4% (42/956) vs.   3% (703/22655) 1.2.0

Modules:
    100% (955/956) vs.   4% (956/22655) BExternal.dll

From that data, we can see that out of 22655 total crash reports yesterday on Firefox 11.0 (beta) builds, 1722 had the ffxtlbr@babylon.comm add-on installed (the version breakdown of the add-on is just FYI, to show that unfortunately not all crashes are on one add-on version), interestingly only 145 of the 956 crash reports with the affected signature have the add-on, so an add-on blocklist would not be effective.
ALL BUT ONE of the affected crashes with this signature have BExternal.dll (unfortunately unversioned) installed though, and that's only 1 difference from the number of total Firefox 11 crashes ports that have this library loaded.

That means that yesterday on Firefox 11.0 and Windows, every crash report but one that had this module loaded did crash with this same signature, which is pretty impressive.

Again, we don't know about Firefox runs that don't crash at all or don't send a report, but from those that crash in any place and sent us a report, basically everyone crashed this way, which dmandelin clearly indicates in bug 715757 comment #19 as "an add-on is doing bad stuff". This IMHO is a very strong argument for blocking this DLL ASAP on 11.0 beta.
(In reply to Kev [:kev] Needham from comment #14)
> Working on getting a contact with Babylon Software. Apologies for the delay,
> this one was flying under my radar.

Kev - has there been response from Babylon? It'd be preferable to give them a heads up prior to blocklisting the DLL.
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #20)
> I realize I'm opening a can of worms here, but do users actually benefit
> from the Babylon Toolbar?

It does appear to provide benefit:

"One click translation application offers over 75 languages to be translated from language to language, Babylon provides precise translations and updated content (e.g. Wikipedia in 21 languages) in a single click, without interrupting your workflow."

If we're interested in moving forward with this blocklist, my preference would be to first hear back from Babylon about updating their software. If that's not possible, let's prepare a blocklist patch and some try builds so that we can determine if we need to block the add-on toolbar at the same time (and verify that the blocklist addition works). We could push this into the FF11 beta 6 build (3/5) depending on our status on Monday.
Keywords: qawanted
It's not clear to me what, if anything, QA can do -- what is the specific request for qawanted?
(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #24)
> It's not clear to me what, if anything, QA can do -- what is the specific
> request for qawanted?

Sorry - errant keyword added.
Keywords: qawanted
Attached patch Patch (v1) — — Splinter Review 
Assignee: nobody → ehsan
Status: NEW → ASSIGNED

        Attachment #602380 -
        Flags: review?(benjamin)

    
  

        Attachment #602380 -
        Flags: review?(benjamin) → review+

    
    

    
  
(In reply to Ehsan Akhgari [:ehsan] from comment #27)
> Try build: https://tbpl.mozilla.org/?tree=Try&rev=a1bd00172de6

Thanks guys - much appreciated! Adding qawanted to do exploratory testing around Babylon's software (comment#2) with this try build. We want figure out whether we need to blocklist any associated add-ons and make sure we're not leaving users in a worse position than before the blocklist.
Keywords: qawanted

    
    

    
  
Howdy all, I've tried contacting babylon several times, and have not received a response. I believe we've satisfied the criteria in attempting to work things through with the vendor, and will inform them of that we'll be blocklisting the DLL (sometimes that generates a response, too).

Should not block this, though.

(In reply to Alex Keybl [:akeybl] from comment #22)
> (In reply to Kev [:kev] Needham from comment #14)
> > Working on getting a contact with Babylon Software. Apologies for the delay,
> > this one was flying under my radar.
> 
> Kev - has there been response from Babylon? It'd be preferable to give them
> a heads up prior to blocklisting the DLL.

    
    

    
  
The SUMO team has found out that the Babylon toolbar (as other Montiera toolbars) are pretty much non desired for our users. Between 42% and 54% of the users who have it installed specifically don't want it.

You can find more info about the survey we run in here: https://etherpad.mozilla.org/babylon-toolbar-survey

    
    

    
  
Bug 721264 is in file for blocking the Babylon Toolbar. Let me know if we want to do this, and when.

Also, should it be blocked for Firefox 10 and above only, or other versions as well? Do we know which versions of the toolbar are causing the crashes?

    
    

    
  
(In reply to Jorge Villalobos [:jorgev] from comment #31)
> Bug 721264 is in file for blocking the Babylon Toolbar. Let me know if we
> want to do this, and when.
> 
> Also, should it be blocked for Firefox 10 and above only, or other versions
> as well? Do we know which versions of the toolbar are causing the crashes?

No decision has been made on the toolbar yet. We've not confirmed it causes crashes at all. The fact that many users don't want it should be a separate discussion, unless the add-on becomes nonfunctional after blocklisting the DLL.

    
    

    
  
Sorry, that reads funny - let me try again. Blocklisting the toolbar should be a separate discussion unless we find that it doesn't function after blocklisting the DLL. We're trying to resolve a topcrasher here, not remove an unpopular add-on.

    
    

    
  
I haven't been able to get the BExternal.ddl to load with Fx11b5 with different types of interactions with the Babylon 9 toolbar. I launch it, take a look at the loaded dlls in firefox.exe, use the toolbar a bit more, but this particular dll doesn't show up in the list. A similar thing happens with the try-server builds, so I can't tell the effect of the block.

    
    

    
  
Tried the TRY build on Windows 7 64-bit. Firefox asks if I want to allow the following add-ons to install:
 * Babylon Translation Activation 1.1
 * Babylon 1.2.0
 * Babylon Spelling and Proofreading 1.0.0.1

Disallowing the installation disables the add-ons but I still get a Babylon icon in my title bar. I can also manually enable the add-ons in about:addons. Allowing the installation enables the add-ons by default.

In Firefox 11.0b5, I get no such opportunity to allow/disallow.

    
    

    
  
(In reply to juan becerra [:juanb] from comment #34)
> I haven't been able to get the BExternal.ddl to load with Fx11b5 with
> different types of interactions with the Babylon 9 toolbar. I launch it,
> take a look at the loaded dlls in firefox.exe, use the toolbar a bit more,
> but this particular dll doesn't show up in the list. A similar thing happens
> with the try-server builds, so I can't tell the effect of the block.

To clarify, you're testing with http://www.babylon.com/redirects/download.cgi?type=100 (Babylon9_setup.exe)? This bug specifically pertains to having the babylon DLL loaded when running Firefox, and then seeing it blocked with the try build. If we're not able to repro BExternal.dll loading at all, there's no point in further testing because we have the wrong installer.

    
    

    
  
(In reply to Alex Keybl [:akeybl] from comment #36)
> (In reply to juan becerra [:juanb] from comment #34)
> > I haven't been able to get the BExternal.ddl to load with Fx11b5 with
> > different types of interactions with the Babylon 9 toolbar. I launch it,
> > take a look at the loaded dlls in firefox.exe, use the toolbar a bit more,
> > but this particular dll doesn't show up in the list. A similar thing happens
> > with the try-server builds, so I can't tell the effect of the block.
> 
> To clarify, you're testing with
> http://www.babylon.com/redirects/download.cgi?type=100 (Babylon9_setup.exe)?
> This bug specifically pertains to having the babylon DLL loaded when running
> Firefox, and then seeing it blocked with the try build. If we're not able to
> repro BExternal.dll loading at all, there's no point in further testing
> because we have the wrong installer.

Correct. I used the Babylon9_setup.exe installer.

    
    

    
  
(In reply to juan becerra [:juanb] from comment #37)
> Correct. I used the Babylon9_setup.exe installer.

They may have renamed their DLL at some point. After installing Babylon 9 I see captlib.dll loading in Firefox, but not bexternal.dll.

Can we try some of the older versions at http://forum.oldversion.com/showthread.php?108-Babylon&p=23831&viewfull=1#post23831 to see if bexternal.dll is loaded with any of them?

    
  
Blocks: 721645

    
    

    
  
I tried versions 3.2, 4.0316, and 5.0 from the list in the link in comment #38. The other ones were dead links. From the ones I tried none loaded bexternal.dll

I tried looking in other places for older versions of this software, but they all point to some version of the latest.

    
    

    
  
(In reply to juan becerra [:juanb] from comment #39)
> I tried versions 3.2, 4.0316, and 5.0 from the list in the link in comment
> #38. The other ones were dead links. From the ones I tried none loaded
> bexternal.dll
> 
> I tried looking in other places for older versions of this software, but
> they all point to some version of the latest.

Thanks Juan. In that case, I'm not comfortable with fixing in FF11 given our lack of understanding of the blocklist's effect. The resolution of this bug is on hold until we find the software that installs BExternal.dll.

          tracking-firefox11:
          + → -

    
    

    
  
Babylon Toolbar uses 10 DLLs: BExternal.dll, IECookieLow.dll, sqlite3.dll, BabyFox.dll, BabylonIEPI.dll, BabylonOfficePI.dll, BContentServer.dll, BContentServerExt.dll, BContentServerLite.dll, captlib.dll (see http://www.threatexpert.com/report.aspx?md5=4d2965e27ebe75c22e0423f705b4e3e0).
So there should be certain conditions to have BExternal.dll loaded.

    
    

    
  
(In reply to Scoobidiver from comment #41)
> Babylon Toolbar uses 10 DLLs: BExternal.dll, IECookieLow.dll, sqlite3.dll,
> BabyFox.dll, BabylonIEPI.dll, BabylonOfficePI.dll, BContentServer.dll,
> BContentServerExt.dll, BContentServerLite.dll, captlib.dll (see
> http://www.threatexpert.com/report.
> aspx?md5=4d2965e27ebe75c22e0423f705b4e3e0).
> So there should be certain conditions to have BExternal.dll loaded.

Perhaps the DLL is loaded post-startup? Let's continue to try to get BExternal.dll to load in QA. May be worth trying to translate selected text and other features of Babylon.

    
    

    
  
(In reply to Alex Keybl [:akeybl] from comment #40)
> Thanks Juan. In that case, I'm not comfortable with fixing in FF11 given our
> lack of understanding of the blocklist's effect. The resolution of this bug
> is on hold until we find the software that installs BExternal.dll.


IMHO, that's the completely wrong way to go. We *know* that a huge number of real crashes for users are caused by this DLL, so we *know* that users have installed it and have problems. The absolute worst case this patch can have is that it has no effect, so the risk is zero in practice. We need to be hugely more aggressive in blocking libraries and add-ons that even *can* have bad side effects for users, we know that the top problems users have with Firefox are all issues with third-party software, and we need to become huge less forgiving for their errors, esp. when it's parties that don't even communicate with us as in this case.

Please reconsider this decision for 11.

In the position we are in without this, I cannot recommend shipping 11 at all, because it will cause a lot of grief for users (and for the CrashKill team, but users are the main concern).

    
    

    
  
From a full code inspection of the 3 addons the Babylon9 installer creates, the dll isn't loaded directly from within the addons.  It would have to be loaded as a plugin, XPCOM component, with ctypes or executed via nsIProcess.  

It appears that communication with the binary part of the toolbar is via a local server component (ports 9876 and 22678), so I'd guess the offending dlls are loaded via that instead.

    
    

    
  
It's not clear to me whether I'm supposed to land this patch on mozilla-central or not...

          tracking-firefox11:
          - → +

    
    

    
  
(In reply to Ehsan Akhgari [:ehsan] from comment #45)
> It's not clear to me whether I'm supposed to land this patch on
> mozilla-central or not...

We'll be discussing this bug in today's channel meeting at 2PM to discuss the risk of blocklisting the DLL without a testable case. We'll comment in this bug afterwards with a decision.

    
  

        Attachment #602380 -
        Flags: approval-mozilla-aurora+

    
    

    
  
Let's land this on m-c and Aurora 12 so that we can see if crash numbers drop on those branches before considering taking this for FF11 (drop dead date 3/9).

    
    

    
  
CCing the bugzilla accounts I found at babylon.com

    
    

    
  
I received a response from Babylon claiming that the crashes are being caused by a previous version of their software. That could explain the difficulty in reproducing the crashes. I'm trying to get information about the crashy version so we can test it.

    
    

    
  
Can someone post up to date crash stats? Are we seeing any change in the numbers? The Babylon people are saying they released a fixed version (I don't know when exactly) and would like to know if that has had any effect.

    
    

    
  
Aurora is in version 12, not 11.

          status-firefox11:
          fixed → ---

          status-firefox12:
          --- → fixed

    
    

    
  
Testing Nightly,Aurora and Stable release of Firefox on Window 7 with Babylon 8 installed. Firefox blacklists the plugin. 

Unfortunately I didn't notice the BExternal.dll file anywhere. Let me know if you need further help.

    
    

    
  
(In reply to Jorge Villalobos [:jorgev] from comment #51)
> Can someone post up to date crash stats? Are we seeing any change in the
> numbers? The Babylon people are saying they released a fixed version (I
> don't know when exactly) and would like to know if that has had any effect.

The CrashInJS signatures are seeing a downward trend since March 4, and have seen a huge drop in yesterday's data.

    
    

    
  
https://hg.mozilla.org/mozilla-central/rev/6151695f9df4
Status: ASSIGNED → RESOLVED
Closed: 12 years ago

          status-firefox13:
          --- → fixed
Resolution: --- → FIXED

    
    

    
  
On Firefox 12 (Aurora), where the blocklist landed, we had ~200 (CrashInJS | GetContextFromObject) crashes daily from March 1 to 4, then a slight dropoff and a sharp decline on 7th to 47. Yesterday (8th) we had none at all, which seems to be the blocklist in effect.

So there, we seem to have a good positive trend due to the fix mentioned in comment #51 and then full success of the blocklist in yesterday's data.


Now, for those versions that don't have the blocklist in place (and where, for other reasons, the signature is CrashInJS | JS_AbortIfWrongThread instead):

On 11, we had 1600-1800 crashes daily from March 1 through 6, a steep decline to 488 on 7th and 47 yesterday. The signature is now ranked #100 (for all of 11) and far out of topcrashers.

On 10, we had ~500-600 crashes after throttling (to 10%, so it's ~5000-6000 in reality) from March 1 through 6, a bit of a decline to 299 on 7th and further significant decline on 7th to 180. The crash rank is now #66, also out of topcrashers.

Looking at those, it looks like the new Babylon version itself had a tremendous effect, even without blocklisting the DLL. On 10, we probably have people adopting to updates a bit more slowly so the decline is not that steep but it's still pretty significant.

Judging from that, I'm unsure if we need to blocklist on 11 as well, it looks to me that Babylon solved this well enough on their side.

    
    

    
  
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #56)
> Judging from that, I'm unsure if we need to blocklist on 11 as well, it
> looks to me that Babylon solved this well enough on their side.

Thanks for looking into this KaiRo! Is there any reason to leave the blocklist on central/aurora in that case?

    
    

    
  
(In reply to Alex Keybl [:akeybl] from comment #57)
> Thanks for looking into this KaiRo! Is there any reason to leave the
> blocklist on central/aurora in that case?

In the light of those developments, it might be reasonable to undo those again. Right now, I'm unsure if we should or not, knowing too little about the whole library and its use myself.

    
    

    
  
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #58)
> (In reply to Alex Keybl [:akeybl] from comment #57)
> > Thanks for looking into this KaiRo! Is there any reason to leave the
> > blocklist on central/aurora in that case?
> 
> In the light of those developments, it might be reasonable to undo those
> again. Right now, I'm unsure if we should or not, knowing too little about
> the whole library and its use myself.

Let's continue to watch FF10 data and if it doesn't plateau with a high correlation to BExternal.dll, we can back out of 12/13.

          status-firefox11:
          --- → wontfix

    
    

    
  
Adding Tomer Cohen of Babylon.

    
    

    
  
I'm assuming that if this needs to be backed out from central/Aurora, somebody will notify me.  :-)

    
    

    
  
(In reply to Gen Kanai [:gen] from comment #60)
> Adding Tomer Cohen of Babylon.
For the record, I am not the same person as tomer@babylon. ☺

Re: https://groups.google.com/forum/?fromgroups#!topic/mozilla.dev.l10n/0TgNk5NMDEQ

Some contact information aggregated from the web:

Technical support and users services: +972-3-5382111
Fax: +972-3-5334080
Email: support@babylon.com , helpdesk@babylon.com

    
    

    
  
I've contacted Babylon Support as well and made sure they are aware of this issue…

    
    

    
  
(In reply to Tomer Cohen :tomer from comment #63)
> I've contacted Babylon Support as well and made sure they are aware of this
> issue…

We're already talking with Babylon, so this is unnecessary. Thanks anyway :)

(In reply to Ehsan Akhgari [:ehsan] from comment #61)
> I'm assuming that if this needs to be backed out from central/Aurora,
> somebody will notify me.  :-)

I think we should back out the block. Alex?

    
  

          status-firefox12:
          affected → ---

          status-firefox13:
          affected → ---
Resolution: FIXED → WONTFIX

    
    

    
  
Marking with [3rd-party-bustage], a new whiteboard tag I'm using to track problems caused by non-AMO add-ons.
Whiteboard: [dll] → [dll][3rd-party-bustage]

    
    

    
  
(In reply to Jorge Villalobos [:jorgev] from comment #64)
> I think we should back out the block. Alex?

I'm late to the party, but yes - I agree that this should be backed out from m-c.
Product: addons.mozilla.org → Toolkit

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: