Closed Bug 1036184 Opened 10 years ago Closed 8 years ago

Blocklist Crossrider framework add-ons

Categories

(Toolkit :: Blocklist Policy Requests, defect)

Product:
Toolkit
Component:
Blocklist Policy Requests
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED WONTFIX

People

(Reporter: kmag, Unassigned)

References

Details

(In reply to Jan de Mooij [:jandem] from comment #13)
> (In reply to Kris Maglione [:kmag] from comment #12)
> > We should probably just block all IDs using this format, and give the
> > Crossrider devs a short window to fix whatever the issue in the framework is
> > for the rest. I'd actually be completely happy to block the framework
> > entirely until the developers come up with a version that passes AMO review.
> > It's caused serious issues in the past, and I very much doubt that many of
> > the add-ons in the wild provide any real value to our users.
> 
> That'd be great, to stop the crashes.

Crossrider add-ons have a lot of issues. The most recent and pressing one is a fairly serious crash correlation. They've also caused serious crashes in the past, are often silently side-installed, and have potentially serious security issues, due to fetching scripts over HTTP and evaluating them in a chrome context.

We should block all add-ons which use the Crossrider framework until the developers have submitted a version which passes review for AMO. We began that process over a year ago, but it quickly stalled.

The most common patterns seem to be:

/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}@[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\.com$/
/^crossriderapp\d+@crossrider\.com$/
mconnor, do you think we should pull this trigger now or wait for signing next cycle?
Flags: needinfo?(mconnor)
Hey Kris and Benjamin,

We have checked our crashes logs that we keep for each browser and we do not see any unusual crashes for Firefox.

Can you please share with us more information regarding the "crash correlation" you are seeing ? we would like to get to the bottom of this with your help and to fix those crashes ASAP.

Shay.
Please see bug 1028902 comment 5 and newer
My thinking is that if it's causing real, serious issues we should blocklist the affected versions.  It's disruptive, sure, but crashing is much worse.  We can still do soft blocks, right?
Flags: needinfo?(mconnor)
Hey, I am from the development team of Crossrider and I can promise you that we will work to solve this as ASAP, as we take this VERY seriously.
Hi Shay, it's much appreciated! The primary issue for me is what to do in the meantime.  If we soft block these add-ons, once updates are available they'll be re-enabled, and users will be able to override the block through the UI if they're willing to make that trade.  Obviously the goal is to fix the issue ASAP, and especially with multiple add-ons we know that'll take time, so we have to do what we think is best for our users.
Hey Mike, Benjamin and Kevin,

I understand your concerns regarding the crashes, but you can be rest assured that this is a top priority for us to find its cause, and our tech team is working around the clock on this.

Also, I can assure you that once any bug will be found (if indeed these crashes are related to our development framework) we can rollout fixes very fast across the board!

It is for the best interest of all (you, us and obviously our developers who work **** their extensions) not to block extensions blindly on the assumption that these crashes might be related to our platform. (We have more than 30,000 different developers using our framework)

If you wish, I would love to go on a phone call or Skype to discuss this.

In any case, we will update you ASAP with any findings.

Shmueli Ahdut (Crossrider CTO)
I'm fairly confident that the vast majority of Crossrider add-on instances are silent side-installs, so the number of developers the block would effect is not currently a mark in its favor. I've been inclined to block it for that reason alone for a long time now. The crashes only make the issue more urgent.
Hey All,

We found a possible issue in our code that can cause these crashes.
To be honest, we didn't succeed to reproduce the crash in our QA machines, and we tried all possible configuration that we can think of.
Anyway, we pushed now the fix to all of the extensions and incremented the version, I guess that it will take affect in the next couple of hours.
If you can keep us posted with the status from your side it will be great.

As Shmueli mentioned, you can be sure that it's our top priority right now.

Shay.
Kris,

On top of the fix that Shay had mentioned we had also found an issue with another component which might be crashing Firefox version 3.6 *only*. (The issue was fixed and pushed to all users)

As for collectively punishing Crossrider developers, that range from thousands of indie developers and many many other startups and companies - this will not be beneficial for the Firefox community, the developers and their end-users. (if you have a specific issue with a specific extension we are more than glad to discuss)

Again, regarding the crash - if you still see anything on your end or if you can share with us more specific information that you may have then it would be very helpful in our goal to remedy this situation.

Shay.
Shay, let's be clear here: this is not a collective punishment (and using phrases that evoke human rights abuses is rather over the top).  Blocklisting all currently impacted versions (that are causing real crashes, in volume, for our users) is not a death blow, but it will require pushing updates for all users.  Once those updates are available, the add-ons would be re-enabled.

If users _really_ want to restore the unstable versions, they'll be able to do it.  I do not think we would be serving our users well by leaving unstable code running on their systems.
Mike,

The fixes were already pushed and all users had been updated already with the new code.

Are you still seeing any crashes issues on your end ? 

If yes, can you please specify on which versions (is it current production versions, or on only future releases such as version 33) ?

If it is still happening and only on future release then I am assuming we still have time to work on this together to resolve all issues.
If all users have updated (which defies general logic, updates generally aren't instantaneous in the wild), then there's no harm in blocklisting old versions of those add-ons.

I don't have the full set of affected versions, I'm simply arguing that unstable software versions should be disabled.  If you've actually pushed updates, it's a non-issue from your end, no?
I don't think that there's currently any way around moving quickly on this block. We have data to clearly indicate that the vast majority of Crossrider extensions are silent side-installs, which is reason enough to block on its own. We also have crash data to say that, not only are these installs that users did not ask for, and may not know about, but they're also making the browser unstable.

There may be legitimate Crossrider developers and users that this block would hurt, but I think that it's unreasonable at this point to ignore the larger issue and let the situation continue as-is.
Hey guys,

To make sure there are no crashes what-so-ever we had disabled the platform from running on Nightly versions altogether (in our case it will take affect on version 33 immediately).

This is the code we had added to the XPI:
var prefManager = Components.classes["@mozilla.org/preferences-service;1"].getService(Components.interfaces.nsIPrefBranch);
if (prefManager.getCharPref("app.update.channel") == "nightly") return;

XPI version have been increased and all Firefox clients will be update in the next few hours.

We are always testing ahead versions of upcoming releases and making sure there are no crashes or any other issues for that matter.

Can you please confirm that you are not seeing any more crashes on your end ?

Thanks,
Shay.
Kris, I don't think it's helpful to repeat the same arguments over and over and antagonize people who are trying to actually be helpful here.

The crashes have been reduced but have not gone away yet. That said, they are on Firefox Nightly 33.0a1 only and it's possible that some code changes in Mozilla code triggered the crashes - even if that could be due to Crossrider code doing unexpected things. It would be good to find out what exactly it is so we can fix things if it's our bug.

Also, yes, silent side-installs are bad and not allowed. Let's treat those as two separate issues though, as the silent side-installing issue affects people on other channels as well while the crashes only affect Nightly. We seem to have a communication avenue with Crossrider, maybe we can actually get the issues on all levels fixed through that.
Thanks Robert,

We are very glad to see that crashings had been reduced and there are positive outcomes to this thread!

We are still working and trying to minimize potential crashes down to zero, so if you (or any one else) can share more information regarding FF versions or anything else that you see on your end that might be helpful for the matter, it will help us greatly in making all the crashing issues go away.

As a side note re installations - we allow FF installations only when the user is *fully* aware and consciously agrees and approves the add-on to be installed. If you are seeing a Crossrider based extension that is not in accordance to the above then please communicate this to us and we will send a cease and desist to the owner of the extension in question (and block them from our platform if necessary).
Based on the crash signatures, look for places where you do: new String(x) or x.length, where x is a string. Does the framework use "new String(..)" somewhere? 

The problem could be somewhere else, of course, but if you've no leads that's where I would start.
(In reply to Shmueli Ahdut from comment #17)
> We are still working and trying to minimize potential crashes down to zero,
> so if you (or any one else) can share more information regarding FF versions
> or anything else that you see on your end that might be helpful for the
> matter, it will help us greatly in making all the crashing issues go away.

I posted some more information about the current state of the crash in bug 1028902, we should track/discuss those Nightly crashes there.

Also, it looks like the out of memory situations in bug 984033 (there is a bug on our side involved there for sure as we should try to not crash there when memory gets low) are more likely to happen with some Crossrider framework add-ons installed, so it may be a good idea to take a look if there are ways that your framework may use a lot of memory and if that can be reduced (may not be the case, just speculation based on a symptom).
Jan - we don't have in our framework code "new String(..)" at all.
Robert - we looked at the related bugs you have mentioned and tested the specific extensions on the nightly version and saw no crashes.
(In reply to Shay Dadosh from comment #15)
> To make sure there are no crashes what-so-ever we had disabled the platform
> from running on Nightly versions altogether (in our case it will take affect
> on version 33 immediately).

If Crossrider disables itself on Firefox Nightly and we think this crash is a regression in Nightly 33, how will we know when the crash has been fixed? Won't these crashes resurface for Crossrider users when 33 goes to Aurora?
status-firefox32: --- → unaffected
status-firefox33: --- → affected
Keywords: regression
I realize that they're trying to be helpful, but the crashes are not the only issue that I'm interested in here, and the other issues I've brought up have not even been touched upon. This is a bug for blocklisting the framework. Crashes are one of the reasons I think that doing so is necessary, but they're not the only 
reason, and the others are just as important.

(In reply to Chris Peterson (:cpeterson) from comment #21)
> If Crossrider disables itself on Firefox Nightly and we think this crash is
> a regression in Nightly 33, how will we know when the crash has been fixed?
> Won't these crashes resurface for Crossrider users when 33 goes to Aurora?

Yes. I think that disabling on nightlies is exactly the wrong solution here. It only prevents us from seeing issues before they begin affecting a larger user base.
(In reply to Kris Maglione [:kmag] from comment #22)
> I realize that they're trying to be helpful, but the crashes are not the
> only issue that I'm interested in here, and the other issues I've brought up
> have not even been touched upon.

Shmueli actually has addressed the silent side-install issue in comment #17. We should follow up on that, sounds like they are willing to help and see it the same way as us.

Were there any other concerns?

(Also, removing the keywords that point this out as anything else than what the bug actually is about).
Keywords: crash, regression, topcrash-win
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #23)
> Shmueli actually has addressed the silent side-install issue in comment #17.
> We should follow up on that, sounds like they are willing to help and see it
> the same way as us.

This is a very common misunderstanding with developers, where they think that having an opt-out in the installer is sufficient and justifies actively bypassing the opt-in in Firefox. We don't allow it, and current policy is to block under those circumstances, since it's likely a large portion of users didn't want the add-on to be installed.

The solution on their side is to use a new ID and perform the installations following our policy. To that I would add the recommendation that they use easily understandable add-on IDs, since the alphanumeric ones make developers much harder to track and we may end up blocking if we can't locate the author.

We're moving forward with the block for the reasons given so far.
Hi Jorge, and everyone,

My name is Koby Menachemi and I'm the CEO of Crossrider. I think it got to the point where you can do an irreversible damage to my company and to all of our developers so I should address these issues myself. 

First, I agree there are 2 separated threads in here so it's better to put things in order:

1. Regarding the crashes, which I consider as the more important issue, we are working hard to get to the bottom of things and to verify our code will run smoothly on the Nightly build as well. As a side note: we really appreciate your help with solving this issue!

2. Regarding the silent installations: if you have an add-on who is installing silently and it's against your policy then simply block it! (btw, to be honest then while Google's policy of not allowing silent installations is very clear, I'm not sure developers understand what is Mozilla's take on it). 

Anyway, since the view/policy of this was changed some time ago, Crossrider does not allow any kind of add-on installations which the user is not *fully* aware of the installation (including the name of the add-on, origin, description, functionality of that add-on, etc.) and we are working with other companies to block unwanted add-ons (besides our own mechanisms of actively monitoring add-ons ourselves).

You can view the policy here: http://crossrider.com/pages/developer_guidelines (I would love to get your feedback, if you have any)

Also, together with all the legal wording we are actively warning developers at the signup process (as an additional confirmation step) from ‘playing games’ by showing them this warning: http://screencast.com/t/FtIhM9u1uIV (just fyi: we are blocking 5-15 extensions/add-ons on a weekly basis!). 

We have reached to Mozilla on several occasions to work together and we never had the same cooperation level we got from companies such as Google or Facebook - in matter of fact, no cooperation at all. 

Just to give you a sense: there are tickets of unwanted add-ons (like this: https://bugzilla.mozilla.org/show_bug.cgi?id=754648) that Facebook contacted us directly and notified us about. Why there was never an open and direct communication between Mozilla and Crossrider is beyond me.  

What I’m trying to say is that treating Crossrider as the bad guy is not only unfair, but completely not true, as we put all of our efforts to maintain a clean environment for all users, just like Mozilla is doing. 

Blocking Crossrider entirely as a platform (rather than blocking specific add-ons) will affect more than 30,000(!) developers who are currently using Crossrider. In other words it's a delegitimation of Crossrider as a platform, as a product, and as a company. I think you should be very careful before taking this direction and do it only after thorough thought and understanding the meanings of your action as it affects so many developers and companies (not just as ‘fair thing’, but also what Mozilla can and can’t do legally; think about all the startups who are using Crossrider and for them it’s their business; an example of a great product that I use personally is http://folloze.com and there are many more).

I can only suggest to meet (even at the corporate level) to actually work together to make sure the Mozilla eco-system will be safe for both end-users and developers alike.

Yours,
Koby
Crossrider, CEO
(In reply to Koby Menachemi from comment #25)
> 2. Regarding the silent installations: if you have an add-on who is
> installing silently and it's against your policy then simply block it!

That's our usual policy, as what we did with bug 754648. However, we're not going to add hundreds of individual blocks when it's clear that there's a pattern here and we will probably miss hundreds more.

> (btw,
> to be honest then while Google's policy of not allowing silent installations
> is very clear, I'm not sure developers understand what is Mozilla's take on
> it). 

Our policy is here: https://developer.mozilla.org/en-US/Add-ons/Add-on_guidelines

> Anyway, since the view/policy of this was changed some time ago, Crossrider
> does not allow any kind of add-on installations which the user is not
> *fully* aware of the installation (including the name of the add-on, origin,
> description, functionality of that add-on, etc.) and we are working with
> other companies to block unwanted add-ons (besides our own mechanisms of
> actively monitoring add-ons ourselves).

If you are going to permanently block *all* add-ons that are silently installed per our policy above (meaning that the Firefox opt-in isn't shown), I'm willing to give you time for that and eventually drop this block request. I'll also need to know exactly what capabilities you have to accomplish this.

> You can view the policy here:
> http://crossrider.com/pages/developer_guidelines (I would love to get your
> feedback, if you have any)

Please email me directly and we can talk about it.

> We have reached to Mozilla on several occasions to work together and we
> never had the same cooperation level we got from companies such as Google or
> Facebook - in matter of fact, no cooperation at all.

I have several discussions with you in my email history. Unfortunately they didn't seem to yield useful results, which is why we didn't accept the Crossrider framework on AMO.

> Just to give you a sense: there are tickets of unwanted add-ons (like this:
> https://bugzilla.mozilla.org/show_bug.cgi?id=754648) that Facebook contacted
> us directly and notified us about. Why there was never an open and direct
> communication between Mozilla and Crossrider is beyond me.

We generally treat add-ons as products created directly by their developers, and we only try to contact the developers about them. We have no reason to assume you have direct control over all the add-ons created with your framework. Since you claim that you do, we will keep that in mind when we run into problems with other Crossrider add-ons in the future.

> What I’m trying to say is that treating Crossrider as the bad guy is not
> only unfair, but completely not true, as we put all of our efforts to
> maintain a clean environment for all users, just like Mozilla is doing. 

If we can't reach an agreement on how to move forward, we will need to continue with the current plan, which is what we believe is best to ensure a good user experience in Firefox. We have data showing that the majority of installs correlated to your platform are silent per our policies, and we can't neither let that slide nor spend an unreasonable amount of time picking and choosing the add-ons that aren't causing these problems.
(In reply to Jorge Villalobos [:jorgev] from comment #26)

> That's our usual policy, as what we did with bug 754648. However, we're not
> going to add hundreds of individual blocks when it's clear that there's a
> pattern here and we will probably miss hundreds more.

You are talking about hundreds and I'm talking about more than 50,000 add-ons that were built using Crossrider.  It does make sense that these hundreds of add-ons are generating most of the installations and therefore your claim about majority of installs are silents, but please note it's not the majority of the addons built with our platform!  Silent installs are not relevant to Crossrider alone, and I hope my example will not sound too far-fetch but think about it as someone will say that there are so many silent installs on FF and rather than adding them one by one to the block list, you can simply block FF as there is a pattern in here. 

> Our policy is here:
> https://developer.mozilla.org/en-US/Add-ons/Add-on_guidelines

Actually it's a good idea to add it as 'reference' that developers need to respect the policies of the browsers.

> If you are going to permanently block *all* add-ons that are silently
> installed per our policy above (meaning that the Firefox opt-in isn't
> shown), I'm willing to give you time for that and eventually drop this block
> request. I'll also need to know exactly what capabilities you have to
> accomplish this.
I'm just like you hunting them one by one. What I'm saying is that I can help you fight it by blocking addons you report to us (I guess it's easier than adding it to your block-list) - unlike me, you can tell whether it was installed silently or not (I have no way of knowing it).  If you think "why to bother if I can simply shut it altogether" then think about the day after where developers will create their own xpi and will do the same (like many others who are not using Crossrider). You will be at the same starting point, but now without a partner to work with.

Thanks for your prompt reply and will email you re the policy and how to take it forward.
(In reply to Koby Menachemi from comment #27)
>Silent installs are not relevant to Crossrider alone, and I 
>hope my example will not sound too far-fetch but think about it 
>as someone will say that there are so many silent installs on 
>FF and rather than adding them one by one to the block list, 
>you can simply block FF as there is a pattern in here. 

It's true that this is not an issue with Crossrider alone, but 
the silent install pattern that we see with Crossrider add-ons 
is overwhelming. The *vast* majority of add-ons installed with 
it are silent installs, which is something that we don't 
currently see with any other framework. For all such instances 
of this magnitude we've dealt with in the past, we've demanded 
similar swift action.

As for the larger problem of silent installs, yes, it's 
something we're planning to deal with as an issue in its own 
right. We're currently in the works of instituting a mandatory 
add-on signing policy, which will require developers to 
explicitly accept an agreement to abide by our installation 
policies.

>I'm just like you hunting them one by one. What I'm saying is that I can 
>help you fight it by blocking addons you report to us

I may be able to get you a list of offending IDs. However, we 
have very strict privacy rules relating to this data, and I'll 
have to clear it with the privacy team before I can give you 
anything.
(In reply to Kris Maglione [:kmag] from comment #28)
> (In reply to Koby Menachemi from comment #27)
> >I'm just like you hunting them one by one. What I'm saying is that I can 
> >help you fight it by blocking addons you report to us
> 
> I may be able to get you a list of offending IDs. However, we 
> have very strict privacy rules relating to this data, and I'll 
> have to clear it with the privacy team before I can give you 
> anything.

Also, I need you to clarify exactly how your blocking capability works. Is the add-on disabled, or does it stay enabled and just stops working? Are users notified in any way? Do you have an example of a Firefox add-on you have blocked that we can test?
Jorge,

On the email thread I've sent you how to report addons to Crossrider and how the blocking will work (I prefer not to put it on public forum so no one will use this information to bypass our mechanism). I suggest you send us the IDs and we can work to make sure we together fix this issue (and fine-tune it if needed).
Blocks: 1043017
See Also: → 1052599
[Tracking Requested - why for this release]:
tracking-b2g: --- → backlog
Please don't change flags without explanation.
tracking-b2g: backlog → ---
Product: addons.mozilla.org → Toolkit
Would it be incorrect to suggest this is past being useful?
Flags: needinfo?(koby)
Flags: needinfo?(kmaglione+bmo)
At this point I don't think there's anything that should be done here. If problems with this framework or add-ons built with it come up again, please file new bugs.
Status: NEW → RESOLVED
Closed: 8 years ago
Flags: needinfo?(koby)
Flags: needinfo?(kmaglione+bmo)
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.