Closed Bug 838864 Opened 7 years ago Closed 7 years ago

keyword.URL reset interferes poorly with some partner add-ons

Categories

(Firefox :: General, defect)

defect
Not set

Tracking

()

RESOLVED FIXED
Tracking Status
firefox19 + verified
firefox20 + verified
firefox21 + verified
firefox22 --- verified

People

(Reporter: Gavin, Assigned: Gavin)

References

Details

Attachments

(1 file)

Joanne pointed out via email that some add-ons that we've created for partners in the past (e.g. http://releases.mozilla.com/bundles/bing/addon/) set the keyword.URL pref directly. This conflicts somewhat with bug 718088's fix.

In builds with these add-ons installed and with bug 718088's fix, we will prompt to reset keyword.URL to the build's default engine (Google). If the user accepts, we will reset keyword.URL, but the rest of the add-ons effects won't be undone, i.e. the customized search engine (e.g. Bing) will remain the search bar "default".

This isn't really optimal, and kind of breaks the add-ons intended functionality. I think this would be less of a problem if we had a more complete reset (if accepting the prompt also affected the search bar).

I don't know how many such partner addons we have, or what their usage is. We should figure that out if possible before making a decision on whether to disable this reset for the 19 cycle.
For context, the Bing Addon has roughly 33k users (per AMO's info)
Since we seem to be in control of these add-ons, can we update them to change the value in the default branch rather than the user branch?

As for the usage, for the two add-ons in http://releases.mozilla.com/bundles/ the usage numbers are relatively low, though I'm not sure I can give them in a public bug.
(In reply to Kris Maglione [:kmag] from comment #2)
> Since we seem to be in control of these add-ons, can we update them to
> change the value in the default branch rather than the user branch?

We should probably change them, yes, though I don't think that's feasible before 19 goes out the door.
Just to clarify, while the usage for these add-ons is relatively low - they are part of a distribution agreement in which there are economic terms.  We have set Bing to the default on all search access points contractually and they should remain that way.  Please disable the prompt on these before launch of 19.
We have an estimated 200 million users affected by keyword.url changes. I'd like to do whatever we can to protect these users and their right to choose a search provider. In the bundled builds do we currently prevent users from changing these values if they have the inclination and technical ability to do so?
Group: mozilla-corporation-confidential
Can we rule out that updating the add-ons before the Firefox 19 release is at all feasible before we decide on whether to push this back a release? The extensions are relatively simple, and shouldn't take much time to update, test, and deploy, and rolling this back would affect a much larger number of users.
(In reply to Joanne Nagel from comment #4)
> Just to clarify, while the usage for these add-ons is relatively low - they
> are part of a distribution agreement in which there are economic terms.  We
> have set Bing to the default on all search access points contractually and
> they should remain that way.  Please disable the prompt on these before
> launch of 19.

and just to add that the addons are bing and msn, the other partner in the bundle directory ripcurl don't use the addon and is no more actively promoted i think
MSN and Bing addons are hosted by us and combined account for 45k ADIs. Here's a list of other partners, amazon, aol, canonical, ebay, mail.ru, msn/bing, rambler, seznam, twitter, united internet-mail.com, 1&1, web.de; yahoo, yandex who might be hosting their own addons leading to legitimate downloads + search default change.

I agree with Joanne that we shouldn't take any action without notifying our partners, there's a lot at stake given Bing is the default partner for Firefox Mobile and we have other high profile projects that will need involvement from these esteemed partners. 

matt - wrt comment #5, can you please cite the source of the 200M number so we can investigate the impact of the change to our partners?
Flags: needinfo?(mgrimes)
Do we have contracts with any of these companies that prevent us from interfering with such add-ons? Changing the keyword.URL setting in ways that are not reversed when the add-ons are uninstalled violates our add-on guidelines and is usually reason enough for blocklisting. While we would certainly notify partners and give them a chance to react before blocklisting their add-ons (as we would do for any other developer), all this change is doing is giving users, especially those who have already uninstalled the add-ons in question, a chance to restore this non-user-visible setting to its default value.

The MSN and Bing add-ons in question also violate these guidelines, but they can easily be fixed such that they a) don't violate our add-on guidelines, b) don't trigger this reset prompt, and c) don't (to the best of my knowledge) violate our contracts, by the simple expedient of changing the preference in the default branch where it won't persist after the add-on has been uninstalled.

The issues with those two add-ons are our responsibility, so I don't think that the reset prompt should hit release until they're fixed. In the case of any similarly afflicted partner add-ons, I'm not as sure.
We get the 200 million number from telemetry. currently on release roughly 51% of users have a non-standard keyword.url. Extrapolated to half our Firefox population this equates to around 200 million.
Flags: needinfo?(mgrimes)
The 200 milllion number is a rough approximation with a lot of baked-in assumptions that may or may not hold, and so let's not over-rotate on it.

I think we all agree on the importance of doing something to address the keyword.URL problem, but we're not doing ourselves any favors if the first step we take is incomplete and has bad partner-related side effects. We're not in much of a position to throw stones at others practices when the add-ons we ourselves developed have some of these issues.

I don't think we can reasonably address the add-on issues prior to the Firefox 19 release, and so I think really don't have much option than to delay this an additional cycle.
Group: mozilla-corporation-confidential
This applies to beta cleanly and is pretty straightforward.
Assignee: nobody → gavin.sharp
Status: NEW → ASSIGNED
Attachment #711481 - Flags: review?(fyan)
Attachment #711481 - Flags: review?(fyan) → review+
agree with :gavin on comment #11, we need to give our partners enough time to react to our changes and not get caught by surprise!
Comment on attachment 711481 [details] [diff] [review]
disable the prompting

[Approval Request Comment]
Bug caused by (feature/regressing bug #): bug 718088
User impact if declined: bad interactions with partner add-ons
Testing completed (on m-c, etc.): none, but this is a simple switch-off
Risk to taking this patch (and alternatives if risky): only risk is that the disabling isn't complete enough, but this is pretty simple logic.
String or UUID changes made by this patch: none
Attachment #711481 - Flags: approval-mozilla-beta?
This isn't about throwing rocks, and I don't think it's a matter of punishing anyone. This is as much in the interests of our partners as it is in ours, since in their custom builds it will restore hijacked keyword URLs to the original values for those builds.

As for updating the MSN and Bing add-ons, I don't think it's unreasonable. The simplest change that will address the issue is one line:

--- bootstrap.js	2013-02-07 14:44:45.011215864 -0800
+++ -	2013-02-07 14:44:56.246448038 -0800
@@ -83,7 +83,7 @@
 
 // Customize the default prefs
 function setPref(pref, value) {
-  let branch = Services.prefs.getBranch("");
+  let branch = Services.prefs.getDefaultBranch("");
   branch.setCharPref(pref, value);
 }

Changing the add-ons to bring them more completely in line with policy is nearly as simple: https://gist.github.com/63f39616238430268ba7

Given the triviality of the add-ons and the changes, I don't think that QAing them in a short timeframe would be problematic. And as add-on update checks are done at startup after an upgrade, I don't think that there are deployment issues.

Pushing this back another 6 weeks, on the other hand, means that we have, even if not 200 million, at least hundreds of thousands of users affected by hijacked keyword URLs that lead to large amounts of SUMO complaints. It also means lost revenue for our partners, who will have had many of their custom builds' keyword URLs hijacked by now.
Re: Given the triviality of the add-ons and the changes, I don't think that QAing them in a short timeframe would be problematic. 

--Agreed these are trivial changes, but at same time, we don't control our partner resources, neither can we hold them on a strict timeline as that will definitely strain the relationship.

Re: Pushing this back another 6 weeks, on the other hand, means that we have, even if not 200 million, at least hundreds of thousands of users affected by hijacked keyword URLs that lead to large amounts of SUMO complaints

--Can u please point me to the data that says this affects hundreds of thousands of users? 
Telemetry isn't a representative sample to make any derivable conclusions for general audience - it's opt-in thereby making it biased set. 

Re: It also means lost revenue for our partners, who will have had many of their custom builds' keyword URLs hijacked by now.

--None of our existing partners have complained so far saying they are having issues. I am not denying the existence of the issue, but we are addressing the wrong end of the stick by simply blocking it. Given partner heads-up and educating them is the correct way, it signifies that we care about our partners and our users.
Group: mozilla-corporation-confidential
(In reply to aphadke from comment #16)
> Re: Given the triviality of the add-ons and the changes, I don't think that
> QAing them in a short timeframe would be problematic. 
> 
> --Agreed these are trivial changes, but at same time, we don't control our
> partner resources, neither can we hold them on a strict timeline as that
> will definitely strain the relationship.

I'm talking about the two add-ons that we control and have the 
power to update. If there are other partner add-ons with these 
issues that we don't control, that might be another matter, but 
I'm still not convinced that we're responsible for making sure 
they're immune to this prompt.

> Re: Pushing this back another 6 weeks, on the other hand, means that we
> have, even if not 200 million, at least hundreds of thousands of users
> affected by hijacked keyword URLs that lead to large amounts of SUMO
> complaints
> 
> --Can u please point me to the data that says this affects hundreds of
> thousands of users? 
> Telemetry isn't a representative sample to make any derivable conclusions
> for general audience - it's opt-in thereby making it biased set. 

I'm using hundreds of thousands as an absolute low end ballpark 
guess here. We have data that show millions of extant installs 
of add-ons which we know change keyword.URL without prompting or 
reverting it. We don't know how many of these are still enabled, 
or how many past instances have been uninstalled. The Babylon 
Toolbar alone has almost 9 million.

The 200 million number may be extravagant, but it's clear that 
this affects a very large number of users.

> Re: It also means lost revenue for our partners, who will have had many of
> their custom builds' keyword URLs hijacked by now.
> 
> --None of our existing partners have complained so far saying they are
> having issues. I am not denying the existence of the issue, but we are
> addressing the wrong end of the stick by simply blocking it. Given partner
> heads-up and educating them is the correct way, it signifies that we care
> about our partners and our users.

My only argument here is that if this does our partners any harm 
(and I don't think it does), it does them just as much good.
(In reply to Kris Maglione [:kmag] from comment #17)
> I'm talking about the two add-ons that we control and have the 
> power to update.

How would we update them? They're not on AMO, as far as I know. We could put them there, I suppose, but I don't know what other implications that would have, and in any case wouldn't help existing users unless we had a more complicated fix (that also reset the user-set pref).

I think the costs of delaying this 6 weeks are being overblown, and the costs of confusing users and third-parties (partners or otherwise) are being underestimated.
Group: mozilla-corporation-confidential
(In reply to :Gavin Sharp (use gavin@gavinsharp.com for email) from comment #18)
> How would we update them? They're not on AMO, as far as I know. We could put
> them there, I suppose, but I don't know what other implications that would
> have, and in any case wouldn't help existing users unless we had a more
> complicated fix (that also reset the user-set pref).

They aren't on AMO, but we can put them there. I don't think there
are any negative implications to doing so, and I think this needs to
be done whether we push this back or not.

I'm not sure why you don't think this would help existing users. If
we change the value in the default branch to match the value that
was previously stored in the user branch, prefHasUserValue will
return false and the effect should be the same as if we'd done it
this way at the start.

> I think the costs of delaying this 6 weeks are being overblown, and the
> costs of confusing users and third-parties (partners or otherwise) are being
> underestimated.

I don't know how to quantify either of these costs. I have good
reason to think, based on the experience of the SUMO reps, that this
issue is causing a lot of trouble for users, who uninstall these
add-ons only to see traces of them in the painfully offensive and
ineffective search result pages that many of these add-ons set. 
How much damage this adds up to in 6 weeks for millions of 
effective users, I don't know.

As for pushing it back and it affecting some unknown number of 
partner add-ons, I'm more in the dark. If someone intentionally 
installed (say) an AOL add-on, which changed their default 
search engine, and after this update they got a message saying:

Firefox is using 'www.aol.com' for searches from the location bar. Would you like to restore the default search (Google)?

I suspect that users who are intentionally using AOL will 
generally click "No, continue using 'www.aol.com'" with no 
further ill will.

This is obviously a problem if we developed the add-on to change 
the settings under contractual obligation (and did it 
incorrectly). I don't think it's the same kind of problem if 
partners developed the add-on on their own.
Where is the data to show the number of users who did not intentionally change their default to one of our partners so we can effectively scope this?  Also, based on real data from a search partner, less than .1% of searches occur in the URL bar - I am unclear why we are prioritizing this change based on this volume.  We need to delay this, gather more data and be sure we are doing what is right by both our users and partners.

Lets gather real data and meet to discuss this further.
The problem is that when you're dealing with potentially 200M affected users, .1% of searches amounts to a lot. We've blocked several add-ons which we know to be almost strictly silently installed by third party installers, which change these settings and do not revert them on uninstall, and which we know to have millions of current installs each (to say nothing of previously uninstalled versions). These users are still affected by the setting changes, and we currently have no way to address the problem other than this change.

As for users who have intentionally changed the default search to search engines of our partners, this is not targeted at them. The message that they will see in now way encourages them to restore the default search engine, simply gives them the option to do so.

I can't say for certain how many users have intentionally installed partner add-ons which make these changes. I know that the two add-ons mentioned amount to ~40K users (who will not be affected by this if we update the add-ons in time). If the numbers are similar for other such (legitimate) add-ons, then it's maybe 100K users who will see a friendly offer to reset their search engine when they perform a location bar search.
I also don't understand why any of those partners or add-ons need to change keyword.URL at all, as with the default setting, the "keyword" searches this pref affects are going through the default search engine anyhow - so if those add-ons change the default search engine (browser.search.defaultenginename IIRC), that should be enough, and I'd expect that to be the right thing to do.
That said, it may potentially not be the right thing to do for the new reset functionality to compare with the default branch of prefs, as we probably even ask for a re-set if the user has changed the default search engine intentionally and did set keyword.URL in some way.
The most exact data we have is this, which is about 6 months old, but still fairly true (in fact it's most likely worse now):

* 50% of our users have a non- standard keyword.URL
* Total known malicious add-ons (most of these add-ons change keyword.url) in Top 100 addons: 25
* Total installs for top 100 add-ons: 410 million
* Total installs of known malicious add-ons for top 100: 106
  Million (26%)
* Conservatively, 20% of add-on users have at least one malicious add-on installed. A more realistic number is 50- 60%.


This 106 million only counts add-ons that are installed right now, and since the vast majority of these add-ons don't revert keyword.url when uninstalled (a policy violation that the Bing add-on is breaking as well) then we know there are more. We've also blocked add-ons that violate this policy, but still that doesn't revert the keyword.url setting. The 106 million is also very conservative, as we are assuming each user has about 4 malicious add-ons. They may have less. So we are at a minimum of 106 million users that have a hijacked keyword.url, most likely 150-200 million. So we aren't just relying on telemetry data, this is a real problem.

Each week roughly 20% of the threads and Me Too votes on SUMO are related to Add-ons that don't revert their settings when uninstalled. We also get a significant portion of our input feedback centered around this. 

I realize that partners might have to change their add-ons, but the reality is this is being overblown. The total number of users with partner extensions is maybe 100k. We have well over 100 million users at a bare minimum with this issue just from malicious add-ons. The partner add-ons (bing and MSN at least) are violating our policies and it is something significant enough we would have blocklisted them for it if they weren't partner add-ons. We aren't going to impact our partners in a significant way, if a user really did mean to install a partner extension, then they want that partner as their default and they will click "No, use Bing" in the dialog. I strongly feel that we should pursue contacting the partners that we have that will be impacted by this (this reaching out should have happened awhile ago when this change hit beta) and update the add-ons in our power before we consider reverting this feature.
Sorry, made a mistake in "The 106 million is also very conservative, as we are assuming each user has about 4 malicious add-ons." It should read:

 The 106 million is conservative as there are many thousands of malicious add-ons that are outside of the top 100

/me needs coffee
(In reply to Kris Maglione [:kmag] from comment #19)
> I'm not sure why you don't think this would help existing users. If
> we change the value in the default branch to match the value that
> was previously stored in the user branch, prefHasUserValue will
> return false and the effect should be the same as if we'd done it
> this way at the start.

prefHasUserValue doesn't work that way. The user-set pref hashtable isn't updated in real time when a value is set in the default branch, and prefHasUserValue only checks the existence of an entry in the hashtable.

That's not an insurmountable problem; we can (and should) absolutely look into fixing this the right way, but we don't have enough time to make sure we can do that before we need to ship the last beta of this cycle (goes to build on Monday).

As distasteful as it is to delay the work further, maintaining the status quo is the safest choice given our time constraints.

Let's take the discussion of numbers and affected users offline; it's all very speculative, and they shouldn't be affecting our decisions at the last minute.
(In reply to :Gavin Sharp (use gavin@gavinsharp.com for email) from comment #25)
> prefHasUserValue doesn't work that way. The user-set pref hashtable isn't
> updated in real time when a value is set in the default branch, and
> prefHasUserValue only checks the existence of an entry in the hashtable.

Well, it doesn't until after a restart, but it's an easy problem 
to solve.

> Let's take the discussion of numbers and affected users offline; it's all
> very speculative, and they shouldn't be affecting our decisions at the last
> minute.

Most of the numbers that we have aren't speculative, but it 
looks like it's too late to do anything about this now in any 
case.
We are ok with delaying this to 20 if we are confident we can come to a satisfactory solution between now and then. This fix is part of a much larger project that does impact millions of users and is costing us adu. We are not ok with letting this project fall off the radar as it has several times before. I am 100% in agreement that we should be very cautious and respectful of our partner relations. We also can't forget our brand promise of "Firefox answers to no one but you." Ultimately we are trying to protect the best interest of our users. We should continue these discussions in the Squeaky meetings.
No disagreement here! Thanks Matt.
That's pretty much my take on the matter. Every time we've gotten traction on doing anything about this issue, it's stalled for one reason or another. It's been over a year since this solution was first proposed, and even longer since we've been talking about other solutions.

It's particularly troubling because Chrome has a much better location bar search experience than we do, and users who use this feature and have had their keyword URLs hijacked by third parties who show more ads than search results and have no qualms with seizure-inducing animations are much more likely to be swayed by that.
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #22)
> I also don't understand why any of those partners or add-ons need to change
> keyword.URL at all, as with the default setting, the "keyword" searches this
> pref affects are going through the default search engine anyhow - so if
> those add-ons change the default search engine
> (browser.search.defaultenginename IIRC), that should be enough, and I'd
> expect that to be the right thing to do.

You may be right. My proposed patch changes both, but I'll file a new bug about fixing these add-ons and we can discuss it there.

> That said, it may potentially not be the right thing to do for the new reset
> functionality to compare with the default branch of prefs, as we probably
> even ask for a re-set if the user has changed the default search engine
> intentionally and did set keyword.URL in some way.

I think this is a corner case that's not worth worrying about.
Depends on: 839629
Attachment #711481 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Mozilla/5.0 (X11; Linux x86_64; rv:19.0) Gecko/20100101 Firefox/19.0

Verified as fixed in Firefox 19 beta 6 (buildID: 20130212082553). 
Installed the bing addon from comment 0 and searched for a word in url bar: The notification is gone.
What are the next steps and actions to get this into 20? User Advocacy has goals around this, plus in the interest of users we want to get this into 20 to reduce the delay (already being delayed for quite some time).
The patch that backed this out only landed in Beta, so unless something changes, this should already be on track to land in 20 release.
Great to hear it Kris! This will be a huge win for our user base.
After installing the add-on from the description, the notification keeps on appearing  with Firefox 20 beta 1 (buildID: 20130220104816) on an Ubuntu 12.04 32-bit machine.
I'm guessing that the add-on specified in the description won't be used anymore, right? It might be replaced with the one proposed in bug 839629, right?
(In reply to Manuela Muntean [:Manuela] [QA] from comment #36)
> After installing the add-on from the description, the notification keeps on
> appearing  with Firefox 20 beta 1 (buildID: 20130220104816) on an Ubuntu
> 12.04 32-bit machine.

I get the same behavior with the same beta build, using Windows 7 64-bit.
Yes, the add-on linked in comment 0 is expected to continue triggering the prompt. The updated versions won't.
In today's channel meeting we agreed this would be landed prior to our final beta (Tues March 26th) - please carry forward the approval-beta and land there in the coming week then update the firefox20 status flag accordingly.
Verified fixed with Firefox 20 beta 7 (buildID: 20130325214615), on: Windows 7 64-bit, Ubuntu 12.04 32-bit and Mac OSX 10.8.2. 

After the installation of the Bing add-on from comment 0 and the search for a word in the URL bar: the notification is gone.
QA Contact: manuela.muntean
So what's the plan here? We missed 19 and there seemed to be some urgency around figuring this out for 20, and then.... nothing. Are we any closer to a solution?
We're close, the remaining steps are listed out here: https://bugzilla.mozilla.org/show_bug.cgi?id=839629#c41 and are actively being worked on.  Afaik this is on track for going into FF21.
(In reply to lsblakk@mozilla.com from comment #44)
> We're close, the remaining steps are listed out here:
> https://bugzilla.mozilla.org/show_bug.cgi?id=839629#c41 and are actively
> being worked on.  Afaik this is on track for going into FF21.

Thanks for the pointer to the dependency bug. I should have looked there first.
Comment on attachment 711481 [details] [diff] [review]
disable the prompting

Re-requesting beta and aurora approval given discussion about our plans for this feature. The current plan is to just disable this across the board and instead focus on shipping bug 738818.
Attachment #711481 - Flags: approval-mozilla-beta?
Attachment #711481 - Flags: approval-mozilla-beta+
Attachment #711481 - Flags: approval-mozilla-aurora?
Comment on attachment 711481 [details] [diff] [review]
disable the prompting

Looks good to land per comment# 46
Attachment #711481 - Flags: approval-mozilla-beta?
Attachment #711481 - Flags: approval-mozilla-beta+
Attachment #711481 - Flags: approval-mozilla-aurora?
Attachment #711481 - Flags: approval-mozilla-aurora+
Keywords: verifyme
No longer depends on: 839629
The QA Verification wanted here is to just confirm that this feature has been disabled on Firefox 21 and Firefox 22, as it was in Firefox 20. Confirmation that the prompt doesn't appear after manually changing keyword.URL and then performing a location bar search should be sufficient.
The notification is still appearing with Firefox 21 beta 3 (build ID: 20130416200523), after changing keyword.URL and then performing a location bar search. 

I've changed the pref manually, and I also installed DuckDuckGo Plus add-on (https://addons.mozilla.org/en-US/firefox/addon/duckduckgo-for-firefox/) that changes the value of this pref.

I did the verification on: Windows 8 32bit, Ubuntu 12.10 32bit and Mac OSX 10.8.3.
(In reply to Manuela Muntean [:Manuela] [QA] from comment #50)
> The notification is still appearing with Firefox 21 beta 3 (build ID:
> 20130416200523), after changing keyword.URL and then performing a location
> bar search. 
Hey Manuela,

The patch just landed on beta yesterday in preparation for Fx 21 Beta 4. So this needs to be verified disabled once we have beta 4 builds , next week :) You will not see the expected behavior in comment # 49 as of current beta .
> 
> I've changed the pref manually, and I also installed DuckDuckGo Plus add-on
> (https://addons.mozilla.org/en-US/firefox/addon/duckduckgo-for-firefox/)
> that changes the value of this pref.
> 
> I did the verification on: Windows 8 32bit, Ubuntu 12.10 32bit and Mac OSX
> 10.8.3.
Verified fixed with Firefox 21 beta 4 (build ID: 20130423212553) on Ubuntu 12.10 32bit and Mac 10.8.3.

After changing the pref manually, the notification doesn't appear anymore.
Verified fixed with Firefox 22 beta 2 (build ID: 20130514181517), on Mac OSX 10.8.3, Ubuntu 12.10 32-bit and Windows 7 64-bit.
mass remove verifyme requests greater than 4 months old
Keywords: verifyme
Hey guys! I just got my search enginge preferences hijacked. No warning, no nothing. Great feature. Really. Seriously, what'se going through someone's brain while deciding to implement such feature? Why even have a feature that allows the search enginge to be changed programatically by third parties in the first place? It makes no sense what so ever. Remove it and make the world a better place. Keep it and gain absolutely nothing but recentment from your users. The choice should be a no brainer. This is the last web page that my Firefox is ever going to display.
We don't have a "feature that allows the search enginge to be changed". We have settings which allow the user to change the search engine, which a lot of third-party software happens to abuse, often through the use of privileged installers and system services which we have little control over. We invest considerable resources in stopping and reverting these changes, and are working to lock down the settings to make hijacking more difficult, but it's not nearly as simple as removing a feature. Please do not immediately assume malice on our part rather than on the part of whatever malware you installed which made the changes.
You need to log in before you can comment on or make changes to this bug.