Closed Bug 633773 Opened 13 years ago Closed 12 years ago

Use Google's HTTPS search by default

Categories

(Firefox :: Search, enhancement)

enhancement
Not set
normal

Tracking

()

RESOLVED FIXED
Firefox 14

People

(Reporter: chris, Assigned: geekboy)

References

Details

(Keywords: perf, privacy, Whiteboard: [sg:want P5][spdy])

Attachments

(1 file, 1 obsolete file)

User-Agent:       Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b11) Gecko/20100101 Firefox/4.0b11
Build Identifier: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b11) Gecko/20100101 Firefox/4.0b11

Firefox directs users to Google searches in 3 ways:

1. The search bar on the right hand side of the navigation bar, for which Google is the default search engine.
2. The about:home page that is built into Firefox, which directs searches to Google.
3. The "I'm feeling lucky" search that occurs if a user enters words into the location bar.

While it would be awesome if Google would use HTTPS by default, the company currently does not do so. However, they are nice enough to offer an encrypted search page (but most users don't know it exists).

Mozilla should direct these 3 search paths to Google's HTTPS search engine (encrypted.google.com).



Reproducible: Always
Version: unspecified → Trunk
Something to look at, but definitely something we'd need to coordinate with Google if we wanted to do so. We'd want to make sure their https interface could take the traffic, and ensure they were aware and ok with it.
If Mozilla is unwilling to use encrypted.google.com by default, could we get it as an additional Search Engine?  That is, keep "Google" as the default option, but also offer "Google SSL" or "Secure Google" as an alternative SE?
(In reply to comment #2)
> If Mozilla is unwilling to use encrypted.google.com by default, could we get it
> as an additional Search Engine?  That is, keep "Google" as the default option,
> but also offer "Google SSL" or "Secure Google" as an alternative SE?

Just to make sure we're clear, it's not something we're unwilling to do, but we'd have to make sure Google is ok with it, and has sufficient capacity to handle the traffic. Firefox sends a considerable amount of traffic Google's way, and switching the user base over to https to a service we don't manage without coordinating and getting approval on first is not something I'd be comfortable doing. 

I'm happy to work with them to see if we can do it, but it's something we'd look at for a future release of Firefox once 4.0 ships.
We would welcome Firefox giving their users the option to use encrypted search. However, at this time we don't feel that our encrypted search offers the features and speed that our users expect and so we wouldn't want it to be the default. We are working towards making encrypted search as fast and complete as unencrypted search, but we're not there yet.
In a December 2009 article with Ars Technica (http://arstechnica.com/microsoft/news/2009/12/mozilla-exec-bing-is-not-popular-enough-for-firefox.ars), Asa Dotzler stated that:

"Mozilla and only Mozilla decides which search services to include in Firefox."

As I gather it (and please let me know if I am wrong), Google does not pay to be the default search engine. It pays to be included in the list of included search engines, and then Mozilla itself picks the default. It is for this reason, AFAIK, that Yahoo remains the default search engine for Firefox users in Japan.

If this is indeed the case, then it shouldn't particularly matter that Adam's colleagues at Google feel that the unencrypted Google site provides the best search experience.

What should matter, is if Mozilla feels that protecting its users data by default is indeed something that is important.

Obviously, if Google tells you that 400 million firefox users visiting the HTTPS search page will crush its servers, then that is a factor you will need to consider, but looking through Adam's message, I don't see him making that claim. Instead, I see him commenting on the fact that the non-SSL search page likely has lower latency, and offers more features than the encrypted search services.

At the very least, Mozilla should make the SSL encrypted search an option in FF 4.0 (and likely, enabled by default in private browsing mode). However, I see a strong case to enable it by default for all users.
Whoops. Minor slip there.

I meant to say:

At the very least, Mozilla should make the SSL encrypted search an option _post_ FF 4.0 (and likely, enabled by default in private browsing mode). However, I see a
strong case to enable it by default for all users.
I'm joining the choir.

It's a necessity nowadays - no less.
For Mozilla and users a like.
For users from the obvious reasons.
For Mozilla - in order to be there.
Up-front in the avant-guarde of user privacy and security.

Users are simply ignorant as to the whole secured search issue.
Mozilla as a leader in user's privacy should take the stand!
Mozilla prides itself in many privacy "firsts". Let's add one more to the list.

I would even go as far as stating a deadline (say 1 year from now), after which users will be discouraged from using non-SSL searches.
Something like having the search-box background coloured [say] red when searching in non-SSL mode, with an X over the provider's icon.
And only allow SSL'ed searches over the "awesom bar" - be it Google, or the next guy!

You know what will happen right?!
This won't go through b/c of too much timidty from uncle Goog.
And then?!
Then Google itself will make Chrome's default search to be SSL'ed.
And taking another privacy first.
It's obvious that's where things are going.
Just like they were the first to turn SSL on by default on their mail service.
Google will not sit idle!
Where gmail went, would where google search goes!
And Mozilla can still be there first!
Based on Google's own reports on this matter, SSL / TLS is not a big computational expense for them, see: http://www.imperialviolet.org/2010/06/25/overclocking-ssl.html

In particular:

"If there's one point that we want to communicate to the world, it's that SSL/TLS is not computationally expensive any more. Ten years ago it might have been true, but it's just not the case any more. You too can afford to enable HTTPS for your users.

In January this year (2010), Gmail switched to using HTTPS for everything by default. Previously it had been introduced as an option, but now all of our users use HTTPS to secure their email between their browsers and Google, all the time. In order to do this we had to deploy no additional machines and no special hardware. On our production frontend machines, SSL/TLS accounts for less than 1% of the CPU load, less than 10KB of memory per connection and less than 2% of network overhead. Many people believe that SSL takes a lot of CPU time and we hope the above numbers (public for the first time) will help to dispel that.

If you stop reading now you only need to remember one thing: SSL/TLS is not computationally expensive any more."

This would suggest to me that it would be acceptable to add SSL / TLS by default.
Severity: enhancement → normal
Status: UNCONFIRMED → NEW
Ever confirmed: true
Today, Google announced that they will be sending signed in Google search users to the HTTPS version of the service by default.
See: http://googleblog.blogspot.com/2011/10/making-search-more-secure.html

Back in March of this year, Google's Adam Langley wrote in this bug that:

"[A]t this time we don't feel that our encrypted search offers the features and speed that our users expect and so we wouldn't want it to be the default."

Presumably, Google now feels that it can deliver the features and speed that its most valuable (signed-in) users expect.

Can we now get the ball rolling on HTTPS by default for Firefox users, or should we wait for Chrome to do it first?
Keywords: privacy
Severity: normal → enhancement
(In reply to Christopher Soghoian from comment #9)
> Presumably, Google now feels that it can deliver the features and speed that
> its most valuable (signed-in) users expect.

That's a big presumption. We can't just make changes to our searches willy-nilly without coordinating with Google. We're an important part of their search traffic, and if we don't work together to make these changes the result is a poor experience for our shared users.

We should get the ball rolling, certainly.
Does Google provide a search suggestions endpoint via HTTPS?  suggestqueries.google.com doesn't work over https, and making search HTTPS by default without secure search suggestions is... well... silly.
(In reply to Masatoshi Kimura [:emk] from comment #12)
> https://clients1.google.com worked which is used by
> <https://www.google.com/>.

I monitored network connections through an autocomplete action and only saw connections to www.google.com.  Because the value will be hard-coded, this is probably a URL we should get from Google so we can be sure it scales, not temporary, and is robust.
We're happy to be offering SSL search for our signed-in users on https://www.google.com, and we've received a lot of positive feedback. We want to make it available on other Google domains as well, but we're still working on that.
Adam can correct me if I've misinterpreted his statement, but reading between the lines, Google's position seems to be:

1. Google will roll out HTTPS by default to signed in users first.
2. Google does not currently want Mozilla to shift its users to Google's encrypted search service by default.
3. If and when Google wants Mozilla to make that switch, Mozilla will be told by Google.

Five years ago, when fighting a much publicized request from the Department of Justice for its customers search queries, Google argued that:

"[S]earch query content can disclose identities and personally identifiable information such as user‐initiated searches for their own social security or credit card numbers, or their mistakenly pasted but revealing text."

If search query information was sensitive enough for Google to fight DOJ's request, and most recently, for Google to tart scrubbing referring headers so that third party sites no longer receive search terms for signed in Google users, then perhaps it should be sensitive enough for Mozilla to protect by default too.

If switching to Google's HTTPS search service by default over the objections of Adam and his colleagues is something that Mozilla doesn't want to do, then perhaps it is time for Mozilla to switch to a different search engine that uses HTTPS by default. I'm sure duckduckgo would appreciate the traffic.
The Google/Mozilla search deal has been signed for another 3 years (reportedly for 300 million per year [1]). From my perspective, it seems pretty screwed up that Mozilla is deriving a massive percentage of its revenue by getting its users to use, by default, a service that is insecure, and leaks user search queries to: 1. anyone watching the network, and 2. other websites via the referrer header.

Currently, Google has gone on record [see comment 14] that they don't want Mozilla to switch Firefox users to HTTPS search right now, even though they have already made the switch for all signed in Google users, irrespective of the browser they are using.

The decision to switch to Google's encrypted search service seems like it is largely in the hands of Mozilla's legal/business dev/executive teams, rather than a technical issue. As such, it is curious that Mozilla's legal/policy folks have been largely silent on this matter.

Mozilla's commitment to privacy is now very much part of its brand. Yet its public statements about privacy [2] are totally inconsistent with the highly profitable current practice of shipping unsafe default search settings.

Continued silence on this matter from those responsible at Mozilla is no longer reasonable.

If Mozilla's upper management decides that Google's preferences (and the big pile of cash that is associated with respecting those preferences) are more important, please at least publicly acknowledge that user privacy comes second.

[1] http://allthingsd.com/20111222/google-will-pay-mozilla-almost-300m-per-year-in-search-deal-besting-microsoft-and-yahoo/
[2] https://firstpersoncookie.wordpress.com/2011/01/12/mozillas-draft-privacy-data-operating-principles/
I think we should aim to make this change in the first release that enables SPDY by default (bug 698229).

What percentage of Google searches (from Firefox users) are from users who are logged into Google?  Currently, those users are getting an extra connection and redirect on every search.
Keywords: perf
Whiteboard: [sg:want P5]
(In reply to Jesse Ruderman from comment #19)
> I think we should aim to make this change in the first release that enables
> SPDY by default (bug 698229).

I was thinking a similar thing - google might feel differently about the default if it is over spdy, where the subresources aren't going to require new (independent) ssl sessions.. I would guess it would be ~6x less ssl terminations for google than http/1 over https.

698229 is misnamed, but that's a different issue.

> What percentage of Google searches (from Firefox users) are from users who
> are logged into Google?  Currently, those users are getting an extra
> connection and redirect on every search.

And their search is being sent in plaintext anyhow (before being redirected). that's really painful.
Whiteboard: [sg:want P5] → [sg:want P5][spdy]
Attached patch Work in Progress (obsolete) — Splinter Review
This is a change that rewrites the HTTP search URLs in Firefox, Firefox Mobile and B2G to use HTTPS. This patch won't be the final product, as we still want to explore adapting it to use the open search stuff (for fewer locale-based or other redirects at the time of search), but this is a start and I'd like feedback about whether or not I missed any bits of code that need to be included. 

The search suggestions are served by clients1.google.com in this patch because suggestqueries.google.com doesn't yet provide the service over SSL.  I'm told this should be remedied soon, but in the meantime, I imagine the behavior is the same.

Also, I am working with some folks at Google to obtain the appropriate URLs and timeline for rolling this out in a way that will be the best quality of service for this privacy enhancement, but here's a first whack.  Did I miss any bits?
Assignee: nobody → sstamm
Status: NEW → ASSIGNED
We'll also need to coordinate fixing up a few l10n plugins along, http://mxr.mozilla.org/l10n-mozilla-aurora/find?text=&string=google[\-_]&regexp=1.

browser.search.siteSearchURL is unused, AFAICT, see bug 482229.
(In reply to Sid Stamm [:geekboy] from comment #21)

> This is a change that rewrites the HTTP search URLs in Firefox, Firefox
> Mobile and B2G to use HTTPS. 

Sid, awesome!

if I actually implemented 723628 would you consider using that to make sure a (ssl) handshake was already active when the search terms were known?
> The search suggestions are served by clients1.google.com in this patch because 
> suggestqueries.google.com doesn't yet provide the service over SSL.  I'm told this 
> should be remedied soon, but in the meantime, I imagine the behavior is the same.

Could we use https://www.google.com/ for search suggestions?  It seems to work, and it would cut down on latency when we actually do the search (cf bug 723628).
Blocks: 567518
(In reply to Patrick McManus [:mcmanus] from comment #23)
> if I actually implemented 723628 would you consider using that to make sure
> a (ssl) handshake was already active when the search terms were known?

Perhaps... sounds like that might be a useful feature (bug 723628) for more than just search too.  We may not want to start the handshake on search box focus, but maybe after a first character is typed (if search suggestions are enabled) or a similar event before the query is selected.
Google's search team is ok with Firefox using https://www.google.com for search suggestions, so please use this endpoint.  Thanks!
(In reply to Mike Graboski from comment #27)
> Google's search team is ok with Firefox using https://www.google.com for
> search suggestions, so please use this endpoint.  Thanks!

awesomeness!
(In reply to Patrick McManus [:mcmanus] from comment #28
> awesomeness!

Just to be clear here, I don't think Mike is implying "Go, turn it on for all and use that endpoint", he is just providing a "yes" to Jesse's comment 24.  I still like your enthusiasm.  :)

And thanks, Mike.  I'll update the patch.
OS: Windows 7 → All
Hardware: x86 → All
http://insidesearch.blogspot.com/2012/03/bringing-more-secure-search-around.html

We should get this finalized and approved quickly!

Sid, you still actively working on this?
Dolske: yep, still working on it.  

Axel: Do you want to tweak those l10n plugins from the mxr query you put in comment 22 (ku, zh-TW, ja-JP), or should I just roll it into my patch?
(In reply to Sid Stamm [:geekboy] from comment #31)
> Axel: Do you want to tweak those l10n plugins from the mxr query you put in
> comment 22 (ku, zh-TW, ja-JP), or should I just roll it into my patch?

Nevermind, I'm not sure how to do that appropriately.  Can you help out with this, Axel?
The best way to track the three other plugins is to file bugs on the individual components in the "Mozilla Localizations" product, and CC me.

It'd be helpful to know if this change also applies to thunderbird and seamonkey, we'll need to be rather explicit in the bugs to clarify the scope.
Attached patch proposed patchSplinter Review
Hey Gavin, here's a first whack at a patch.  Can you take a look?
Attachment #594856 - Attachment is obsolete: true
Attachment #605197 - Flags: review?(gavin.sharp)
Comment on attachment 605197 [details] [diff] [review]
proposed patch

At least the suggestion parts of this conflict with bug 557890, which requested that a different URL be used for the suggestion JSON. Seems like we need to sort that out.

The rest of this patch looks good to me, but I haven't looked at it in-depth. Can do that once I'm back on Wednesday.

Generally sign-off for these kinds of changes go through Kev, but I know you've been talking to Google folks - I assume you've coordinated with him already.
Attachment #605197 - Flags: review?(gavin.sharp) → feedback+
Thanks for the rapid feedback, Gavin.  Please take a deeper look when you can and meanwhile I'll run it through try.  

Comment 27 suggests using the same domain for both endpoints is acceptable, but I'll confirm.  This may mean we don't need to use clients1.google.com for bug 557890, but I'll ask someone from Google to confirm that in the bug.

Yes, Kev and I have been in constant communication about this and we discussed it earlier today; he's on board with the change.

Axel: I think we want this to apply to Thunderbird and Seamonkey too.  I'll file follow-up bugs for the other components as soon as the endpoints are confirmed and this patch is r+ed and ready to land.
Looks fine on try, though I don't think our automated tests do much searching.  

Link to test builds, if you wanna play with it yourself: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/sstamm@mozilla.com-4750ed3d79fb/
(In reply to Sid Stamm [:geekboy] from comment #36)

I have updated bug 557890 indicating that it is ok to use https://www.google.com instead of clients1.google.com as a top-level domain for search suggestions.
Attachment #605197 - Flags: review?(gavin.sharp)
Attachment #605197 - Flags: review?(gavin.sharp) → review+
Keywords: checkin-needed
Depends on: 736260
Depends on: 736261
Depends on: 736262
Hey, Axel: I filed bug 736262, bug 736261, and bug 736260 to address the localized plugins you pointed out in comment 22.  Please let me know if I missed anything...
https://hg.mozilla.org/mozilla-central/rev/36fd3090b006
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
A nice side-effect is that we're now using SPDY for all searches ...
"Christopher Capozziello for the Open Society Foundations" on his homepage yet he advocates closing the internet and making it difficult for webmasters like myself to understand what people are looking for.

The fact that ANY ONE entertained this idea is ludicrous! Mozilla has advocated for a free and open internet which has only prospered because of it's open nature.

Now Mozilla is actively all too happy to be working to close the internet.

This has NOTHING in any justifiable way to do with privacy! Webmasters like myself rely on referral headers to improve our content. Removing that does NOT protect people's privacy. The only effect it has is to make people more dependent on Google's tools.

This is a introduced bug, NOT an enhancement.

Anyone and everyone advocating for this willingly has completely lost face and should be banned from further damaging Firefox and the internet as a whole. REAL people do not want to participate in Google's war to invade people's privacy and the fact ANY ONE here would just privacy to justify this shows just how low corporations are willing to go to control the internet.
Using security and encrypted connections isn't equal to using a closed internet...
Encrypting search pages and denying webmasters the ability to see what people used to find our sites is EXACTLY the kind of thing that closes the internet.

This makes webmasters dependent on using Google's tools and closes the web off from people who do not have access to Google's tools.

Since Google emphasizes writing good content along with it's very dominate if not outright monopoly on the search market that effectively allows it to dictate through it's services what people should or should not generate content wise.

That effectively controls and manipulates the internet and thereby reduces the openness of the web.

This coming from the same company that "customizes" search results based on data such geographic information which puts it in the position to manipulate opinion of events.

Everyone participating here explicitly knows that non-technical people aren't going to know what impact they will have on webmasters when Firefox is updated. Ignorance of a topic does NOT equate to agreement or willingness.

The fact that I see absolutely no opposition to this skimming through the comments creates some grave concerns. The few things that have been brought up were highly manipulated in order to create a sense of objectivity further showing the agenda and bias in forcing this unwanted "feature" on to Firefox users and webmasters alike. Firefox only gained prominence because of webmasters such as myself yet the people I see here are browser developers and individuals. This means the process in which important decisions are made about the browser are NOT being reviewed by those who have initially supported Mozilla and made Firefox success.

For those absent of any agenda I urge you to strongly reconsider this issue. Slippery-slope tactics is EXACTLY how any open system slowly comes under the control of those who wish to control it.
Take this to a newsgroup, please. Bugs are not a place for long debates that aren't actually focused on the technical issues of the bug.
John A. Bilicki III, Virtual_ManPL: as abillings said, this is a fine discussion you're having but please take it over to mozilla.dev.platform.  That is a better forum for this discussion and you will get more engagement there than on a closed bug.

http://www.mozilla.org/about/forums/#dev-platform
Depends on: 738871
@Sid Stamm, John Bilicki III, and Virtual_ManPL - additional comments posted here:  http://groups.google.com/group/mozilla.dev.platform/browse_thread/thread/23bb13fb75ae8bb1#
Sid, when going through the l10n bugs, on which repos should we land those? Most locales work on aurora only, and I see folks landing on beta even. Meaning fx 13 or even 12.
If possible, we should land on the same repo as the trunk patch, which is nightly right now.  It would be best for everyone involved if we coordinated it so all the HTTPS changes were in the same train and release.
https is blocked in China.

This means that search is now broken for all users in mainland china (tourists, expats, citizens, who prefers other languages than Chinese) of locales updated to adhere to this new standard (I am assuming the Chinese locale won't be updating to this!).

Before redirecting users to https an IP-GEO check should be performed and users should be redirected appropriately. E.g, users from Mainland China (despite their locale) should be redirected to the insecure version of Google search.

I am assuming this bug is also responsible for users being redirected from http://google.com to the non-functioning (within mainland China) https://google.com, when manually typing in the address bar.
Normally Google redirect requests from China to http://google.com.hk but not anymore for Firefox users.

I can use Google fine from Chrome and Safari from Mainland China, but Google is essentially broken in Firefox after this feature got implemented.

I think it is important to ensure that Firefox works not just in the country you installed it in, but also when you bring you laptop with you on travels. I also think it is important that users can choose to use Firefox in whichever language they desire. Therefore we must ensure that all locales works all over the world, and sadly the Chinese government isn't helping.
(In reply to Mads from comment #51)
This landed on Central (i.e. Nightlys) 4 Months ago and rode the Train through Aurora and Beta.
Wouldn't there have been Bugs filed regarding that Issue (what I can't confirm nor deny) in that Timeframe?
(In reply to Mads from comment #51)
File a Localization bug https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Localizations each localization is its own build and can have custom search engines (zh-CH can have different search engines from en-US).
Please don't. If our product has a problem for people that happen to be in china, we need a cross-localization solution.
(In reply to Kevin from comment #53)

The Chinese localization does not use Google as default, so it is a non-issue for people using the Chinese localization. However, people who live in China and do not read Chinese (tourists and expats, mostly) as well as Chinese who prefer English or some other language will have an issue.

(In reply to XtC4UaLL from comment #52)

Personally I had been wondering why the awesome bar stopped working on the beta and thought it was The Great Firewall or just Google screwing up. Today I read Firefox 14 release announcement and realized this is a Firefox issue.
This might be why no-one noticed. Maybe Mozilla China did notice, but didn't consider the implications for tourists and other users of different locales.

-----

I have tested with the newest zh-CN release from firefox.com.cn and found no issues (e.g. no redirection to https. It also uses baidu.com instead of Google anyway...) and the newsest en-US release from firefox.com with issues using the awesome bar and the search bar (both instances redirected to the blocked https://google.com) in both cases using a new profile.

I have also found that new profiles don't redirect manual requests for google.com to https and it must be a setting in my default profile. I apologize.

I am currently in Shanghai, but will leave tonight and cannot re-test.
I filed bug 774836 about the issue of Google Searches not working in China.
This is to protect people's privacy right?

So with Google having 85-90% of the WORLD's TOTAL search queries no one here in their right mind would EVER imagine that the corrupt regime that took over America and it's information gathering agencies such as the CIA would EVER consider infiltrating Google? As a for-profit corporation no one here thinks Google keeps records of everything that happens on their servers for years?

http://gs.statcounter.com/#search_engine-ww-monthly-201106-201206

This bug BLATANTLY and undeniably aids Google. Google in turn says to webmasters to generate more content. If Google withholds search terms from webmasters that effectively censors people's searches and allows Google and any information gathering agencies to completely dictate what webmasters DO see for incoming search terms completely allowing them to heavily dictate what they desire to be generated content wise.

THIS IS the EXPLICIT place to warn developers working on Firefox what they are participating in! Telling me to take privacy concerns away from the eyes of developers unwittingly censoring the web only shows blatant and unretractable admission of intentional participation in this agenda.

Why would I take this discussion to Google when you'd AIDING Google? I'm not stupid and the discussions there have proven utterly pointless since Mozilla has proven it will do as it damn well pleases in the face of overwhelming opposition.
A couple people complaining on Bugzilla is not overwhelming opposition.  You were already told to take this conversation elsewhere.
(In reply to John A. Bilicki III from comment #57)

> Why would I take this discussion to Google when you'd AIDING Google? I'm not
> stupid and the discussions there have proven utterly pointless since Mozilla
> has proven it will do as it damn well pleases in the face of overwhelming
> opposition.

You weren't told to take this to Google. You were told to take it to the mozilla.dev.platform newsgroup. You can access that via Google Groups or, if the tinfoil hat gets in the way, you can go to the news server directly at news.mozilla.org using Thunderbird or another NNTP client per http://www.mozilla.org/about/forums/. THAT is where discussion on this issue takes place, not in an ever longer string of comments in a bug. It is a very public, open place to discuss things.
Blocks: 1039004
Blocks: 1039003
Flags: needinfo?(chris)
You need to log in before you can comment on or make changes to this bug.