380785 - Add Wikipedia to the default search engine list

Status: VERIFIED FIXED

(Firefox :: Search, defect, P2)

Description

[Basil Hashem [:baz]]

Comment 1

[Rafael Ebron (:rebron)]

What's the rationale?  

For historical purposes, we didn't put Wikipedia in at first because they didn't have a search engine they were leveraging Yahoo and Google.

We added Answers.com in place of Dictionary.com and it also provided Thesaurus, Wikipedia, and other resources for answers.  I don't know what the usage is/was for Answers.com but still a pretty good service deal or no deal.

I suspect "Answers" doesn't scream dictionary/thesauraus/wikipedia so usage might have been low?  Don't know.  

Also note that "dict" was a keyword search for Answers.com in Firefox 1.5, not sure about 2.0.

Comment 2

[Basil Hashem [:baz]]

Rafael,

Thanks for the background.

The rationale is:
- Wikipedia has grown to become a relatively accurate, pretty valuable, mostly impartial, localized resource
- Wikipedia is a non-commercial entity that aligns well with Mozilla's approach
- Wikipedia continues to grow and expand in multiple languages/locales
- Wikipedia's knowledge base is larger (and growing) compared to answers.com
- Including both answers.com seems redundant

I think we should try to get some stats on answers.com searches if that's possible and how often the wikipedia search plugin has been downloaded off AMO.

Comment 3

[Basil Hashem [:baz]]

AMO data shows that the Wikipedia search engine is the most downloaded search plugin from that site.

The following Firefox locales include answers.com:
af, ar, be, da, el, en-GB, en-US, en-ZA, es-AR, eu, ga-IE, hu, ka,  
ku, lt, mk, mn, nr, nso, sq, ss, st, sv-SE, tn, tr, ts, ve, xh, zu

The following Firefox locales already include Wikipedia:
zh-TW, sl, sk, ru, ro, pt-PT, pt-BR, pl, nn-NO, nl, nb-NO, lt, it, fy-NL, fr, fi, eu, es-ES, de, ca, ar

1) We need to obtain permission from Wikipedia to use their search plugin and send traffic to their site (Mic Berman is reaching out)
2) Generate a list of locales that Wikipedia supports and map them to Firefox locales
3) For each locale, we should endeavor to use a search plugin that includes Search Suggest capability.

Comment 4

[Basil Hashem [:baz]]

Nom'ing as blocking Firefox 3

Comment 5

[Basil Hashem [:baz]]

Sorry, I meant wanted-firefox3, not blocking.

Comment 6

[Jesse Ruderman]

I use Wikipedia more often than i use a dictionary, but I still need a dictionary sometimes...

Comment 7

[Basil Hashem [:baz]]

Firefox ships with "dict" keyword in en-US by default. If you type "dict pernicious" in the URL bar, you can quickly get definitions. Also, Google (assuming that that is your default engine includes a "define" keyword for quickly getting definitions.

Comment 8

[Axel Hecht [:Pike]]

We don't ship with any keyword search URLs in our bookmarks.html anymore for a bit now, at least since fx2.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

I personally use the Answers.com and Wikipedia search engines side by side. Yes, Answers.com does include Wikipedia in their results, but I think it's a bad idea removing the dictionary/thesaurus abilities from the search bar.

I also think that having a search engine for a dictionary/thesaurus is infinitely more discoverable than a dict keyword. I'm willing to bet that the vast majority of the Firefox audience doesn't even know what keywords are, let alone that there may be some default ones included out of the box.

Comment 10

[Pavel Franc - Mozilla.cz]

Czech (cs) Firefox doesn't ship neither with answers.com nor Wikipedia. However, I like the idea of replacing one current Czech search plugin with Wikipedia - mainly  due to the fact 1 and 3 from comment #2.

Comment 11

[Ahmet Serkan Tıratacı]

I think that if you are good at googling, you already get a link to Wikipedia on the first page of Google search result. Also, adding Wikipedia to and removing Answers.com from search bar wouldn't be a better experience for users while Answers.com already include Wikipedia content.

Comment 12

[Uri Bernstein]

(In reply to comment #2)
> The rationale is:
> - Wikipedia has grown to become a relatively accurate, pretty valuable, mostly
> impartial, localized resource
> - Wikipedia is a non-commercial entity that aligns well with Mozilla's approach
> - Wikipedia continues to grow and expand in multiple languages/locales
> - Wikipedia's knowledge base is larger (and growing) compared to answers.com
> - Including both answers.com seems redundant

I agree with all of these points, save for the last one.
While including Wikipedia sounds like a great idea (especially for locales that have their own Wikipedias), I don't see why it has to come at the price of dropping Answers.com. Answers.com has, on top of Wikipedia, dozens, perhaps hundreds, of other sources - most notably general-purpose, as well as domain-specific, dictionaries.

Just one example:
Looking up "interest" on Answers.com (http://www.answers.com/interest) gives you, among others:
- A definition from the American Heritage Dictionary.
- Several definitions from Barron's financial dictionaries (which, I believe are not available elsewhere on the web).
- A legal definition from the Thomson Gale's Law Encyclopedia.
- Entries from the Columbia Encyclopedia and the Britannica Concise Encyclopedia.
- Translations to over a dozen languages.
- And, of course, the Wikipedia entry.

What this means for the user is more information, and more choice, which is something I think we are trying to promote.

[Full disclosure: I'm a former employee of Answers Corp., and as such I had a part in creating Answers.com. Although I no longer work at that company, I still enjoy their product as a user].

Comment 13

[Dwayne Bailey]

(In reply to comment #3)

> The following Firefox locales include answers.com:
> af, ar, be, da, el, en-GB, en-US, en-ZA, es-AR, eu, ga-IE, hu, ka,  
> ku, lt, mk, mn, nr, nso, sq, ss, st, sv-SE, tn, tr, ts, ve, xh, zu

These locales: af, en-ZA, nr, nso, ss, st, tn, ts, ve, xh, zu
are not attached to answers.com, we put those in to match en-US while we focus on translation of the UI.  Users in South Africa would most probably prefer Wikipedia.

Comment 14

[Mark Tyndall]

(In reply to comment #3, and echoing comment #13)

> The following Firefox locales include answers.com:
> af, ar, be, da, el, en-GB, en-US, en-ZA, es-AR, eu, ga-IE, hu, ka,  
> ku, lt, mk, mn, nr, nso, sq, ss, st, sv-SE, tn, tr, ts, ve, xh, zu

en-GB includes answers.com because Rafael requested it (bug 315385).  I'd be exceedingly surprised if non-trivial numbers of UK users were using Answers.com.

Comment 15

[Rafael Ebron (:rebron)]

> I agree with all of these points, save for the last one.
> While including Wikipedia sounds like a great idea (especially for locales that
> have their own Wikipedias), I don't see why it has to come at the price of
> dropping Answers.com. Answers.com has, on top of Wikipedia, dozens, perhaps
> hundreds, of other sources - most notably general-purpose, as well as
> domain-specific, dictionaries.

I agree with Uri here and I think you should rename this bug to just adding Wikipedia to the search engine list.  Removing Answers.com or another search engine from the default list should be a separate bug if necessary.

I don't think you have enough information to arbitrarily drop Answers.com from the list like this, and they do provide much more than just "user-generated" encyclopedia info.  

I recommend that we keep Answers.com until you have real data that says otherwise or they do something "evil".  Have they done anything bad to piss off our users that we don't know about?  Barring that, Answers.com should stay.  

Also, it should be noted that they've been good citizens with their promotion of Firefox and development of Firefox extensions.  It's not like they're Yahoo! or anything (oops, did I say that outloud?).

Comment 16

[Basil Hashem [:baz]]

This bug has been a bit stale and we need to make a decision. Here's my recommendation.

0. I've retitled the bug to say "Add Wikipedia to default search engine list" - removed reference to answers.com.
1. Where Wikipedia does not appear in a Firefox locale, add it (assuming there's good localized content and the localizers agree that it's worthwhile).
2. If Wikipedia is there, ensure that we use the "Search Suggest" version of the search plugin, if it's supported by Wikipedia for that language.
3. It should be a localizer decision as to where in the list Wikipedia appears. Localizers should be aware of various contractual constraints with regard to Yahoo and Google.

We won't remove the answer.com plugin for now and will try to work with the company to get data about degree of Firefox usage (per locale).

Any questions?

Comment 17

[Jesse Ruderman]

Does Wikipedia have a search mode where if an article (or redirect) exists, it goes there, but if it doesn't, it searches instead?  I don't find Wikipedia's search very useful, so I usually either (1) guess at the article name (en.wikipedia.org/wiki/foo) or (2) search Google for 'wikipedia foo'.

Comment 18

[:Gavin Sharp [email: [e-mail]]]

(In reply to comment #17)
> Does Wikipedia have a search mode where if an article (or redirect) exists, it
> goes there, but if it doesn't, it searches instead?

That's essentially the difference between the "Go" and the "Search" buttons in it's search form, as far as I can tell, and as far as I know that search plugin uses the "Go" mode which behaves as you describe (http://en.wikipedia.org/w/index.php?title=Special:Search&search=<input>, for example; "Cow" sends me directly to http://en.wikipedia.org/wiki/Cow, but "fowoo" sends me to http://en.wikipedia.org/wiki/Special:Search?search=foowo).

Comment 19

[Basil Hashem [:baz]]

I can't tell what algorithm the search suggest version of Wikipedia is using, but it seems to do what I would expect and want. It seems to collect pure alphabetic results...but not quite.

E.g. I do a search for "Milo" in the search plugin, I get search suggestions that don't appear when I do a "Go" search for "Milo" on the website. The nice part is if you provide something disambiguated, e.g. "Milos Forman", you'll get the exact page and don't need to worry if it was a Go vs. Search search.

Comment 20

[Aryeh Gregor (:ayg) (no longer with Mozilla)]

According to http://bugzilla.wikimedia.org/show_bug.cgi?id=7288#c3 , autosuggest as used by the search plugin is a pure alphabetic operation at present (SELECT title FROM page WHERE title LIKE 'string%' ORDER BY title LIMIT 10).  This will probably be improved at some point in the future.  Brief testing shows that it seems like current Answers.com-based prefix searching is similarly alphabet-based.

[Full disclosure 2: I'm a MediaWiki developer and Wikipedia editor.  ;)]

Comment 21

[Mic Berman]

per comment #16, clarification on item #3, after google, yahoo the order is alphabetical so wikipedia is typically last

clarification question: it seems from comment #16 we are moving to add to default list en-US? if this is true will this be in place for Firefox beta2? it's not currently in beta 1

Comment 22

[John Lilly]

on comment #21:

it's not generally alphabetical -- creative commons usually comes last, for example. myself, i'd prefer that wikipedia be close to answers, as they're similar types of search engines.

on the 2nd part of that comment, we should put it into the builds as soon as we can.

Comment 23

[:Gavin Sharp [email: [e-mail]]]

(In reply to comment #22)
> it's not generally alphabetical -- creative commons usually comes last, for
> example.

That's kind of surprising to me... Are you talking about en-US, or some other locale? The search service code does sort engines alphabetically after it's positioned the ones that are explicitly positioned [1], and the only ones we explicitly position in en-US are Yahoo and Google [2].

A new profile with a current branch (2.0.0.x) en-US build does indeed show: [Google, Yahoo, Amazon, Answers, Creative Commons, eBay], by default.

[1] http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/browser/components/search/nsSearchService.js&rev=1.1.2.73&mark=2402,2440-2450#2402
[2] http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/browser/locales/en-US/chrome/browser-region/region.properties&rev=1.9.2.15&mark=4-6

Comment 24

[John Lilly]

gavin, you're totally right. my mis-remembering. i think it's because it's what i *wanted* to be the case, not actual case, since alpha ordering only makes a little bit of sense. 

Comment 25

[Mike Beltzner [:beltzner, not reading bugmail]]

Nominating for blocking as per comment 22.

I think that ordering search plugins explicitly instead of alphabetically should be a new bug, but I kind of support the idea. Logical groupings of search type probably make more sense, especially if we're explicitly positioning the top 1 or 2.

Comment 26

[Mike Connor [:mconnor]]

So, yeah, let's do this.  Might as well do explicit ordering in this bug, perhaps as follows?

Google
Yahoo!
Wikipedia
Answers.com
Amazon
Ebay
Creative Commons

Wikipedia has an opensearch descriptor with search suggest hooked up, we should be able to snag that with permission.

Comment 27

[John Lilly]

that's what i would probably do -- i view these groupings as (1) general search, (2) reference, (3) commerce, and (4) other. (fwiw, i'd probably make commerce 2nd, but don't really care.)

i don't think we particularly need to rearrange ordering for fx3, but it's a nice to have. 

Comment 28

[Axel Hecht [:Pike]]

I always preferred alphabetic ordering unless there is a good reason not to. I personally find that easy to navigate, and it gives adding new engines a somewhat well defined behaviour.

Say, I add 4 engines to 7, then the the first two might look like it'd just add to the bottom of the list, and then you'd realize that some of them actually are ordered by name.

Explicit ordering for all engine isn't dead-on easy to implement in an l10n regime, too. Currently we just use two entries, with an option for more if hooked up in firefox-l10n.js, IIRC. Gavin might be able and awake enough to explain this better.

I guess if we go for extensive ordering, it'd make sense to drop the alphabetic sorting completely, and do something better for the initial ordering and profile migration on major update.

Comment 29

[Dave Townsend [:mossop]]

Attachment: Add wikipedia search

Not totally sure what is needed here but this patch adds the wikipedia search engine to the en-US locale. Do we need to generate patches for the other locales or do localisers handle that?

Left the ordering thing, that appears to have been spun off to bug 407922

Comment 30

[Axel Hecht [:Pike]]

Comment on attachment 298434 [details] [diff] [review]
Add wikipedia search


>diff -r 92c1ba245c80 browser/locales/en-US/searchplugins/wikipedia.xml
>--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
>+++ b/browser/locales/en-US/searchplugins/wikipedia.xml	Tue Jan 22 11:07:40 2008 +0000
>@@ -0,0 +1,13 @@
>+<SearchPlugin xmlns="http://www.mozilla.org/2006/browser/search/">
>+<ShortName>Wikipedia</ShortName>

If we make <ShortName> "Wikipedia (English)", too, then wp shouldn't offer a new plugin when you're on the site.

>+<Description>Wikipedia (English)</Description>
>+<InputEncoding>UTF-8</InputEncoding>
>+<Image width="16" height="16">data:image/x-icon;base64,AAABAAEAEBAQAAEABAAoAQAAFgAAACgAAAAQAAAAIAAAAAEABAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAEAgQAhIOEAMjHyABIR0gA6ejpAGlqaQCpqKkAKCgoAPz9%2FAAZGBkAmJiYANjZ2ABXWFcAent6ALm6uQA8OjwAiIiIiIiIiIiIiI4oiL6IiIiIgzuIV4iIiIhndo53KIiIiB%2FWvXoYiIiIfEZfWBSIiIEGi%2FfoqoiIgzuL84i9iIjpGIoMiEHoiMkos3FojmiLlUipYliEWIF%2BiDe0GoRa7D6GPbjcu1yIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</Image>
>+<Url type="application/x-suggestions+json" method="GET"
>+     template="http://en.wikipedia.org/w/api.php?action=opensearch&amp;search={searchTerms}" />

Use a template with <Param>s here, too?

>+<Url type="text/html" method="GET" template="http://en.wikipedia.org/w/index.php">
>+  <Param name="title" value="Special:Search"/>
>+  <Param name="search" value="{searchTerms}"/>
>+</Url>

I prefer the template to be the Special:Search page, too, and please add a sourceid=Mozilla-search?

>+<SearchForm>http://en.wikipedia.org/wiki/Special:Search</SearchForm>
>+</SearchPlugin>

The current version we're using in l10n looks like the italian one, http://mxr.mozilla.org/l10n/source/it/browser/searchplugins/wikipedia-it.xml.

And yes, we'll handle the versions on l10n individually.

Comment 31

[Dave Townsend [:mossop]]

Attachment: rev 2

Addresses Pike's comments. Meant to mention the ShortName thing, it's a bit irritating that we have to put it like that since it looks odd in the search UI.

Comment 32

[Dave Townsend [:mossop]]

Not going to make the string freeze unless gavin reviews and someone checks in for me.

Comment 33

[:Gavin Sharp [email: [e-mail]]]

Comment on attachment 298439 [details] [diff] [review]
rev 2

I assume the Wikipedia folks are aware that we'll be adding them by default? Could mean increased load, especially for the search suggest URI.

Comment 34

[Mic Berman]

Brion Vibber from Wikipedia is aware of this particularly for international locales. He was the last one to comment on the correct url for search suggest in http://wiki.mozilla.org/Firefox3/L10n_Requirements#Wikipedia

I am double checking we also have their explicit permission for en-US (this was not explicit in our last email exchange from November 2007). I will comment in the bug as soon as I have it (expect this by Monday at the latest).

Comment 35

[Mic Berman]

one more note: i may be being paranoid on the explicit part about en-US and I don't want to put this at risk of missing string freeze if at all possible. 

So, some additional information: from email exchange on this topic in July 2007, I believe John was in touch with Wikipedia as well about this. And, the following is my email to Brion and Wikipedia - following this Brion edited the page above:

----- Original Message -----
From: "Brion Vibber" <[e-mail]>
To: "Wikipedia information team" <[e-mail]>
Cc: "Michal Berman" <[e-mail]>
Sent: Tuesday, October 16, 2007 5:34:19 PM (GMT-0500) America/New_York
Subject: Re: [Ticket#2007101610016823] firefox

Added a quick note at
http://wiki.mozilla.org/Firefox3/L10n_Requirements#Wikipedia

-- brion vibber (brion @ wikimedia.org)
Chief Technology Officer, Wikimedia Foundation, Inc.

Wikipedia information team wrote:
> Dear Michal Berman,
>
> Thank you for your mail.
>
> Michal Berman <[e-mail]> wrote:
>
>> Hi
>> I'm working on Firefox internationalization and would like to talk with
>> someone about the technical requirements to enable search suggest
>> internationally i.e., to give our localizers the right code string to include
>> in their builds such that wikipedia is included and enabled for search suggest
>> across our 40+ locales.
>> here's the link specifically i'm working on:
>> http://wiki.mozilla.org/Firefox3/L10n_Requirements#Wikipedia
>> Thanks very much
>> Mic
>>
>> --
>> Michal Berman
>> Mozilla Corporation
>> International localization
>> 650-903-0800x242
>>
>>
>
> Thank you very much for your offer.  Would you mind bringing this up on our
> public Wikitech-l mailing list?
> <http://lists.wikimedia.org/mailman/listinfo/wikitech-l>  If not, you can
> contact our Chief Technical Officer, Brion Vibber, privately at
> [e-mail].
>
> Thank you for contacting us regarding this matter.
>
> Yours sincerely,
> Casey Brown
>

Comment 36

[Tomer Cohen :tomer]

A feature in Wikipedia I'm really missing is the ability to fallback into other languages. If the locale Wikipedia don't have a page for certain keyword (that's the case for some keywords in the Hebrew Wikipedia), search also the English Wikipedia. I know about sites which already does it for Wikipedia (maybe even answers.com), but missing it in the first place.

I know it has nothing to do with Mozilla, but it might make better experience to our users if implemented by Wikimedia. 

Comment 37

[Brion Vibber]

Just to confirm -- we will be very happy to see the traffic.

(On the language question above -- this would be cool; there may be some experimental such tools but we don't have I'd built in so far.)

Comment 38

[Reed Loden [:reed]]

Checking in browser/locales/en-US/searchplugins/list.txt;
/cvsroot/mozilla/browser/locales/en-US/searchplugins/list.txt,v  <--  list.txt
new revision: 1.5; previous revision: 1.4
done
RCS file: /cvsroot/mozilla/browser/locales/en-US/searchplugins/wikipedia.xml,v
done
Checking in browser/locales/en-US/searchplugins/wikipedia.xml;
/cvsroot/mozilla/browser/locales/en-US/searchplugins/wikipedia.xml,v  <--  wikipedia.xml
initial revision: 1.1
done

Comment 39

[Robert Accettura [:raccettura]]

For some historical context this has been something many have wanted for years (I personally wanted it in bug 269198, bug 242586).

Awesome.

Comment 40

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED using:

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b4pre) Gecko/2008020504 Minefield/3.0b4pre

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b4pre) Gecko/2008020504 Minefield/3.0b4pre

-and-

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b4pre) Gecko/2008020504 Minefield/3.0b4pre

(For en-US, with alphabetical sorting.)