Closed Bug 380785 Opened 17 years ago Closed 16 years ago

Add Wikipedia to the default search engine list

Categories

(Firefox :: Search, defect, P2)

defect

Tracking

()

VERIFIED FIXED
Firefox 3 beta3

People

(Reporter: baz, Assigned: mossop)

References

Details

(Keywords: late-l10n, Whiteboard: SRCH-001e)

Attachments

(1 file, 1 obsolete file)

 
What's the rationale?  

For historical purposes, we didn't put Wikipedia in at first because they didn't have a search engine they were leveraging Yahoo and Google.

We added Answers.com in place of Dictionary.com and it also provided Thesaurus, Wikipedia, and other resources for answers.  I don't know what the usage is/was for Answers.com but still a pretty good service deal or no deal.

I suspect "Answers" doesn't scream dictionary/thesauraus/wikipedia so usage might have been low?  Don't know.  

Also note that "dict" was a keyword search for Answers.com in Firefox 1.5, not sure about 2.0.
Rafael,

Thanks for the background.

The rationale is:
- Wikipedia has grown to become a relatively accurate, pretty valuable, mostly impartial, localized resource
- Wikipedia is a non-commercial entity that aligns well with Mozilla's approach
- Wikipedia continues to grow and expand in multiple languages/locales
- Wikipedia's knowledge base is larger (and growing) compared to answers.com
- Including both answers.com seems redundant

I think we should try to get some stats on answers.com searches if that's possible and how often the wikipedia search plugin has been downloaded off AMO.
AMO data shows that the Wikipedia search engine is the most downloaded search plugin from that site.

The following Firefox locales include answers.com:
af, ar, be, da, el, en-GB, en-US, en-ZA, es-AR, eu, ga-IE, hu, ka,  
ku, lt, mk, mn, nr, nso, sq, ss, st, sv-SE, tn, tr, ts, ve, xh, zu

The following Firefox locales already include Wikipedia:
zh-TW, sl, sk, ru, ro, pt-PT, pt-BR, pl, nn-NO, nl, nb-NO, lt, it, fy-NL, fr, fi, eu, es-ES, de, ca, ar

1) We need to obtain permission from Wikipedia to use their search plugin and send traffic to their site (Mic Berman is reaching out)
2) Generate a list of locales that Wikipedia supports and map them to Firefox locales
3) For each locale, we should endeavor to use a search plugin that includes Search Suggest capability.
Component: PRD Change Request → Search
Product: Documentation → Firefox
Target Milestone: --- → Firefox 3 M7
Nom'ing as blocking Firefox 3
Assignee: basil → nobody
OS: Mac OS X → All
QA Contact: prd.change → search
Hardware: PC → All
Whiteboard: [wanted-firefox3]
Version: unspecified → Trunk
Sorry, I meant wanted-firefox3, not blocking.
I use Wikipedia more often than i use a dictionary, but I still need a dictionary sometimes...
Firefox ships with "dict" keyword in en-US by default. If you type "dict pernicious" in the URL bar, you can quickly get definitions. Also, Google (assuming that that is your default engine includes a "define" keyword for quickly getting definitions.
We don't ship with any keyword search URLs in our bookmarks.html anymore for a bit now, at least since fx2.
I personally use the Answers.com and Wikipedia search engines side by side. Yes, Answers.com does include Wikipedia in their results, but I think it's a bad idea removing the dictionary/thesaurus abilities from the search bar.

I also think that having a search engine for a dictionary/thesaurus is infinitely more discoverable than a dict keyword. I'm willing to bet that the vast majority of the Firefox audience doesn't even know what keywords are, let alone that there may be some default ones included out of the box.
Czech (cs) Firefox doesn't ship neither with answers.com nor Wikipedia. However, I like the idea of replacing one current Czech search plugin with Wikipedia - mainly  due to the fact 1 and 3 from comment #2.
I think that if you are good at googling, you already get a link to Wikipedia on the first page of Google search result. Also, adding Wikipedia to and removing Answers.com from search bar wouldn't be a better experience for users while Answers.com already include Wikipedia content.
(In reply to comment #2)
> The rationale is:
> - Wikipedia has grown to become a relatively accurate, pretty valuable, mostly
> impartial, localized resource
> - Wikipedia is a non-commercial entity that aligns well with Mozilla's approach
> - Wikipedia continues to grow and expand in multiple languages/locales
> - Wikipedia's knowledge base is larger (and growing) compared to answers.com
> - Including both answers.com seems redundant

I agree with all of these points, save for the last one.
While including Wikipedia sounds like a great idea (especially for locales that have their own Wikipedias), I don't see why it has to come at the price of dropping Answers.com. Answers.com has, on top of Wikipedia, dozens, perhaps hundreds, of other sources - most notably general-purpose, as well as domain-specific, dictionaries.

Just one example:
Looking up "interest" on Answers.com (http://www.answers.com/interest) gives you, among others:
- A definition from the American Heritage Dictionary.
- Several definitions from Barron's financial dictionaries (which, I believe are not available elsewhere on the web).
- A legal definition from the Thomson Gale's Law Encyclopedia.
- Entries from the Columbia Encyclopedia and the Britannica Concise Encyclopedia.
- Translations to over a dozen languages.
- And, of course, the Wikipedia entry.

What this means for the user is more information, and more choice, which is something I think we are trying to promote.

[Full disclosure: I'm a former employee of Answers Corp., and as such I had a part in creating Answers.com. Although I no longer work at that company, I still enjoy their product as a user].
(In reply to comment #3)

> The following Firefox locales include answers.com:
> af, ar, be, da, el, en-GB, en-US, en-ZA, es-AR, eu, ga-IE, hu, ka,  
> ku, lt, mk, mn, nr, nso, sq, ss, st, sv-SE, tn, tr, ts, ve, xh, zu

These locales: af, en-ZA, nr, nso, ss, st, tn, ts, ve, xh, zu
are not attached to answers.com, we put those in to match en-US while we focus on translation of the UI.  Users in South Africa would most probably prefer Wikipedia.
(In reply to comment #3, and echoing comment #13)

> The following Firefox locales include answers.com:
> af, ar, be, da, el, en-GB, en-US, en-ZA, es-AR, eu, ga-IE, hu, ka,  
> ku, lt, mk, mn, nr, nso, sq, ss, st, sv-SE, tn, tr, ts, ve, xh, zu

en-GB includes answers.com because Rafael requested it (bug 315385).  I'd be exceedingly surprised if non-trivial numbers of UK users were using Answers.com.
> I agree with all of these points, save for the last one.
> While including Wikipedia sounds like a great idea (especially for locales that
> have their own Wikipedias), I don't see why it has to come at the price of
> dropping Answers.com. Answers.com has, on top of Wikipedia, dozens, perhaps
> hundreds, of other sources - most notably general-purpose, as well as
> domain-specific, dictionaries.

I agree with Uri here and I think you should rename this bug to just adding Wikipedia to the search engine list.  Removing Answers.com or another search engine from the default list should be a separate bug if necessary.

I don't think you have enough information to arbitrarily drop Answers.com from the list like this, and they do provide much more than just "user-generated" encyclopedia info.  

I recommend that we keep Answers.com until you have real data that says otherwise or they do something "evil".  Have they done anything bad to piss off our users that we don't know about?  Barring that, Answers.com should stay.  

Also, it should be noted that they've been good citizens with their promotion of Firefox and development of Firefox extensions.  It's not like they're Yahoo! or anything (oops, did I say that outloud?).
This bug has been a bit stale and we need to make a decision. Here's my recommendation.

0. I've retitled the bug to say "Add Wikipedia to default search engine list" - removed reference to answers.com.
1. Where Wikipedia does not appear in a Firefox locale, add it (assuming there's good localized content and the localizers agree that it's worthwhile).
2. If Wikipedia is there, ensure that we use the "Search Suggest" version of the search plugin, if it's supported by Wikipedia for that language.
3. It should be a localizer decision as to where in the list Wikipedia appears. Localizers should be aware of various contractual constraints with regard to Yahoo and Google.

We won't remove the answer.com plugin for now and will try to work with the company to get data about degree of Firefox usage (per locale).

Any questions?
Summary: Remove answers.com from Search Engine list, Add Wikipedia → Add Wikipedia to the default search engine list
Does Wikipedia have a search mode where if an article (or redirect) exists, it goes there, but if it doesn't, it searches instead?  I don't find Wikipedia's search very useful, so I usually either (1) guess at the article name (en.wikipedia.org/wiki/foo) or (2) search Google for 'wikipedia foo'.
(In reply to comment #17)
> Does Wikipedia have a search mode where if an article (or redirect) exists, it
> goes there, but if it doesn't, it searches instead?

That's essentially the difference between the "Go" and the "Search" buttons in it's search form, as far as I can tell, and as far as I know that search plugin uses the "Go" mode which behaves as you describe (http://en.wikipedia.org/w/index.php?title=Special:Search&search=<input>, for example; "Cow" sends me directly to http://en.wikipedia.org/wiki/Cow, but "fowoo" sends me to http://en.wikipedia.org/wiki/Special:Search?search=foowo).
I can't tell what algorithm the search suggest version of Wikipedia is using, but it seems to do what I would expect and want. It seems to collect pure alphabetic results...but not quite.

E.g. I do a search for "Milo" in the search plugin, I get search suggestions that don't appear when I do a "Go" search for "Milo" on the website. The nice part is if you provide something disambiguated, e.g. "Milos Forman", you'll get the exact page and don't need to worry if it was a Go vs. Search search.
According to http://bugzilla.wikimedia.org/show_bug.cgi?id=7288#c3 , autosuggest as used by the search plugin is a pure alphabetic operation at present (SELECT title FROM page WHERE title LIKE 'string%' ORDER BY title LIMIT 10).  This will probably be improved at some point in the future.  Brief testing shows that it seems like current Answers.com-based prefix searching is similarly alphabet-based.

[Full disclosure 2: I'm a MediaWiki developer and Wikipedia editor.  ;)]
Whiteboard: [wanted-firefox3] → [wanted-firefox3] SRCH-001e
per comment #16, clarification on item #3, after google, yahoo the order is alphabetical so wikipedia is typically last

clarification question: it seems from comment #16 we are moving to add to default list en-US? if this is true will this be in place for Firefox beta2? it's not currently in beta 1
on comment #21:

it's not generally alphabetical -- creative commons usually comes last, for example. myself, i'd prefer that wikipedia be close to answers, as they're similar types of search engines.

on the 2nd part of that comment, we should put it into the builds as soon as we can.
(In reply to comment #22)
> it's not generally alphabetical -- creative commons usually comes last, for
> example.

That's kind of surprising to me... Are you talking about en-US, or some other locale? The search service code does sort engines alphabetically after it's positioned the ones that are explicitly positioned [1], and the only ones we explicitly position in en-US are Yahoo and Google [2].

A new profile with a current branch (2.0.0.x) en-US build does indeed show: [Google, Yahoo, Amazon, Answers, Creative Commons, eBay], by default.

[1] http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/browser/components/search/nsSearchService.js&rev=1.1.2.73&mark=2402,2440-2450#2402
[2] http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/browser/locales/en-US/chrome/browser-region/region.properties&rev=1.9.2.15&mark=4-6
gavin, you're totally right. my mis-remembering. i think it's because it's what i *wanted* to be the case, not actual case, since alpha ordering only makes a little bit of sense. 
Nominating for blocking as per comment 22.

I think that ordering search plugins explicitly instead of alphabetically should be a new bug, but I kind of support the idea. Logical groupings of search type probably make more sense, especially if we're explicitly positioning the top 1 or 2.
Flags: blocking-firefox3?
Blocks: 407922
So, yeah, let's do this.  Might as well do explicit ordering in this bug, perhaps as follows?

Google
Yahoo!
Wikipedia
Answers.com
Amazon
Ebay
Creative Commons

Wikipedia has an opensearch descriptor with search suggest hooked up, we should be able to snag that with permission.
Flags: blocking-firefox3? → blocking-firefox3+
Priority: -- → P3
Target Milestone: Firefox 3 M7 → Firefox 3 M11
that's what i would probably do -- i view these groupings as (1) general search, (2) reference, (3) commerce, and (4) other. (fwiw, i'd probably make commerce 2nd, but don't really care.)

i don't think we particularly need to rearrange ordering for fx3, but it's a nice to have. 
I always preferred alphabetic ordering unless there is a good reason not to. I personally find that easy to navigate, and it gives adding new engines a somewhat well defined behaviour.

Say, I add 4 engines to 7, then the the first two might look like it'd just add to the bottom of the list, and then you'd realize that some of them actually are ordered by name.

Explicit ordering for all engine isn't dead-on easy to implement in an l10n regime, too. Currently we just use two entries, with an option for more if hooked up in firefox-l10n.js, IIRC. Gavin might be able and awake enough to explain this better.

I guess if we go for extensive ordering, it'd make sense to drop the alphabetic sorting completely, and do something better for the initial ordering and profile migration on major update.
Priority: P3 → P2
Flags: wanted-firefox3+
Whiteboard: [wanted-firefox3] SRCH-001e → SRCH-001e
Attached patch Add wikipedia search (obsolete) — Splinter Review
Not totally sure what is needed here but this patch adds the wikipedia search engine to the en-US locale. Do we need to generate patches for the other locales or do localisers handle that?

Left the ordering thing, that appears to have been spun off to bug 407922
Assignee: nobody → dtownsend
Status: NEW → ASSIGNED
Attachment #298434 - Flags: review?(gavin.sharp)
Comment on attachment 298434 [details] [diff] [review]
Add wikipedia search


>diff -r 92c1ba245c80 browser/locales/en-US/searchplugins/wikipedia.xml
>--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
>+++ b/browser/locales/en-US/searchplugins/wikipedia.xml	Tue Jan 22 11:07:40 2008 +0000
>@@ -0,0 +1,13 @@
>+<SearchPlugin xmlns="http://www.mozilla.org/2006/browser/search/">
>+<ShortName>Wikipedia</ShortName>

If we make <ShortName> "Wikipedia (English)", too, then wp shouldn't offer a new plugin when you're on the site.

>+<Description>Wikipedia (English)</Description>
>+<InputEncoding>UTF-8</InputEncoding>
>+<Image width="16" height="16">data:image/x-icon;base64,AAABAAEAEBAQAAEABAAoAQAAFgAAACgAAAAQAAAAIAAAAAEABAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAEAgQAhIOEAMjHyABIR0gA6ejpAGlqaQCpqKkAKCgoAPz9%2FAAZGBkAmJiYANjZ2ABXWFcAent6ALm6uQA8OjwAiIiIiIiIiIiIiI4oiL6IiIiIgzuIV4iIiIhndo53KIiIiB%2FWvXoYiIiIfEZfWBSIiIEGi%2FfoqoiIgzuL84i9iIjpGIoMiEHoiMkos3FojmiLlUipYliEWIF%2BiDe0GoRa7D6GPbjcu1yIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</Image>
>+<Url type="application/x-suggestions+json" method="GET"
>+     template="http://en.wikipedia.org/w/api.php?action=opensearch&amp;search={searchTerms}" />

Use a template with <Param>s here, too?

>+<Url type="text/html" method="GET" template="http://en.wikipedia.org/w/index.php">
>+  <Param name="title" value="Special:Search"/>
>+  <Param name="search" value="{searchTerms}"/>
>+</Url>

I prefer the template to be the Special:Search page, too, and please add a sourceid=Mozilla-search?

>+<SearchForm>http://en.wikipedia.org/wiki/Special:Search</SearchForm>
>+</SearchPlugin>

The current version we're using in l10n looks like the italian one, http://mxr.mozilla.org/l10n/source/it/browser/searchplugins/wikipedia-it.xml.

And yes, we'll handle the versions on l10n individually.
Attached patch rev 2Splinter Review
Addresses Pike's comments. Meant to mention the ShortName thing, it's a bit irritating that we have to put it like that since it looks odd in the search UI.
Attachment #298434 - Attachment is obsolete: true
Attachment #298439 - Flags: review?(gavin.sharp)
Attachment #298434 - Flags: review?(gavin.sharp)
Whiteboard: SRCH-001e → [has patch] SRCH-001e
Not going to make the string freeze unless gavin reviews and someone checks in for me.
Keywords: late-l10n
Comment on attachment 298439 [details] [diff] [review]
rev 2

I assume the Wikipedia folks are aware that we'll be adding them by default? Could mean increased load, especially for the search suggest URI.
Attachment #298439 - Flags: review?(gavin.sharp) → review+
Brion Vibber from Wikipedia is aware of this particularly for international locales. He was the last one to comment on the correct url for search suggest in http://wiki.mozilla.org/Firefox3/L10n_Requirements#Wikipedia

I am double checking we also have their explicit permission for en-US (this was not explicit in our last email exchange from November 2007). I will comment in the bug as soon as I have it (expect this by Monday at the latest).
one more note: i may be being paranoid on the explicit part about en-US and I don't want to put this at risk of missing string freeze if at all possible. 

So, some additional information: from email exchange on this topic in July 2007, I believe John was in touch with Wikipedia as well about this. And, the following is my email to Brion and Wikipedia - following this Brion edited the page above:

----- Original Message -----
From: "Brion Vibber" <brion@wikimedia.org>
To: "Wikipedia information team" <info-en@wikimedia.org>
Cc: "Michal Berman" <mic@mozilla.com>
Sent: Tuesday, October 16, 2007 5:34:19 PM (GMT-0500) America/New_York
Subject: Re: [Ticket#2007101610016823] firefox

Added a quick note at
http://wiki.mozilla.org/Firefox3/L10n_Requirements#Wikipedia

-- brion vibber (brion @ wikimedia.org)
Chief Technology Officer, Wikimedia Foundation, Inc.

Wikipedia information team wrote:
> Dear Michal Berman,
>
> Thank you for your mail.
>
> Michal Berman <mic@mozilla.com> wrote:
>
>> Hi
>> I'm working on Firefox internationalization and would like to talk with
>> someone about the technical requirements to enable search suggest
>> internationally i.e., to give our localizers the right code string to include
>> in their builds such that wikipedia is included and enabled for search suggest
>> across our 40+ locales.
>> here's the link specifically i'm working on:
>> http://wiki.mozilla.org/Firefox3/L10n_Requirements#Wikipedia
>> Thanks very much
>> Mic
>>
>> --
>> Michal Berman
>> Mozilla Corporation
>> International localization
>> 650-903-0800x242
>>
>>
>
> Thank you very much for your offer.  Would you mind bringing this up on our
> public Wikitech-l mailing list?
> <http://lists.wikimedia.org/mailman/listinfo/wikitech-l>  If not, you can
> contact our Chief Technical Officer, Brion Vibber, privately at
> brion@wikimedia.org.
>
> Thank you for contacting us regarding this matter.
>
> Yours sincerely,
> Casey Brown
>
A feature in Wikipedia I'm really missing is the ability to fallback into other languages. If the locale Wikipedia don't have a page for certain keyword (that's the case for some keywords in the Hebrew Wikipedia), search also the English Wikipedia. I know about sites which already does it for Wikipedia (maybe even answers.com), but missing it in the first place.

I know it has nothing to do with Mozilla, but it might make better experience to our users if implemented by Wikimedia. 
Keywords: checkin-needed
Just to confirm -- we will be very happy to see the traffic.

(On the language question above -- this would be cool; there may be some experimental such tools but we don't have I'd built in so far.)
Checking in browser/locales/en-US/searchplugins/list.txt;
/cvsroot/mozilla/browser/locales/en-US/searchplugins/list.txt,v  <--  list.txt
new revision: 1.5; previous revision: 1.4
done
RCS file: /cvsroot/mozilla/browser/locales/en-US/searchplugins/wikipedia.xml,v
done
Checking in browser/locales/en-US/searchplugins/wikipedia.xml;
/cvsroot/mozilla/browser/locales/en-US/searchplugins/wikipedia.xml,v  <--  wikipedia.xml
initial revision: 1.1
done
Status: ASSIGNED → RESOLVED
Closed: 16 years ago
Keywords: checkin-needed
Resolution: --- → FIXED
Whiteboard: [has patch] SRCH-001e → SRCH-001e
For some historical context this has been something many have wanted for years (I personally wanted it in bug 269198, bug 242586).

Awesome.
Verified FIXED using:

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b4pre) Gecko/2008020504 Minefield/3.0b4pre

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b4pre) Gecko/2008020504 Minefield/3.0b4pre

-and-

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b4pre) Gecko/2008020504 Minefield/3.0b4pre

(For en-US, with alphabetical sorting.)
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.