270120 - Yahoo search plugin does not work with Unicode characters

Status: RESOLVED FIXED

(Firefox :: Search, defect)

Description

[Panos Stokas]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0

Greek words in Yahoo search toolbar are encoded incorrectly in the resulting
URI. As a result, Yahoo does not show any find when typing greek words in the
search toolbar.

Reproducible: Always
Steps to Reproduce:
1. Select Yahoo on the search toolbar.
2. Copy "αναζήτηση" to the search toolbar and hit enter.

Actual Results:  
http://search.yahoo.com/search?p=%3F%3F%3F%3F%3F%3F%3F%3F%3F&ei=UTF-8&rls=org.mozilla:en-US:official&fr=moz2

Expected Results:  
http://search.yahoo.com/search?p=%CE%B1%CE%BD%CE%B1%CE%B6%CE%AE%CF%84%CE%B7%CF%83%CE%B7&ei=UTF-8&fr=moz2&fl=0&x=wrt

Comment 1

[Ivan Ičin]

It does not work with Serbian characters either, so title should be updated to:
Yahoo search plugin does not work with Unicode characters. I don't have right to
change this, so, Panos, if you are reading this please update the title.

As this affects using of Firefox in every language other than English, I guess
that it is candidate to block 1.1

Comment 2

[:Gavin Sharp [email: [e-mail]]]

This is a bug in the yahoo toolbar, not in Firefox.

Comment 3

[Ivan Ičin]

Gavin, reporter obviously meant Yahoo search plugin, but he was not able to
express himself as he is from Greece.

So, there is such bug in Firefox Yahoo search plugin for Unicode characters.

Comment 4

[Panos Stokas]

Ivan, "reporter" was quite able to express himself.

Gavin, the problem is in the Search component in the Firefox product. "Search:
Internet search from the Toolbar Search field"

Guys, please make it easier for bug reporters to report bugs and don't hassle
them with rude comments or invalidate their reports.

Comment 5

[Ivan Ičin]

(In reply to comment #4)
> Ivan, "reporter" was quite able to express himself.

OK, let's not quarrel, but let's be constructive to solve this bug.

First good step would be to update your summary. Don't take it as an insult, but
I also understood that it points to Yahoo toolbar, not to Firefox bug. Gavin
understood it also the same way. We were probably only two men that came to this
report. Is it so terrible to change it to something more friendly so that
everyone can understand that without effort? I gave you example of acceptable
summary in comment #2.

After that you should add in keywords intl, so to mark that this bug deals with
international issue. There are some people that track bugs like that, so it will
improve chances for the bug to be resolved.

Finally, IMO in this bug blocking-aviary 1.1 flag should be changed to ?. That
means that this bug is a candidate to block release of Firefox 1.1 until it is
solved. I really think that it is major bug. Actualy, before bug was closed, I
changed flag to this, and that's why Gavin came here.

Comment 6

[:Gavin Sharp [email: [e-mail]]]

*** This bug has been marked as a duplicate of 177691 ***

Comment 7

[Panos Stokas]

Ivan, I agree.

Gavin, this bug is not a duplicate of 177691.

Comment 8

[Asaf Romano (gone)]

Attachment: add missing attributes

we have sent "UTF-8" as the encoding we use (see <input name="ei"
value="UTF-8">), but sent is as something else...

Comment 9

[Phil Ringnalda (:philor)]

We may or may not send it as something else: it looks to me like if the .src
doesn't include a queryCharset (queryEncoding seems to refer to the charset of
the returned HTML, for the search sidebar's benefit) then we encode the query in
the user's default encoding as selected in Options - Languages, which is
something we probably don't ever want to do. creativecommons.src wants utf-8
also, while amazondotcom.src, dictionary.src and eBay.src will only tolerate
ISO-8859-1. If your default encoding is UTF-8 rather than ISO-8859-1, then Yahoo
and Creative Commons will work, and the three shopping searches will fail
(though they'll fail for Panos no matter what, since we can't convert Greek to
ISO-8859-1).

Comment 10

[Panos Stokas]

Default encoding UTF-8, charset auto detect ON or OFF: still same behavior on yahoo.

Besides, I don't think default encoding has any impact on the search plugin or
search results (considering they are HTML >= 4.0) as per
http://ppewww.ph.gla.ac.uk/~flavell/charset/quick.en.html

Comment 11

[Phil Ringnalda (:philor)]

Well, I was half-right, not bad for me. queryEncoding should only be used if
queryCharset is missing, in which case it needs to be one of the integers listed
in InternetSearchDataSource::MapEncoding. However, I say *should* because a
friend who wrote a plugin that used both ="EUC-JP" says it worked fine on
Windows, but when he switched to using a Mac, it broke and fell back to
ISO-8859-1 until he removed the queryEncoding="EUC-JP", leaving just the
queryCharset. That both puzzles and worries me.

Comment 12

[Jungshik Shin]

*** Bug 271357 has been marked as a duplicate of this bug. ***

Comment 13

[Jungshik Shin]

Comment on attachment 171109 [details] [diff] [review]
add missing attributes

This should be fine. For any search engine that accepts UTF-8 encoded URIs
(IRIs) and returns the result in UTF-8, we should use that. 

For search engines that don't work with UTF-8, yet, I'll investigate what
querchCharset and queryEncoding do.

Comment 14

[Jungshik Shin]

So I took a look. queryEncoding is not necessary and specifying queryCharset is
sufficient. Moreover, for mozilla, the result charset has to be specified like this:

<interpret 
    browserResultType="result" 
    charset = "UTF-8"
    resultListStart="<div id=web>" 
    resultListEnd="MORE SPONSOR RESULTS" 
    resultItemStart="<li>" 
>

Asaf, would you update your patch? 

What's observed by Phil's friend on Mac is indeed very strange. The spurrious
'queryEncoding' shouldn't matter as long as 'queryCharset' is present.

Comment 15

[Asaf Romano (gone)]

Comment on attachment 171109 [details] [diff] [review]
add missing attributes

Yeah, later today. Thanks jshin.

Comment 16

[Asaf Romano (gone)]

Attachment: better patch

Comment 17

[Mike Connor [:mconnor]]

Comment on attachment 173092 [details] [diff] [review]
better patch

wow, docs on mozdev aren't so good...

Comment 18

[Jungshik Shin]

I filed a bug at mozdev bugzilla about the documentation. See
http://bugzilla.mozdev.org/show_bug.cgi?id=9098

Comment 19

[:Gavin Sharp [email: [e-mail]]]

*** Bug 283828 has been marked as a duplicate of this bug. ***