246848 - Bad encoding by searching using bookmark keywords

Status: RESOLVED DUPLICATE of bug 258223

(Firefox :: Bookmarks & History, defect)

Description

[Jan Krcal]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7) Gecko/20040614 Firefox/0.9
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7) Gecko/20040614 Firefox/0.9

When I write in the location bar "g search_term" where g is keyword for google
search and search term includes non-ASCII character, the browser replaces in the
addressbar those characters with something like '%E8' (different substitution
for every character, naturally). The result is that google shows '?' instead of
every such character. So searching of i.e. czech words really doesn't work...

Reproducible: Always
Steps to Reproduce:
1.
2.
3.

Comment 1

[Jan Krcal]

I've got some additional info...

firefox substitutes non-ASCII characters with their hexadecimal value in the
local encoding (iso-8859-2 in my case), for characters that don't exist in the
local encoding (if there are such characters) it uses unicode values.

Google uses UTF-8 by default. 
Well, I could change my local encoding to UTF-8 (just in linux, I don't know if
it's possible in windows) or (that's the easier way) add "&ie=iso-8859-2" into
my Google Quick Search bookmark's location so Google uses my local encoding.

But some pages might not allow me to set the enconding such way...
What about remembering the encoding to use for every bookmark (that has a keyword)?

Comment 2

[Robert Wall]

Duplicate of bug 123006?

Comment 3

[Jan Krcal]

Well, it's related to the bug 123006, but I think it's not duplicate, because in
that bug the problem of "what characters on what conditions should be encoded"
is discussed.

As far as I know all the characters I talk about are encoded in quick search
nowadays. (I mean special non-english characters like some czech characters or
german Umlauts and many chracters in onther languages, not the characters that
have special meaning in URL like '&', '#', etc.) 

This bug is related to THE WAY those characters are encoded. IMHO, it's a
different problem.

Comment 4

[Walter Wlodarski]

Wrong detection of charset?

I replaced http://www.google.com/search?&q=%s&sourceid=firefox by
http://www.google.com/search?q=%s&sourceid=firefox&ie=iso-8859-1&oe=utf-8
to make it work with French characters (יאח...)

But for Japanese and other Asian languages (麻雀・モントリオール), it needs
http://www.google.com/search?q=%s&sourceid=firefox&ie=utf-8&oe=utf-8 to work fine.

It seems to me, the problem lies on Google side.

Comment 5

[Jan Krcal]

I don't think so...

I guess your local system encoding is iso-8859-1. Firefox uses this encoding for
French characters, but it can't use it for Japanese characters logically so it
automatically uses utf-8 instead.

How could Google tell which one you (Firefox) chose unless you say it explicitly?

Comment 6

[Met - Martin Hassman]

Well, I think this isn't browser bug. (You cannot set for every bookmark keyword
its special encoding).

Jan, can you try these definition of google keyword?

http://www.google.com/search?q=%s&ie=windows-1250&oe=UTF-8&hl=cs&btnG=Vyhledat+Googlem&lr=
or
http://www.google.com/search?q=%s&ie=iso-8859-2&oe=UTF-8&hl=cs&btnG=Vyhledat+Googlem&lr=

Did you still see the problem?

Comment 7

[Jan Krcal]

(sorry for being so late with reply, I was on holiday)

Yes, your definition of google keyword works well. But only for characters that
are included in this encoding and only if the specified encoding is my system
default encoding.

I still don't belive that current state is ideal. First of all I have to change
default settings to make it work (and I have to understand it). Secondly I'm
still limited to the specified character set.

Well, I agree that it really isn't a nice solution to set for every bookmark
keyword its special encoding.

Is there any drawback of using UTF-8 instead of system default encoding?

Comment 8

[Christian Bischoff]

If I type a keyword with a parameter in the location bar for the page
http://dict.leo.org and use a special character (e. g. a German umlaut like ü)
it isn't showed correctly on the page.

Keyword searches for Google work. Also the most other sites. I found 2 other
with the same behaviour as for LEO described:
http://www.dwds.de/cgi-bin/portalD.pl?search=%s
http://asta-killer.com/?srch=%s&submit=+search 

In Version 1.0 PR I didn't have this problem. I described it more detailed here:
http://forums.mozillazine.org/viewtopic.php?t=170087&highlight=

After reading the related Bug 270515 I tried a workaround as described in this
tread: http://forums.mozillazine.org/viewtopic.php?t=164217&highlight (replacing
a line in browser.js)

Now all the above pages work for me well. I only have a problem with amazon.de:

I open www.amazon.de and click with the right mouse button on the search field
(on the right of "SCHNELLSUCHE"). I choose "Add a Keyword for this Search".
Special character in Keyword Searches for amazon.de aren't shown correctly.

The special characters of the amazon page itself are only displayed correclty if
I set "View -> Character Encoding -> Auto-Detect -> Universal".

Something that may be related to the problem:
If I search in amazon.de by Keyword Search in a blank tab, it works. But if I
have a tab with already another page open, then type my keyword + parameter and
hit Alt+Enter, the Amazon Page displays: "Fehler im Browser-Programm?" (Error in
Browser). It suggests to go one page back or click on the shopping cart.

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0

Comment 9

[Adam Guthrie]

*** Bug 311173 has been marked as a duplicate of this bug. ***

Comment 10

[u81239]

I think I encountered this bug as well:

http://www.animenewsnetwork.com/encyclopedia/

This website has a search field. If I type "めぐみ林原" in it, the resulting
page will say "Search: めぐみ林原", showing that the search is working correctly.

However, if I search for the same string using a keyword bookmark, e.g. "ann め
ぐみ林原", it shows "Search: めぐみ林原" - clearly the wrong encoding.

The proper way to solve this would be to automatically store the used encoding
as well when creating a keyboard bookmark by right-clicking on a form field and
choosing �Add keyword for this search�. In this case, it�s Latin-1, so it should
(iirc) submit the Unicode characters as %uXXXX instead. On a Japanese site, it
should for example use SJIS, if that is the encoding of the page. The current
implementation is quite English-centric.


~Grauw

Comment 11

[u81239]

Bleh, Bugzilla could use some Unicode love as well. Even with its current
Latin-encoding, this should just work.

Anyway, the entities that show up are the characters of the name of the Japanese
singer Megumi Hayashibara. The problem should show with any Japanese character
though.


~Grauw

Comment 12

[Phil Ringnalda (:philor)]

Fixed, as best we could, in bug 258223 - if you need a quicksearch bookmark to use an encoding other than what we send by default, add &mozcharset=euc-kr or what have you to the bookmark URL.

*** This bug has been marked as a duplicate of 258223 ***

Comment 13

[Mike Connor [:mconnor]]

sorry for bugspam, long-overdue mass reassign of ancient QA contact bugs, filter on "beltznerLovesGoats" to get rid of this mass change