417766 - Quicksearch bookmarks use UTF8 encoding, which is incompatible with many sites that worked in Fx2

Status: RESOLVED DUPLICATE of bug 317472

(Firefox :: Bookmarks & History, defect)

Description

[Szabolcs Horvát]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b4pre) Gecko/2008021504 Minefield/3.0b4pre
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b4pre) Gecko/2008021504 Minefield/3.0b4pre

When searching with quicksearch bookmarks, the word that is substituted for %s is encoded as UTF8 in Minefield, while a one-byte encoding was used in Firefox 2.  This change makes quicksearch bookmarks incompatible with many sites.

Three out of four dictionaries I use regularly are affected, so I suspect that generally a significant portion of website will be affected.

Examples:

For testing with latin-2 encoding:
http://dict.sztaki.hu/dict_search.php?L=ENG%3AHUN%3AEngHunDict&W=%s&T=1&P=1

For testing with latin-1 encoding:
http://www.tritrans.net/cgibin/translate_en.cgi?spraak=Norsk&Fra=%s&sourceid=Mozilla-search
http://www.dokpro.uio.no/perl/ordboksoek/ordbok.cgi?OPP=%s&ordbok=bokmaal&alfabet=n&renset=j

Reproducible: Always

Steps to Reproduce:
1. Create a quicksearch bookmark from one of the above URLs by entering it in the "Location" field of the "New bookmark" dialog.  Add a keyword for the bookmark.

2. Search for a word using the bookmark (i.e. type the keyword in the address bar, followed by a space and the word).  Words for testing latin-1 characters: år, bær.  Words for testing latin-2 characters: őr, tűr.
Actual Results:  
Note that the words are encoded as UTF8, therefore the dictionary didn't recognize them.

Expected Results:  
The words should have been encoded with a one-byte encoding, just like with Firefox 2, so that the dictionaries recognize them. (I don't know how does Fx2 know that it should select latin-2 over utf-8 for ő and ű, and latin-1 for å and æ, but it does do this.)

Comment 1

[Szabolcs Horvát]

Maybe my analysis of the problems source above was incorrect.

In simple terms, the problem is:

I cannot find any way to define correctly working quicksearch bookmarks for the following websites in Firefox 3:

http://dict.sztaki.hu/
http://www.tritrans.net/
http://www.dokpro.uio.no/ordboksoek.html

All of these sites are compatible with Firefox 2's quicksearch bookmarks, so this can be considered a regression.


Notes about the sites:

It is important the the characters ő and ű should work for dict.sztaki.hu (for Hungarian -> English).  These characters are present in ISO-8859-2, but not in ISO-8859-1.  
The tritrans.net site is tricky to play with because unless the correct URL is used (see the original bug report), it complains about the referrer.

Neither right clicking the search box and selecting "add keyword for this search", nor editing the bookmark URL directly produces correctly working quicksearch bookmarks.

Comment 2

[Phil Ringnalda (:philor)]

Did you by any chance edit your Fx2 bookmarks, replacing the default %s, which sends the search UTF-8 encoded, with %S, which uses your OS's encoding? Or (much, much less likely that you would have forgotten), did you add &mozcharset=iso-8859-2 to the end of the bookmarked URLs?

If it wasn't either of those, then you must have been depending on a LAST_CHARSET saved with a separate normal bookmark of the page, and this a duplicate of bug 317472.

Comment 3

[Szabolcs Horvát]

(In reply to comment #2)
> Did you by any chance edit your Fx2 bookmarks, replacing the default %s, which
> sends the search UTF-8 encoded, with %S, which uses your OS's encoding? Or
> (much, much less likely that you would have forgotten), did you add
> &mozcharset=iso-8859-2 to the end of the bookmarked URLs?

No, I have not.  I didn't even know about these possibilities.  If I had known about them, I wouldn't have been panicking.  They are a fine workaround for the moment.  Can you please tell me where they are documented?  I can not find any reference to them.

The bookmarks have been imported from Firefox 2, and the URL is exactly the same in Firefox 3 as in Firefox 2.  I have not edited it.  If I compare the two, they are exactly the same.  (You can see the URLs above.)

> If it wasn't either of those, then you must have been depending on a
> LAST_CHARSET saved with a separate normal bookmark of the page, and this a
> duplicate of bug 317472.

I'm not sure I understand that bug completely, but to me it seems that it is about something different ...

Anyway ... there is one more thing to note:

If I right click in the search field and choose "add a keyword for this search", the quicksearch bookmark that is produced will work correctly (use the correct encoding) in Firefox 2, without any additional effort.  But NOT in Firefox 3 (where UTF-8 is used regardless of the page's encoding).  (I use a US English version of Window XP if this matters.)  From a user's point of view this is a regression: something that has worked before but is broken now.

Comment 4

[Phil Ringnalda (:philor)]

"documentation"? What a funny looking word, what's that? ;)

It seems to be documented in one Korean and one German blog post, and as a fixme on the talk page for the article where it should be be, http://kb.mozillazine.org/Talk:Using_keyword_searches

But, this _is_ bug 317472 (despite the summary, the patch is mostly about fixing just this) - in 2.x, when you right-click, Add a keyword for this search, it saves the page's charset in the bookmark's LAST_CHARSET attribute, and then uses it in getShortcutOrURI(), which looks for mozcharset, then asks the bookmarks service for the LAST_CHARSET, then falls back to UTF-8 for %s and your system charset for %S if it doesn't find either of those. Places doesn't yet have a way to ask for LAST_CHARSET, so the only thing that worked for people who hadn't read bug 258223 no longer works.