Last Comment Bug 647403 - Firefox 4 submits string encoded with Big5 as UTF-8
: Firefox 4 submits string encoded with Big5 as UTF-8
Status: RESOLVED FIXED
:
Product: Core
Classification: Components
Component: Internationalization (show other bugs)
: unspecified
: All Windows 7
: -- normal (vote)
: mozilla11
Assigned To: Masatoshi Kimura [:emk]
:
:
Mentors:
http://libcat.hkpl.gov.hk/webpac_cjk/...
Depends on:
Blocks: 333859 443588 472669
  Show dependency treegraph
 
Reported: 2011-04-01 22:00 PDT by Jeff Tong
Modified: 2011-11-11 09:48 PST (History)
8 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
patch (7.11 KB, patch)
2011-04-05 05:15 PDT, Masatoshi Kimura [:emk]
no flags Details | Diff | Splinter Review
patch (7.17 KB, patch)
2011-11-06 14:52 PST, Masatoshi Kimura [:emk]
bzbarsky: review+
Details | Diff | Splinter Review
patch for check in (7.35 KB, patch)
2011-11-07 06:32 PST, Masatoshi Kimura [:emk]
no flags Details | Diff | Splinter Review
patch for l10n-central/zh-TW (423 bytes, patch)
2011-11-09 07:15 PST, Peter Pin-Guang Chen [:petercpg] (MozTW.org)
no flags Details | Diff | Splinter Review
patch for l10n-central/zh-TW, v2 (412 bytes, patch)
2011-11-09 07:56 PST, Peter Pin-Guang Chen [:petercpg] (MozTW.org)
l10n: review+
VYV03354: review+
Details | Diff | Splinter Review

Description Jeff Tong 2011-04-01 22:00:15 PDT
User-Agent:       Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0) Gecko/20100101 Firefox/4.0
Build Identifier: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0) Gecko/20100101 Firefox/4.0

The site is Hong Kong public library material search page which is encoded with traditional Big5 code (Traditional Chinese). Any Chinese character input in the page is treated as UTF-8. As a result, nothing was found with the browser.

Reproducible: Always

Steps to Reproduce:
1. enter any Traditional Chinese character
Actual Results:  
There is a warning message that no result is found.
Comment 1 Simon Montagu :smontagu 2011-04-02 13:19:17 PDT
I couldn't reproduce this bug. For example, I searched for 春秋 and was sent to the URL http://libcat.hkpl.gov.hk/webpac_cjk/wgbroker.exe?201104030406530250811+-access+top.books-page+search+open+BT+%ACK%AC%EE%23%23A:NONE%23NONE:NONE::%23%23

%ACK%AC%EE is equivalent to 春秋 encoded in Big5

Is network.standard-url.encode-query-utf8 set to true for you in about:config?
Comment 2 Jeff Tong 2011-04-03 07:53:42 PDT
I have just checked. It is set to true but I swear I have never touched it and the problem can be reproduced on Win7 and WinXP machine.
Comment 3 Boris Zbarsky [:bz] (still a bit busy) 2011-04-04 22:54:13 PDT
> but I swear I have never touched it 

And none of your extensions did either?
Comment 4 Jeff Tong 2011-04-04 23:16:16 PDT
I have found the problem since beta release.

I always tested whether the problems solved on different machines shortly after installation. None of extension was added until few days ago.
Comment 5 Simon Montagu :smontagu 2011-04-04 23:48:08 PDT
Are you using the localized version of Firefox? The zh-tw localization sets the pref to true.
Comment 6 Masatoshi Kimura [:emk] 2011-04-05 00:15:30 PDT
Probably we should make a pref based on bug 393246 with fewer side effects for Localizers...
As I explained in bug 552273 comment #17, legacy Western sites are already "broken" on Japanese (and perhaps Taiwan?) locales. So bug 393246 patch has viatually zero side effect on those locales.
Comment 7 Masatoshi Kimura [:emk] 2011-04-05 05:15:12 PDT
Created attachment 524051 [details] [diff] [review]
patch

I've added "browser.fixup.use-utf8" pref which is usable for localizers on some non-Western locales. If the pref is set to true, CreateFixupURI will always use UTF-8 as a default encoding. Only the location bar is affected by the pref.
I've also removed "network.standard-url.encode-query-utf8" pref because we will be unable to turn on the pref forever due to Web compat curse and RFC3987bis will formalize the current behavior. Now the query will be always encoded in the document charset.
Comment 8 Jeff Tong 2011-04-05 14:46:33 PDT
(In reply to comment #5)
> Are you using the localized version of Firefox? The zh-tw localization sets the
> pref to true.

Sure. Traditional Chinese version (zh-tw).
Comment 9 Jeff Tong 2011-06-22 07:00:23 PDT
I have just tested Firefox 5.0 Traditional Chinese version. I am disappointed that the problem still remains.
Comment 10 Masatoshi Kimura [:emk] 2011-11-06 14:52:38 PST
Created attachment 572345 [details] [diff] [review]
patch
Comment 11 Masatoshi Kimura [:emk] 2011-11-06 14:53:42 PST
Comment on attachment 572345 [details] [diff] [review]
patch

Unbitrotted and switching the reviewer
Comment 12 Boris Zbarsky [:bz] (still a bit busy) 2011-11-06 18:43:04 PST
Comment on attachment 572345 [details] [diff] [review]
patch

Please parenthesize the "aFixupFlags & FIXUP_FLAG_USE_UTF8" bit.

Also please make sure the l10n impact of this gets flagged; localizations that set the old pref probably need to set the new one.

r=me
Comment 13 Masatoshi Kimura [:emk] 2011-11-06 20:13:41 PST
> Also please make sure the l10n impact of this gets flagged; localizations that set the old pref probably need to set the new one.
How can I flag for the l10n impact?

zh-TW is the only locale which modified the pref as far as I know.
https://mxr.mozilla.org/l10n-central/search?string=encode-query-utf8
Tim, please replace "network.standard-url.encode-query-utf8" with "browser.fixup.use-utf8" when this patch is landed. It will fix this bug and will not regress bug 443588.
Comment 14 Masatoshi Kimura [:emk] 2011-11-06 20:15:38 PST
It's unwise to land this just before the aurora merge (meaning l10n freeze).
Comment 15 Masatoshi Kimura [:emk] 2011-11-07 06:32:45 PST
Created attachment 572449 [details] [diff] [review]
patch for check in
Comment 17 Peter Pin-Guang Chen [:petercpg] (MozTW.org) 2011-11-09 07:15:46 PST
Created attachment 573190 [details] [diff] [review]
patch for l10n-central/zh-TW

double check before landing
Comment 18 Axel Hecht [:Pike] 2011-11-09 07:38:18 PST
Peter, anyone you want to review your change in particular?

Also, you probably want a different comment, as you're not working around something, you're merely adjusting a setting to a good local default, I guess?
Comment 19 Peter Pin-Guang Chen [:petercpg] (MozTW.org) 2011-11-09 07:56:21 PST
Created attachment 573198 [details] [diff] [review]
patch for l10n-central/zh-TW, v2

(In reply to Axel Hecht [:Pike] from comment #18)
> Peter, anyone you want to review your change in particular?
> 
> Also, you probably want a different comment, as you're not working around
> something, you're merely adjusting a setting to a good local default, I
> guess?

Maybe you Pike, or emk could review for this? sorry I did not add requestee. 

And that's why I need to make sure in case there's still something missing.
Comment 20 Masatoshi Kimura [:emk] 2011-11-09 14:23:42 PST
Comment on attachment 573198 [details] [diff] [review]
patch for l10n-central/zh-TW, v2

You don't have to change the comment. This is still a workaround for bug 443588.
This bug will be fixed regardless of your settings.
Otherwise looks good.
Comment 21 Masatoshi Kimura [:emk] 2011-11-09 14:55:50 PST
Comment on attachment 573198 [details] [diff] [review]
patch for l10n-central/zh-TW, v2

Ah, Axel is right about this is no longer a workaround because bug 443588 is Traditional Chinese specific.
Change the comment to "Fix bug 443588".
Comment 22 Peter Pin-Guang Chen [:petercpg] (MozTW.org) 2011-11-09 15:14:53 PST
Landed: http://hg.mozilla.org/l10n-central/zh-TW/rev/6b71288595d2
Comment 23 Marco Bonardo [::mak] 2011-11-10 03:13:07 PST
https://hg.mozilla.org/mozilla-central/rev/0246515e5fda

Note You need to log in before you can comment on or make changes to this bug.