647403 - Firefox 4 submits string encoded with Big5 as UTF-8

Status: RESOLVED FIXED

(Core :: Internationalization, defect)

Description

[Jeff Tong]

User-Agent:       Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0) Gecko/20100101 Firefox/4.0
Build Identifier: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0) Gecko/20100101 Firefox/4.0

The site is Hong Kong public library material search page which is encoded with traditional Big5 code (Traditional Chinese). Any Chinese character input in the page is treated as UTF-8. As a result, nothing was found with the browser.

Reproducible: Always

Steps to Reproduce:
1. enter any Traditional Chinese character
Actual Results:  
There is a warning message that no result is found.

Comment 1

[Simon Montagu :smontagu]

I couldn't reproduce this bug. For example, I searched for 春秋 and was sent to the URL http://libcat.hkpl.gov.hk/webpac_cjk/wgbroker.exe?201104030406530250811+-access+top.books-page+search+open+BT+%ACK%AC%EE%23%23A:NONE%23NONE:NONE::%23%23

%ACK%AC%EE is equivalent to 春秋 encoded in Big5

Is network.standard-url.encode-query-utf8 set to true for you in about:config?

Comment 2

[Jeff Tong]

I have just checked. It is set to true but I swear I have never touched it and the problem can be reproduced on Win7 and WinXP machine.

Comment 3

[Boris Zbarsky [:bzbarsky]]

> but I swear I have never touched it 

And none of your extensions did either?

Comment 4

[Jeff Tong]

I have found the problem since beta release.

I always tested whether the problems solved on different machines shortly after installation. None of extension was added until few days ago.

Comment 5

[Simon Montagu :smontagu]

Are you using the localized version of Firefox? The zh-tw localization sets the pref to true.

Comment 6

[Masatoshi Kimura [:emk]]

Probably we should make a pref based on bug 393246 with fewer side effects for Localizers...
As I explained in bug 552273 comment #17, legacy Western sites are already "broken" on Japanese (and perhaps Taiwan?) locales. So bug 393246 patch has viatually zero side effect on those locales.

Comment 7

[Masatoshi Kimura [:emk]]

Attachment: patch

I've added "browser.fixup.use-utf8" pref which is usable for localizers on some non-Western locales. If the pref is set to true, CreateFixupURI will always use UTF-8 as a default encoding. Only the location bar is affected by the pref.
I've also removed "network.standard-url.encode-query-utf8" pref because we will be unable to turn on the pref forever due to Web compat curse and RFC3987bis will formalize the current behavior. Now the query will be always encoded in the document charset.

Comment 8

[Jeff Tong]

(In reply to comment #5)
> Are you using the localized version of Firefox? The zh-tw localization sets the
> pref to true.

Sure. Traditional Chinese version (zh-tw).

Comment 9

[Jeff Tong]

I have just tested Firefox 5.0 Traditional Chinese version. I am disappointed that the problem still remains.

Comment 10

[Masatoshi Kimura [:emk]]

Attachment: patch

Comment 11

[Masatoshi Kimura [:emk]]

Comment on attachment 572345 [details] [diff] [review]
patch

Unbitrotted and switching the reviewer

Comment 12

[Boris Zbarsky [:bzbarsky]]

Comment on attachment 572345 [details] [diff] [review]
patch

Please parenthesize the "aFixupFlags & FIXUP_FLAG_USE_UTF8" bit.

Also please make sure the l10n impact of this gets flagged; localizations that set the old pref probably need to set the new one.

r=me

Comment 13

[Masatoshi Kimura [:emk]]

> Also please make sure the l10n impact of this gets flagged; localizations that set the old pref probably need to set the new one.
How can I flag for the l10n impact?

zh-TW is the only locale which modified the pref as far as I know.
https://mxr.mozilla.org/l10n-central/search?string=encode-query-utf8
Tim, please replace "network.standard-url.encode-query-utf8" with "browser.fixup.use-utf8" when this patch is landed. It will fix this bug and will not regress bug 443588.

Comment 14

[Masatoshi Kimura [:emk]]

It's unwise to land this just before the aurora merge (meaning l10n freeze).

Comment 15

[Masatoshi Kimura [:emk]]

Attachment: patch for check in

Comment 16

[Dão Gottwald [:dao]]

http://hg.mozilla.org/integration/mozilla-inbound/rev/0246515e5fda

Comment 17

[Peter Pin-Guang Chen [:petercpg] (MozTW.org)]

Attachment: patch for l10n-central/zh-TW

double check before landing

Comment 18

[Axel Hecht [:Pike]]

Peter, anyone you want to review your change in particular?

Also, you probably want a different comment, as you're not working around something, you're merely adjusting a setting to a good local default, I guess?

Comment 19

[Peter Pin-Guang Chen [:petercpg] (MozTW.org)]

Attachment: patch for l10n-central/zh-TW, v2

(In reply to Axel Hecht [:Pike] from comment #18)
> Peter, anyone you want to review your change in particular?
> 
> Also, you probably want a different comment, as you're not working around
> something, you're merely adjusting a setting to a good local default, I
> guess?

Maybe you Pike, or emk could review for this? sorry I did not add requestee. 

And that's why I need to make sure in case there's still something missing.

Comment 20

[Masatoshi Kimura [:emk]]

Comment on attachment 573198 [details] [diff] [review]
patch for l10n-central/zh-TW, v2

You don't have to change the comment. This is still a workaround for bug 443588.
This bug will be fixed regardless of your settings.
Otherwise looks good.

Comment 21

[Masatoshi Kimura [:emk]]

Comment on attachment 573198 [details] [diff] [review]
patch for l10n-central/zh-TW, v2

Ah, Axel is right about this is no longer a workaround because bug 443588 is Traditional Chinese specific.
Change the comment to "Fix bug 443588".

Comment 22

[Peter Pin-Guang Chen [:petercpg] (MozTW.org)]

Landed: http://hg.mozilla.org/l10n-central/zh-TW/rev/6b71288595d2

Comment 23

[Marco Bonardo [:mak]]

https://hg.mozilla.org/mozilla-central/rev/0246515e5fda