845743 - Firefox 19.0 does not support BIG5-HKSCS encoding. Version 18.0 does

Status: RESOLVED FIXED

(Core :: Internationalization, defect)

Description

[arbiant]

User Agent: Mozilla/5.0 (Windows NT 6.1; rv:18.0) Gecko/20100101 Firefox/18.0
Build ID: 20121231071231

Steps to reproduce:

does not show HK special character although webpage set with charset=BIG5-HKSCS


Actual results:

鰂 show as 鰂


Expected results:

should show as 鰂 under charset=BIG5-HKSCS. It works on firefox 18.0 but not version 19.0

Comment 1

[Loic]

Attachment: Testcase

WFM here with the testcase.

Comment 2

[arbiant]

Attachment: 3rd character is a big5-hkscs.

The 3rd character is a big5 hkscs character: 鰂 (show correct on firefox v18.0 and Google Chrome). show in Firefox v19 as 裵 (big5).

sorry, my earlier explanation is not quite write. when we enter in 鰂 inside a form (charset=Big5-HKSCS), when server received it translate as 鰂 because Firefox 19.0 treat Big5-HKSCS as Big5, as it not part of Big5 that is why server received it as 'unicode' 鰂

Comment 3

[arbiant]

sorry, mean: my earlier explanation is not quite right.

Comment 4

[Jonathan Kew [:jfkthame]]

Comment on attachment 718896 [details]
Testcase

This testcase isn't useful, as the file is actually in UTF-16LE (despite its charset claim), and moreover it represents the Chinese character with a Unicode numeric entity, so the page's own encoding is not relevant.

Comment 5

[Jonathan Kew [:jfkthame]]

Comment on attachment 718907 [details]
3rd character is a big5-hkscs.

This testcase displays correctly for me (with FF18 on Windows, or with Nightly on OS X) but only if I explicitly choose the Big5-HKSCS encoding from the Character Encoding submenu; when it is initially opened, it displays complete junk: "­»´ä‘o³½¯F ". Maybe because bugzilla serves the testcase with

  Content-Type:text/html; name="test-big5-hkscs.html"; charset=windows-1252

which is clearly the wrong charset for the file.

(Possibly the behavior varies if you're running a localized Chinese version of Firefox, or have your system locale set to Chinese?)

Comment 6

[Henri Sivonen (:hsivonen)]

IIRC, the Encoding Standard unifies big5 and big5-hkscs based on research about existing content and based on IE not recognizing the big5-hkscs label.

Reporter, is this breaking a real site somewhere that works in other browsers?

(If this is about a page you authored yourself, please use UTF-8 instead of a legacy encoding.)

Comment 7

[arbiant]

just need to open the attached file test-big5-hkscs.html on FF19.
the question is, why it works on FF18, and does not works in FF19? 
something might has set wrongly there. 

If Firefox accept this as a bug and do not want to fixed this, we will just use Google Chrome and forget about FF.

Jonathan Kew, as explained it works for FF18 but not FF19.
please try on FF19. you will see the 3rd character is wrong.

Comment 8

[Henri Sivonen (:hsivonen)]

Reporter, is this breaking a Web site somewhere? Which site? Is that site maintained by you?

Does the page work in a Hong Kong-localized version of IE? Does it work in Taiwan-localized or en-US-localized IE?

Comment 9

[Henri Sivonen (:hsivonen)]

Probably caused by bug 801402.

Comment 10

[arbiant]

it's on our internal CRM. long history, data originated big5-hkscs from old desktop database system.
then today our user realize something wrong and reported this problem. 
we don't use IE and only use FF.
if FF don't support HKSCS, we have no choice but to go to google chrome. 
yes, we are looking at converting the data to utf.
by then, we will just use available solution.

just one question, why FF19 suddenly stop support HKSCS?

Comment 11

[Masatoshi Kimura [:emk]]

(In reply to Henri Sivonen (:hsivonen) from comment #9)
> Probably caused by bug 801402.

Yes, I changed it because Encoding Standard defines big5-hkscs as just an alias of big5.

Reporter, please file a spec bug and convince a spec editor of the demand of big5-hkscs as a separate encoding.
https://www.w3.org/Bugs/Public/enter_bug.cgi?product=WHATWG&component=Encoding

Comment 12

[Simon Montagu :smontagu]

I think we should fix this and not wait for the Encoding Standard.

Comment 13

[Jonathan Kew [:jfkthame]]

I'd agree.

(FWIW, if we were going to "unify big5 and big5-hkscs" - which may not be a good idea after all - it seems a bit odd to still have Big5-HKSCS available in the menu, distinct from Big5, yet -not- to recognize it in the charset metadata. I'd have expected that either we support that charset just as we used to, or we eliminate it completely. Continuing to "support" it via the UI yet mapping it to Big5 when it occurs in charset metadata seems a bit confusing.)

Comment 14

[Masatoshi Kimura [:emk]]

Attachment: Support Big5-HKSCS as a separate encoding again

Annevk sees here, so the spec will be updated shortly.

Comment 15

[Masatoshi Kimura [:emk]]

(In reply to Jonathan Kew (:jfkthame) from comment #13)
> (FWIW, if we were going to "unify big5 and big5-hkscs" - which may not be a
> good idea after all - it seems a bit odd to still have Big5-HKSCS available
> in the menu, distinct from Big5, yet -not- to recognize it in the charset
> metadata. I'd have expected that either we support that charset just as we
> used to, or we eliminate it completely. Continuing to "support" it via the
> UI yet mapping it to Big5 when it occurs in charset metadata seems a bit
> confusing.)

I filed bug 805374 to match them.

Comment 16

[Masatoshi Kimura [:emk]]

This revert should be propagated as soon as possible.

Comment 17

[arbiant]

thanks

Comment 18

[Simon Montagu :smontagu]

Comment on attachment 718991 [details] [diff] [review]
Support Big5-HKSCS as a separate encoding again

Review of attachment 718991 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks

Comment 19

[Henri Sivonen (:hsivonen)]

If there are sighting of broken public Web content or information about other broken intranets, please make it known here instead of just concluding that the bug has been reported already.

Comment 20

[Masatoshi Kimura [:emk]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/4cc81e04c718

Comment 21

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/4cc81e04c718

Comment 22

[Masatoshi Kimura [:emk]]

Comment on attachment 718991 [details] [diff] [review]
Support Big5-HKSCS as a separate encoding again

[Approval Request Comment]
Bug caused by (feature/regressing bug #): bug 801402
User impact if declined: Some web pages encoded in Big5-HKSCS will not work correctly.
Testing completed (on m-c, etc.): on m-c
Risk to taking this patch (and alternatives if risky): very low
String or UUID changes made by this patch: none

Comment 23

[Henri Sivonen (:hsivonen)]

(In reply to Masatoshi Kimura [:emk] from comment #22)
> User impact if declined: Some web pages

Well, evidence so far is of one intranet app, only.

Comment 24

[Jonathan Kew [:jfkthame]]

http://w3techs.com/technologies/details/en-b5hkscs/all/all lists a few public websites using Big5-HKSCS.

Not every such page is necessarily affected, as most codepoints in Big5-HKSCS will be the same when interpreted as Big5 (AIUI), but there may be occasional characters that appear incorrectly.

Comment 25

[Anne (:annevk)]

Those pages were studied as far as I know. See also:

http://lists.w3.org/Archives/Public/public-whatwg-archive/2012Apr/thread.html#msg42

It's not entirely clear to me how we support big5 though, if we implemented the changes suggested in the Encoding Standard or if we kept our old weird variant.

Comment 26

[Masatoshi Kimura [:emk]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/00e8a50057cc
https://hg.mozilla.org/releases/mozilla-beta/rev/f200205f2950

Comment 27

[Virgil Dicu [:virgil] [QA]]

Verified the issue for Firefox 19.0 (20130215130331). By using the testcase from comment 2, the 3rd character is 裵. 

Verified the fix for Firefox 20.0 Beta 3 (20130305164032), the 3rd character is 鰂, as expected.

Comment 28

[Simona Badau, Desktop QA]

Mozilla/5.0 (Windows NT 6.1; rv:21.0) Gecko/20100101 Firefox/21.0
Build ID: 20130423212553

Verified as fixed on Firefox 21 beta 3: the shown character is:  鰂.

Comment 29

[Tracy Walker [:tracy]]

mass remove verifyme requests greater than 4 months old