307648 - International/accented character support for extensions/JavaScript is broken

Status: RESOLVED INVALID

(Firefox :: Toolbars and Customization, defect)

Description

[Nem Mondomel]

User-Agent:       Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.8b4) Gecko/20050908 Firefox/1.4

The international character support for extensions and/or JavaScript is somehow 
broken between Deer Park Alpha 2 and Firefox 1.4 (1.5). Extensions using 
international chars in inline JS-constants can display these strings (messages) 
using for ex. an alert() call properly in Firefox 1.0.x and Deer Park alpha 2 
versions, but in FF 1.4/1.5b1 the browser displays question marks in place of 
the accented/non-US-ASCII characters. The problem seems to affected JS-constant 
strings only - strings hardcoded inside XUL forms seem to display correctly, 
though. If you try to display the same string in a plain HTML page using alert
() (and not from inside an extenions), it also works correctly.

Reproducible: Always

Steps to Reproduce:
1. Download and install the extension specified at the homepage URL
2. Press the rightmost button (labeled "Belépés")
3. Check the contents of the displayed messagebox

Actual Results:  
In FF 1.4/1.5b1 "?" characters are displayed where some accented chars should 
be shown. US-ASCII chars are displayed correctly, though.

Expected Results:  
Display accented/international characters correctly

The problem is present in the newest FF 1.5b1 (which has 1.4 in it's user-agent 
string as version) in the US-EN version (no other version available now), but 
I've also tested it previously with yesterdays 1.4 nightly build in HU-hu 
(hungarian) language version, which had the bug, too. Previous FF versions 
(1.0.x, DPa2) display the international strings correctly in both EN-us and HU-
hu language versions.

Comment 1

[Simon Montagu :smontagu]

Can you attach a minimized testcase?

Comment 2

[Nem Mondomel]

Attachment: Minimal extension demonstrating the bug

Click the "Click me!" button on the toolbar to get the messagebox shown. In
Firefox versions before 1.4 it displays accented chars correctly, but in FF
1.4+ only "?" marks are shown in place of international chars.

Comment 3

[Simon Montagu :smontagu]

I think this change was caused by the fix to bug 241739, but this is not a
regression: the previous behaviour was a bug, even if it gave the expected
results. You should either use encode your source files in UTF-8 or correctly
declare whatever other encoding you are using.

Comment 4

[Christian :Biesinger (don't email me, ping me on IRC)]

note also bug 297625

hmm... it looks like XUL doesn't currently support specifying the charset for
<script>. that's not ideal... but you should be able to set the charset of the
XUL file itself, like <?xml version="1.0"? encoding="ISO-8859-1">


Anyway, this bug as written is invalid. but someone should maybe write on
developer.mozilla.org about this...

Comment 5

[Nem Mondomel]

(In reply to comment #3)
> I think this change was caused by the fix to bug 241739, but this is not a
> regression: the previous behaviour was a bug, even if it gave the expected
> results. You should either use encode your source files in UTF-8 or correctly
> declare whatever other encoding you are using.
If I encode source files in UTF-8, it will display incorrecly in previous FF 
versions, right?

Comment 6

[Christian :Biesinger (don't email me, ping me on IRC)]

hm, yeah... you could use unicode escapes (\u12ab)
or .properties files :)

Comment 7

[Nem Mondomel]

(In reply to comment #6)
> hm, yeah... you could use unicode escapes (\u12ab)
> or .properties files :)
I used originally String.fromCharCode(), but \u will be far more compact and 
readable. Thanks.

Comment 8

[Christian :Biesinger (don't email me, ping me on IRC)]

(this change is now documented at
http://developer.mozilla.org/en/docs/International_characters_in_XUL_JavaScript)

Comment 9

[Jungshik Shin]

Thanks for your write-up. Just one thing:

> UTF-8, which can represent almost all characters in the world

Perhaps, it's me but 'virtually' appears to reflect the status of the Unicode
better than    'almost'. 

Comment 10

[Christian :Biesinger (don't email me, ping me on IRC)]

ok, made that change. feel free to do such changes yourself :)