208363 - mozilla -compose cannot interpret UTF-8 String correctly

Status: RESOLVED FIXED

(SeaMonkey :: General, defect)

Description

[Joe]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4b) Gecko/20030507
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4b) Gecko/20030507

mozilla -compose commandline allow user to add body and subject to the message
compose window, but currently it cannot correctly display bodies and subjects
encoded with UTF-8. 

Reproducible: Always

Steps to Reproduce:
1.run the following commandline:
$mozilla -compose body="some UTF-8 String"
2.
3.

Actual Results:  
If the UTF-8 String contains for example chinese characters, those characters
won't be displayed correctly

Expected Results:  
correctly displays all the characters encoded by UTF-8

Not sure which charset is assumed by the mozilla commandline. the String objects
and conversions in nsAppRunner are some kind of old fashion. I will submit a
patch for handling all the arguments input from commandline as UTF-8 String.

Comment 1

[Joe]

Attachment: accept commandline arguments as UTF-8 String

This may not be a good enough solution, just paste for further discussions.

Comment 2

[Joe]

-compose commandline works fine with the patch.
The key point here is whether treating all the commandline arguments as UTF-8
String will affect behaviors of other current commands. 
Personally I think make UTF-8 the default "commandline charset" is reasonable.

Comment 3

[Joe]

add i18n expert to the cc list

Comment 4

[Dan Mosedale (:dmosedale, :dmose)]

Comment on attachment 124965 [details] [diff] [review]
accept commandline arguments as UTF-8 String

I'm not 100% sure this is the right fix.  Should we really be using the system
native charset rather than UTF-8?  However, it seems unlikely to be worse than
what we've got now, so if darin and smontagu are OK with it,
sr=[e-mail]

Comment 5

[Simon Montagu :smontagu]

Comment on attachment 124965 [details] [diff] [review]
accept commandline arguments as UTF-8 String

We should definitely be using the system default charset. What we do now
(implicitly assuming ISO-8859-1) is wrong, but forcing UTF-8 may cause a
regression for some people.

Comment 6

[Darin Fisher]

yeah, assuming UTF-8 is way wrong.  you should use mozilla's uconv library to
convert from the platform/native charset to unicode.  see nsIUnicodeEncoder,
nsIPlatformCharset, and nsICharsetConverterManager2.

Comment 7

[Joe]

OK, I understand. But I think there should be some way for the commandline to
accept utf8 String directly. Assume the condition that the system default
charset is ISO-8859-1, while I pass a utf8 String contains Chinese characters, I
could lost the charset information.

what about a switch like -utf8? with this switch, commandline would accept
arguments as utf8 String. or maybe there should be a switch and arg like:
-charset iso-8859-1 or -charset utf8.

What's your opinion?

Comment 8

[Darin Fisher]

if you want to pass an URL with UTF-8 characters through the command line then
if the platform charset is not UTF-8 you really must URL escape the non-ASCII
bytes.  i don't think it is worth it to add a switch to specify the command line
charset.  the command line charset should simply match the platform charset
(i.e., whatever nsIPlatformCharset would provide).

Comment 9

[Alec Flett]

as discussed with smontagu, the command line has a well-understood contract that
arguments are in the platform charset. If an application is passing around UTF8
command line arguments, then it is the application that is broken..

However, we have our own bugs as this conversation describes. We should be going
through the nsICharsetConverterManager to convert from the platform charset when
processing an argument. One approach to reduce the amount of code in
nsAppRunner.cpp is to make nsICommandLineService give out unicode or UTF8
strings, thereby sheilding the consumers of nsICommandlineService from any kind
of string conversions..

I'd personally vote for nsICommandLineService to give out UTF8 (rather than
unicode) and use jshin's nsIUTF8ConverterService that he'll be landing soon in
bug 162765. That bug also has UTF8 detection routines which we could use to
detect UTF8 if it is (incorrectly) passed in on the command line.

Comment 10

[Joe]

In my case of using mozilla commandline, I launched mozilla -compose
commandline from Java code, where the body and subject arguments would be
encoded by UTF-8. I don't it's incorrect for Java application to pass UTF-8
strings, the limitation of commandline to platform charset, would badly
affect the ability of multi-language supported application, which was trying to
launch mozilla by commandline.

Maybe I should make the question much simpler: How could I pass, say, chinese
characters, to mozilla by commandline when the platform charset is for example
ISO-8859? 
Any suggestions?

BTW: the commandline I used looks like:
$mozilla -compose to=somebody@somewhere,subject=hi,body=Hello

Comment 11

[Darin Fisher]

Joe: have you tried URL escaping the non-ASCII bytes?

mozilla -P test -compose
to=somebody@somewhere,subject=hi%20how%20are%20you,body=Hello

results in a subject that reads "hi how are you"

i'm not sure exactly how moz will treat unescaped bytes, but perhaps it can be
made to work.

Comment 12

[Simon Montagu :smontagu]

Another workaround for the command line is to find (or write) a tool which
converts UTF-8 to URL-encoding, and then use a command line like

./mozilla -compose body=$(urlencode <random utf-8 string>)

But if you are doing this from Java you can do the URL-encoding internally and
you won't need that :-)

Comment 13

[Darin Fisher]

looks like the compose command line handler only recognizes certain escape
sequences like %20, but not others.  i'm not sure why.  we need to ask someone
on the mailnews team.

Comment 14

[Jungshik Shin]

I just came across this bug while looking for other bugs :-). Anyway, I'll try 
to implement what Alec suggested in comment #9. 
 

Comment 15

[Asa Dotzler [:asa]]

*** Bug 229016 has been marked as a duplicate of this bug. ***

Comment 16

[Andrew Schultz]

*** Bug 291294 has been marked as a duplicate of this bug. ***

Comment 17

[Simon Montagu :smontagu]

Attachment: Convert command line arguments from native charset to unicode

Following discussion on #seamonkey, we probably don't want to go to the trouble of modifying nsICommandLineService if it's soon going to be superseded by the toolkit command line handler. This patch just NS_CopyNativeToUnicodes the command line arguments.

Comment 18

[[e-mail]]

Comment on attachment 230481 [details] [diff] [review]
Convert command line arguments from native charset to unicode

This is the right thing to do by comparison with DoCommandLines().

Comment 19

[Simon Montagu :smontagu]

Checked in.

Comment 20

[Robert Kaiser]

I wouldn't block the release by this, but I think it's fine if it goes in :)

Comment 21

[[e-mail]]

This doesn't work on the branch, there's a dependency (bug 58523?)
Backing out for now.

Comment 22

[[e-mail]]

Comment on attachment 230481 [details] [diff] [review]
Convert command line arguments from native charset to unicode

Rerequesting approval bearing in mind that this patch depends on one in bug 305949 or 312287 (I can't remember which).