52248 - signature encoding

Status: RESOLVED WORKSFORME

(MailNews Core :: Internationalization, defect, P3)

Description

[Richard]

Some conversion is applied to international characters in signature file when
 it is attached and signature text gets mangled.  For example I have signature
 in UTF-8 and when I attempt to use it text appears to be wrong even if I have
 UTF-8 as default encoding for sending messages.

Mozilla/5.0 (X11; U; FreeBSD 4.1-STABLE i386; en-US; m18) Gecko/20000911.
XFree 4.0.1
GTK 1.2.8

Comment 1

[Katsuhiko Momoi]

I am sure that this problem exists but I don't know if
we should fix it. 

Currently, we assume that the signature text file is
in the system encoding. So the assumption is that if you
set your locale to a UTF-8 one, this should work. 
If not, the text will be mangled. 

Try the following experiments, please:

1. Instead of designating the signature in the Mail/News account
   settings dialog, don't set up any signature file. 
   Now simply use the attach function in the Mail Composer to add  
   a sigature text. Does it have a problem when you receive it?
   I doubt that it would. 

2. Instead of C or Posix locale, use a legitimate UTF-8 locale
   that is properly installed on your system. Now use the Acct
   setting to designate the signature file. Use it under
   UTF-8 send default option. This should work. If this does not,
   the it would be a bug. 

Comment 2

[Richard]

1. Sending file as attachement is OK though it gets application/octet-stream type
2. Admittedly UTF-8 support in freebsd libc isn't perfect. Though
 x programs don't complain about unsupported locale if I set LANG=lt_LT.UTF-8.
 freebsd libc itself (though you can use libutf8) doesn't have wchar functions
 or nl_langinfo(CODESET), only Plan9 rune functions.
 When I run mozilla with LANG=lt_LT.UTF-8 and  LC_* unset I can input utf-8 but
 mozilla shows each byte as separate character. setlocale is OK:
 # env -i LANG=lt_LT.UTF-8 LC_CTYPE= ./libc_locale_test
 setlocale (LC_ALL, "") returns lt_LT.UTF-8
 setlocale (LC_COLLATE, "") returns lt_LT.UTF-8
 setlocale (LC_CTYPE, "") returns lt_LT.UTF-8
 setlocale (LC_MESSAGES, "") returns lt_LT.UTF-8
 setlocale (LC_MONETARY, "") returns lt_LT.UTF-8
 setlocale (LC_NUMERIC, "") returns lt_LT.UTF-8
 setlocale (LC_TIME, "") returns lt_LT.UTF-8
 The same happens even if I replace /usr/share/locale/lt_LT.UTF-8/LC_CTYPE
 with symlink to iso8859-1 locale.

 When I set LC_CTYPE=C in adition I can input utf-8 characters and they show up
 normally - except signature file that always shows wrong like utf-8 on iso8859-1
 terminal even if select file again from scratch. setlocale() also works:
 $ env -i LANG=lt_LT.UTF-8 LC_CTYPE=C ./libc_locale_test
 setlocale (LC_ALL, "") returns
lt_LT.UTF-8/C/lt_LT.UTF-8/lt_LT.UTF-8/lt_LT.UTF-8/lt_LT.UTF-8
 setlocale (LC_COLLATE, "") returns lt_LT.UTF-8
 setlocale (LC_CTYPE, "") returns C
 setlocale (LC_MESSAGES, "") returns lt_LT.UTF-8
 setlocale (LC_MONETARY, "") returns lt_LT.UTF-8
 setlocale (LC_NUMERIC, "") returns lt_LT.UTF-8
 setlocale (LC_TIME, "") returns lt_LT.UTF-8

  Can't figure out what logic mozilla uses to get locale or signature
 file encoding.  Many other unicode aware programs get it from $LANG codeset
 part after dot.

Comment 3

[nhottanscp]

For unix, please look at nsUNIXCharset::GetCharset in nsUNIXCharset.cpp. That is 
used for getting a system charset.
http://lxr.mozilla.org/seamonkey/source/intl/uconv/src/nsUNIXCharset.cpp#108

Comment 4

[Katsuhiko Momoi]

As to problem in 1 (application/octet-stream), would it help
if you name the signature file with the .txt extension, e.g.
mysignature.txt?

Comment 5

[Katsuhiko Momoi]

QA contact to marina. Thanks!

Comment 6

[Richard]

After looking at that source I have added
 locale.all.lt_LT.UTF-8=UTF-8 to res/unixcharset.properties
 and now both input,display and signature looks normaly with
 LANG=lt_LT.UTF-8 and LC_CTYPE unset (still wonder why UTF-8 input&display
worked before with LC_CTYPE=C LANG=lt_LT.UTF-8 ).
 But it really should work in more general way!, for example
 locale.all.*.UTF-8=UTF-8
 Current res/unixcharset.properties completely misses everchanging XFree
 locales & aliases not to mention that user can always add his own one.

 About attachment type: yes, adding .txt extention changes mime type to text/plain.

Comment 7

[nhottanscp]

Reassign to ftang, the current issue is about unixcharset.properties.

Comment 8

[Katsuhiko Momoi]

Sorry, Unix issue. The QA contact should go to ji.

Comment 9

[Frank Tang]

Richard- do you want to rewrite nsUNIXCharset.cpp to make it more generic ?
Give me a list of name/value pair that we should add to unixcharset.properties 
untill someone fix the nsUNIXCharet.cpp.

Comment 10

[Richard]

OK, I'm making the patch for it now.

Comment 11

[Richard]

Attachment: Here is locale list from XFree86 4.01

Comment 12

[Richard]

Attachment: and here is locale aliases list

Comment 13

[Richard]

Attachment: Here a patch for your review

Comment 14

[Richard]

Attachment: Here are aditional entries for unixcharset.properties. Not sure if this completely matches Mozilla's supported unix charset list.

Comment 15

[Frank Tang]

rch- Please send your patch to brendan or waterson for approval. Thanks
shanjian- can you review and verify the patch ? Thanks.

Comment 16

[Shanjian Li]

I think the patch is OK. Please ask waterson or brendan for approval.
(Sorry about the late response.) 

Comment 17

[Richard]

I have sent that patch to brendan two weeks ago.

Comment 18

[Chris Waterson]

a=waterson

Comment 19

[Richard]

Hi all,

If everybody approves this can somebody commit this patch?

	Thank you.

Comment 20

[Frank Tang]

reassign to bstell

Comment 21

[Frank Tang]

Do we still need this? I don't think so. Move it to future. 

Comment 22

[kill this account]

--> ftang

Comment 23

[Frank Tang]

bulk move NEW FUTURE bug to ASSIGN

Comment 24

[Boris 'pi' Piwinger]

*** Bug 172770 has been marked as a duplicate of this bug. ***

Comment 25

[Ilmar Kerm]

I have the same problem with mozilla 1.4a and... windows xp.
And I can't set windows system encoding to utf8, so i should be able to select
signatures encoding in mozilla.

Comment 26

[Mike Cowperthwaite]

Ilmar Kerm, for plain-text, UTF-8 sigs under Windows, see bug 153855.

Comment 27

[Andrew Church]

*** Bug 240281 has been marked as a duplicate of this bug. ***

Comment 28

[Nicolas Dietrich [:nidi]]

problem still exists in 
- thunderbird 0.7a (Linux / 20040614)
- seamonkey Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8a2) Gecko/20040601

Comment 29

[Frank Tang]

what a hack. I have not touch mozilla code for 2 years. I didn't read these bugs
for 2 years. And they are still there. Just close them as won't fix to clean up.

Comment 30

[Travis Chase]

Mass Reassign Please excuse the spam

Comment 31

[Travis Chase]

Mass Re-opening Bugs Frank Tang Closed on Wensday March 02 for no reason, all
the spam is his fault feel free to tar and feather him

Comment 32

[Travis Chase]

Reassigning Franks old bugs to Jungshik Shin for triage - Sorry for spam

Comment 33

[JongAm Park]

*** Bug 320600 has been marked as a duplicate of this bug. ***

Comment 34

[Mike Cowperthwaite]

(In reply to comment #0)
> Some conversion is applied to international characters in signature file when
> it is attached and signature text gets mangled.  For example I have signature
> in UTF-8 and when I attempt to use it text appears to be wrong even if I have
> UTF-8 as default encoding for sending messages.

Bug 201071 added automatic recognition of UTF-8 in a sig file, otherwise defaulting to the system default encoding.  Is that patch sufficient to address this bug?  (Altho the bug is marked Mac, it was also fixed in Windows and, I assume, all platforms.)

Comment 35

[JongAm Park]

It is not solved even for the Windows.
Even when you set the outgoing mail's default encoding to UTF-8, the content of  signature file in UTF-8 mode is displayed broken.

I wish this could be solved soon.

(In reply to comment #34)
> (In reply to comment #0)
> > Some conversion is applied to international characters in signature file when
> > it is attached and signature text gets mangled.  For example I have signature
> > in UTF-8 and when I attempt to use it text appears to be wrong even if I have
> > UTF-8 as default encoding for sending messages.
> 
> Bug 201071 added automatic recognition of UTF-8 in a sig file, otherwise
> defaulting to the system default encoding.  Is that patch sufficient to address
> this bug?  (Altho the bug is marked Mac, it was also fixed in Windows and, I
> assume, all platforms.)
> 

Comment 36

[Mike Cowperthwaite]

(In reply to comment #35)
> It is not solved even for the Windows.
> Even when you set the outgoing mail's default encoding to UTF-8, the content
> of signature file in UTF-8 mode is displayed broken.
> 
> I wish this could be solved soon.

And you tested that with *which* version of software?

Comment 37

[JongAm Park]

It is the version 1.5.0.8 (20061128).

(In reply to comment #36)
> (In reply to comment #35)
> > It is not solved even for the Windows.
> > Even when you set the outgoing mail's default encoding to UTF-8, the content
> > of signature file in UTF-8 mode is displayed broken.
> > 
> > I wish this could be solved soon.
> 
> And you tested that with *which* version of software?
> 

Comment 38

[Raffael Luthiger]

(In reply to comment #36)
> And you tested that with *which* version of software?
> 
I tested it with Thunderbird 1.5.0.9 (20061207). Still here...

Comment 39

[Simon Montagu :smontagu]

It works in Thunderbird 2.0.0.14 (20080421) but not 1.5.0.14 (20071210), which is strange because bug 201071 was supposed to have been checked into both trunk and branch.