201071 - signature file containing accented characters has become corrupted

Status: VERIFIED FIXED

(MailNews Core :: Internationalization, defect, P1)

Description

[Marc-Etienne VARGENAU]

User-Agent:       Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.4b) Gecko/20030406
Build Identifier: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.4b) Gecko/20030406

My signature file is the following :

-- 
Marc-Etienne Vargenau http://vargenau.free.fr
Membre du Cercle généalogique norvillois http://www.cgnorvillois.org
Mes données sont indexées sur Généanet http://www.geneanet.org

When I create a new message, all 'é' in the signature are displayed as 
a black square.
When trying to send the message, I get an alert telling that the file
contains charcters not in the character set.



Reproducible: Always

Steps to Reproduce:
1. create a signature file (with a text editor) containing accented characters
2. in the preferences, select the file to be the signature file
3. compose a new message
4. send it


Actual Results:  
All accented charcters in the signature are displayed as 
a black square.
When trying to send the message, I get an alert telling that the file
contains charcters not in the character set.

Expected Results:  
The accented characters should have been displayed correctly.
The message should have been sent.

Comment 1

[marina]

confirmin the bug: the corruption happens with plain text signature file (
create it in TextEdit application). Html signature is OK

Comment 2

[Rafael Ebron (:rebron)]

I18N triage: nsbeta1+/adt2

Comment 3

[nhottanscp]

Is this Mac specific?

Comment 4

[marina]

yes, it's Mac specific

Comment 5

[nhottanscp]

So the file name does not have to be non-ASCII?
Is this a regression (from 7.02)?

Comment 6

[marina]

that is correct, the signature file name doesn't have to be necessarily
non-ascii, i believe that this is a regression

Comment 7

[marina]

actually this is not a regression... i see the same problem with rtf format file
in 7.02 rtm

Comment 8

[Marc-Etienne VARGENAU]

I am the reporter.

It *is* a regression in Mozilla.
I do not know about Nestscape.

Regards,

Comment 9

[marina]

Marc-Etienne, what MacOSX version are you using? 10.1.5 or higher? Also, whenthe
regression started? i see you're using 04/06 builds.

Comment 10

[Marc-Etienne VARGENAU]

I use Mac OS X 10.2.6

The signature file works in Mozilla 1.3
It does not in Mozilla 1.4beta.

Regards,

Marc-Etienne

Comment 11

[Christophe Franco]

I can confirm the bug is still there in Mozilla 1.4 final for MacOS X, not in
Mozilla 1.4 final for Windows.

Comment 12

[marina]

the bug was filed for MacOSX, it is still there because it didn't get fixed...

Comment 13

[Marc-Etienne VARGENAU]

Hello,

Is someone doing something to correct this very annoying bug ?

Regards,

Marc-Etienne

Comment 14

[Jungshik Shin]

What encoding (character encoding) is your signature file in? If it's MacOS
10.x, it might be in UTF-8. What character encoding did you use for outgoing
emails when this happened? 
Does changing the char. encoding(View | Character Coding) for outgoing message
to UTF-8 solve the problem? 

I wish I had a Mac to fix this problem, but I don't.  

Comment 15

[Marc-Etienne VARGENAU]

The signature file is a "plain text" file created with BBEDIT.

The char. encoding for outgoing message is ISO 8859-1.

The signature file works in Mozilla 1.3
It does not in Mozilla 1.4

Regards.

Comment 16

[Jungshik Shin]

Thanks for the reply, but you didn't answer any of my questions. I knew your
outgoing message is in ISO-8859-1. The fact that you used BBEdit to create your
signature file didn't answer my question about the encoding of your signature
file created with BBEdit. Being run under Mac OS X, BBEdit may use UTF-8 or
UTF-16BE as the default character coding. Can you attach your signature file
here for me to take a look? In addition, what happens if you set the encoding of
your outgoing message to UTF-8 instead of ISO-8859-1? 

 

Comment 17

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Jungshik:

I think that we should determine the charset by the native charset of the OS and
locale. But we are using file system charset for the text file reading.
On ohter OS, file system charset and default text file charset is same. But on
MacOS X, it is not so. Because if on MacOS X, the file system charset is always
UTF-8.

Comment 18

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Attachment: proposal patch rv1.1

I cannot test this patch. Because I don't have Mac.

Comment 19

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

The patch works fine. It tested by Bugzilla-jp's tester.

Comment 20

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Attachment: Patch rv2.0

This patch makes customizable the charset of signature file by pref.

Comment 21

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Please block the next release.
In Japan, some Mac users are confused by this problem.
We hope that this bug should be fixed before Thunderbird 1.1.

Comment 22

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 189534 [details] [diff] [review]
Patch rv2.0

I have a better idea for backward compatibility.
I'll create a new patch.

Comment 23

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Attachment: Patch rv3.0

This patch allows both UTF-8 and system default encodings for signature files.
So, this has compatibility for older builds on Mac and we support UTF-8 encoding on all platforms.

Comment 24

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

This patch has low risk. We should go to Tb2.0.

Comment 25

[Jungshik Shin]

Comment on attachment 215850 [details] [diff] [review]
Patch rv3.0

>Index: mailnews/base/util/nsMsgI18N.cpp

>+const char * nsMsgI18NTextFileCharset()
>+{
>+  /* Get a charset used for the file. */
>+  static nsCAutoString textFileCharset;

  I don't think we need caching here. This is not perf-critical at all. 



>+  if (sigEncoding.IsEmpty()) {
>+    if (IsUTF8(nsDependentCString(readBuf))) {
>+      sigEncoding.Assign("UTF-8");

Does |readBuf} contain the entire signature file? If not, it can go wrong, can't it?

Comment 26

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Attachment: Patch rv4.0

Updating.

|nsMsgCompose::LoadDataFromFile| is used only for signature file. but the name is not only for signature file. Therefore, I add the new parameter that allow/deny the  text file as UTF-8.

Comment 27

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Attachment: Patch rv4.0

Comment 28

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

jshin:

Could you review this?

Comment 29

[Jungshik Shin]

Comment on attachment 217151 [details] [diff] [review]
Patch rv4.0

sorry for the delay. 

cf. bug 322566. The patch there may have to be changed to use this.

Comment 30

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

checked-in to trunk and 1.8 branch, thanks.

Comment 31

[Christian :Biesinger (don't email me, ping me on IRC)]

+     kPlatformCharsetSel_PlainTextInFile = 7

so... why does this not need any changes to actual nsPlatformCharset implementations?

Comment 32

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

(In reply to comment #31)
> +     kPlatformCharsetSel_PlainTextInFile = 7
> 
> so... why does this not need any changes to actual nsPlatformCharset
> implementations?

I need only system default charset. The all implementations returns it if the param is not unknown. Therefore, I need suitable param value.

Comment 33

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

(In reply to comment #32)
> if the param is not unknown.

Sorry, "if the param is unknown."

Comment 34

[Hiro]

verified.

Mac OS X 10.3.9
version 2 alpha 1 (20060411)

Windows XP SP1
version 2 alpha 1 (20060410)