115096 - In reply composition, RFC2047 encoded characters of Subject are mis-interpreted if not in charset of body

Status: VERIFIED FIXED

(MailNews Core :: MIME, defect)

Description

[Jean-Marc Desperrier]

There's a bit of i18n involved, but it more seems like a problem in MIME.

I read a news message with the header in ISO-8859-15 encoded after RFC2047,
but the body in ISO-8859-1.

Inside the content of the header, there's 6 Euro symbols.

When answering the message, in the title of the message in the composing
windows, the 6 Euro symbols are replaced by 6 neutral monetary symbol.

If I then send the message, it's sent in ISO-8859-1, header and content, and the
title has the 6 Euro replaced by 6 neutral monetary symbols.

The main difference between ISO-8859-1 and ISO-8859-15 is that the code 
point that was neutral monetary symbol in ISO-8859-1 is now the Euro
symbol in ISO-8859-15.

It sounds like when the data inside the title is decoded, it is not
tagged as being ISO-8859-15, and gets converted to unicode using the 
charset of the body instead of using the charset that was inside the
RFC2047 encoding.

I join the original message as an attachment.

Comment 1

[Jean-Marc Desperrier]

Attachment: the message that causes the problem

Comment 2

[Jean-Marc Desperrier]

Setting the default encoding of the folder to ISO-8859-15 instead of ISO-8859-1
changes nothing to the problem.

Comment 3

[Boris 'pi' Piwinger]

I see it in Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.9+) Gecko/20020426.

pi

Comment 4

[Jean-Francois Ducarroz]

reassign to nhotta

Comment 5

[nhottanscp]

For reply, the charset of the original mail body is used for both header and body.
But I think this is fixed by bug 109342 because it implemented a fallback
mechanism for charset conversion failure.

Comment 6

[Jean-Marc Desperrier]

Reopening. This is still not corrected in 2003012908.

Reproduction Steps :
- Open message in mailer 
- The title has 6 euro symbols
- Create a reply (CTRL-R)
- The title in the message composition windows has 6 neutral monetary symbol

The original sample was using euro characters, so it looked like it was a euro
problem. In fact, it has nothing to see with Euro.

I have a new sample of the problem, that has in the subject :
Subject: Re: Raw UTF8 - =?windows-1251?Q?=CA=E8=F0=E8=EB=EB=E8=F6=E0?=
and a content type of :
Content-Type: text/plain; charset=us-ascii; format=flowed

With the reproduction step before the title is initially :
Re: Raw UTF8 - Кириллица

And in the reply composition window, the title is changed to : 
Re: Raw UTF8 - Кириллица

The charset of the original data was windows-1251, but in the title of the reply
composition windows, it is interpreted as if it were something else.

Comment 7

[Jean-Marc Desperrier]

Attachment: New sample with cyrillic data

Comment 8

[Igor Velkov]

I can confirm - not only for UTF. Every time, when mail body and subject have
differnt charsets, making reply converts subect as string with charset like in
mail body.

If subject is russian (koi80r, or windows-1251), message body is 8859-1 then
subject converts to the junk.

Comment 9

[Boris 'pi' Piwinger]

*** Bug 191885 has been marked as a duplicate of this bug. ***

Comment 10

[Boris 'pi' Piwinger]

This bug means, data is lost (namely all the information in the subject of the
reply). Setting keyword. Dataloss by definition is critical.

This bug really is bad for MailNews. Such an important release as upcoming 1.4
must not have it. Requesting blocking.

pi

Comment 11

[Asa Dotzler [:asa]]

mozilla1.4 shipped. unsetting blocking1.4 request.

Comment 12

[Boris 'pi' Piwinger]

This is very elementary MIME functionality. Asking for new blocking.

pi

Comment 13

[Asa Dotzler [:asa]]

bienvenu or sspitzer, can one of you take a look at this and see if there's a
quick fix we could get for beta?

Comment 14

[David :Bienvenu]

well, one possibility would be to use nsIMsgHdr::GetMime2DecodedSubject, which
is what the view code does, and that seems to work. Not sure why because the
code looks very similar. I'd have to step through to see what's different.

Comment 15

[David :Bienvenu]

Attachment: proposed fix

using GetMime2DecodedSubject seems to fix the problem and cleans up the code a
bit.

Comment 16

[David :Bienvenu]

Attachment: proposed fix without reply on top sig quoting

whoops, don't need the reply on top sig quoting stuff.

Comment 17

[(not reading, please use [e-mail] instead)]

Comment on attachment 129382 [details] [diff] [review]
proposed fix without reply on top sig quoting

r/a=sspitzer for 1.5 beta

Comment 18

[(not reading, please use [e-mail] instead)]

Comment on attachment 129382 [details] [diff] [review]
proposed fix without reply on top sig quoting

r/a=sspitzer for 1.5 beta

Comment 19

[David :Bienvenu]

fix checked in.

Comment 20

[Jean-Marc Desperrier]

Verified/Fixed with 2003081104 Windows 2000

Can't believe this bug that stayed for so long was that easy to fix.
I should have upped the priority early.

Watching the patch, I can't help, but think that this kind of errors occurs
mostly because the string/MIME API is too complex, and a better conceived API
would reduce the probility of such errors. 
I hope to be able to put my money where my mouth is someday, and suggest
changes, but that would mean a major rewrite.

Comment 21

[Boris 'pi' Piwinger]

FWIW: Also verifying:
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5b) Gecko/20030818

pi

Comment 22

[Asa Dotzler [:asa]]

Comment on attachment 129382 [details] [diff] [review]
proposed fix without reply on top sig quoting

This is not going to make 1.4.1.  Please re-request aproval after 1.4.1 ships
if you'd like to get this in for 1.4.2.