41009 - "Edit Message As New" is quoting non-ASCII mails with no or wrong MIME charset info as garbage

Status: VERIFIED FIXED

(MailNews Core :: Internationalization, defect, P2)

Description

[ji]

Build: 2000053008 comm build

"Message | Edit Message As New" is quoting Ja mails as garbage.

Steps of reproduce:
1. Unzip the smoketest folder at the above URL and put the folder under
   your testing account.
2. Go to the 2nd mail and select Message | Edit Message As New.
   You'll see the Japanese characters in the mail body is quoted as garbage.

Comment 1

[ji]

Attachment: Attached a screen shot.

Comment 2

[ji]

UTF8 mail looks fine (the 4th mail in the smoketest folder).
This problem is observed with the 2nd mail, which is in iso-2022-jp encoding.

Comment 3

[Katsuhiko Momoi]

We also don't see a checkmark against an encoding menu
when it fails like this. 

Comment 4

[nhottanscp]

Reassign to selmer.
I think the charset conversion (from mail charset to UTF-8) is missing.

Comment 5

[scottputterman]

reassigning to rhp, cc'ing jefft (I can never remember who worked on this).
Please reassign if I got it wrong.

Comment 6

[rhp (gone)]

Putting it on the heap.

- rhp

Comment 7

[rhp (gone)]

*** Bug 43084 has been marked as a duplicate of this bug. ***

Comment 8

[rhp (gone)]

Ok, I see what the problem is here and I have a fix for it now in my tree.

- rhp

Comment 9

[rhp (gone)]

per selmer and mail/news PDT.

Comment 10

[Katsuhiko Momoi]

It is curiosu that people who don't use Japanese mail are marking
a critical bug like this as "correctness". 
If you have a fix, I want it in the product. This is a feature a lot of users
including myself use often/daily to send out mail with slightly different 
text or wording. 
This feature does not work for anything other than ASCII or Latin 1 mail
right now. 

That is not "correctness". That is called "broken" for all
languages which don't use Latin 1 encoding.

Renominating for nsbeta2.
If you have a fix, please put it in the product.

Comment 11

[rhp (gone)]

Who are you yelling at here? I have a fix and I'll check it into whatever tree 
you want, but I'm starting on the tip.

- rhp

Comment 12

[Katsuhiko Momoi]

Rich, I apologize. 
I didn't realize that this bug has not been nominated for nsbeta2
before this. I had thought it had already been done so.
I would like to see this done for beta2. So I'm nominating it now.

Comment 13

[rhp (gone)]

Ok, this is fixed now on the tip, but I will just hold this here in case we 
want this checked in on the branch.

- rhp

Comment 14

[Jay Patel [:jay]]

Putting on [dogfood-] radar.  Not critical to everyday use. Putting on 
[nsbeta2-] radar. Not critical to beta2. Editing drafts should be working, and 
this is a subtle feature.  There is a workaround by saving as a draft and 
opening the draft for editing.

Comment 15

[rhp (gone)]

Updating status.

- rhp

Comment 16

[rhp (gone)]

Per selmer and mail/news PDT.

- rhp

Comment 17

[rhp (gone)]

Checked this fix into the tip.

- rhp

Comment 18

[Katsuhiko Momoi]

There is no workaround to this.
Saving as draft and trying to edit it has exactly the same problem.

Comment 19

[Katsuhiko Momoi]

As the suggested workaround simply does not work for this problem,
I'm requesting reconsideration.

Comment 20

[rhp (gone)]

Can we please end the churn on this one bug.

- rhp

Comment 21

[leger]

Not a major feature. [nsbeta2-] for branch.  If fixed in Trunk, marked FIXED to 
verify.

Comment 22

[rhp (gone)]

Yes, this is fixed in the trunk. Checked in the changes 7/26.

- rhp

Comment 23

[Daniel Veditz [:dveditz]]

Adding keyword to bugs with nsbeta3 triage value in status whiteboard so 
tracking queries will not be misled

Comment 24

[Katsuhiko Momoi]

** Observed with 8/10/2000 Win32 M18 build **

We finally shipped Beta 2 and I now have a bit of room to see what
happened to this bug.
I guess I need to re-open this bug.

The fix seems to be working for the following case:

1. When the message contains charset info in the headers.

It does not work for the following type of cases:

2. The message is displaying correctly because the view default charset works OK
   for that message but it has no MIME charset info in the headers.
3. The message contains incorrect MIME charset info in the headers but
   was corrected by a manual charset override and now it shows correctly and
   the menu checkmark also is correct.

In 2 and 3, "Message as New" fails for both the header and body, neither
of which has MIME charset info.

Comment 25

[rhp (gone)]

Uh, at this point I don't think we are going to be able to handle the 
situations where messages are not labled properly or labeled improperly. 

Opinions from others?

- rhp

Comment 26

[Katsuhiko Momoi]

Some facts to consider:

1. Both types of messages do exist.
2. A user will have a hard time understanding why a message 
   we can diaplay properly cannot be edited as new properly. 

Comment 27

[rhp (gone)]

I understand that they do exist...but the issue is that we are at beta3 now and 
lots of stuff that does exist/occur is being Latered. How did 4.x handle this 
issue? Are we worse than that?

- rhp

Comment 28

[Katsuhiko Momoi]

Talking from an outward appearance point of view,
4.x can handle 2 above with no problem, and 3 above
with partial success probably. In 3, body can
be corrected but subject header may not be easily
correctable. 
So, yes, we are worse off than 4.7x from a phenomenal
point of view.

Comment 29

[rhp (gone)]

Can you point me to specific messages that exhibit #2 and #3 behavior.

Thanks.

- rhp

Comment 30

[Katsuhiko Momoi]

Attachment: A file .zip format. The test messages for #2 above. Set the default Msg vuiew encoding to ISO-2022-JP before using this mailbox file.

Comment 31

[Katsuhiko Momoi]

These messages have no MIME charset info either in headers or for body parts.
But they can be viewed with ISO-2022-JP as the default viewing charset set in
the Pref dalog.

Comment 32

[Katsuhiko Momoi]

Attachment: A file i n .zip format for #3. This file contains several messages which bear incorrect body MIME charset. The header MIME is correct. Correct it with the charset menu.

Comment 33

[Katsuhiko Momoi]

The 2nd file contains messages whose body (not subject header) contains incorrect
charset info. Corect the body view looking at the headers -- some are Japanese
messages. Then try Edit as New.

Comment 34

[lchiang]

P2 per mail triage

Comment 35

[lchiang]

second pass: - per mail triage unfortunately. Workaround is to copy/paste 
contents of mail msg.

Comment 36

[rhp (gone)]

for my sorting...

Comment 37

[rhp (gone)]

Just some personal sorting.

- rhp

Comment 38

[scottputterman]

sorry for the extra email. Removing mail2 keyword.

Comment 39

[scottputterman]

marking nsbeta1- as per I18N triage.

Comment 40

[Katsuhiko Momoi]

Mass change of QA contact to ji for the bugs filed by her.

Comment 41

[ji]

Since this problem only happens with the mails with no or wrong MIME charset 
info, changed the summary accordingly.

Comment 42

[scottputterman]

reassigning to ducarroz

Comment 43

[Jaime Rodriguez, Jr.]

this has been cut a few times during the seamonkey dev cycle. Request we fix it
this time, since it is marked as P1 | Major. please change target milestone to
M0.9.1

Comment 44

[Jaime Rodriguez, Jr.]

renominating for rtm.

Comment 45

[ji]

Modified the summary since the problem is not only for JA mails, it's a problem 
for all the non-ASCII mails with wrong or no MIME charset info.

Comment 46

[marina]

*** Bug 85764 has been marked as a duplicate of this bug. ***

Comment 47

[marina]

Attachment: this is a KOI8-U mail that illustrate the problem

Comment 48

[Denis Antrushin]

Attachment: (quick!) fix, v0.5

Here's a patch, that show source of problem: we're checking only message
headers
for charset.
Patch fixes this, but my feeling is that we just should always use
mdd->mailcharset (it's the same as comp fields' charset), but I need
confirmation from Jean-Francois.

Also, it seems to me that it's possible to simplify
charset->unicode->utf8->unicode body conversion in some cases, need to 
think more carefully...

Comment 49

[Denis Antrushin]

Attachment: (alpha) patch, v0.6

I think that mdd->mailcharset should have precedence over mime headers info
(cause it reflects charset override, isn't?).
Also I tried to remove extra string conversions from unicode to utf8 and back.

Comment 50

[Denis Antrushin]

I got an impression, that mdd->mailcharset is always the same as
bodyCharset - both is set by
MimeHeaders_get_parameter (mdd->messageBody->type, "charset", NULL, NULL);
Strange...

Comment 51

[Denis Antrushin]

Uff! msg window charset override is set, then 
mdd->mailcharset is set to override charset,
although I don't know where :-)

Comment 52

[scottputterman]

Moving out, but if the patch is good, we should get it in.

Comment 53

[nhottanscp]

This is fixed now (probably by bug 66098), please verify.

Comment 54

[ji]

Verified as fixed.