Last Comment Bug 3979 - [BETA] Mail- Content-Type: charset of reply and forward is always us-ascii
: [BETA] Mail- Content-Type: charset of reply and forward is always us-ascii
Product: MailNews Core
Classification: Components
Component: Internationalization (show other bugs)
: Trunk
: x86 Windows NT
P3 normal (vote)
: M14
Assigned To: rhp (gone)
: Katsuhiko Momoi
: 6995 (view as bug list)
Depends on: 28055
Blocks: 7228
  Show dependency treegraph
Reported: 1999-03-18 14:54 PST by nhottanscp
Modified: 2008-07-31 01:22 PDT (History)
6 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---


Description User image nhottanscp 1999-03-18 14:54:10 PST
Content-Type: charset of the original message should be used.
Assign to me (for now).
Comment 1 User image nhottanscp 1999-04-16 15:46:59 PDT
Requested to add charset name support to nsIMessage.
Comment 2 User image nhottanscp 1999-04-16 15:46:59 PDT
Requested to add charset name support to nsIMessage.
Comment 3 User image David :Bienvenu 1999-04-16 15:55:59 PDT
I've added it, but there's no code that sets it to anything other than the empty
string. I'm drowning in IMAP and will be until after M6.
Comment 4 User image nhottanscp 1999-04-19 16:01:59 PDT
Changed to use the new code. Use the value when the string is not empty.
Comment 5 User image nhottanscp 1999-04-23 17:50:59 PDT
Moving to M6, this will not work until nsIMessage change.
Comment 6 User image nhottanscp 1999-05-05 10:48:59 PDT
Reassinging to
Comment 7 User image David :Bienvenu 1999-05-05 10:52:59 PDT
What more do you want me to do? I've added the methods to nsIMessage. I'm
guessing you want me to parse every message for its content type and set the
field appropriately. This is of course impossible for IMAP and News...didn't we
used to rely on libmime having parsed the message in the past?
Comment 8 User image nhottanscp 1999-05-05 11:22:59 PDT
Reassinging to
I don't know who is responsible for reply implementation.
Comment 9 User image Phil Peterson 1999-05-05 13:39:59 PDT
Reassigning to rhp
Comment 10 User image nhottanscp 1999-05-24 15:01:59 PDT
*** Bug 6995 has been marked as a duplicate of this bug. ***
Comment 11 User image rhp (gone) 1999-06-15 16:48:59 PDT
Will look at for M8.
Comment 12 User image rhp (gone) 1999-07-05 22:46:59 PDT
How should this work? Should I just set the charset of the reply to the same
charset of the replied to message? This seems like we could have problems doing
that, but just guessing here.

- rhp
Comment 13 User image nhottanscp 1999-07-06 09:29:59 PDT
For reply and forward, the charset of the original message main body should be
Comment 14 User image rhp (gone) 1999-08-10 19:42:59 PDT
Won't get to this before tonight...kind of a tough problem.

- rhp
Comment 15 User image bobj 1999-08-11 11:54:59 PDT
Naoki,  As a temporary improvement, can we just omit the charset parameter
in the headers for replies and forwards until we get this fixed?
Comment 16 User image nhottanscp 1999-08-11 12:20:59 PDT
It is not usual to send messages without Content-type charset.
I don't think we get workaround by sending without charset, do we?
Comment 17 User image Katsuhiko Momoi 1999-08-11 12:53:59 PDT
Rich, are you proposing to omit the charset parameter from all parts including the 1st of the multi-part bodies?
If so, that wouldn't be a good idea,
Comment 18 User image rhp (gone) 1999-08-11 13:02:59 PDT
I'm not suggesting we change anything...I just latered the bug because this is
not a trivial fix and then bobj and naoki had the other comments.

- rhp
Comment 19 User image Katsuhiko Momoi 1999-08-11 13:53:59 PDT
Sorry, Rich. My question should have been aimed at bobj.
Comment 20 User image nhottanscp 1999-08-11 14:52:59 PDT
I talked to bobj and momoi, bob's suggestion depended the charset override
feature which is not working yet (5938). So we should not omit a charset label
from reply.
Comment 21 User image Phil Peterson 1999-08-19 16:13:59 PDT
Does our 4.x product do what's being suggested here; i.e. use the charset from
the message being replied to, rather than the charset of the author of the

Effectively, I'm asking whether this is an RFE or a regression.
Comment 22 User image nhottanscp 1999-08-20 10:52:59 PDT
I don't know about 'RFE' but this is not a regression of 4.x.
I think 4.x depended on the charset menu selection to decide the charset of
sending out messages.
But this is needed for 5.0.
In 4.x, reply/forward quoted message cound not be displayed unless the charset
menu is set properly.
5.0 uses Content-type charset and usually the user doesn't have to set charset
menu for viewing. Reply/forward also does not require to set charset menu for
viewing/editing but sending out message needs to specify a charset. For initial
charset for reply/forward, it is appropriate to use the charset of the original
message than the current charset menu selection.
I assume confusions if we rely on the menu selection for reply/forward charset
(e.g. sending out with a wrong charset by accident and the receiver cannot see
the message at all).
BTW, there is also a bug of no menu check mark.
Comment 23 User image Phil Peterson 1999-08-20 11:04:59 PDT
Well, two thoughts:

> For initial charset for reply/forward, it is appropriate to use the charset of
> the original message than the current charset menu selection.

I don't understand this. If the reply isn't in the same character set as the
original message, it seems like the charset on the reply is wrong -- i.e.
doesn't reflect the data which is actually in the message.

Second, Rich tells me that changing the MIME parser so that we have the original
character set hanging around in memory is a major change. It's not likely we'll
attempt that before he returns, if at all. TFV -> M13.
Comment 24 User image nhottanscp 1999-08-20 11:41:59 PDT
>I don't understand this. If the reply isn't in the same character set as the
>original message, it seems like the charset on the reply is wrong -- i.e.
>doesn't reflect the data which is actually in the message.
That case (e.g. replying to French message by Japanese), it is users
responsibility to reset the charset. There is also a risk that the original
message may not be mapped to the sending charset if sending out as different
charset than original unless send as unicode (e.g. UTF-8). For that, we're
planning for replacing those unmapped characters as entities (but that's for
HTML only).
Comment 25 User image rhp (gone) 1999-11-02 09:30:59 PST
I'll investigate further when time allows.

- rhp
Comment 26 User image Phil Peterson 1999-12-17 10:25:59 PST
If there's an i18n interop issue here, we should look at it for B1.
Comment 27 User image rhp (gone) 1999-12-17 15:55:59 PST
For reply we do, but not for forward.

- rhp
Comment 28 User image nhottanscp 1999-12-17 16:15:59 PST
So, how much extra work needed or performance increase for getting Content-Type
of the main body in case of we're not quoting?
Comment 29 User image rhp (gone) 2000-01-15 15:11:59 PST
this is a test
Comment 30 User image rhp (gone) 2000-01-21 08:35:30 PST
I have an idea how to fix this problem, but I need to understand one thing. Do 
we need to use the charset of the mail message, or the part that is displayed 
as the primary body part. Please tell me the first one :-) Also, I could use a 
message that has a non us-ascii charset for the top level mail message itself. 
Can you post an example I can work from.


- rhp
Comment 31 User image rhp (gone) 2000-01-21 12:12:25 PST
Finally, after putting my brain into this one, I think I have it fixed. Now, 
when you reply to a non US-ASCII email, you will reply in that charset. 
Forwarding INLINE also has this same behavior, but if you forward as an 
attachment, I do not do this. The reason is that the attachment is an 
encapsulated message/rfc822 message that should render properly since it has 
all of the appropriate charset information inside of the message structure.

I will check this in when the tree opens for M14.

- rhp
Comment 32 User image Katsuhiko Momoi 2000-01-21 12:22:21 PST
Does this do something like:

1. For single part msg --> relfect the content-type charset info.
2. For multi-part msg --> reflect first body par charset

for reply/forward inline?
Comment 33 User image rhp (gone) 2000-01-21 12:34:21 PST
Not currently. This increases the complexity to have to determine the type of 
message and decend to the main body part to pull out the charset. Is is this 
the right thing to do???

- rhp
Comment 34 User image Katsuhiko Momoi 2000-01-21 12:42:05 PST
For multi-part, if you are not descending into the body to get the reply/forward-inline charset,
where do you get the charset info? 
Comment 35 User image rhp (gone) 2000-01-21 12:56:46 PST
I get it from the RFC822 message header. If one isn't specified it will default 
to the users default settings.

- rhp
Comment 36 User image Katsuhiko Momoi 2000-01-21 13:51:05 PST
Rich, I remember discussing some of this with nhotta. The spec has been 
published to this page:

"Charset of the original message (main body) is used." for charset for reply/forward mail.

Naoki argued above also that we should not use the default charset set by the user if
there is a charset info already in the mail. We would not go crazy like descending into 2nd, 3rd parts
for multi-part msgs. The 1st (main) part should be sufficient. 
Thus, falling back to user's default fro multi-art is not what the spec would call for.

Could we wait until Monday when Naoki is back to settle this? I'm working on Messenger
charset menu related specs and some of them may have bearing on this issue.
Comment 37 User image rhp (gone) 2000-01-22 08:04:32 PST
This makes sense. I will see if I can handle this situation...I have some ideas 
how to make this happen.

- rhp
Comment 38 User image rhp (gone) 2000-01-22 17:16:50 PST
Good news. I think after some more hacking that I am able to decend into the 
body part of a multipart/ message and get the charset to be used for replies 
and forwards.

We'll need more testing, but so far, I think this is working.

- rhp
Comment 39 User image rhp (gone) 2000-01-24 20:09:07 PST
Just checked in the fix for this problem.

- rhp
Comment 40 User image rhp (gone) 2000-01-24 20:17:40 PST
Marking fixed.
Comment 41 User image Katsuhiko Momoi 2000-02-03 13:36:05 PST
I'm in the process of looking at this fix. It's going to take a bit of time for
we need to cover a variety of cases.
There seems to be one problem now. On our smoketest mailbox, the 1st msg is 
Latin 1 but using HTML entities and so marked as US-ASCII. The last msg is in UTF-8
and marked as so. These two will not go out reflecting the original charset label
if you have the following setting in prefs.js.

user_pref("intl.character_set_name", "iso-2022-jp");

I tried, Trad. Chinese (Big5), Simplified Chinese (gb2312), Korean (EUC-KR),
Czech (ISO-8859-2), Russian (KOI8-R), and they are OK even with the above setting.

Naoki, the above is not the mail default setting, is it? Could you remind me what the mail default
setting formual is? I'll also try that later.
Comment 42 User image nhottanscp 2000-02-03 13:39:27 PST
Momoi san, your setting for the mail send default charset is correct.
And that setting should not affect the behavior of reply/forward charset 
Comment 43 User image Katsuhiko Momoi 2000-02-03 13:42:18 PST
I should add that when you remove: user_pref("intl.character_set_name", "iso-2022-jp");
reply msgs work as intended by this fix. 
By the way, forward/inline seems to have this problem with the above special setting.
Comment 44 User image Katsuhiko Momoi 2000-02-03 13:44:22 PST
Correction: "By the way, forward/inline seems to have this problem with the above special setting."

to: By the way, forward/inline DOES NOT seem to have this problem with the above special setting.
Comment 45 User image Katsuhiko Momoi 2000-02-03 13:55:35 PST
Re-opening it since there is a limited but nonetheless real problem with some
msg charset headers when the mail default charset setting exists in prefs.js.
Comment 46 User image rhp (gone) 2000-02-03 15:38:45 PST
Ok, here is the problem. We make a call to nsMsgI18NGetDefaultMailCharset() in 
the compose back end when we create a message. This is the charset we will use 
unless there is a non us-ascii specified charset on the mail message. If we 
have a non-usascii, it shows up on the content type of the quoting operation, 
but other than that, we don't see any and we don't change it.

I'm looking to see where exactly that is stripped off the line.

- rhp

Comment 47 User image rhp (gone) 2000-02-03 17:30:46 PST
I have a fix that Naoki is reviewing.

- rhp
Comment 48 User image rhp (gone) 2000-02-03 23:09:38 PST
Ok, I think I have a fix for this one now. Will try to get checked in tonight.

- rhp
Comment 49 User image leger 2000-02-04 05:16:57 PST
Due to Beta indication in Summary, putting beta1 into keyword field.
Comment 50 User image rhp (gone) 2000-02-04 17:24:14 PST
Checked in this fix today.

- rhp
Comment 51 User image Katsuhiko Momoi 2000-02-11 00:03:46 PST
Half-way through. So far tried replying to some 16 different
language msgs with some charset buried in the main body part.
Also tried the problem ones with default charset set to ISO-2022-JP.
So far none failed. I will move on to forard/inline to finish
the verification.
Comment 52 User image Katsuhiko Momoi 2000-02-16 15:37:13 PST
At this point, I cannot verifyu this until Bug 28055 is fixed.
Comment 53 User image Katsuhiko Momoi 2000-02-21 21:04:02 PST
** Checked with 2/21/2000 Win32 build **

This feature is working very well with the above build.
I checked with some 17 different charsets -- some in
single part msgs and others in the first part of
multipart msgs. We are now able to reflect the original
charset regardless of the mail-send default charset setting
with all these test msgs.

This is in accord with our current spec.

Marking it verified/fixed.

Note You need to log in before you can comment on or make changes to this bug.