10940 - JPN Plain text Mail send is broken

Status: VERIFIED FIXED

(MailNews Core :: Composition, defect, P1)

Description

[Katsuhiko Momoi]

** Observed with 7/30/99 Win32 M9 build **

This was not a problem with pre-necko 7/28/99 Win32 M9 build.
The problem is that the body content of Japanese mail is not
really Japanese.

This could be a composer problem.

1. Start New Mail compose window. (Plain text mail)
2. Set the Character set menu to Japanese (ISO-2022-JP)
3. Type in something in Japanese into body. Subject area does
   not take Japanese yet.
4. Send it and look at it with 4.x or 5.0 Messenger.
5. Note that instead of Japanese characters, you only see dots.

Naoki is on vacation. So I'm sending this over to Jean-F first and cc'ing
Tague.

Comment 1

[Katsuhiko Momoi]

This was working pre-Necko, and so I'll mark this a Necko bug.

Comment 2

[lchiang]

changing the component to composition.  I think mscott wanted to get rid of the
Necko component in the Mailnews product.

ducarroz is on vacation.  Noticed that this is marked a blocker.  Will cc: rhp
for any ideas.

Comment 3

[Katsuhiko Momoi]

Milestone set to M9. Really cannot do much without this working.

Comment 4

[Jean-Francois Ducarroz]

Rich, could you take a look at it please. Does the character set is correctly set in the sent message?

Comment 5

[rhp (gone)]

I was looking at this...question, for step #3 how can I type Japanese
characters?

- Rich

Comment 6

[rhp (gone)]

I verified that the charset sent for the main mail message is ISO-2022-JP.
If I can get japanese characters input, I'll look into it further.

- rhp

Comment 7

[bobj]

Kat Momoi in IQA should be able to help you on this.  I'll let him know.

Comment 8

[tague]

I've looked at this, and as far as I can tell Japanese mail is completely
broken.  I can't get as far as Momoi-san, I'm no Japanese input what-so-ever --
no dots or anything.

From tracing it, something seems to have interfered with the installation of the
TextEvent handlers for the Ender widget.  Text events aren't getting handled by
the observer.  This problem is unique to the mail compose and isn't
reproduceable in the HTML or plaintext editor components.

Comment 9

[Katsuhiko Momoi]

I'm wondering if this is not a Charset menu problem.
I tried today's Win32 build and the first time I attempted to
send JPN mail body, it succeeded. I tried twice more since then
but I got the same result as originally reported for this bug,
i.e. just dots instead of real characters.
But one thing I noticed about the 1st time I tried and succeeded is
that when I went to select Japanese (ISO-2022-JP) before
sending the msg out, I had difficulty selecting this menu item --
I select it but the focus seems to go elsewhere when I let go
of the selection. Finally I think I succeeded in setting it
right and then sent it out. This was the message which showed OK.
The other 2 times I tried the same but did not observe the same
kind of selection problem in the first place, but the results were
both bad,

Comment 10

[Katsuhiko Momoi]

To re-cap the problem, the input via the Japanese IME is showing
correctly as they are entered into the mail body, but
as the mail is sent out, the characters arrive as dots.
Here's what the headers and body look like with the problem:

----------------------------------------------------------
To: [e-mail], [e-mail]
Subject: JPN test #3 8/2/99
Content-Type: text/plain; charset=ISO-2022-JP
Content-Transfer-Encoding: 7bit


...................

----------------------------------------------------------

Note the charset header is correct but the conversion from the
input to ISO-2022-JP does not seem to occur properly . It' as if we
are not really engaging Unicode-to-ISO-2022-JP converter at all but
rather Unicode-to-default (ISO-8859-1) converter as the msg is
assembled and sent out.

Comment 11

[Katsuhiko Momoi]

Rich, you need Japanese Windows to do this input at step #3 since
it requires Japanese Input method which comes with Japanese
Windows.
Copy/paste has other problems and cannot be used for this either.
One possibility is to hard-code a send data/string in the
Message Send file. If you tell me which line of the MsgSend file
I can modify, I can send you a string to try.

Comment 12

[Katsuhiko Momoi]

It turns out that there is another condition which must be
met to produce this bug.
When you open the Plain Text mail compose window, there is a pre-set
line of data which is read in from "defaultTextBody.html". It's
my habit to erase this line completely before I start inputting
Japanese data. That's when this bug occurs.
This bug does not seem to occur under any one of the following
conditions:

1. If you simply add Japanese text to the pre-set English string.
2. If you erase the pre-set string but add ASCII string data before
   or after the Japanese string portion.

Because there is a workaround, I'm lowering the status from Blocker to
Critical but it should be still M9.

Comment 13

[Katsuhiko Momoi]

Jay, in your Smoketest, do you erase the pre-set data line?

Comment 14

[nhottanscp]

I debugged two cases using a Japanese text (three characters 0x3042, 0x3044,
0x3046).
The reason of the failure in the second case is because of an appended character
0x0A after the three Japanese characters.
The code 0xA0 (nbsp) cannot be mapped to Japanese charset (ISO-2022-JP), the
unicode converter fails. Then we fall back to use nsAutoCString(msgBody) then
the text turned to a garbage.
We need to know why Ender appends this extra character for the second case
(can Ender replace nbsp to space in case of plain text?). Also we might want to
improve the conversion not to abort in case of this kind of error.

Case1: Added Japanese
0B219480  42 30 44 30 46 30 2D 00 2D 00  B0D0F0-.-.
0B21948A  2D 00 20 00 54 00 68 00 69 00  -. .T.h.i.
0B219494  73 00 20 00 6D 00 65 00 73 00  s. .m.e.s.
0B21949E  73 00 61 00 67 00 65 00 20 00  s.a.g.e. .
0B2194A8  77 00 61 00 73 00 20 00 73 00  w.a.s. .s.
0B2194B2  65 00 6E 00 74 00 20 00 62 00  e.n.t. .b.
0B2194BC  79 00 20 00 4D 00 65 00 73 00  y. .M.e.s.
0B2194C6  73 00 65 00 6E 00 67 00 65 00  s.e.n.g.e.
0B2194D0  72 00 20 00 35 00 2E 00 30 00  r. .5...0.

Case2: Erased original text
0B4B9F00  0D 00 0A 00 42 30 44 30 46 30  ....B0D0F0
0B4B9F0A  A0 00 0D 00 0A 00 00 00 CD CD  ........??

Comment 15

[Jay Patel [:jay]]

I sometimes delete the preset data line, but not always.  Today i did the line,
and the problem was still there (build 1999080408).

Comment 16

[Jean-Francois Ducarroz]

nhotta,
do you have any idea from where comes the extra 0x00A0?
Does the converted should be able to take care of this ascii value, maybe by converting it to 0x0020 instance of just
fail?
Does the proposed path for bug 6672 about the 0x0A fix also this bug?

Comment 17

[nhottanscp]

I don't know where that A0 is comming from (possibly inside editor since it's
not typed by the user).
I think 8865 will be the solution for 6672. But 8865 is for HTML, I am not sure
if that also handles plain text.
Adding [e-mail] for his opinion.

Comment 18

[bobj]

Where does the HTML get converted to plain text?  Should that conversion
replace &nbsp with spaces instead of 0xA0?

Comment 19

[Jean-Francois Ducarroz]

I still don't understand why we get a &nbsp in the output. The user never type a
&nbsp and the default plain text page (defaultTextBody.html) doesn't contain
&nbsp!

Comment 20

[Jean-Francois Ducarroz]

The extra nbsp is added by ender when the body is empty (select all/delete).
At this point we have to problem:

1) Encoding failed because entity isn't working yet for Japanese (bug 8865)
2) For some reason, the extra nbsp isn't removed when the use type something in
the body as is the case with a US system (at least on Windows NT)

The work around today is to not delete the default body. As we have a work
around and as HTML japanese composition now works (with the same problem if you
delete the default body), I lower the severity of this bug to normal.

I have also fixed the problem that tague saw about plain text composition that
was totally dead. Now you can again compose in plain text mode.

I reassing the bug to tague as the real problem here is the non-support on
entities.

Comment 21

[bobj]

Is there a separate Ender bug filed about the extra nbsp that isn't removed
when the user types something in?

Comment 22

[tague]

Since there is a know work-around, I'm not going to do a hack to get this to
work.  The real fix for this problem is slated for m10.

Comment 23

[tague]

moving to m12.  i'm having too many other problems with typing plain text
japanese mail (focus not taking, characters not being displayed) which are
making this extremely difficult to reproduce.

i can't get japanese text into the text field reliably once the messenger
signature is deleted.  since there is a reasonably work around, i'm going to
move this out to m12 until the focus and typing issues get fixed.

Comment 24

[nhottanscp]

This is going to be taken care by charset conversion fall back in message
compose. The change is not to abort the conversion in case of failure but skip
the unconvertable character and proceeed the conversion.

Comment 25

[nhottanscp]

Changed the summary since this is not a Necko bug and plain text only.

Comment 26

[nhottanscp]

M11

Comment 27

[nhottanscp]

Removing 8865 dependency, currently planed solution (skipping unmapped
characters) does not depends on 8865.

Comment 28

[nhottanscp]

The change was checked in, skip unconvertable chars while converting from
unicode. The change is applied to both plain and html body (so #6672 may be
verified and file a separate bug for entity specific issue).
I also applied the same change for message header (checked in today).

Later, add an option (by pref) to skip or convert to utf-8 instead.

Comment 29

[Katsuhiko Momoi]

** Checked with 9/16/99 Win32 M11 build **

The problem still exists with this build.
If you erase the pre-set characters and start fresh in the body,
no text gets sent out. (If there is some condition which is causing this
for this particular build, let me know.)
I'm re-opening this for the time being.

Comment 30

[Katsuhiko Momoi]

The result was for Plain Text only. HTML mail does to have this
problem.

Comment 31

[nhottanscp]

My change to skip convertable chars is working (e.g. Ascii+Japanese text sent as
ISO-8859-1 only sends out Ascii, Japanese chars are skipped).
The remaining problem is the same problem as #13677. After erasing the template
text, if any Ascii is typed followed by Japanese then it sends out the entire
text correctly. But typing text starting with Japanese doesn't send at all
because Ender generates an empty sting.
Put 13677 to depend.

Comment 32

[nhottanscp]

13677 is fixed now

Comment 33

[Katsuhiko Momoi]

Mrina, please verify this. I'll be off a week starting Monday.

Comment 34

[Katsuhiko Momoi]

We have not had a problem in plain text message send for a long time.
Marking the fix verified with 2/19/2000 Win32 build.