13677 - JPN Subject input disappears upon send (HTML & plain text mail)

Status: VERIFIED FIXED

(MailNews Core :: Internationalization, defect, P1)

Description

[Katsuhiko Momoi]

** Observed with 9/13/99 Win32 M10 build **

This bug was discovered while looking at Bug 13454.
There are some conditions under which you can succesffully
input Japanese strings into the Subject field without
the initial ASCII space. Under such conditions,
when you send that mail. The received mail lacks the
Subject header completely --  it's not found in the mail source file.

To avoid this problem, you need to insert an ASCII space
before you begin Japanese input into the subject line.
(Any ther ASCII character would be OK to begin the line.)

Comment 1

[Katsuhiko Momoi]

Raised priority to P1 and set TM to M11.

Comment 2

[Katsuhiko Momoi]

I have been more or less successful in inputting Japanese
into the Subject field without any prior ASCII space when
I first input into the main body window. (This is not
always true but works most of the time.)

Comment 3

[Katsuhiko Momoi]

The same problem occurs with Plain Text mail. So
I corrected the summary line.

Comment 4

[nhottanscp]

I think this is the same problem as 10940.
I will change the code to skip unmappable unicode chars on conversion.
Later we should either encode as UTF-8 or encode with multiple charset labels.

Comment 5

[nhottanscp]

Attachment: call stack for nsMsgCompFields::SetSubject, subject value is empty string...

Comment 6

[nhottanscp]

I think my change (in MIME encoder) is necessary and I will check in.
But currently, there is a problem before getting to the encoder. In
nsMsgCompFields::SetSubject, the vaule getting from JS is an empty string in
case the subject starts with Japanese text. Works fine if I put 'a' before
Japense text. Need more investigation.

Comment 7

[nhottanscp]

I put dump statements in GenericSendMessage of MsgComposeCommands.js.

Recipients2CompFields(msgCompFields);
msgCompFields.SetSubject(document.getElementById("msgSubject").value);
dump("subject = " + document.getElementById("msgSubject").value + "\n");
dump("subject = " + msgCompFields.GetSubject() + "\n");

They dump nothing when the subject starts with Japanese. If the text starts with
Ascii followed by Japanese then it dumps Ascii plus dots (for Japanese).
For me, it doesn't seem like a messenger but gfx widget problem.

I'll try more generic test writing a simple html with js.

Comment 8

[nhottanscp]

Attachment: a simple form with a text widget

Comment 9

[nhottanscp]

Reassign to Ender since this looks like a generic gfx widget problem.
It is reproducable with an attached html.
Try following 4 Japanese characters in case no IME installed on the machine.
\u98a8\u6797\u706b\u5c71

Comment 10

[buster]

sounds like an output

Comment 11

[Akkana Peck]

This is a DUP of 8427, which is currently waiting on a decision from I18n
regarding how the unicode converter is supposed to work.

*** This bug has been marked as a duplicate of 8427 ***

Comment 12

[nhottanscp]

#8427 sched: [DOGFOOD] make copy work with Japanese characters
Is this bug really related to copy?

Comment 13

[Akkana Peck]

Copy/Paste, Send, and Save all go through the same output converters, which rely
on the unicode converters.

Comment 14

[nhottanscp]

akkana,
Question: Do you have any idea why this works if the text starts with Ascii
followed by Japanese characters?

Comment 15

[Akkana Peck]

It may be a case of the sink stream getting tricked into using a unicode
converter in that case where normally it might not.

Comment 16

[Akkana Peck]

The copy/paste problem in the sink stream has been fixed -- it no longer uses
ToNewCStrings except in the all-else-failed case -- and the copy/paste problem
is actually in the xpwidget clipboard code.  That being the case, that means
that this bug isn't a dup after all.  Reopening.

The problem is that I haven't been able to reproduce this problem myself.  I
have no way of typing Japanese input on a Linux box, and my NT box isn't set up
for Japanese.  nhotta is working on a way that I'll be able to reproduce it on
Linux so I can try to find out what the problem is.  It might be that that one
pathological case in the output sink is the culprit ... but it might just as
well be that the problem is somewhere else entirely.

Comment 17

[nhottanscp]

Attachment: widget with hard coded Japanese text no IME needed to reproduce

Comment 18

[nhottanscp]

I attached a new html file with two test cases which have hard coded Japanese
texts so no IME is needed to reproduce.
But they're actually working fine, the problem does not occure like it does with
IME.
I realized that whenever I type Japanese text, it is appended a white space like
extra character. When I prepend or append any ascii chars, that white space is
gone then my script works (as well as mail send). If I append ascii after
Japanese then remove appended ascii later then I was able to have Japanese text
with no white space appended. With that text, my test html worked no problem (I
assume it's the same for mail header).
So I assume that extra white space is something to do with this problem. Who is
appending that white space and why the widget behave differently with it? It
looks like only reproducable by IME (not by setting Japanese by JS code).

Comment 19

[Akkana Peck]

I spent a fair amount of time yesterday trying to reproduce this in the editor
... and I came to the conclusion that the problem probably isn't in the
nsHTMLToTXTSinkStream, but somewhere else.  The gfx text widget code also looks
pretty solid (I saw no CString stuff where it shouldn't be).  At this point, I'm
guessing maybe the mime code that gets the contents of the subject line, but I
can't trace it without being able to type Japanese characters.

I'm afraid I have to send this back to nhotta to track down where the Japanese
characters are being lost.

Comment 20

[nhottanscp]

I debugged with Japanese NT and it seems like something got wrong
around nsTextEditRules when doing Japanese text.
Reassign to Akkana, I can help to debug but I cannot fix the problem with my
very limited knowledge about Ender.

Paste from mail,
Naoki Hotta wrote:
> Inside OutputToString, mRules->WillDoAction sets cancel to true then
> do aOutputString = *(ruleInfo.outString); and returns (if the text is
> Japanese only).
>
> In nsTextEditRules::WillOutputText, it got a case like below to set
> aOutString as empty.
>
>     else if (mBogusNode)
>     { // this means there's no content, so output null string
>       *aOutString = "";
>       *aCancel = PR_TRUE;
>     }

Comment 21

[Akkana Peck]

Rules, and this one in particular, are Joe's.

Comment 22

[Joe Francis]

the real deal here is that none of the ime stuff goes through the rules code at
all, so the bogusNode never gets deleted when text is inserted.  If you type
"normal" text first, that goes through the rules code and causes the bogusNode to
be removed, which is why things work in that case.

I'll have to override SetCompositionString() in nsHTMLEditor, just so I can get
access to the rules system (nsEditor doesn't have access).  Then I'll have to add
in a mRules->WillDoAction()/ mRules->DidDoAction() sandwhich to give the rules a
chance to remove the bogs node.

I should be able to get this checked in tonight.

Comment 23

[Joe Francis]

fixed

Comment 24

[Katsuhiko Momoi]

Over to marina for verification.

Comment 25

[nhottanscp]

There is a blocking problem with today's build.
windows/32bit/x86/1999-09-20-12-M11
tested with the second attachment (09/13/99 16:21).

The typed Japanese characters do not appear in the text field. Also the field
itself is not selectable (and editable) until I reload the page.
The Dos prompt shows dump as below (three lines for three JA chars I typed).

Candidate window position: x=35, y=35
Candidate window position: x=35, y=35
Candidate window position: x=35, y=35

I am not sure if this is a result of this bug fix or something else.

Comment 26

[nhottanscp]

Added teruko because this is not a mail/news specific problem.

Comment 27

[Joe Francis]

does anyone know if this is fixed or not?  I didnt get to test it (dont have a
machine set up for japanese text).

Comment 28

[nhottanscp]

As I mentioned before, Japanese input to text widget has been broken (cannot
input any Japanaese at all) since 9/20 build.
I don't know if that is caused by the fix for this bug or not. But it is
possible because the check in of this was on 9/17 (Friday). I can check that by
reverting the change on my local machine. Joe, could you send me files that I
can revert your change?

Comment 29

[Katsuhiko Momoi]

** Checked with 2/17/2000 Win32 build **

We have not had a problem with JPN subject headers for a long time.
I still don't see a problem with the above build.
Marking it verified/fixed.