Beginning on October 25th, 2016, Persona will no longer be an option for authentication on BMO. For more details see Persona Deprecated.
Last Comment Bug 13677 - JPN Subject input disappears upon send (HTML & plain text mail)
: JPN Subject input disappears upon send (HTML & plain text mail)
Product: MailNews Core
Classification: Components
Component: Internationalization (show other bugs)
: Trunk
: x86 Windows 98
: P1 critical (vote)
: M11
Assigned To: Joe Francis
: marina
Depends on:
Blocks: 10940
  Show dependency treegraph
Reported: 1999-09-13 12:48 PDT by Katsuhiko Momoi
Modified: 2008-07-31 01:22 PDT (History)
7 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---

call stack for nsMsgCompFields::SetSubject, subject value is empty string... (3.49 KB, text/plain)
1999-09-13 15:10 PDT, nhottanscp
no flags Details
a simple form with a text widget (526 bytes, text/html)
1999-09-13 16:21 PDT, nhottanscp
no flags Details
widget with hard coded Japanese text no IME needed to reproduce (852 bytes, text/html)
1999-09-14 22:48 PDT, nhottanscp
no flags Details

Description Katsuhiko Momoi 1999-09-13 12:48:51 PDT
** Observed with 9/13/99 Win32 M10 build **

This bug was discovered while looking at Bug 13454.
There are some conditions under which you can succesffully
input Japanese strings into the Subject field without
the initial ASCII space. Under such conditions,
when you send that mail. The received mail lacks the
Subject header completely --  it's not found in the mail source file.

To avoid this problem, you need to insert an ASCII space
before you begin Japanese input into the subject line.
(Any ther ASCII character would be OK to begin the line.)
Comment 1 Katsuhiko Momoi 1999-09-13 12:49:59 PDT
Raised priority to P1 and set TM to M11.
Comment 2 Katsuhiko Momoi 1999-09-13 12:50:59 PDT
I have been more or less successful in inputting Japanese
into the Subject field without any prior ASCII space when
I first input into the main body window. (This is not
always true but works most of the time.)
Comment 3 Katsuhiko Momoi 1999-09-13 12:55:59 PDT
The same problem occurs with Plain Text mail. So
I corrected the summary line.
Comment 4 nhottanscp 1999-09-13 13:40:59 PDT
I think this is the same problem as 10940.
I will change the code to skip unmappable unicode chars on conversion.
Later we should either encode as UTF-8 or encode with multiple charset labels.
Comment 5 nhottanscp 1999-09-13 15:10:59 PDT
Created attachment 1646 [details]
call stack for nsMsgCompFields::SetSubject, subject value is empty string...
Comment 6 nhottanscp 1999-09-13 15:20:59 PDT
I think my change (in MIME encoder) is necessary and I will check in.
But currently, there is a problem before getting to the encoder. In
nsMsgCompFields::SetSubject, the vaule getting from JS is an empty string in
case the subject starts with Japanese text. Works fine if I put 'a' before
Japense text. Need more investigation.
Comment 7 nhottanscp 1999-09-13 15:50:59 PDT
I put dump statements in GenericSendMessage of MsgComposeCommands.js.

dump("subject = " + document.getElementById("msgSubject").value + "\n");
dump("subject = " + msgCompFields.GetSubject() + "\n");

They dump nothing when the subject starts with Japanese. If the text starts with
Ascii followed by Japanese then it dumps Ascii plus dots (for Japanese).
For me, it doesn't seem like a messenger but gfx widget problem.

I'll try more generic test writing a simple html with js.
Comment 8 nhottanscp 1999-09-13 16:21:59 PDT
Created attachment 1652 [details]
a simple form with a text widget
Comment 9 nhottanscp 1999-09-13 16:35:59 PDT
Reassign to Ender since this looks like a generic gfx widget problem.
It is reproducable with an attached html.
Try following 4 Japanese characters in case no IME installed on the machine.
Comment 10 buster 1999-09-13 16:45:59 PDT
sounds like an output
Comment 11 Akkana Peck 1999-09-13 16:50:59 PDT
This is a DUP of 8427, which is currently waiting on a decision from I18n
regarding how the unicode converter is supposed to work.

*** This bug has been marked as a duplicate of 8427 ***
Comment 12 nhottanscp 1999-09-13 16:56:59 PDT
#8427 sched: [DOGFOOD] make copy work with Japanese characters
Is this bug really related to copy?
Comment 13 Akkana Peck 1999-09-13 16:57:59 PDT
Copy/Paste, Send, and Save all go through the same output converters, which rely
on the unicode converters.
Comment 14 nhottanscp 1999-09-13 17:04:59 PDT
Question: Do you have any idea why this works if the text starts with Ascii
followed by Japanese characters?
Comment 15 Akkana Peck 1999-09-13 17:07:59 PDT
It may be a case of the sink stream getting tricked into using a unicode
converter in that case where normally it might not.
Comment 16 Akkana Peck 1999-09-14 18:36:59 PDT
The copy/paste problem in the sink stream has been fixed -- it no longer uses
ToNewCStrings except in the all-else-failed case -- and the copy/paste problem
is actually in the xpwidget clipboard code.  That being the case, that means
that this bug isn't a dup after all.  Reopening.

The problem is that I haven't been able to reproduce this problem myself.  I
have no way of typing Japanese input on a Linux box, and my NT box isn't set up
for Japanese.  nhotta is working on a way that I'll be able to reproduce it on
Linux so I can try to find out what the problem is.  It might be that that one
pathological case in the output sink is the culprit ... but it might just as
well be that the problem is somewhere else entirely.
Comment 17 nhottanscp 1999-09-14 22:48:59 PDT
Created attachment 1678 [details]
widget with hard coded Japanese text no IME needed to reproduce
Comment 18 nhottanscp 1999-09-14 23:01:59 PDT
I attached a new html file with two test cases which have hard coded Japanese
texts so no IME is needed to reproduce.
But they're actually working fine, the problem does not occure like it does with
I realized that whenever I type Japanese text, it is appended a white space like
extra character. When I prepend or append any ascii chars, that white space is
gone then my script works (as well as mail send). If I append ascii after
Japanese then remove appended ascii later then I was able to have Japanese text
with no white space appended. With that text, my test html worked no problem (I
assume it's the same for mail header).
So I assume that extra white space is something to do with this problem. Who is
appending that white space and why the widget behave differently with it? It
looks like only reproducable by IME (not by setting Japanese by JS code).
Comment 19 Akkana Peck 1999-09-15 10:19:59 PDT
I spent a fair amount of time yesterday trying to reproduce this in the editor
... and I came to the conclusion that the problem probably isn't in the
nsHTMLToTXTSinkStream, but somewhere else.  The gfx text widget code also looks
pretty solid (I saw no CString stuff where it shouldn't be).  At this point, I'm
guessing maybe the mime code that gets the contents of the subject line, but I
can't trace it without being able to type Japanese characters.

I'm afraid I have to send this back to nhotta to track down where the Japanese
characters are being lost.
Comment 20 nhottanscp 1999-09-15 13:48:59 PDT
I debugged with Japanese NT and it seems like something got wrong
around nsTextEditRules when doing Japanese text.
Reassign to Akkana, I can help to debug but I cannot fix the problem with my
very limited knowledge about Ender.

Paste from mail,
Naoki Hotta wrote:
> Inside OutputToString, mRules->WillDoAction sets cancel to true then
> do aOutputString = *(ruleInfo.outString); and returns (if the text is
> Japanese only).
> In nsTextEditRules::WillOutputText, it got a case like below to set
> aOutString as empty.
>     else if (mBogusNode)
>     { // this means there's no content, so output null string
>       *aOutString = "";
>       *aCancel = PR_TRUE;
>     }
Comment 21 Akkana Peck 1999-09-15 16:00:59 PDT
Rules, and this one in particular, are Joe's.
Comment 22 Joe Francis 1999-09-15 18:01:59 PDT
the real deal here is that none of the ime stuff goes through the rules code at
all, so the bogusNode never gets deleted when text is inserted.  If you type
"normal" text first, that goes through the rules code and causes the bogusNode to
be removed, which is why things work in that case.

I'll have to override SetCompositionString() in nsHTMLEditor, just so I can get
access to the rules system (nsEditor doesn't have access).  Then I'll have to add
in a mRules->WillDoAction()/ mRules->DidDoAction() sandwhich to give the rules a
chance to remove the bogs node.

I should be able to get this checked in tonight.
Comment 23 Joe Francis 1999-09-17 16:32:59 PDT
Comment 24 Katsuhiko Momoi 1999-09-17 17:13:59 PDT
Over to marina for verification.
Comment 25 nhottanscp 1999-09-20 13:53:59 PDT
There is a blocking problem with today's build.
tested with the second attachment (09/13/99 16:21).

The typed Japanese characters do not appear in the text field. Also the field
itself is not selectable (and editable) until I reload the page.
The Dos prompt shows dump as below (three lines for three JA chars I typed).

Candidate window position: x=35, y=35
Candidate window position: x=35, y=35
Candidate window position: x=35, y=35

I am not sure if this is a result of this bug fix or something else.
Comment 26 nhottanscp 1999-09-20 14:12:59 PDT
Added teruko because this is not a mail/news specific problem.
Comment 27 Joe Francis 1999-09-25 05:31:59 PDT
does anyone know if this is fixed or not?  I didnt get to test it (dont have a
machine set up for japanese text).
Comment 28 nhottanscp 1999-09-27 10:12:59 PDT
As I mentioned before, Japanese input to text widget has been broken (cannot
input any Japanaese at all) since 9/20 build.
I don't know if that is caused by the fix for this bug or not. But it is
possible because the check in of this was on 9/17 (Friday). I can check that by
reverting the change on my local machine. Joe, could you send me files that I
can revert your change?
Comment 29 Katsuhiko Momoi 2000-02-19 13:36:41 PST
** Checked with 2/17/2000 Win32 build **

We have not had a problem with JPN subject headers for a long time.
I still don't see a problem with the above build.
Marking it verified/fixed.

Note You need to log in before you can comment on or make changes to this bug.