** Observed with 7/30/99 Win32 M9 build ** This was not a problem with pre-necko 7/28/99 Win32 M9 build. The problem is that the body content of Japanese mail is not really Japanese. This could be a composer problem. 1. Start New Mail compose window. (Plain text mail) 2. Set the Character set menu to Japanese (ISO-2022-JP) 3. Type in something in Japanese into body. Subject area does not take Japanese yet. 4. Send it and look at it with 4.x or 5.0 Messenger. 5. Note that instead of Japanese characters, you only see dots. Naoki is on vacation. So I'm sending this over to Jean-F first and cc'ing Tague.
This was working pre-Necko, and so I'll mark this a Necko bug.
Component: Internationalization → Necko
Component: Necko → Composition
Summary: JPN Mail send is broken → Necko: JPN Mail send is broken
changing the component to composition. I think mscott wanted to get rid of the Necko component in the Mailnews product. ducarroz is on vacation. Noticed that this is marked a blocker. Will cc: rhp for any ideas.
Milestone set to M9. Really cannot do much without this working.
Rich, could you take a look at it please. Does the character set is correctly set in the sent message?
I was looking at this...question, for step #3 how can I type Japanese characters? - Rich
I verified that the charset sent for the main mail message is ISO-2022-JP. If I can get japanese characters input, I'll look into it further. - rhp
Kat Momoi in IQA should be able to help you on this. I'll let him know.
I've looked at this, and as far as I can tell Japanese mail is completely broken. I can't get as far as Momoi-san, I'm no Japanese input what-so-ever -- no dots or anything. From tracing it, something seems to have interfered with the installation of the TextEvent handlers for the Ender widget. Text events aren't getting handled by the observer. This problem is unique to the mail compose and isn't reproduceable in the HTML or plaintext editor components.
I'm wondering if this is not a Charset menu problem. I tried today's Win32 build and the first time I attempted to send JPN mail body, it succeeded. I tried twice more since then but I got the same result as originally reported for this bug, i.e. just dots instead of real characters. But one thing I noticed about the 1st time I tried and succeeded is that when I went to select Japanese (ISO-2022-JP) before sending the msg out, I had difficulty selecting this menu item -- I select it but the focus seems to go elsewhere when I let go of the selection. Finally I think I succeeded in setting it right and then sent it out. This was the message which showed OK. The other 2 times I tried the same but did not observe the same kind of selection problem in the first place, but the results were both bad,
To re-cap the problem, the input via the Japanese IME is showing correctly as they are entered into the mail body, but as the mail is sent out, the characters arrive as dots. Here's what the headers and body look like with the problem: ---------------------------------------------------------- To: email@example.com, firstname.lastname@example.org Subject: JPN test #3 8/2/99 Content-Type: text/plain; charset=ISO-2022-JP Content-Transfer-Encoding: 7bit ................... ---------------------------------------------------------- Note the charset header is correct but the conversion from the input to ISO-2022-JP does not seem to occur properly . It' as if we are not really engaging Unicode-to-ISO-2022-JP converter at all but rather Unicode-to-default (ISO-8859-1) converter as the msg is assembled and sent out.
Rich, you need Japanese Windows to do this input at step #3 since it requires Japanese Input method which comes with Japanese Windows. Copy/paste has other problems and cannot be used for this either. One possibility is to hard-code a send data/string in the Message Send file. If you tell me which line of the MsgSend file I can modify, I can send you a string to try.
It turns out that there is another condition which must be met to produce this bug. When you open the Plain Text mail compose window, there is a pre-set line of data which is read in from "defaultTextBody.html". It's my habit to erase this line completely before I start inputting Japanese data. That's when this bug occurs. This bug does not seem to occur under any one of the following conditions: 1. If you simply add Japanese text to the pre-set English string. 2. If you erase the pre-set string but add ASCII string data before or after the Japanese string portion. Because there is a workaround, I'm lowering the status from Blocker to Critical but it should be still M9.
Jay, in your Smoketest, do you erase the pre-set data line?
I debugged two cases using a Japanese text (three characters 0x3042, 0x3044, 0x3046). The reason of the failure in the second case is because of an appended character 0x0A after the three Japanese characters. The code 0xA0 (nbsp) cannot be mapped to Japanese charset (ISO-2022-JP), the unicode converter fails. Then we fall back to use nsAutoCString(msgBody) then the text turned to a garbage. We need to know why Ender appends this extra character for the second case (can Ender replace nbsp to space in case of plain text?). Also we might want to improve the conversion not to abort in case of this kind of error. Case1: Added Japanese 0B219480 42 30 44 30 46 30 2D 00 2D 00 B0D0F0-.-. 0B21948A 2D 00 20 00 54 00 68 00 69 00 -. .T.h.i. 0B219494 73 00 20 00 6D 00 65 00 73 00 s. .m.e.s. 0B21949E 73 00 61 00 67 00 65 00 20 00 s.a.g.e. . 0B2194A8 77 00 61 00 73 00 20 00 73 00 w.a.s. .s. 0B2194B2 65 00 6E 00 74 00 20 00 62 00 e.n.t. .b. 0B2194BC 79 00 20 00 4D 00 65 00 73 00 y. .M.e.s. 0B2194C6 73 00 65 00 6E 00 67 00 65 00 s.e.n.g.e. 0B2194D0 72 00 20 00 35 00 2E 00 30 00 r. .5...0. Case2: Erased original text 0B4B9F00 0D 00 0A 00 42 30 44 30 46 30 ....B0D0F0 0B4B9F0A A0 00 0D 00 0A 00 00 00 CD CD ........??
I sometimes delete the preset data line, but not always. Today i did the line, and the problem was still there (build 1999080408).
nhotta, do you have any idea from where comes the extra 0x00A0? Does the converted should be able to take care of this ascii value, maybe by converting it to 0x0020 instance of just fail? Does the proposed path for bug 6672 about the 0x0A fix also this bug?
I don't know where that A0 is comming from (possibly inside editor since it's not typed by the user). I think 8865 will be the solution for 6672. But 8865 is for HTML, I am not sure if that also handles plain text. Adding email@example.com for his opinion.
Where does the HTML get converted to plain text? Should that conversion replace   with spaces instead of 0xA0?
I still don't understand why we get a   in the output. The user never type a   and the default plain text page (defaultTextBody.html) doesn't contain  !
Assignee: ducarroz → tague
Severity: critical → normal
Status: ASSIGNED → NEW
Depends on: 8865
The extra nbsp is added by ender when the body is empty (select all/delete). At this point we have to problem: 1) Encoding failed because entity isn't working yet for Japanese (bug 8865) 2) For some reason, the extra nbsp isn't removed when the use type something in the body as is the case with a US system (at least on Windows NT) The work around today is to not delete the default body. As we have a work around and as HTML japanese composition now works (with the same problem if you delete the default body), I lower the severity of this bug to normal. I have also fixed the problem that tague saw about plain text composition that was totally dead. Now you can again compose in plain text mode. I reassing the bug to tague as the real problem here is the non-support on entities.
Is there a separate Ender bug filed about the extra nbsp that isn't removed when the user types something in?
Since there is a know work-around, I'm not going to do a hack to get this to work. The real fix for this problem is slated for m10.
moving to m12. i'm having too many other problems with typing plain text japanese mail (focus not taking, characters not being displayed) which are making this extremely difficult to reproduce. i can't get japanese text into the text field reliably once the messenger signature is deleted. since there is a reasonably work around, i'm going to move this out to m12 until the focus and typing issues get fixed.
This is going to be taken care by charset conversion fall back in message compose. The change is not to abort the conversion in case of failure but skip the unconvertable character and proceeed the conversion.
Summary: Necko: JPN Mail send is broken → JPN Plain text Mail send is broken
Changed the summary since this is not a Necko bug and plain text only.
Removing 8865 dependency, currently planed solution (skipping unmapped characters) does not depends on 8865.
Status: ASSIGNED → RESOLVED
Last Resolved: 19 years ago
Resolution: --- → FIXED
The change was checked in, skip unconvertable chars while converting from unicode. The change is applied to both plain and html body (so #6672 may be verified and file a separate bug for entity specific issue). I also applied the same change for message header (checked in today). Later, add an option (by pref) to skip or convert to utf-8 instead.
** Checked with 9/16/99 Win32 M11 build ** The problem still exists with this build. If you erase the pre-set characters and start fresh in the body, no text gets sent out. (If there is some condition which is causing this for this particular build, let me know.) I'm re-opening this for the time being.
The result was for Plain Text only. HTML mail does to have this problem.
My change to skip convertable chars is working (e.g. Ascii+Japanese text sent as ISO-8859-1 only sends out Ascii, Japanese chars are skipped). The remaining problem is the same problem as #13677. After erasing the template text, if any Ascii is typed followed by Japanese then it sends out the entire text correctly. But typing text starting with Japanese doesn't send at all because Ender generates an empty sting. Put 13677 to depend.
Status: ASSIGNED → RESOLVED
Last Resolved: 19 years ago → 19 years ago
Resolution: --- → FIXED
13677 is fixed now
Mrina, please verify this. I'll be off a week starting Monday.
We have not had a problem in plain text message send for a long time. Marking the fix verified with 2/19/2000 Win32 build.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.