Conversion failure: char corruption

VERIFIED FIXED in M5

Status

MailNews Core
Internationalization
P3
normal
VERIFIED FIXED
19 years ago
9 years ago

People

(Reporter: Katsuhiko Momoi, Assigned: rhp (gone))

Tracking

Trunk
x86
Windows NT

Firefox Tracking Flags

(Not tracked)

Details

(URL)

(Reporter)

Description

19 years ago
** Observed with 3/16/99 Mozilla Win32 build **

Send yourself 2 msgs:

1. HTML and Plain text
2. Both should contain a Japanese Subject line: "Kore ha Nihongo no Meeru desu."
3. The same wording as the subject line should be in the body text.
4. Send yourself these 2 msgs. Receive them and display them.

Note that both have problems of character corruption right after
"Me" in "Meeru". Apparently the long vowel symbol is causing a problem.

Should check if this is a general conversion failure from JIS
to Unicode in this area.

Updated

19 years ago
Target Milestone: M3

Comment 1

19 years ago
momoi, please clearify the sending is done on 5.0 or 4.5 ? Is this is a sending
problem or a receiving problem ? If the problem is receiving, then the target
should be M3, if the problem is sending, then mark it M4. I mark it as M3 for
now, please change the target ASAP. Thanks.
(Reporter)

Comment 2

19 years ago
The original msgs were sent by Comm4.51 and are correctly displayed
by the same client.This seems to be either a display/conversion problem
or parsing problem by 5.0, i.e. mail headers and body may not be
parsed correctly when non-ASCII characters are there.TM should
be M3m therefore.

Comment 3

19 years ago
I propose to test the exact same text by creating a ISO-2022-JP meta-tagged
HTML. This way, we can verify if this is a converter bug or somewhere in the
messenger.
(Reporter)

Updated

19 years ago
(Reporter)

Comment 4

19 years ago
This does not seem to be a straight conversion failure.
I see other kinds of corruption when messages get longer.
By the way, I see that the problem string I described above
gets displayed OK in some msgs (typically a 1-line msg) but get
corrupted in longer ones.
1-line display test for the layout is at the above URL. The browser
has no problem with it.

Updated

19 years ago
Assignee: nhotta → rhp
Target Milestone: M3 → M4

Comment 5

19 years ago
The problem does not happen to all the japanese characters.
Looks like it happens for some specific characters (of iso-2022-jp) which
confilicts with html (e.g. 0x213C where 3C is '<').
Since we have already decided to change libmime to use unicode, the problem will
be solved when the change is available. Even in the new code when to convert to
unicode is important, we should consider this to avoid the problem like this
bug.
Rich, can this be ready be M4?

Comment 6

19 years ago
Adding myself to cc-list.
(Assignee)

Updated

19 years ago
Status: NEW → ASSIGNED
(Assignee)

Comment 7

19 years ago
My plate is really full for M4 already. I am doing some major reworking of
the output routines for libmime and I am not sure I will be able to address
this issue without some assistance.

- rhp

Comment 8

19 years ago
We need to review where to put the unicode converter in the new code. I18n eng.
can do that help, also possible to do a simple test (by using the characters
which had problems in M3) when the code is checked in.
(Assignee)

Updated

19 years ago
Target Milestone: M4 → M5
(Assignee)

Comment 9

19 years ago
Just updating this bug. I probably won't be able to do any major investigation
in this area before M4. Naoki, since we are not generating XML for headers and
HTML for the body, we need to investigate the header issue to see the behavior
now. (Also, if the emitters need to be exporting charset information, I could
use a little help on what we should output.)

- rhp

Comment 10

19 years ago
I saw this is working in my local build with Rich's change.
There is a bug in UTF-8 encoder (#4968) so the text is truncated.

Comment 11

19 years ago
Rich, would you change the status to 'FIXED'?
(Assignee)

Updated

19 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 19 years ago
Resolution: --- → FIXED
(Reporter)

Updated

19 years ago
Status: RESOLVED → REOPENED
(Reporter)

Comment 12

19 years ago
** Checked with 4/16/99 Win32 build **

Now that we can see non-ASCII body, I'm able to check on
this problem. I sent the same test string which failed originally
(see the URL above), and the result indicates that there is
still the same problem remaining in processing isio-2022-jp charcters
containg 0x3C in the first or the 2nd byte.
As far as I can see this is not a truncation problem reported
elsewhere.
Reopening it ...
(Reporter)

Comment 13

19 years ago
The display of the test string works on headers.
So maybe I'm just seeing the efects of the truncation bug in the body,
but this is hard to tell without seeing the rest of the string. There
is some indication that I'm seeing a messed-up HTML structure.
I'm going to leave the status as is but we need to look into this
further.

Comment 14

19 years ago
I tried some of the test cases sent from momoi by using my local build.
My build has a hack to work around the UTF-8 truncation (#4968).
I saw the message body without truncation and looks fine execpt some character
display as boxes (but this can be seen in the thread pane).
So I think the new code is working fine. But the verification is not possible
because of the blocking bug (#4968).

Updated

19 years ago
Resolution: FIXED → ---

Comment 15

19 years ago
I checked in an error handling code which will reallocate the buffer in case the
converter's estimate length was incorrect thus avoid the truncation.
It will be included in the next build. Please verify this bug again with that
build.

Comment 16

18 years ago
4968 was FIXED 4/19.
(Reporter)

Updated

18 years ago
Status: REOPENED → RESOLVED
Last Resolved: 19 years ago18 years ago
Resolution: --- → FIXED
(Reporter)

Updated

18 years ago
Status: RESOLVED → VERIFIED
(Reporter)

Comment 17

18 years ago
** Checked with 4/20/99 Win32 build **

Now that theh truncation bug has been fixed, I can now see
all the text in the mail body part.
The parsing failure does not occur with the original
problem string I used to file this bug. There are other
similar strings in othe mail messages in my mailbox and
none show this type of parsing failures.

Marking the fix verified.
Product: MailNews → Core
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.