Persona is no longer an option for authentication on BMO. For more details see Persona Deprecated.
Last Comment Bug 4925 - Japanese thread pane headers get extraneous materials on re-start
: Japanese thread pane headers get extraneous materials on re-start
Product: MailNews Core
Classification: Components
Component: Backend (show other bugs)
: Trunk
: x86 Windows NT
: P3 normal (vote)
: M5
Assigned To: davidmc
: Katsuhiko Momoi
Depends on:
  Show dependency treegraph
Reported: 1999-04-09 23:34 PDT by Katsuhiko Momoi
Modified: 2008-07-31 01:21 PDT (History)
6 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---


Description Katsuhiko Momoi 1999-04-09 23:34:40 PDT
** Observed with 4/9/99 Win32 5.0 build **

Please re-assing to this someone appropriate in Mail/News.

Here's how the problem occurs:

1. Prepare a mailbox with a number of msgs with Japanese
   headers. The headers should be of varying length -- anywhere
   from 5-6 characters to over 80 characters.

2. Erase the existing summary file, i.e. ",msf" file.
3. Now start Messenger. This will re-create the summary of headers.
4. At this point examine thread pane headers and see that they are
   displayed correctly.
5. Now quit Messenger and then re-start Messenger.
6. Exmaine the headers again. This time, some of the headers
   (particularly ones longer than certain length) now include
   extraneous materials, i.e. $0D$0A$09

7. There is actually no truncation of headers. It's just that these
   extra strings are inserted.

8. After this, you will see these extra strings until you delete
   the summary (".msf") file.
Comment 1 Katsuhiko Momoi 1999-04-09 23:35:59 PDT
TM set to M5.
Comment 2 nhottanscp 1999-04-12 09:50:59 PDT
Question: Did you see $0D$0A$09 in .msf file or display?
Comment 3 Katsuhiko Momoi 1999-04-12 11:38:59 PDT
The extraneous chatracters show up in the thread pane display.
Comment 4 nhottanscp 1999-04-12 11:57:59 PDT
Question: I think .msf is a plain text file. So, $0D$0A$09 were not in the .msf
file, just confirming.
$0D$0A$09 are CR, LF, and a tab. Where did you see those characters inserted?
Did you see a line break after the first line and a tab in the following line? I
think a header is displayed in one line and truncated if longer.
Comment 5 Katsuhiko Momoi 1999-04-12 12:43:59 PDT
Now that we can examine the source for messages, I looked at it in the
View pane part and found that these extraneous characters are
there in the MIME-encoded format. It seems that this problem occurs only with
longer headers and where there should be line breaking, we are
actually MIME-encoding these characters in the following way:


This string represents the Japanese period plus the extraneous characters.
Comment 6 nhottanscp 1999-04-12 12:53:59 PDT
So, this is an encoding problem.
1) The header was generated by Mozila, correct?
2) What do you actually see in the thread pane of Mozilla?
3) How is that displayed in 4.5 thread pane?
Comment 7 Katsuhiko Momoi 1999-04-12 13:54:59 PDT
Here's on example display in Japanese:

HTML: 今度はうまく行くでしょう。これは日本語のメールです。

This is what I see in Communicator 4,5x thread pane, but under the current
Mozilla, I see:

HTML: 今度はうまく行くでしょう。これ$0D$0A$09は日本語のメールです。

Note the insertion of the extraneous characters. In the source, this is
represented as:


Note that the header folding is taking place actually.
Comment 8 nhottanscp 1999-04-12 14:43:59 PDT
One more question, the source do you mean in .msf file or in the locale file in
Berkeley format?
Comment 9 Katsuhiko Momoi 1999-04-12 15:02:59 PDT
Well, I had thought that the problem is in the MIME-encoded part
but actually it is not. What Iquoted above is from my Inbox file. When
I looked in the Inbox.msf file, I saw:


So maybe this is a summary file creation problem? Are these characters
supposed to be there, but we should not display them?
Comment 10 nhottanscp 1999-04-12 16:39:59 PDT
Reassigning to bienvenu, adding putterman to cc.
I think this is a summary file issue.
This is not Japanese specific. We saw it happens in a folded Latin1 header too.
BTW, what's the spec here, we used to truncate it for a display? Do we support
line breaks in the thread pane in 5.0 instead?
Comment 11 David :Bienvenu 1999-04-12 16:45:59 PDT
No, Mork is escaping the strings to include $0D$0A, etc, as it should. I'm not
sure how I should handle this. I wouldn't think we should be giving the DB lines
with CRLF's in them...
Comment 12 David :Bienvenu 1999-04-12 20:54:59 PDT
David, are you unescaping yarns when you give them back?
Comment 13 davidmc 1999-04-13 14:15:59 PDT
I should be unescaping yarns, but it's possible I messed it up after
it was working earlier.  I'll check after I finish pulling the tree.
Comment 14 davidmc 1999-04-13 15:02:59 PDT
It looks like I was never unescaping bytes encoded as hex in Mork,
so once they were written as $xx, it stayed that way when read later.

I wrote the code just now to unescape it correctly, but I still don't
have an operative build yet.  Someone else can check it in before me
if necessary and one can't wait until I build and hopefully run for
the first time today.  If I'm unable to run today, I'll note this here
later in case someone needs the fix faster.
Comment 15 davidmc 1999-04-20 12:26:59 PDT
Fixed checked in yesterday by changing morkParser::ReadValue() to convert
next two hex bytes after '$' into the next source byte.  Resolving FIXED
pending independent verification.
Comment 16 Katsuhiko Momoi 1999-04-20 18:47:59 PDT
** Checked with 4/20/99 Win32 build **

With this new build, under the same conditions which prompted me
to file this bug (see steps 1-6 above), there no longer are any
extraneous strings inserted.
This problem used to occur with Latin 1 headers as well as Japanese
ones. All the problems I obserevd with the non-ASCII headers in
my mailbox are gone now.

Marking the fix verified.

Note You need to log in before you can comment on or make changes to this bug.