5660 - Unable to process Non-MIME 8-bit headers properly

Status: VERIFIED FIXED

(MailNews Core :: MIME, defect, P3)

Description

[Katsuhiko Momoi]

** Observed with 4/28/99 Win32 build **

For the thread pane Subject display, we are now decoding and
displaying MIME-encoded 8-bit and CJK headers. But sometimes
the Subject may contain raw 8-bit characters.
For these cases, I either see XML parser errors or
weird character substitutions (e.g. A Korean character for a
French 8-bit character.)

I understand that these are not going to be handled until M6.
This bug is filed then as a reminder to check on this type of
headers.

Comment 1

[nhottanscp]

Accepting bug, set to M6.

Comment 2

[nhottanscp]

Adding [e-mail] to cc.
This is going be controlled by the charset menu. See the related bug 5938.

Comment 3

[nhottanscp]

I realize that the thread pane subject view is a separate issue that I am doing
with Rich. I'll take a look and see what we should change.

Comment 4

[nhottanscp]

This is related to the folder charset (set through charset menu UI), #3966.

Comment 5

[nhottanscp]

Moving to M7, MIME decoder needs to get mail folder charset (see #3966).
Regarding the original report, there are two parts. The thread pane display is
the one which is going to be fixed in M7. The second problem of XML parser error
is a part of #5938.
By the way, I can see Latin1 Non-MIME 8-bit headers (prorbably Latin1
converter is used as the default). But I think that's only for Latin1.

Comment 6

[nhottanscp]

Just for my memo, I should enable the code for this bug.

Index: mailnews/mime/src/comi18n.h
===================================================================
RCS file: /cvsroot/mozilla/mailnews/mime/src/comi18n.h,v
retrieving revision 1.11
diff -c -r1.11 comi18n.h
*** comi18n.h	1999/06/03 01:08:23	1.11
--- comi18n.h	1999/06/11 00:57:56
***************
*** 39,45 ****
   *
   *
   * @param header      [IN] A header to decode.
!  * @param charset     [OUT] Charset name (in C string) from a MIME header is
set.
   *                    Caller should allocate at least 65 bytes (kMAX_CSNAME +
1) for a charset name.
   * @return            Decoded buffer (in C string) or return NULL if the
header is not MIME encoded.
   */
--- 39,46 ----
   *
   *
   * @param header      [IN] A header to decode.
!  * @param charset     [IN/OUT] Input default charset or empty string to be
used for non MIME labeled header,
!  *                    After the decode charset name (in C string) from a MIME
header is set.
   *                    Caller should allocate at least 65 bytes (kMAX_CSNAME +
1) for a charset name.
   * @return            Decoded buffer (in C string) or return NULL if the
header is not MIME encoded.
   */
Index: mailnews/mime/src/comi18n.cpp
===================================================================
RCS file: /cvsroot/mozilla/mailnews/mime/src/comi18n.cpp,v
retrieving revision 1.28
diff -c -r1.28 comi18n.cpp
*** comi18n.cpp	1999/06/03 03:12:07	1.28
--- comi18n.cpp	1999/06/11 00:57:56
***************
*** 1729,1735 ****
--- 1729,1742 ----
    // No MIME encoded, duplicate the input and put us-ascii as a charset
    if (*header == '\0' || !intlmime_is_mime_part2_header(header)) {
      result = PL_strdup(header);
+ #if 1
      PL_strcpy(charset, "us-ascii");
+ #else
+     // Put us-ascii as a charset if the default charset is not specified.
+    if (*charset == '\0') {
+       PL_strcpy(charset, "us-ascii");
+    }
+ #endif
    }
    else {
      char *header_copy = PL_strdup(header);  // avoid to modify the original
string

Comment 7

[nhottanscp]

3966 got fixed last night, I updated the tree and apply my changes but the
testing failed. I think I have one more place to change (nsMimeConverter.cpp).
Regarding the fix of 3966, I saw the folder charset is passed down to MIME
decoder so I can say that it's fixed. But for this bug, I need more time to
change the code and to test. Currently, all Non MIME 8 bit headers are
interpreted as Latin1 (so we can see Latin1 Non MIME and Non MIME Japanese is
very rare).
Moving target to M8.

Comment 8

[nhottanscp]

Fix checed in mime/src/comi18n.cpp rev=1.30, mime/src/nsMimeConverter.cpp
rev=1.6.
Note for verification:
Charset menu change does not cause reload thread pane (8206). To see the change,
click other folder then click back.
Because Latin1 has been working, something other charset should also be tested.

Comment 9

[Katsuhiko Momoi]

** Checked with 6/25/99 Win32 M8 build **

I looked at some Latin msg headers which contain 8-bit characters without MIME-encoding.
They all failed to display correctly in the message viewing window thougg they displayed OK
in the thread pane. I tried switching to some other folder and come back to the msgs I wanted to
display but this did not work, either.  Maybe it's not easy to reset the folder encoding?

The error message was something like: "XML Error in file 'file://c:/temp/tempMessage.eml?header=only', Line Number: 5,
Col Number: 739, Description: mismatched tag"


I'm re-opening this bug for now ...

Comment 10

[Katsuhiko Momoi]

Correction:

I was looking at only Latin 1 examples. That's not going to really
reveal the fix. So I tried Latin 2 non-MIME headers.
By changing the encoding to Latin 2 and higlighting another
folder and then back, I was able to display Latin 2 characters
OK which were not displayed correctly initially in the thread pane.

However, the message viewing window's subject header displays
incorrectly. Unlike the case of Latin 1 headers, I didn't get
XML parser error but still it displays incorrectly.

My initial bug report was about not being able to deal with non-MIME
8-bit headers correctly, and it implicitly referred to both thread pane and
message window problems -- note my mention of XML parser error
in such cases which you get in the message viewing window.

So, even though the thread pane display now correctly reflects
the folder charset as in this Latin 2 message, the viewing window
problem needs to be resolved.

Comment 11

[nhottanscp]

Message view pane problem has been filed separately (5938 - charset override).
That bug should apply for both header and body. Can we mark this FIXED and add
comments to 5938?

Comment 12

[Katsuhiko Momoi]

Thanks fo reminding me of that bug.
Let me now mark the fix here verified.