132671 - Double-byte mails in Sent/Trash/Draft/Unsent folder are not imported correctly (outlook)

Status: VERIFIED FIXED

(MailNews Core :: Internationalization, defect)

Description

[ji]

Build: 03/20 win32 build

Now bug 100867 is fixed, double-byte mails in Inbox/Trash/Draft/Unsent can be
imported correctly. But the mails in Sent folder are displayed garbled after import.

Steps to reproduce:
1. With Outlook XP Simplified Chinese version, have Chinese mails stored in
Inbox/Trash/Draft/Unsent/Sent folders in POP3 account.
2. Launch Mail with the trunk build.
3. Select File | Import to import the mails from Outlook XP, you can see after
import, Chinese mails in Inbox/Trash/Draft/Unsent shows alright. But the mails
including both subject and mail body in Sent folder are displayed garbled. 
Screenshot to follow.

Comment 1

[ji]

Attachment: A screenshot on Simplified Chinese XP.

Comment 2

[nhottanscp]

Reassign to putterman.

Comment 3

[scottputterman]

reassigning to Cavin

Comment 4

[ji]

Nominating for nsbeta1 since it causes data corruption.

Comment 5

[Cavin Song]

> mail body in Sent folder is displayed garbled
>
I thnk this can be resolved by selecting the Auto-Detect|Japanese via 
View|Character Coding... menu item. Reporter, can you verify this?

> subject in Sent folder is displayed garbled
>
I do see that the encoding of the subject is not correct for some reason. 
Looking into the problem.

Comment 6

[ji]

Attachment: A Simplified Chinese mail in Sent folder after import from Outlook

In above mail, both sender and mail body don't have charset info. For mail
body, auto-detector can be used for display. We also need to fix sender problem
as well. Now applying folder charset can't make sender displayed correctly.

Comment 7

[Cavin Song]

Yes, both sender and subject have the same problem.

Comment 8

[Michael Buckland]

Discussed at mail news bug meeting.  Decided to remove RTM from this bug.  This
means fixing this bug is currently not planned for RTM.

Comment 9

[nhottanscp]

nominate for nsbeta1

Comment 10

[Katsuhiko Momoi]

Using US version of Outlook Express and English version of
Netscape 7 under Win 2000 US/JPN locale, I could not confirm this
problem. However, my test cases have all the OE folder names in
English. We need to re-test this with the following typical 
environments for Japanese also:

1. JPN Win 2000 with JPN version of Outlook Express & JPN Netscape 7.
2. JPN Win XP with JPN version of Outlook Express & JPN Netscape 7.

** Note in both cases, the Outlook folder names should be in Japanese.

Comment 11

[ji]

This problem only exists when importing mail from Outlook, it doesn't happen
when importing from Outlook Express. Checked on Japanese XP, after import from
Outlook XP, Ja mails in Sent/Trash/Draft/Unsent are corrupted.

Comment 12

[ji]

Attachment: A screenshot on Japanese XP.

Comment 13

[Katsuhiko Momoi]

As of July, 2002, we have the following stats from the Internet Association
of Japan annual survey results:

Percentage of Japanese users for top 3 Mail programs 
in descending order:

Outlook Express: 75.5%
Outlook: 10.3%
Post Pet (by Sony): 7.7%

So this problem does affect a significant user base (10.3%) of mail
in Japan.

Comment 14

[ji]

Checked this on Japanese ME with Outlook XP installed.
Only mails in Sent folder got corrupted (attachment problem is reported
seperately in bug 174402.) Open the message source for the imported mail, the
subject is encoded in Shift JIS, and the sender is not MIME encoded at all. The
original charset for the subject and sender is iso-2022-jp.

Comment 15

[Jeesun Cho]

Attachment: On JA Win XP machine, importing mails from Draft folder(Outlook) also have the same problem

Comment 16

[ji]

yes, that's what I saw on Japanese XP too. (Comment #11) It looks like the
behaviour is different on different windows platforms.

Comment 17

[Cavin Song]

Naoki and I were able to reproduce the problem and the content type of test msg
in the Sent folder is multipart/alternative.

The reason why Outlook does not work is that we use MAPI to retrieve the header
and body info and MAPI returns the following content-type header:

Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_0009_01C273A8.66981CA0"

which indicates that the body contains multiple body parts. However, the body
data returned from MAPI contains only the original Japanese chars we entered in
the compose window. In other words the body does not contain multiple body parts
with different content types (as it should). For this reason we end up with a
msg without the “Content-Type:” header and thus without the charset info.

The reason why it works for OE is that it doesn’t use MAPI to retrieve header
and body info, instead it interprets the msg stored in the OE msg store to get
the data it needs.  In this case the retrieved body data does contain two body
parts (as expected) which contain content type info with correct charset.

I need to try more test msgs (ie, some with attachments) to see if I can figure
out a way to fix it but this is what I've found so far.

Comment 18

[Cavin Song]

Another case that has the same problem is when a message does not have content 
type header at all.

So to fix the two cases we've found so far we need to default content type to 
"text/html" or "text/plain" depends on the type of body data returned by MAPI.

Comment 19

[Cavin Song]

Attachment: Proposed patch, v1

Set default content type for msgs without a content type header or with a
"multipar/alternative" content type but the body data contains only text.

Comment 20

[nhottanscp]

Comment on attachment 103787 [details] [diff] [review]
Proposed patch, v1

r=nhotta

Comment 21

[(not reading, please use [e-mail] instead)]

looks good, one comment:

1)

+      nsCAutoString body(msg.GetBody());
+      if (body.Find(msg.GetMimeBoundary()) <=0)
+        needDefaultCType = PR_TRUE;

This will copy the body unnecessarily.

you should do something like:

const char *body = msg.GetBody();
if (!body || !strstr(body,msg.GetMimeBoundary())
  needDefaultCType = PR_TRUE;

You probably check if msg.GetMimeBoundary() is null (or non-empty), too.

Comment 22

[Cavin Song]

Good suggestion. So I changed:

+    if (msg.GetBodyLen() && msg.GetMimeBoundaryLen())
+    {
+      // Comments here.
+      nsCAutoString body(msg.GetBody());
+      if (body.Find(msg.GetMimeBoundary()) <=0)
+        needDefaultCType = PR_TRUE;
+    }

to:

+    // Comments here.
+    const char *body = msg.GetBody();
+    const char *boundary = msg.GetMimeBoundary();
+    if (body && boundary && !strstr(body, boundary))
+      needDefaultCType = PR_TRUE;

Comment 23

[Cavin Song]

Attachment: Proposed patch, v2

Incorporated comment (using a better way to check boundary in the body data).

Comment 24

[(not reading, please use [e-mail] instead)]

looks good.

I think we can simplify the implementation of SetDefaultContentType.

Consider:

+{
+  cType.Truncate()
+
+  // double check that the comment is correct
+  if (msg.GetMimeContentLen()) {
+    if (nsCRT::strcasecmp(msg.GetMimeContent(), "multipart/alternative"))
+      return;
+
+    // double check that the comment is correct
+    const char *body = msg.GetBody();
+    const char *boundary = msg.GetMimeBoundary();
+    // is this right?
+    // notice we return on !body or ! boundry
+    // or if we find it boundry in the body.
+    if (!body || !boundary || strstr(body, boundary))
+      return;
+  }
+  
+  cType = msg.BodyIsHtml() ? "text/html" : "text/plain";
+}

If that will work, can you switch to that, and make sure the comments are correct?

once you do that, sr=sspitzer

Comment 25

[Cavin Song]

Yes the code can be simplied. nhotta actually memtioned this to me. The reason I
used 'needDefaultCType' to qualify all cases is that if we have to add another
case ot two later it'll be easier. Right now we only know two cases but more may
need to be added. This code is used to workaround some MAPI issues.

Comment 26

[Cavin Song]

Adding [fixedish1] to status whiteboard.

Comment 27

[ji]

On the build containing the fix, the problem described in the original report is
fixed.
BTW, after mail import is finished, there are two subfolders generated under
Outlook Mail folder: Personal  Folders and Saved Folders. Personal Folders
contains all the regular folders, like Inbox, Sent, Trash....and etc. But Saved
Folders contains Trash folder. What's "Saved Folders" for? (this also exists
before the fix, it's not related to the fix.)

Comment 28

[Cavin Song]

I'm not sure what "Saved Folders" is but some external users had told me about
it. I don't see that on my machine and I don't think it's on nhotta's machine
either.

Comment 29

[ji]

I''m using Japanese Outlook XP. Could it be related to the language/version of
the outlook ?

Comment 30

[Cavin Song]

Fix checked in to the trunk.

Comment 31

[ji]

verified as fixed using 2003042105-trunk build.