558653 - HTML e-mail with embedded (and attached) pictures missing Content-ID after import from Outlook 2003

Status: RESOLVED DUPLICATE of bug 207156

(MailNews Core :: Import, defect)

Description

[Jorg K (CEST = GMT+2)]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 GTB6 (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4

HTML e-mail with embedded pictures looks poorly after import from Outlook 2003.

I imported a large PST file and some of the pictures embedded in the HTML, they are not shown properly but get displayed only as an attachment.

I will try to include a screen shot.

Is this related to Bug 415045? But this one says that the attachment isn't imported at all. Here it is imported, only badly.

Here's the relevant bit of the message:

<br>
</a></b></font>
<img src="cid:[e-mail].0" width=155 height=80 alt="[]">

</b>&nbsp; </body>
</html>

--------------090008060207040502080208
Content-Type: image/jpeg;
 name="ANEL-sig.jpg"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="ANEL-sig.jpg"

/9j/4AAQSkZJRgABAQIHsAewAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRof
Hh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwh
MjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAAR

Basically the Content-ID is missing. Here's an example which doesn't come from an import and where the embedding works:

<br>
<img src="cid:[e-mail]" alt=""><br>
<br>
</body>
</html>

--------------070601030700080802000508
Content-Type: image/png;
 name="moz-screenshot-1.png"
Content-Transfer-Encoding: base64
Content-ID: <[e-mail]>
Content-Disposition: attachment;
 filename="moz-screenshot-1.png"

iVBORw0KGgoAAAANSUhEUgAAAFwAAACHCAIAAAAUdKnjAAAgAElEQVR4nFyYVVSbefu17+dJ
gjvFSoGWUkpLnbpSoLi7Q4K7e5AYJJBgSQgeIHhwdyhWCvWpTm2snXakM3WZTvMd0Jl3/t9a


Reproducible: Always

Comment 1

[Jorg K (CEST = GMT+2)]

Attachment: Here you can see how the e-mail is displayed

Comment 2

[Joe Sabash [:JoeS1]]

Jorg,
It would be helpful to know what related the CID to the attachment in the original
outlook mail.
In other words, did the PST import break that relationship, or was that relationship one that TB just cannot recognize.

CID:XXXX decoding has some known weakness.

Comment 3

[Jorg K (CEST = GMT+2)]

Hi,

well, I tried to fix the problem editing the mail file in "vi", it's only text.

I noticed, that unless you put multipart/related, it ain't working.

Below is the same message in TB and the headers, as well as I can get them in Outlook. TB first, note it doesn't say "related".

From - Sat, 10 Apr 2010 07:42:51
Return-path: <[e-mail]>
Envelope-to: [e-mail]
Delivery-date: Fri, 09 Apr 2010 17:09:04 -0700
Received: from cpe-124-191-11-14.ewqd1.win.bigpond.net.au ([124.191.11.14] helo=henk.henk.com.au)
	by danu.lunarservers.com with esmtps (TLSv1:AES256-SHA:256)
	(Exim 4.69)
	(envelope-from <[e-mail]>)
	id 1O0OFz-0005EY-J5
	for [e-mail]; Fri, 09 Apr 2010 17:09:04 -0700
X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9
Date: Sat, 10 Apr 2010 10:08:59 +1000
To: =?iso-8859-1?Q?J=F6rg?= Knobloch <[e-mail]>
From: Henk van Leeuwen <[e-mail]>
Subject: OK: Updated: CD listings - to be automated
In-Reply-To: <AA2932F3F07E4C29B79F26FA23F66286@toad>
References: <5B58881003D047009AE98C09C86A57BA@toad>
 <[e-mail]>
 <CDC3D74188C74F3385C7239813265F13@toad>
 <[e-mail]>
 <322233605EA943958E0130C47DE6117B@toad>
 <8A2A5971970343D9A8BF0F998865930E@toad>
 <AA2932F3F07E4C29B79F26FA23F66286@toad>
Mime-Version: 1.0
Content-Type: multipart/mixed;
 boundary="------------070606000407070309060901"
X-ACL-Warn: {

This is a multi-part message in MIME format.
--------------070606000407070309060901
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: 8bit

<html>
<body>
<b>OK Jorg - No probs. Glad I have the hang of it now - Cheers - Thanks -
Henk<br><br>

</b>At 01:28 AM 10/04/2010, you wrote:<br>
<blockquote type=cite class=cite cite=""><font size=2 color="#0000FF">Up
to you. With this system I spend about 10 minutes on each update as long
as you provide a good list. JK.<br>
</font><br>

<dl><hr>

<dd><font face="Tahoma" size=2>From:</b> Henk van Leeuwen
[<a href="mailto:[e-mail]" eudora="autourl">
mailto:[e-mail]</a>] <br>

<dd>Sent:</b> Friday, 9 April 2010 17:23<br>

<dd>To:</b> Jörg Knobloch<br>

<dd>Subject:</b> ok: Updated: CD listings - to be automated<br>
</font><br>

<dd>OK - Thanks again Jorg. I will have a few new EGEA cds but they can
wait until I have say 6 entries. Ciao - Henk</b><br>
</blockquote>
<x-sigsep><p></x-sigsep>

</dl><br>
Henk van Leeuwen<br>

<b>Australia Northern Europe Liaisons<br>
</b>unit 5, 2A Tiuna Grove<br>
Elwood Vic 3184<br>
Australia<br>
ph: + 61 3 9525 7022<br>
<font color="#800080"><a href="mailto:[e-mail]" eudora="autourl">
mailto:[e-mail]</a> <br>
<b><a href="http://www.henk.com.au/" eudora="autourl">www.henk.com.au<br>
<br>
</a></b></font>

<img src="cid:[e-mail].0" width=155 height=80 alt="[]">
</b>&nbsp; </body>
</html>

--------------070606000407070309060901
Content-Type: image/jpeg;
 name="ANEL-sig.jpg"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="ANEL-sig.jpg"

/9j/4AAQSkZJRgABAQIHsAewAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRof
Hh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwh
(stuff deleted)
--------------070606000407070309060901--

Here the headers from Outlook:

Return-path: <[e-mail]>
Envelope-to: [e-mail]
Delivery-date: Fri, 09 Apr 2010 17:09:04 -0700
Received: from cpe-124-191-11-14.ewqd1.win.bigpond.net.au ([124.191.11.14] helo=henk.henk.com.au)
	by danu.lunarservers.com with esmtps (TLSv1:AES256-SHA:256)
	(Exim 4.69)
	(envelope-from <[e-mail]>)
	id 1O0OFz-0005EY-J5
	for [e-mail]; Fri, 09 Apr 2010 17:09:04 -0700
X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9
Date: Sat, 10 Apr 2010 10:08:59 +1000
To: =?iso-8859-1?Q?J=F6rg?= Knobloch <[e-mail]>
From: Henk van Leeuwen <[e-mail]>
Subject: OK: Updated: CD listings - to be automated
In-Reply-To: <AA2932F3F07E4C29B79F26FA23F66286@toad>
References: <5B58881003D047009AE98C09C86A57BA@toad>
 <[e-mail]>
 <CDC3D74188C74F3385C7239813265F13@toad>
 <[e-mail]>
 <322233605EA943958E0130C47DE6117B@toad>
 <8A2A5971970343D9A8BF0F998865930E@toad>
 <AA2932F3F07E4C29B79F26FA23F66286@toad>
Mime-Version: 1.0
Content-Type: multipart/related;
	type="multipart/alternative";
	boundary="=====================_149820703==.REL"
X-ACL-Warn: {

=================================

And here the body of the message:

<html>
<body>
<b>OK Jorg - No probs. Glad I have the hang of it now - Cheers - Thanks -
Henk<br><br>
</b>At 01:28 AM 10/04/2010, you wrote:<br>
<blockquote type=cite class=cite cite=""><font size=2 color="#0000FF">Up
to you. With this system I spend about 10 minutes on each update as long
as you provide a good list. JK.<br>
</font><br>

<dl><hr>

<dd><font face="Tahoma" size=2>From:</b> Henk van Leeuwen
[<a href="mailto:[e-mail]" eudora="autourl">
mailto:[e-mail]</a>] <br>

<dd>Sent:</b> Friday, 9 April 2010 17:23<br>

<dd>To:</b> Jörg Knobloch<br>

<dd>Subject:</b> ok: Updated: CD listings - to be automated<br>
</font><br>

<dd>OK - Thanks again Jorg. I will have a few new EGEA cds but they can
wait until I have say 6 entries. Ciao - Henk</b><br>
</blockquote>
<x-sigsep><p></x-sigsep>

</dl><br>
Henk van Leeuwen<br>
<b>Australia Northern Europe Liaisons<br>
</b>unit 5, 2A Tiuna Grove<br>
Elwood Vic 3184<br>
Australia<br>
ph: + 61 3 9525 7022<br>
<font color="#800080"><a href="mailto:[e-mail]" eudora="autourl">
mailto:[e-mail]</a> <br>
<b><a href="http://www.henk.com.au/" eudora="autourl">www.henk.com.au<br>
<br>
</a></b></font>
<img src="cid:[e-mail].0" width=155 height=80 alt="[]">
</b>&nbsp; </body>
</html>

Comment 4

[Joe Sabash [:JoeS1]]

Can you see the actual Mime part in the original outlook message.
Here is a sample from an outlook sender in my inbox:
------=_NextPart_000_0019_01CAD6F1.B48F19E0
Content-Type: image/jpeg;
	name="image006.jpg"
Content-Transfer-Encoding: base64
Content-ID: <image006.jpg@01CAD6F1.B3E82B80>

/9j/4AAQSkZJRgABAQAAAQABAAD/4AAITlhQTAAB/9sAQwALCAgKCAcLCgkKDQwLDREcEhEPDxEi

I guess the multi-part related was lost somehow, just how that happed is the question.

Comment 5

[Jorg K (CEST = GMT+2)]

I don't know how familiar you are with Outlook, in fact, or how familiar I am with it ;-) To my knowledge there is now way to get to the "raw" message. That's why I hate it and want to migrate off Outlook.

Just to set things straight here: I am talking about messages imported from an Outlook PST file. The messages in question actually originate from an Eudora client and were received with Outlook which displayed them correctly.

In your text above you are referring to a message you received from an Outlook client, so I'm a bit confused.

I also have plenty of messages sent with Outlook which contained embedded pictures and which got imported just as badly. I will attach a PST file with two messages. I hope that helps. If not, do let me know how I should extract the messages from Outlook for you to disseminate.

Comment 6

[Jorg K (CEST = GMT+2)]

Attachment: Outlook 2003 PST containing messages with pictures embedded in HTML. TB fails to import properly.

Three messages: One received from an Eudora client. One received from an OE client a sent messaged.

Comment 7

[Jorg K (CEST = GMT+2)]

Oops. Bad English:
Three messages in the enclosed PST file:
- one received from a Eudora client.
- one received from an OE client.
- one outgoing sent/message.
All are HTML with embedded pictures, all are badly imported from Outlook.

Comment 8

[Jorg K (CEST = GMT+2)]

Bummer. Another type that contorts the meaning:
In Outlook there is **NO** way to get to the "raw" message.

Comment 9

[Jorg K (CEST = GMT+2)]

**typo**. Boy, being dyslexic doesn't help ;-((

Comment 10

[Joseph]

This is kind of related to the bug I created: Bug 556674 , but it might differ in that the other is a problem with the treatment of Content-ID and this might be the fact that Thunderbird strips the Content-ID from the email. I was going to make a bug on this, but this will do...

You may want to check RFC 2392 (http://tools.ietf.org/html/rfc2392 which replaces RFC 2111 which I refer too) which talk about the usage of content-id.

Comment 11

[WADA:World Anti-bad-Duping Agency]

Jorg K(bug opener), can you check next?
(1) Import Outlooks's PST file by Outlook Express first.
    (Outlook Express knows well about Outlook's funny internals than Tb.) 
(2) Then, import mails from Outlook Express using Thunderbird.
    (Tb knows far well about Outlook Express than about Outlook.)
Is mail in comment #3 imported as multipart/related, as image part with Content-Id:? Or imported as multipart/mixed?

If original(imported from Outlook Express) is multipart/mixed, it is possibly issue of bug 556674 upon import(loss of Content-Id: if multipart/mixed) as Joseph says. If Content-Id: header is correctly specified, Outlook probably displayes multipart/mixed mail as if multipart/related mail, and displays the image part as attachment because image part is really in multipart/mixed, as Windows Mail does do. See bug 556674 comment #19 by Joseph, please.

Comment 12

[Jorg K (CEST = GMT+2)]

(1) I imported the Outlook Mail with Outlook Express:
Result: Outlook Express does the same bad job as TB. The images do not show in the HTML but show as an attachment below the message.

(2) Do I need to proceed with this?

The e-mail shown in comment #3 is exactly shown as imported by TB. In TB the e-mail arrives as multipart/mixed.

When looking at the mail headers with Outlook, they show multipart/related:

The following four lines were already posted in comment #3, they show the headers as retrieved with Outlook:
Mime-Version: 1.0
Content-Type: multipart/related;
    type="multipart/alternative";
    boundary="=====================_149820703==.REL"

Sadly there is no way to get to the full message in Outlook.

I submitted a PST file with the messages in question as attachment. Can you not look at that? When I extract strings out of the binary PST file, I find multipart/related, so that indicates to me that there is really a multipart/related header in the e-mail.

Also, I get a lot of e-mail from the same person using the embedded picture in his signature. Here are some headers I received recently that show that the Eudora client he is using produces the correct headers.

To me the question is whether when TB is retrieving e-mail from Outlook, it is retrieving the correct headers and should therefore migrate the e-mail correctly.

Sample headers as received from the same Eudora client follow:

Mime-Version: 1.0
Content-Type: multipart/related;
	type="multipart/alternative";
	boundary="=====================_207383421==.REL"
X-ACL-Warn: {

--=====================_207383421==.REL
Content-Type: multipart/alternative;
	boundary="=====================_207383421==.ALT"

--=====================_207383421==.ALT
Content-Type: text/plain; charset="iso-8859-1"; format=flowed
Content-Transfer-Encoding: quoted-printable

--=====================_207383421==.ALT
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

--=====================_207383421==.ALT--

--=====================_207383421==.REL
Content-Type: image/jpeg; name="ANEL-sig.jpg";
 x-mac-type="4A504547"; x-mac-creator="4A565752"
Content-ID: <[e-mail].0>
Content-Transfer-Encoding: base64
Content-Disposition: inline; filename="ANEL-sig.jpg"

/9j/4AAQSkZJRgABAQIHsAewAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0a

Comment 13

[Joseph]

(In reply to comment #12)
I'm unable to import from my version of Outlook (no comment on the version), so I can't test.  I tested the format in comment #12, and it works fine in Thunderbird.  If confirmed, it seems that not only the Content-ID is stripped, but that the Content-Type is changed to multipart/mixed.  Maybe it's due to the 'type="multipart/alternative";' part, maybe not, but either way, they both change the email to where it doesn't handle as it would if sent directly to Tb.  Can you import to a different client to see if the Content-Type and/or Content-ID might be changed by Outlook.  Probably not, but it helps to check... I'm not sure which client to try.  See if Opera Mail will import from Outlook.

Comment 14

[Jorg K (CEST = GMT+2)]

Opera Mail does not import from Outlook, only from Outlook Express.

Comment 15

[Joseph]

Attachment: Test Email directly to Tb (with names changed)

Comment 16

[Joseph]

Attachment: Email to Outlook from Gmail, then imported to Tb (names changed)

Comment 17

[Joseph]

I was able to test with Outlook (the problem was the user), and the two attachments show the difference in the same email.  The first one is directly from Gmail with content-type multipart/related.  While the second has been changed to multipart/mixed and the Content-ID is gone.  I don't want to put into confirmed yet until I can test on another client to see if the problem is Outlook or Tb.  If Tb gets its information directly Outlook without windows converting it, then there is not much else to check.

Comment 18

[Jorg K (CEST = GMT+2)]

Yep. These two attachments are exactly what I am seeing as well, only that my e-mail comes from Eudora as originating client.

But we still don't know how Outlook stores the receive e-mail internally and we also don't know what Outlook returns via the SimpleMAPI interface that is used to retrieve the e-mail from Outlook.

To check that, the person working on the migration would have to use the PST I supplied and run the migration with some debug turned on.

Comment 19

[Joseph]

Attachment: Gmail to Outlook (2002) and imported from Windows Mail then output as EML

Comment 20

[Joseph]

With this last attachment, I imported to Windows Mail from Outlook (2002).  I could already tell there was a change on viewing the email.  I could see that the image was as an attachment.  Upon outputting the EML and viewing, you can see the changes from Outlook to Windows Mail are similar from Outlook to Thunderbird.  I don't think the problem is Thunderbird, but Outlook (depending on how Tb imports).

Comment 21

[Joe Sabash [:JoeS1]]

(In reply to comment #20)
> With this last attachment, I imported to Windows Mail from Outlook (2002).  I
> could already tell there was a change on viewing the email.  I could see that
> the image was as an attachment.  Upon outputting the EML and viewing, you can
> see the changes from Outlook to Windows Mail are similar from Outlook to
> Thunderbird.  I don't think the problem is Thunderbird, but Outlook (depending
> on how Tb imports).

Except that those last 2 attachments did not contain an html part at all
Just plaintext.

Comment 22

[Joseph]

(In reply to comment #21)
> Except that those last 2 attachments did not contain an html part at all
> Just plaintext.

Those are what I got from an import from Outlook 2002 to Tb and Windows Mail.  The first one is closer to the original with proper Content-Type and Content-ID.  They all came from Gmail.  The first directly to Tb while the other two went to Outlook and were pulled from other applications.

Comment 23

[Joseph]

(In reply to comment #22)

Oh, the reason for this was that the original bug was a problem with import from Outlook to Thunderbird.  I imported to Windows Mail also, because I needed another client to test the import to see the outcome.  I loaded Eudora 7.1.0.9, but can't get it to recognize Outlook on this box even when it is the default.  I can't figure out how to load even straight from the PST, so there is some conflict with those.  If you can test with another client to make verify more that the problem is Outlook, then that would help.

Comment 24

[Jorg K (CEST = GMT+2)]

If I understand your comments correctly you sent and e-mail from Gmail and received it with Outlook 2002, then imported to TB and the multipart/related and the Content-ID were gone.

I imported mail from Outlook 2003 (new unicode PST format) and the multipart/related and the Content-ID were also gone.

What else do you want to do to confirm the problem?

Whether the problem is
a) there is not way for TB to retrieve the e-mail from Outlook correctly
   because the interface MS provide is just not good enough
or
b) TB just gets is wrong
we can't decide without actually doing a conversion from Outlook to TB with a debug version and seeing what is actually retrieved from Outlook and how TB processes it.

Until such time someone can do this, IMHO any further comments on this bug are fruitless.

Even in the case of a), TB could be made smarter to realise that even if the data comes out of Outlook as multipart/mixed, it could still be transferred to TB storage as multipart/related with the proper content IDs.

Once again, if TB were to grab some of Outlook's market, it would have to deal with the conversion correctly, regardless of how bad the data comes out of Outlook.

Comment 25

[Joseph]

(In reply to comment #24)
> Once again, if TB were to grab some of Outlook's market, it would have to deal
> with the conversion correctly, regardless of how bad the data comes out of
> Outlook.

It probably is dealing with the conversion as best it can, but like you said we can't know without looking at a debug.  Oh, I got Eudora 7 working here was the source for the email it received from Outlook...

They (WinMail,Tb,Eudora) are all either getting or choosing the first part in the multipart/alternative and removing the Content-ID.  I think they are getting it like that, but I'm unsure.  It seems Eudora got the multipart/related part of the header, but they all could have gotten that and changed it due to the format of the message from Outlook.

-----------------------------------------------------------------------

From ???@??? Fri Apr 10 11:02:36 1998
MIME-Version: 1.0
Received: by 10.100.105.1 with HTTP; Mon, 19 Apr 2010 07:13:38 -0700 (PDT)
In-Reply-To: <[e-mail]>
References: <[e-mail]>
Date: Mon, 19 Apr 2010 11:13:38 -0300
Delivered-To: [e-mail]
Message-ID: <[e-mail]>
Subject: Fwd: testing again
From: JL <[e-mail]>
To: [e-mail]
Content-Type: multipart/related; boundary=0016368e1ef23f52000484979114


--(null)
Content-Type: text/plain; charset="us-ascii"


---------- Forwarded message ----------
From: JL <[e-mail]>
Date: Mon, Apr 19, 2010 at 2:04 AM
Subject: testing again
To: [e-mail], C <[e-mail]>


---There should be an image BELOW this line---
image
---There should be an image ABOVE this line---



--(null)
Content-Type: image/jpeg; name="image.jpg";
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="image.jpg"


/9j/4AAQSkZJRgABAQEASABIAAD/2wBDABsSFBcUERsXFhceHBsgKEIrKCUlKFE6PTBCYFVlZF9V
(... the rest of the base64 for the image deleted ...)
--(null)--

--=====================main_12747808==_--

Comment 26

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #15)
> Test Email directly to Tb (with names changed)
(In reply to comment #16)
> Email to Outlook from Gmail, then imported to Tb (names changed)

Joseph, thanks for checking. Problem is now clear, because we could see original mail data by your test.
Can you check next additionally to see "original data" in Outlook?
(1) Save the mail in .eml file by Outlook.
(2) Upload the mail by Outlook to Gmail via Gmail IMAP.
    (2-1) At Gmail, delete original mail.
          Gmail Web  : Delete (moved to Trash)
          Gmail IMAP : Move mail(in [Gmail]/Sent Mail) to [Gmail]/Trash.
          This step is required because Gmail won't keep uploaded data
          at step 2-2 if this step is skipped.
    (2-2) At Outlook, copy the mail to Gmail's Inbox.
(3) Forward as attachment to Gmail account by Outlook.
Is orignal mail data kept? (Content-Type:, mail structure, Content-ID:)

(In reply to comment #25)
> Message-ID: <[e-mail]>
> Subject: Fwd: testing again
> Content-Type: multipart/related; boundary=0016368e1ef23f52000484979114
> 
>(snip) 
> --(null)
> Content-Type: text/plain; charset="us-ascii"
> 
> ---------- Forwarded message ----------
> Subject: testing again
>(snip) 
> --(null)
> Content-Type: image/jpeg; name="image.jpg";
> Content-Transfer-Encoding: base64
> Content-Disposition: attachment; filename="image.jpg"
>(snip) 
> --(null)
>
> --=====================main_12747808==_--

Is this mail source obtained by next?
  (1) Inline forward of test mail to Gmail account by Outlook.
  (2) Receive or download the inline forwarded mail by Eudora7
  (3) View mail source by Eudora7
Even though multipart/related mail, mail body part is text/plain?
boundry of "--(null)" or "--...mai..." even though boundary=0016368...?
Or some parts in multipart/alternative part in multipart/mixed part in multipart/related mail?

Comment 27

[Jorg K (CEST = GMT+2)]

OK. I understand Wada wants to get closer to the E-Mail that is kept inside Outlook. Here is what I did.

Using my own IMAP server (it comes with my webhosting) I moved the message in question from Outlook to the IMAP server. Then I received it with TB to the TB inbox via POP.

Result: The message is correct with multipart/related etc.

The entire message is shown below.

From - Tue Apr 20 09:59:59 2010
X-Account-Key: account1
X-UIDL: UID23350-1176703520
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:                                                                                 
Return-Path: <[e-mail]>
From: "Henk van Leeuwen" <[e-mail]>
To: =?iso-8859-1?Q?J=F6rg_Knobloch?= <[e-mail]>
References: <5B58881003D047009AE98C09C86A57BA@toad> <[e-mail]> <CDC3D74188C74F3385C7239813265F13@toad> <[e-mail]> <322233605EA943958E0130C47DE6117B@toad> <8A2A5971970343D9A8BF0F998865930E@toad> <AA2932F3F07E4C29B79F26FA23F66286@toad>
Subject: OK: Updated: CD listings - to be automated
Date: Sat, 10 Apr 2010 02:08:59 +0200
Message-ID: <2893E3D9DDD3487790D0AA19CFDA977A@toad>
MIME-Version: 1.0
Content-Type: multipart/related;
	boundary="----=_NextPart_000_0000_01CAE06F.7A3ADC70"
X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9
Thread-Index: AcrYQgeTnusiW7GMTtWX/1pvGOq4Lw==
In-Reply-To: <AA2932F3F07E4C29B79F26FA23F66286@toad>
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5579
X-OlkEid: FF240020A7E290764070754F9F69E7F82E36303E

This is a multi-part message in MIME format.

------=_NextPart_000_0000_01CAE06F.7A3ADC70
Content-Type: multipart/alternative;
	boundary="----=_NextPart_001_0001_01CAE06F.7A3ADC70"


------=_NextPart_001_0001_01CAE06F.7A3ADC70
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 8bit

OK Jorg - No probs. Glad I have the hang of it now - Cheers - Thanks -
Henk

At 01:28 AM 10/04/2010, you wrote:


Up to you. With this system I spend about 10 minutes on each update as
long as you provide a good list. JK.



  _____

From: Henk van Leeuwen [ <mailto:[e-mail]>
mailto:[e-mail]]


Sent: Friday, 9 April 2010 17:23


To: Jörg Knobloch


Subject: ok: Updated: CD listings - to be automated



OK - Thanks again Jorg. I will have a few new EGEA cds but they can wait
until I have say 6 entries. Ciao - Henk







Henk van Leeuwen
Australia Northern Europe Liaisons
unit 5, 2A Tiuna Grove
Elwood Vic 3184
Australia
ph: + 61 3 9525 7022
 <mailto:[e-mail]> mailto:[e-mail]
www.henk.com.au <http://www.henk.com.au/>

[]


------=_NextPart_001_0001_01CAE06F.7A3ADC70
Content-Type: text/html;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<html>

<body>
<b>OK Jorg - No probs. Glad I have the hang of it now - Cheers - Thanks =
-
Henk<br><br>
</b>At 01:28 AM 10/04/2010, you wrote:<br>
<blockquote type=3Dcite class=3Dcite cite=3D""><font size=3D2 =
color=3D"#0000FF">Up
to you. With this system I spend about 10 minutes on each update as long
as you provide a good list. JK.<br>
</font><br>

<dl><hr>

<dd><font face=3D"Tahoma" size=3D2>From:</b> Henk van Leeuwen
[<a href=3D"mailto:[e-mail]" eudora=3D"autourl">
mailto:[e-mail]</a>] <br>

<dd>Sent:</b> Friday, 9 April 2010 17:23<br>

<dd>To:</b> J=F6rg Knobloch<br>

<dd>Subject:</b> ok: Updated: CD listings - to be automated<br>
</font><br>

<dd>OK - Thanks again Jorg. I will have a few new EGEA cds but they can
wait until I have say 6 entries. Ciao - Henk</b><br>
</blockquote>
<x-sigsep><p></x-sigsep>

</dl><br>
Henk van Leeuwen<br>
<b>Australia Northern Europe Liaisons<br>
</b>unit 5, 2A Tiuna Grove<br>
Elwood Vic 3184<br>
Australia<br>
ph: + 61 3 9525 7022<br>
<font color=3D"#800080"><a href=3D"mailto:[e-mail]" =
eudora=3D"autourl">
mailto:[e-mail]</a> <br>
<b><a href=3D"http://www.henk.com.au/" =
eudora=3D"autourl">www.henk.com.au<br>

<br>
</a></b></font>
<img src=3D"cid:[e-mail].0" =
width=3D155 height=3D80 alt=3D"[]">
</b>&nbsp; </body>
</html>

------=_NextPart_001_0001_01CAE06F.7A3ADC70--

------=_NextPart_000_0000_01CAE06F.7A3ADC70
Content-Type: image/jpeg;
	name="ANEL-sig.jpg"
Content-Transfer-Encoding: base64
Content-ID: <[e-mail].0>

/9j/4AAQSkZJRgABAQIHsAewAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0a
HBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIy
MjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCABQAJsDASIA

Comment 28

[Jorg K (CEST = GMT+2)]

This only proves that internally Outlook knows about the relatedness of message and attachment. It proves that when Outlook moves the message to an IMAP server, it maintains the multipart/related header and the Content-ID:

What this test does NOT prove is how Outlook delivers the e-mail to TB upon conversion on the Simple MAPI interface.

Once again, instead of second guessing and doing all sorts of experiments "outside the box", it would be useful to look at the conversion process in debug mode to look "inside the box".

Comment 29

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #27)
> Using my own IMAP server (it comes with my webhosting) I moved the message in
> question from Outlook to the IMAP server. Then I received it with TB to the TB
> inbox via POP.
> Result: The message is correct with multipart/related etc.

Jorg K, thanks for testing and valuable result.

A simple workaround has been found.
  Upload all mails to IMAP server by Outlook.
This is same as mail data import method from other mailers to Apple Mail 2 or later, which is Apple's official recomendation after version up to Apple Mail 2.

If quota at IMAP server is small, and if you can use fast connection, and if you can accept that Google's system as mail server reads all your mail data, and if total mail data size is not so huge(less than several 10 GB), use of free Gmail account(s) for data migration is a good workaround.
This workaround is applicable to charset related issues around import from old Outlook such as Outlook 2002.

Temporary use of local hMailServer(free mail server software for MS Win) may be another workaround.
> http://en.wikipedia.org/wiki/HMailServer
> http://www.hmailserver.com/

Confirming per attached mail data files and pasted (long) mail data.

Comment 30

[Jorg K (CEST = GMT+2)]

Well, this is a very cumbersome workaround. I have several GB of mail to migrate and they sit in various folders. It would be a very labour intensive job to upload all this to an IMAP server and then receive it again. This might be OK for a few messages, but not for bulk conversion.

Therefore I think the bug should be addressed in TB to allow it to easily import Outlook e-mail correctly even for the novice user.

Can someone point me to the source code, just as a matter of interest. If it's not too hard to set up, I can try looking at it myself.

Comment 31

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #30)
> Can someone point me to the source code, just as a matter of interest.
> If it's not too hard to set up, I can try looking at it myself.

Sorry but I don't know.
Can you search bug via "Advanced Search" for Product=Mail&News Core, Component=Import, summary contains "outlook", sumary doesn't contain "express", resolution=fixed? If patch is attached to such bugs, changed sources are usually source relevant to "import from outlook".

Comment 32

[WADA:World Anti-bad-Duping Agency]

Quick summary of current findings.
1. Original mail is;
   multipart/related
     multipart/alternative
        text/plain
        text/html, <img src="cid:...">
     image/xxx, Content-Id: <...>
   Gmail also generated this structure. 
2. If mail is uploaded to IMAP server by Outlook 2003, mail structure and
   mail headers(Content-Type:, Content-Id:) are correctly kept.
3. If mail data is imported from Outlook 2003/2002 by Tb 3, Outlook Express 6,
   Windows Mail, Eudora 7,
   3-1. mail is changed to multipart/mixed
   3-2. multipart/alternative is lost
   3-3. Content-Id: of image/xxx part is lost.
   3-4. "Which of text/plain and text/html in multipart/alternative part
         is imported" seems to depend on mailer who executed import.

Comment 33

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #30)
> Therefore I think the bug should be addressed in TB to allow it to easily
> import Outlook e-mail correctly(snip)

Jorg K, do you believe that it's easy even though MS's Outlook Express 6 and MS's Windows Mail also fail to import such mails in expected structure, with expected headers?
Is there any way to export mail data to .eml file or some general format files by Outlook?
There are some tools to export pst file data of Outlook(it looks many of them are not charge free). Search Google for pst2eml, or eml2mbox, or "Outlook pst file export", or "Outlook mail storage file" etc., please.
Can you utilize such tools?

Comment 34

[Jorg K (CEST = GMT+2)]

Well, perhaps we misunderstand each other.

The bug should be addressed in TB to allow TB to easily import Outlook e-mail correctly. That means, it need to be easy for the user. The user wants to push a button and get the job done. They don't want any work-arounds, let alone difficult ones, where e-mail is moved to an IMAP server.

To actually code it in TB may be hard, very hard or even impossible. But one would need inside knowledge to determine that. The fact that Outlook Express fails just shows, the Microsoft is not interested to support migration from Outlook to Outlook Express. Why should they? They want to sell MS Office and Outlook.

Outlook doesn't export to any decent format, sorry.

Yes, there are many different tools, mostly very costly, to convert e-mail between the various formats. I don't want to use those tools. I simple want that TB becomes a better product and does a better job compared to, for example, Outlook express.

I appreciate your comments and suggestions. But can we please stop this discussion now and address the real problem. We are already at comment #33.

In my opinion:
- someone with knowledge of the TB code needs to look at the problem.
- we need to stop comparing TB with other clients which also do a bad job.
- we need to stop looking for further work-arounds, there are plenty already.
- we don't want to look at third party tools to do the job.

This is a shortcoming of TB which needs to addressed and fixed if possible.

Comment 35

[Joseph]

Code for all of Mozilla can be found: https://developer.mozilla.org/en/Viewing_and_searching_Mozilla_source_code_online

You can look at http://mxr.mozilla.org/comm-central/source/mailnews/import/ for all the import code.

Outlook exports via mapi32.dll normally (you can see where they load it: http://mxr.mozilla.org/comm-central/source/mailnews/import/outlook/src/MapiApi.cpp), and other than that (ie using the pst) requires a lot more work that Microsoft will probably not give help with.  Not saying they would help much with other things, but at least with the libraries you can find a way to get them to work on them if you can make them believe it affects their users (well, sooner or later).

As far at the output choosing only one in the multipart/alternative, you can see them address it here: http://mxr.mozilla.org/comm-central/source/mailnews/import/outlook/src/nsOutlookMail.cpp#356 .  I also found someone that made a resource (for free) for MAPI Utils, and he addresses the same issue of getting the HTML from MAPI: http://www.wischik.com/lu/programmer/mapi_utils.html ... Not sure if he tries to get the Content-ID at all, but his code might be worth looking into.  He copywrited it, but also says "The code is free and anyone can do with it whatever they like, including incorporating it in commercial products."  Even so, I'm not sure if it's useful.

reply to WADA in comment #26

I am currently unsure how to get Outlook 2002 to spit out eml or the raw data outside of it's own pst.  I was unsure of the other parts of your request, but I think I get where you are going.  Delete the data and get Outlook to repopulate it on the google server to see if there is a change in the data.  I've been meaning to clear the inbox for a while, so that shouldn't be a problem.  Hold up...  Hmm, Outlook 2002 doesn't relay support IMAP.  I want to try a few other things, hold one...  Hmm, I'm unable to get it to put onto the server with pop unless I send as a message.  Ok, forwarded message then pulled from server with Tb.  I'll upload the EML.  The format hasn't changed much.


As far as the odd data to Eudora, I'm guessing that the boundaries where supposed to be replace by Eudora, but I don't know the interface.  The other two mails that were from Outlook transferred fine, but they were not multipart emails.

Comment 36

[Joseph]

Attachment: message forwarded from Outlook to gmail and imported into Tb

You can see the Content-ID has changed, but so did the cid-url which is expected since the Content-ID is supposed to be globally unique.

Comment 37

[Jorg K (CEST = GMT+2)]

Code review. As far as I can see, Outlook import NEVER creates multipart/related messages.

Let's read it together:

http://mxr.mozilla.org/comm-central/source/mailnews/import/outlook/src/nsOutlookMail.cpp#382

nsOutlookMail::ImportMailbox

This fetches headers, body and attachments. It never looks at the MIME headers in the body that it retrieves. Then it calls compose.SendTheMessage

http://mxr.mozilla.org/comm-central/source/mailnews/import/outlook/src/nsOutlookMail.cpp#504

This we find here:

http://mxr.mozilla.org/comm-central/source/mailnews/import/outlook/src/nsOutlookCompose.cpp#571

That does a bunch of stuff and then calls m_pSendProxy->CreateAndSendMessage(

http://mxr.mozilla.org/comm-central/source/mailnews/import/outlook/src/nsOutlookCompose.cpp#649

Take a look at the arguments:
first argument (aEditor) is passed as null, The related part is passes as null as well.

http://mxr.mozilla.org/comm-central/source/mailnews/compose/public/nsIMsgSend.idl#194. 

To my eye, this is very lazily coded and never worked for multipart/related.

Who's in charge of the Outlook import code-wise??

Comment 38

[Jorg K (CEST = GMT+2)]

There is more beauty in the import code:

http://mxr.mozilla.org/comm-central/source/mailnews/import/outlook/src/nsOutlookCompose.cpp#494

Looks like multipart is actively ignored.

Comment 39

[Joseph]

(In reply to comment #38)
> There is more beauty in the import code:
> 
> http://mxr.mozilla.org/comm-central/source/mailnews/import/outlook/src/nsOutlookCompose.cpp#494
> 
> Looks like multipart is actively ignored.

I think they ignore it because of the way it is sent to them.  If you look at the format that Eudora spit out for the multipart message it received, you can kind of see what they were dealing with.  I went through some of the code like you did, and didn't see any reference to Content-ID.  I'm not even sure if Outlook sent it to them.  If you want you can CVS (or whatever they use) the code and change it yourself.  I pulled the code once because I didn't realize that I could just search it online.  Either way, if you have an idea how it should be written then make the changes yourself.  You can see what how the functions are set up, plus you can track through other functions to make sure you cause any major flaws, and make the changes (if you have the time).  Then try to get them implemented.  I say not make any major flaws because sometimes code in large applications that isn't on par is made up for by other code and fixing it can create a problem with that code, but you probably already know that.

If you want to be a developer for Thunderbird check out https://developer.mozilla.org/en/Thunderbird , just remember that a lot of people do work for free on this "lazily coded" application and sometimes don't have time to sit down and spend the time it takes to make it polished.  They have to make money too...  just sayin'

Comment 40

[Jorg K (CEST = GMT+2)]

I'm setting up a machine with MS V++ 2005 Express to look at the problem.

I'm also interested to fix bug 378008, bug 547119 and perhaps bug 309932. The former two are very simple and can be fixed with 3-5 additional lines of code.

Comment 41

[Jorg K (CEST = GMT+2)]

Just for those who are interested: I did a bit of debugging, and here is what TB retrieves from Outlook via the MAPI interface. So the multipart/related header is maintained and there is no reason, why the message should be migrated badly:

MAPI returned these headers:
============================

Return-path: <[e-mail]>
Envelope-to: [e-mail]
Delivery-date: Fri, 09 Apr 2010 17:09:04 -0700
Received: from cpe-124-191-11-14.ewqd1.win.bigpond.net.au ([124.191.11.14] helo=henk.henk.com.au)
	by danu.lunarservers.com with esmtps (TLSv1:AES256-SHA:256)
	(Exim 4.69)
	(envelope-from <[e-mail]>)
	id 1O0OFz-0005EY-J5
	for [e-mail]; Fri, 09 Apr 2010 17:09:04 -0700
X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9
Date: Sat, 10 Apr 2010 10:08:59 +1000
To: =?iso-8859-1?Q?J=F6rg?= Knobloch <[e-mail]>
From: Henk van Leeuwen <[e-mail]>
Subject: OK: Updated: CD listings - to be automated
In-Reply-To: <AA2932F3F07E4C29B79F26FA23F66286@toad>
References: <5B58881003D047009AE98C09C86A57BA@toad>
 <[e-mail]>
 <CDC3D74188C74F3385C7239813265F13@toad>
 <[e-mail]>
 <322233605EA943958E0130C47DE6117B@toad>
 <8A2A5971970343D9A8BF0F998865930E@toad>
 <AA2932F3F07E4C29B79F26FA23F66286@toad>
Mime-Version: 1.0
Content-Type: multipart/related;
	type="multipart/alternative";
	boundary="=====================_149820703==.REL"
X-ACL-Warn: {


MAPI returned this body:
========================

<html>
<body>
<b>OK Jorg - No probs. Glad I have the hang of it now - Cheers - Thanks -
Henk<br><br>
</b>At 01:28 AM 10/04/2010, you wrote:<br>
<blockquote type=cite class=cite cite=""><font size=2 color="#0000FF">Up
to you. With this system I spend about 10 minutes on each update as long
as you provide a good list. JK.<br>
</font><br>

<dl><hr>

<dd><font face="Tahoma" size=2>From:</b> Henk van Leeuwen
[<a href="mailto:[e-mail]" eudora="autourl">
mailto:[e-mail]</a>] <br>

<dd>Sent:</b> Friday, 9 April 2010 17:23<br>

<dd>To:</b> Jörg Knobloch<br>

<dd>Subject:</b> ok: Updated: CD listings - to be automated<br>
</font><br>

<dd>OK - Thanks again Jorg. I will have a few new EGEA cds but they can
wait until I have say 6 entries. Ciao - Henk</b><br>
</blockquote>
<x-sigsep><p></x-sigsep>

</dl><br>
Henk van Leeuwen<br>
<b>Australia Northern Europe Liaisons<br>
</b>unit 5, 2A Tiuna Grove<br>
Elwood Vic 3184<br>
Australia<br>
ph: + 61 3 9525 7022<br>
<font color="#800080"><a href="mailto:[e-mail]" eudora="autourl">
mailto:[e-mail]</a> <br>
<b><a href="http://www.henk.com.au/" eudora="autourl">www.henk.com.au<br>
<br>
</a></b></font>
<img src="cid:[e-mail].0" width=155 height=80 alt="[]">
</b>&nbsp; </body>
</html>

Comment 42

[Joseph]

(In reply to comment #41)
> Just for those who are interested: I did a bit of debugging, and here is what
> TB retrieves from Outlook via the MAPI interface. So the multipart/related
> header is maintained and there is no reason, why the message should be migrated

Ok, were you able to get both (or all) parts of the body when getting a multipart/alternative?

Is the content-ID of the images (or other parts of the multipart/related) maintained in some way?

If you can get these working, then maybe you should go ahead and start work on the fix.

Comment 43

[Jorg K (CEST = GMT+2)]

Only one part was delivered. I promise, all I saw is logged above in comment #41. But we don't need the alternate parts, we only need the related parts. Take a look at this bug https://bugzilla.mozilla.org/show_bug.cgi?id=250878 that tells you that MAPI only ever delivers HTML anyway.

As far as I saw, there is no code to retrieve the Content-IDs. So the change will need to be:

1) retrieve the Content-IDs (I hope there is a MAPI call to do it),
   store them in the object.
2) not obliterate the multipart/related header and call the function that
   creates the imported message in TB (as discussed above) with the correct
   arguments.

Sadly it's Sunday and I have some other bugs on my plate, but it will happen ...

Comment 44

[Joseph]

(In reply to comment #43)
> Only one part was delivered. I promise, all I saw is logged above in comment
> #41. But we don't need the alternate parts, we only need the related parts.
> Take a look at this bug https://bugzilla.mozilla.org/show_bug.cgi?id=250878
> that tells you that MAPI only ever delivers HTML anyway.
> 
> As far as I saw, there is no code to retrieve the Content-IDs. So the change
> will need to be:
> 
> 1) retrieve the Content-IDs (I hope there is a MAPI call to do it),
>    store them in the object.
> 2) not obliterate the multipart/related header and call the function that
>    creates the imported message in TB (as discussed above) with the correct
>    arguments.
> 
> Sadly it's Sunday and I have some other bugs on my plate, but it will happen
> ...

If you can get all parts to an multipart/alternative especially if it only returns text (which is shouldn't) might be helpful.  If it's not possible or rather no practical way, then it shouldn't be a problem, but it would be good to retrieve the email as close as possible to the original.

If you can't get the Content-ID with the MAPI, then maybe maybe they made a conscience decision to change the header to multipart/mixed due to the fact that true multipart/related requires the Content-ID (if I remember correctly).  If there is no way to practically get that part, then there may have to be a discussion about leaving it as multipart/related when it's in a sense not.  Unfortunately, there is no way to guess from the cid used in the link, because they may be wrong (and maybe purposely) or there may be many.

Comment 45

[Jorg K (CEST = GMT+2)]

Attachment details are retrieved here:

http://mxr.mozilla.org/comm-central/source/mailnews/import/outlook/src/MapiMessage.cpp#622

You can see that the Content-ID is not retrieved.

According to this page
http://msdn.microsoft.com/en-us/library/bb446117.aspx
the content ID can also be retrieved (PR_ATTACH_CONTENT_ID).

So there is no problem to fix this bug. Sadly, or fortunately, I have fixed some other bugs which changed code in MapiMessage.cpp and nsOutlookCompose.cpp. So I will wait until those changes are committed to the tree before I'll submit more changes.

Comment 46

[Jorg K (CEST = GMT+2)]

The other changes are checked in now, so work on this is starting now.

Comment 47

[Jorg K (CEST = GMT+2)]

Attachment: This is NOT a patch to be checked in. It's just to illustrate a problem

Comment 48

[Jorg K (CEST = GMT+2)]

The submitted patch shows, how the content ID for an attachment can be retrieved through the MAPI interface and be stored in the internal structures.

However, I get to a point at 
http://mxr.mozilla.org/comm-central/source/mailnews/import/outlook/src/nsOutlookCompose.cpp#591
where I have no idea how to pass the retrieved content ID on.

There is no place for it in the nsMsgAttachedFile structure, which is defined here:
http://mxr.mozilla.org/comm-central/source/mailnews/compose/public/nsIMsgSend.idl#127

Somehow we need to pass the content ID on before calling CreateAndSendMessage at
http://mxr.mozilla.org/comm-central/source/mailnews/import/outlook/src/nsOutlookCompose.cpp#688

HELP!!

Comment 49

[David :Bienvenu]

I could be wrong, but I think to fit in with the way import works, you'd need to insert the embedded image into the editor, and let the nsMsgSend code figure out its own content-id. In other words, this is a limitation of how import works. If you edit a draft message with embedded images, does it work? If so, you might try to figure out how that works, since it's a similar problem.

Comment 50

[Jorg K (CEST = GMT+2)]

Neil suggested to me on irc to take a look at the Eudora import "for inspiration"

Indeed, this does something with "embedded images" - investigation continues.
http://mxr.mozilla.org/comm-central/source/mailnews/import/eudora/src/nsEudoraEditor.cpp#750

Comment 51

[Jorg K (CEST = GMT+2)]

The import from Microsoft Outlook has had many bugs. Mike Kaganski has totally reworked the import to fix all major bugs, ie. bug 207156 and this bug 558653 (which are the parents of many duplicates). He has made a fantastic effort. I have worked with Mike over the last few months and I have tested his changes. This week I have tested Mike's latest changes on my Outlook data from eleven years (1999-2010). I imported 2 GB worth of data in on hit without any problem.

I urge you to review and accept this patch as soon as possible. If ever TB wants to make way into Outlook territory, this patch is an absolute MUST HAVE.

===

This bug will be resolved when the changes to bug 207156 are accepted.