564755 - last Boundary missing on Multipart Message (Wrong/smaller RFC822.SIZE is returned by MS Exchange Server 2003)

Status: RESOLVED DUPLICATE of bug 92111

(Thunderbird :: Message Reader UI, defect)

Description

[stephan.helas]

User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.3) Gecko/20100423 Ubuntu/10.04 (lucid) Firefox/3.6.3 (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.9) Gecko/20100423 Lightning/1.0b1 Thunderbird/3.0.4

An Attachment is not always displayed on Multipart Messages. Sometimes the Mail keeps empty. The last Boundary String is not displayed in the Source Window, so thunderbird might think the E-Mail isn't complete. After several Reloads, the Attachment is displayed as the last Boundary apears in the Source Code.


Sample Source Code:

This is a multi-part message in MIME format.

------=_NextPart_000_19E646A_01CAF020.9FA08CA1
Content-Disposition: inline
Content-Length: 90
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset="iso-8859-1"

Dies ist eine automatisch generierte E-Mail, bitte antworten Sie nicht =
auf diese E-Mail.


------=_NextPart_000_19E646A_01CAF020.9FA08CA1
Content-Disposition: attachment;
	filename="monthly_2010-05-10.xls"
Content-Transfer-Encoding: base64
Content-Type: application/vnd.ms-excel;
	name="monthly_2010-05-10.xls"

0M8R4KGxGuEAAAAAAAAAAAAAAAAAAAAAPgADAP7/CQAGAAAAAAAAAAAAAAAE
AAAAlAEAAAAAAAAAEAAA/v///wAAAAD+////AAAAAJUBAACWAQAAlwEAAJgB
AAD/////////////////////////////////////////////////////////
...
...
...
////////////////////////////////////////////////////////////
////////////////////////////////////////////////////////////
/////////////////////////////////////////////////////w==



Reproducible: Sometimes




The E-Mail was created by Perl MIME:lite and Send to Exchange Server. Other Mail-Clients show the Attachment properly.

Comment 1

[Ludovic Hirlimann [:Usul]]

Can you provide the complete headers ? Is this mulitpart/mixed , multipart/related ? Having a complete example would help for analysis.

Comment 2

[stephan.helas]

The mail is generated by Perl and uses "multipart/mixed". After reloading the message, the attachment is shown in thunderbird. 

Attached you will find the complete Header of a perl-generated Mail and the Test-Script. i have deleted the From-Lines on Top.

Comment 3

[stephan.helas]

Attachment: attachment for the generated mail

Comment 4

[stephan.helas]

Attachment: perl script for generating email

Comment 5

[stephan.helas]

Attachment: This Header is shown by Using CTRL+U on Preview

Comment 6

[stephan.helas]

Attachment: Screenshot before preview

Comment 7

[stephan.helas]

Attachment: Screenshot on preview

Comment 8

[Ludovic Hirlimann [:Usul]]

I think this is a duplicate of bug 149771 - what do you think ?

Comment 9

[stephan.helas]

i'm not sure. It seams so. I'll check the beta.

Comment 10

[WADA:World Anti-bad-Duping Agency]

> Bug summary : last Boundary missing on Multipart Message

Questions to avoid my misunderstanding.
Are you intensionally testing no close boundary line case by mail data of no close boundary? Does your perl script generate close boundary line?
Is mime-type of message/rfc822 for attached "attachment for the generated mail"(test.xls) intentional?
Which is problem of this bug?
(a) Close boundary is not shown by View/Message Source even though it exists in
    original sent mail.
(b) Nothing is displayed if no close boundary.
(c) Exchange server sometimes doesn't send close boundary line.
(d) other
If (b), it's not bug of a mail client, because RFC never defines how mail of corrupted message header is shown to user, although quirks for torelance with bad header is better to be implemented.

Does Exchange server return close boundary correctly?
Exchange server has well known RFC violation on rfc822.size(Bug 92111) for long time. If Exchange server returns smaller rfc822.size, loss of last part of a mail occurs. As you say next, It's suspected.
> After several Reloads, the Attachment is displayed as the last Boundary apears in the Source Code.

Does problem occur with next pref? (restart Tb after change to avoind enexpected problem, please.)
> mail.imap.mime_parts_on_demand=false
> mail.server.default.mime_parts_on_demand=false

Comment 11

[stephan.helas]

> Questions to avoid my misunderstanding.
> Are you intensionally testing no close boundary line case by mail data of no
> close boundary? Does your perl script generate close boundary line?
> Is mime-type of message/rfc822 for attached "attachment for the generated
> mail"(test.xls) intentional?

I can't reproduce the Behavior easily. Most E-Mails are handled correct. I have seen the full Message on other Clients and can say for sure that the last boundary is transferd to the client (ngrep). But somehow thunderbird show an empty Mail-Preview. 


> Which is problem of this bug?
> (a) Close boundary is not shown by View/Message Source even though it exists in
>     original sent mail.

Not always. Mostly the last boundary is shown in ource view, but sometimes not. I'm not sure if thunderbird retransmit the Message on Source view or not. It seams to depend on the download status. 

> (b) Nothing is displayed if no close boundary.

A generated Mail with the perl script is never displayed the first time. neitgher in Preview or in mail-window. The boundary is transferred but maybe swallowed by TB. i can't say for sure because the boundary is mostly shown in source view but not always.

> (c) Exchange server sometimes doesn't send close boundary line.
no, the mail is transferred completly. 

> (d) other
> If (b), it's not bug of a mail client, because RFC never defines how mail of
> corrupted message header is shown to user, although quirks for torelance with
> bad header is better to be implemented.
> 
> Does Exchange server return close boundary correctly?
> Exchange server has well known RFC violation on rfc822.size(Bug 92111) for long
> time. If Exchange server returns smaller rfc822.size, loss of last part of a
> mail occurs. As you say next, It's suspected.
> > After several Reloads, the Attachment is displayed as the last Boundary apears in the Source Code.
> 
> Does problem occur with next pref? (restart Tb after change to avoind
> enexpected problem, please.)
> > mail.imap.mime_parts_on_demand=false
> > mail.server.default.mime_parts_on_demand=false

After setting this, the test mail is shown correct in preview. it may be an exchange problem but i wonder that a lot of other clients (outlook, evolution, OWA, Python Imap Client) read the mail without this problem. imho it could also be a problem with the parsing while using the demand mode. 

One idea. if you are right then the rfc822.size sended by exchange must be recalculated by tb. once the mail header is cached by exchange the mail seams to be correct. i don't know how to reproduce or evince that. i can't delete the cached headers from tb.

Comment 12

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #11)
> > Does problem occur with next pref? (restart Tb after change to avoind
> > enexpected problem, please.)
> > > mail.imap.mime_parts_on_demand=false
> > > mail.server.default.mime_parts_on_demand=false
> After setting this, the test mail is shown correct in preview.

I can't imagine other than phenome of Bug 92111(RFC violation of Exchange).

Do you set Folder Properties/Synchronization, offline-use=On? Or offline-use=off?
If offline-use=on, next can be explained.
> After several Reloads, the Attachment is displayed as the last Boundary apears in the Source Code.

Other workaround of problem of "Nothing is displayed if close boundary is lost" may be "View/Display Attachments/Inline" for relativery large multipart/xxx mails. As mail body part(text/plain or text/html part) only is downloaded first, loss of some header data due to wrong RFC822.SIZE possibly doesn't occur even if wrong RFC822.SIZE is returned. Problem is probably popular data corruption upon download of attachment only.

> it may be an exchange problem but i wonder that a lot of other clients
> (outlook, evolution, OWA, Python Imap Client) read the mail without this problem.
> imho it could also be a problem with the parsing while using the demand mode. 
> One idea. if you are right then the rfc822.size sended by exchange must be
> recalculated by tb.

Have you read and understood Bug 92111? (See also Bug 390795, and dups of Bug 403032, Bug 403039)
Get IMAP log and check flow, and see incorrect or correct RFC822.SIZE value returned from Exchange server by yourself, please.
> https://wiki.mozilla.org/MailNews:Logging

> once the mail header is cached by exchange the mail seams to be correct.
> i don't know how to reproduce or evince that.

What action by Exchange server do you mean by your "the mail header is cached by exchange"?

By the way, Bug 92111 was opened on 2001-07-24. Why MS doesn't fix bug of RFC violation by Exchange yet? Or MS already fixed bug but very old Exchange Servers are still used by many companies or old Exchange server is used without sufficient maintanance by many companies?

Comment 13

[WADA:World Anti-bad-Duping Agency]

(Correction of comment #12)
It should be next.
> Other workaround of problem of "Nothing is displayed if close boundary is lost"
> may be View/Display Attachments Inline=OFF (snip)

Comment 14

[WADA:World Anti-bad-Duping Agency]

If offline-use=on, see also Bug 518702(fixed by Tb 3.0) and Bug 534835(fixed by Tb 3.1).

Comment 15

[WADA:World Anti-bad-Duping Agency]

What is version of your MS Exchange server? Newest or beta instead of old Exchange server?

Comment 16

[WADA:World Anti-bad-Duping Agency]

You could reproduce problem many times, even though not always, with test mail generated by your Perl script?

Comment 17

[WADA:World Anti-bad-Duping Agency]

Tb knows close boundary line or not depends on Tb's fetch.

(1) If folder of offline-use=off, or if mail is not downloaded yet by auto-sync for folder of offline-use=on, text/plain part only is fetched upon mail view, because of mime_parts_on_demand=true and multipart/mixed mail, and because attachment part is not displayed in inline, then mail data is saved in Disk Cache.
> Subject: big mail
> Content-Type: multipart/mixed; boundary="------------060901010509010104000400"
>
> --------------060901010509010104000400
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
> Content-Transfer-Encoding: 7bit
>
> zzz
> --------------060901010509010104000400
> X-Mozilla-IMAP-Part: 2
> Content-Type: application/pdf; name="Pharo-PBE1-2009-10-28.pdf"
> Content-Transfer-Encoding: base64
> Content-Disposition: attachment; filename="Pharo-PBE1-2009-10-28.pdf"
>
> This body part will be downloaded on demand.
> --------------060901010509010104000400--
See files in Disk Cache and IMAP log attached to Bug 565852, please.

(2) If View/Message Source is executed at this step, fetch for whle mail data is executed, but it's probably not saved in Disk Cache as valid downloaded whole mail data.

(3) After that, if offline-use=on, fetch of whole mail data is downloaded by auto-sync sooner or later and saved in offline-store. file
Close boundary is probably written in offline-store file always at (3) in your case, then close boundary line is displayed after a while by View/Source.

I guess parameters used in fetch at (3) by Tb is different from one used at (2), or rfc822.size at (2) is different from one at (3), or header data returned by Exchange at (1), (2), (3) are different.

Comment 18

[WADA:World Anti-bad-Duping Agency]

Your Exchange was Microsoft Exchange Server 2003.  
> x-mimeole: Produced By Microsoft Exchange V6.5
> http://support.microsoft.com/kb/158530

Comment 19

[stephan.helas]

(In reply to comment #12)
>What action by Exchange server do you mean by your "the mail header is cached
>by exchange"?

Sorry, i didn't mean Exchange but the Thunderbird Cache for Headers. 


(In reply to comment #13)
> Other workaround of problem of "Nothing is displayed if close boundary is lost"
> may be View/Display Attachments Inline=OFF (snip)

I don't use the Offline Mode.

(In reply to comment #15)
>What is version of your MS Exchange server? Newest or beta instead of old
>Exchange server?

This is Exchange 2k3 with latest patches. So it may be an Exchange Bug, but remember: all other Clients are working (including Thunderbird 2 btw.)

Comment 20

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #19)
> I don't use the Offline Mode.

I'm never talking about Offline Mode(Work Offline). I'm talking abour "Offline Use option"(Folder Properties, Synchronization).

Fetch log example by Tb 3.2xpre(Gmail IMAP).
Same FETCH command is used by View/Source and auto-sync.
(without .PEEK when View/Source because already read mail)

[1] folder of offline-use=OFF
(1) New mail arrives, first fetch of heaers
> S-ABCDEFG/A:SendData: 17 UID fetch 7 (BODYSTRUCTURE) 
> S-ABCDEFG/A:CreateNewLineFromSocket: * 1 FETCH (UID 7 BODYSTRUCTURE (("TEXT" "PLAIN" ("CHARSET" "ISO-8859-1" "FORMAT" "flowed") NIL NIL "7BIT" 42 1 NIL NIL NIL)("APPLICATION" "X-XPINSTALL" ("NAME" "ie_view-1.4-fx+mz-win.xpi") NIL NIL "BASE64" 65358 NIL ("ATTACHMENT" ("FILENAME" "ie_view-1.4-fx+mz-win.xpi")) NIL) "MIXED" ("BOUNDARY" "------------030107030908040806010308") NIL NIL)) 
> S-ABCDEFG/A:CreateNewLineFromSocket: 17 OK Success 
> S-ABCDEFG/A:SendData: 18 UID fetch 7 (BODY.PEEK[HEADER] BODY.PEEK[1.MIME] BODY.PEEK[2.MIME]) 
> S-ABCDEFG/A:CreateNewLineFromSocket: * 1 FETCH (UID 7 BODY[HEADER] {681} 
> S-ABCDEFG/A:STREAM:OPEN Size: 0: Begin Message Download Stream
> S-ABCDEFG/A:CreateNewLineFromSocket: Return-Path: <[e-mail]> 
>(snip)
> S-ABCDEFG/A:CreateNewLineFromSocket: Subject: mail-001-multipart-mixed-with-attachment 
> S-ABCDEFG/A:CreateNewLineFromSocket: Content-Type: multipart/mixed; boundary="------------030107030908040806010308" 
> S-ABCDEFG/A:CreateNewLineFromSocket:  BODY[2.MIME] {181} 
> S-ABCDEFG/A:CreateNewLineFromSocket: Content-Type: application/x-xpinstall; name="ie_view-1.4-fx+mz-win.xpi" 
> S-ABCDEFG/A:CreateNewLineFromSocket: Content-Transfer-Encoding: base64 
> S-ABCDEFG/A:CreateNewLineFromSocket: Content-Disposition: attachment; filename="ie_view-1.4-fx+mz-win.xpi" 
> S-ABCDEFG/A:CreateNewLineFromSocket:  BODY[1.MIME] {96} 
> S-ABCDEFG/A:CreateNewLineFromSocket: Content-Type: text/plain; charset=ISO-8859-1; format=flowed 
> S-ABCDEFG/A:CreateNewLineFromSocket: Content-Transfer-Encoding: 7bit 
> S-ABCDEFG/A:CreateNewLineFromSocket: 18 OK Success 
(2) Click mail
> S-ABCDEFG/A:SendData: 19 UID fetch 7 (BODY.PEEK[1]) 
> S-ABCDEFG/A:CreateNewLineFromSocket: * 1 FETCH (UID 7 BODY[1] {42} 
> S-ABCDEFG/A:CreateNewLineFromSocket: mail-001-multipart-mixed-with-attachment 
> S-ABCDEFG/A:CreateNewLineFromSocket: 19 OK Success 
(3) View/Message Source
> S-ABCDEFG/A:SendData: 21 UID fetch 7 (UID RFC822.SIZE BODY[]) 
[2] folder of offline-use=ON, automatic download by auto-sync
> S-ABCDEFG/B:SendData: 36 UID fetch 4 (UID RFC822.SIZE BODY.PEEK[]) 

As your test case is small mail, mime_parts_on_demand=true/false is irrelevant and next is probably issued by Tb, and wrong(smaller) RFC822.SIZE is probably returned by your Exchange server(issue of Bug 92111).  
> UID fetch x (UID RFC822.SIZE BODY.PEEK[]) or BODY[] 
Your case is probably next.
(View/Source before auto-sync)
Quirks by Bug 518702 doesn't work on Disk Cache because Disk Cache is not managed by IMAP's mail data handling code(Disk Cache is similar to HTTP cache of browser).
(Download by auto-sync for folder of offline-use=On)
Quirks by Bug 518702(ignore returned RFC822.SIZE) works on data in offline-store. So, View/Source shows close boundary.

> This is Exchange 2k3 with latest patches. So it may be an Exchange Bug, (snip)

I don't know your Exchange 2k3 really has bug explained in Bug 92111 or not. However, Exchange's bug is assumed, all funny phenomena you saw can be explained.

> but remember: all other Clients are working (including Thunderbird 2 btw.)

For (including Thunderbird 2 btw.) part.
 
Really Tb2 can be torelant with Exchange's bug of Bug 92111? "Short data of last attachment" wont't occur with Tb 2 when wrong(short) RFC822.SIZE is returned?
"Nothing is displayed" with your test case(close boundary happens to be cut by shorter RFC822.SIZE) may be Tb 3 speciific phenomenon.

For "all other Clients are working" part.

If so, why next pages exist in Web? Have you read Web pages found by Google search for "ms exchange rfc822.size"?
> http://josefsson.org/nnimap/buggy-imap-servers.html
> http://lumbgaps.blogspot.com/2009/04/exchange-2007-imap-rfc822size-and.html
>   Pine/Alpine looks to create next option. 
>     Set-CASMailbox "username" -ImapUseProtocolDefaults:$false
>     -ImapEnableExactRFC822Size:$true
Some mailers forunately don't know how to use RFC822.SIZE value, some mailer have quirks for Exchange's bug, and, some mailers have to obey MS, ...

Next opion and/or feature of ignoring RFC822.SIZE returned from server(when MS Exchange or some others, if it can be known by greeting) may be required. 
> quirks.ignore_wrong_RFC822_SIZE_by_MS_Exchange_since_2001=true
If MS Exchange returns nnn(WRONG_VALUE) as RFC822.SIZE value in FETCH response, such option nor check of greeting is not required. :-)

Comment 21

[WADA:World Anti-bad-Duping Agency]

Bug 449416 looks request for torelance with the MS's intensional RFC violation in Exchange.

Comment 22

[stephan.helas]

(In reply to comment #20)
> (In reply to comment #19)
> > I don't use the Offline Mode.
> 
> I'm never talking about Offline Mode(Work Offline). I'm talking abour "Offline
> Use option"(Folder Properties, Synchronization).

sorry, that's what i meant.

> For "all other Clients are working" part.
> 
> If so, why next pages exist in Web? Have you read Web pages found by Google
> search for "ms exchange rfc822.size"?
> > http://josefsson.org/nnimap/buggy-imap-servers.html
> > http://lumbgaps.blogspot.com/2009/04/exchange-2007-imap-rfc822size-and.html

i have referred to the clients i used for testing.

Comment 23

[WADA:World Anti-bad-Duping Agency]

[e-mail], if you can reproduce problem(lost close boundary) with mail data genrated by your script, get IMAP log for next and check returned RFC822.SIZE value, please.
> https://wiki.mozilla.org/MailNews:Logging
1. Create two IMAP folders, a1(offline use=off), a2(offline use=on).
2. Start Tb with logging enabled.
3. copy the mail to a1, view mail, view/messages source
4. copy the mail to a2, wait for download by auto-sync.
   after download to offline-store file, view mail, view/message source.

By the way, if you don't set offline-use=on, I think you see Bug 565852 on multipart/mixed mail with Tb 3.0. You didn't see such phenomenon?

Comment 24

[WADA:World Anti-bad-Duping Agency]

There was already known workaround. See comments pointed in Bug 92111 #87, please.
Closing as DUP of Bug 92111. If different problem, re-open, please.

Comment 25

[WADA:World Anti-bad-Duping Agency]

Woops, Bug 92111 Comment #87.