390795 - attachments corruption for IMAP account (truncated due to estmated/incorrect RFC822.SIZE even though server is not MS Exchange)

Status: RESOLVED DUPLICATE of bug 92111

(MailNews Core :: Networking: IMAP, defect)

Description

[Ivan Kadomin]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6
Build Identifier: version 2.0.0.6 (20070728)

I have properly configured IMAP account NONE of my folders are marked for off-line use. I do not use off-line working.
When I'm getting new e-mail items, it happens sometimes that I cant open attachments. Associated application (e.g. MS Word or Adobe Reader) reports that file is corrupted or cant be opened. If there are a number of attachments, I usually cant open the last one. However I'm able to download attachments through web-interface on my e-mail server, and files downloaded in this way can be opened without any problems. I realized that in such situations attachment files saved from Thunderbird are shorter than the original files downloaded through web-interface or by another e-mail client.
By pressing "Ctrl+U" combination I can see the e-mail message source and an attachment base64 encoded and it seems that it is not completely downloaded.

I experience the above problem since the earlier versions. The same situation happens on different machines, running under Windows XP Professional Edition SP2 and Windows 2003 Server Standard Edition SP1.

Note that I do NOT interrupt downloading of the message. Deleting IMAP mail cache also as rebuilding index doesn't help. After the same e-mail is downloaded I cant open attachment and I cant open it after saving.

Reproducible: Always

Steps to Reproduce:
1. Send an e-mail with attachment to yourself (I sent an e-mail with 80 Kb .pdf attached)
2. Close Thunderbird if opened
3. Delete IMAP mail cache for the appropriate profile and IMAP account (I deleted all .msf files and containing directory)
4. Open Thunderbird, select Inbox for appropriate IMAP account
5. After message headers are downloaded, select the header of the mail you just have sent. Do not click headers of any other messages before, select exactly the one you sent. Do not interrupt downloading message.
6. Try to open attachment, in my case Adobe Reader reported that file is corrupted
7. Save attachment using "Save As" feature and compare its filesize with original file attached. In case of my 80 Kb PDF it was about 1-2 Kb shorter.
Actual Results:  
Message is downloaded, but trying to open attachment by clicking it, or by using "Open" or "Save As" causes error in associated application. Message source  (Ctrl+U) looks truncated.

Expected Results:  
Message should be downloaded fully. User should be able to open attachments.

Comment 1

[WADA:World Anti-bad-Duping Agency]

> 7. Save attachment using "Save As" feature and compare its filesize with
> original file attached. In case of my 80 Kb PDF it was about 1-2 Kb shorter.

If download of the attachment completed with no error(both externally and internally), it may be loss of CR/LF because attachment as text or quoted printable instead of Base64.
(Q1) Mail form what mailer on what OS?
(Q2) What mail header is written for the attachment?

> Message source  (Ctrl+U) looks truncated.

It sounds same phenomenon as Bug 330829. This situation is being analyzed deeper in bug 384819.
DUP of them?

Comment 2

[WADA:World Anti-bad-Duping Agency]

Sorry but I missed your "an attachment base64 encoded".
Looks to be similar phenomenon to bug 384819. I could see some bugs when IMAP server is Exchange Server during bugzilla search with "summary contains IMAP & download".
MS's Exchange Server?

Comment 3

[Ivan Kadomin]

My outgoing SMPT server was MS Exchange, but I don't think that it is root of the problem, because of later I can download valid attachment through web-interface.

I'm not sure IMAP server is MS Exchange, perharps not. I don't know how to determine it. For testing I used my own account on the free e-mail service http://www.mail.ru and its IMAP service is imap.mail.ru

In bug 384819 the source of problem was interruption of message downloading, but I didn't it. Reindexing or clearing cache helped in that bug, in my case it doesn't.

I think it is not DUP, but more likely extension.


Below is sample message source (Ctrl+U), if it may help someway:


From [e-mail] Sat Aug 04 12:54:59 2007
Return-path: <[e-mail]>
Received: from [213.128.220.224] (port=30189 helo=[213.128.220.224])
	by mx5.mail.ru with asmtp 
	id 1IHFOM-0000YO-00
	for [e-mail]; Sat, 04 Aug 2007 12:53:47 +0400
Message-ID: <[e-mail]>
Date: Sat, 04 Aug 2007 13:53:40 +0500
From: Ivan Kadomin <[e-mail]>
User-Agent: Thunderbird 2.0.0.6 (Windows/20070728)
MIME-Version: 1.0
To:  [e-mail]
Subject: test
Content-Type: multipart/mixed;
 boundary="------------060200080805050905070205"

This is a multi-part message in MIME format.
--------------060200080805050905070205
Content-Type: text/plain; charset=KOI8-R; format=flowed
Content-Transfer-Encoding: 7bit

body text

--------------060200080805050905070205
Content-Type: application/pdf;
 name="2 =?KOI8-R?Q?=E9=EE=E6=EF=F2=ED-=F0=E9=F3=F8=ED=EF-=D2=D5=D3-02-05-2007=2E?==?KOI8-R?Q?pdf?="
Content-Disposition: inline;
 filename*0*=KOI8-R''%32%20%E9%EE%E6%EF%F2%ED%2D%F0%E9%F3%F8%ED%EF%2D%D2%D5;
 filename*1*=%D3%2D%30%32%2D%30%35%2D%32%30%30%37%2E%70%64%66
Content-Transfer-Encoding: base64

JVBERi0xLjQNJeLjz9MNCjExNSAwIG9iaiA8PC9MaW5lYXJpemVkIDEvTCAxODk4ODgvTyAxMTcv
RSAxNzk4ODUvTiAxL1QgMTg3NTQwL0ggWyAxODEwIDMxM10+Pg1lbmRvYmoNICAgICAgICAgICAg

.......... I truncated a number of base64 lines except last two, ...........
.......... there are nothing more in message source after them.  ...........

MDQnMDAnKS9fQWRIb2NSZXZpZXdDeWNsZUlEKC0xNTU1MjQ3MzQ3KS9Db21wYW55KP7/BBwEEwQi
BCMpL1NvdXJjZU1vZGlmaWVkKEQ6MjAwNzA1MDMxMjM0NDApL1RpdGxlKP7/BB

I attached 185 Kb PDF, when I did "Save As..." it became 183 Kb. It looks like not all base64 lines are present and there is no usual ending line, such as:
--------------060200080805050905070205--

Comment 4

[WADA:World Anti-bad-Duping Agency]

Problem when IMAP of MS Exchange is Bug 92111, and it looks to be different issue from Bug 384819(interrupted download). 
Get IMAP protocol log for step 1 through step 5 in your comment #0. 
And see flow, and check attachment related data such as length.
  http://kb.mozillazine.org/Session_logging_for_mail/news
  http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap
  http://www.mozilla.org/projects/netlib/http/http-debugging.html
I think following is helpful for first analysis.
  SET NSPR_LOG_MODULES=imap:5
If log for your problem is obtained, attach the log(do NOT paste log) to this bug(text/plain, if size is accepted).

Comment 5

[Ivan Kadomin]

Attachment: imap protocol log

Comment 6

[WADA:World Anti-bad-Duping Agency]

Attachment: Part of log of FETCH BODYSTRUCTURE and FETCH of pdf data

Unable to say whether MS Exchange server or not from log, but it looks to be same phenomenon as Bug 92111.

Comment 7

[WADA:World Anti-bad-Duping Agency]

Confirming based on IMAP protocol log.

To Ivan Kadomin(bug opner):
Will workaround of Bug 92111 Comment #45 resolve your problem?
( user_pref(mail.server.serverXX.fetch_by_chunks, false); )

Comment 8

[Ivan Kadomin]

Yes, disabling fetching by chunks resolved the problem.

Situation is questionable, looks that problem is likely to Bug 92111, except I can't determine exactly, but I more than 80% sure that http://www.mail.ru is not MS Exchange based. There are millions of accounts on this free e-mail web-service, it is very popular in .ru zone, I suppose they use something for Unix platforms, some other indirect factors tend to make me think so.

I will try to determine whether they are MS Exchange based to mark it as Bug 92111 DUP or to leave it as another case.

Comment 9

[WADA:World Anti-bad-Duping Agency]

This bug is better to be kept independently if www.mail.ru uses Unix platform(then never MS Exchange), because Bug 92111 refers to MS Exchange case only and is already sufficiently messy, but this bug is very clean. I think setting dependency is sufficient and is useful for other users who will experience same problem as yours.
Adding "truncated" in summary for ease of search.

Comment 10

[Wayne]

It looks like Thunderbird has an IMAP attachment corruption problem.  These appear to be related to the same bug:

https://bugzilla.mozilla.org/show_bug.cgi?id=390795
https://bugzilla.mozilla.org/show_bug.cgi?id=352581
https://bugzilla.mozilla.org/show_bug.cgi?id=301541

http://groups.google.com/group/Gmail-Problem-solving/browse_thread/thread/28f5c7328725fa6e/67f95b8605703a48?hide_quotes=no 

Comment 11

[Sergey Yakovlev]

I have the same bug, but only with internal messanges... I think It's problem with TB & MS Exchange Server.

Comment 12

[Wayne]

The problem is not limited to exchange server.  I experienced the problem with IMAP and gmail, and the google groups link above shows similar problems from others with IMAP and gmail.  Of other reports I linked above, only one actually reported the problem while using Exchange.  All these reports have in common:

  1) Downloading mail with attachments
  2) corrupt attachment
  3 [review]) IMAP
  4) Using TBIRD 1.5 and newer

Another observation:
  - The problem happens with Exchange and gmail IMAP, perhaps other servers.

This bug is definitely related to IMAP, I haven't had any problems with tbird and POP3.

You will find a work around reported in two of the links above:

"I solved this, with a new value in about:config for Thunderbird
(Preferences->Advanced->General->Configuration Editor) or, if you
prefer, in pref.js:

user_pref(mail.server.serverXX.fetch_by_chunks, false)

where XX is the server for imap gmail."

Comment 13

[Wayne]

Others IMAP attachment reports that may be related:

https://bugzilla.mozilla.org/show_bug.cgi?id=262716
https://bugzilla.mozilla.org/show_bug.cgi?id=330829

Comment 16

[WADA:World Anti-bad-Duping Agency]

It has been found that "Gmail IMAP" has this problem, and above two DUP bugs are report when Gmail IMAP. I hope no flood of DUP bugs by Gmail IMAP...   

Comment 17

[Andreas Korthaus]

Perhaps it's a "Gmail IMAP" problem, but other software like Apple Mail has no problems with Google IMAP attachments. The attachments I can't open using Thunderbird, can be opened using Apple Mail.

Comment 18

[WADA:World Anti-bad-Duping Agency]

Seting dependency to Bug 92111(MS Exchange case), because this bug is kept open instead of DUPing to Bug 92111, as I wrote in comment #9, for ease of search & ease of problem analysis.

Comment 19

[jesse]

I have this issue now with Seamonkey 1.1.7, and seems like it really only started happening in the last several minor releases.
 Perhaps this bug is related to 215005.  The corruption does seem to have predilection for pdf files.

Comment 22

[WADA:World Anti-bad-Duping Agency]

See Bug 92111 Comment #45 for workaround by next setting.
> user_pref(mail.server.serverXX.fetch_by_chunks, false);

Comment 24

[Schmidma]

We can confirm the problem also with the Cyrus Mailserver using IMAP.
The Clients are Windows XP SP3 Clients with Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.9) Gecko/20100317 Lightning/1.0b1 Thunderbird/3.0.4

Comment 25

[Nelson Bolyard (seldom reads bugmail)]

I'm running TB 3.0.5 on Mac OS/X Leopard.  I have several IMAP accounts, 
one of which is served by servers administered by GoDaddy.  No Exchange 
servers involved in this account.

Yesterday, my wife sent me a very important document as an attachment.  
It was a single page PDF file.  The PDF file size is just under a megabyte,
about 1.33 MB when base64 encoded.  

Evidently TB truncated it when TB downloaded it from the IMAP server.  

The final multi-part boundary at the end of the final part of the multi-part 
message does not appear in the viewed message source.  When I attempt to 
open the attachment, with Acrobat reader, Acrobat tells me the document has 
been damaged.  

The MIME structure of this message is
   Multi-part/mixed
      Multi-part/alternative
         text/plain
         text/html
      Application/pdf

The last of these parts is truncated.

I have tried and tried to get TB to re-download this message, but it will not.

This IMAP account is configured in TB to NOT "keep messages for this account 
on this computer".  I don't want it to store messages for this account on this 
computer's hard drive, at least not when TB is not running.  Despite that, TB appears to keep the messages for all the IMAP folders for this account in a single file named 
~/Library/Thunderbird/Profiles/XXXXXXXX.default/global-messages-db.sqlite

That file contains the truncated copy of this mail message.

I suspect that if I could get TB to forget that copy, it might download a complete copy.  But it's not clear how to get TB to forget the copy of that
message, or of all messages that belong in that IMAP folder, from that db.
I tried "rebuild index" to no avail.

I wasted damn near a whole day trying to read that message.  Finally I gave 
up and had her call and read it to me.  I was later able to read the pdf file
with an OLD copy of SeaMonkey on another computer.  

I want to know:
- how to get TB to expunge that truncated copy of that message
- how to get TB to download an untruncated copy and print it
- how to get TB to STOP storing copies of the messages in the IMAP folders
for this account in global-messages-db.sqlite

Comment 26

[Nelson Bolyard (seldom reads bugmail)]

My wife sent the message using Microsoft Office Outlook 12.0, which sounds 
very much like bug 301541.
It was a PDF attachment (sounds like bug 352581)

The attachment is NOT corrupt on the IMAP server.  
The copy downloaded by TB on my MAC is corrupt.

Comment 27

[David :Bienvenu]

did you try clearing your disk cache? preferences, advanced, network and disk space, clear now.

Comment 28

[Jörg Bäuerle]

I'm having the same problem with Thunderbird and my IMAP-account from the German mail provider GMX.COM.

An attached video file, size 1.64 MB, ended up as a 345 KB attachment in Thunderbird. When I download it via my mail provider's web interface though, the attached file is 1.64 MB and everything's fine.

Comment 29

[Jörg Bäuerle]

Oh sorry, I forgot: I'm running Thunderbird 3.0.6 under Windows XP Professional Service Pack 3.

Comment 31

[Martin Wildam]

Thunderbird 3.1.10 on Ubuntu 10.04 - same problem.

Workaround as described here worked: http://allanmcrae.com/2010/05/thunderbird-vs-excange-imap/

The link describes that Exchange Server does behave standard conform and it is a fault of Thunderbird - this time.

Comment 32

[Martin Wildam]

Another bug also mentions that settings - and two more:
https://bugzilla.mozilla.org/show_bug.cgi?id=434054#c9

Comment 33

[[e-mail]]

The problem persist in TB 6.0.2 with Mac OS X 10.6.7: 95% of attachments will be corrupt when downloaded from IMAP accounts. But if you download it from webmail, everything fine. Doesn't matter if PDF, ZIP, Doc, etc, etc.

Two more identical bugs are 560346 and 434054.

Thanks

Andrea Moretti from Italy

Comment 34

[:Irving Reid (No longer working on Firefox)]

This was fixed by bug 92111; should we dupe this to that bug to close it, or resolve this bug separately?

Comment 35

[David :Bienvenu]

I think I'd dup this bug to that bug, but change the subject of the other bug to say it's not just Exchange servers.