92111 - IMAP: Attachments not fully downloaded (due to incorrect/smaller RFC822.SIZE by MS Exchange, gmail, and possibly others)

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Artem Saveliev]

Using _both_ Netscape Communicator 4.76 and Mozilla 0.9.2 I have last attachment
in the list truncated (if it's only one attachment than it happens only
sometimes). That happens to all types of attachments (pictures - you can see
them truncated, Word documents). Outlook Express views/downloads thouse files
normaly. Same happened in older Communicators (4.7 etc) and Mozillas (M18 I think).
Using Microsoft Exchange 5.5 Build 2653.23 SP4 with default Internet Mail
Service settings.

Comment 1

[Scott MacGregor]

.9.3 is due out in about a week. Can you test against that? We fixed some
attachment discovery bugs between .9.2 and .9.3. 

 At: 
http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap

you'll find instructions for generating an imap log which will show us
information about why we may not be showing your attachment. Just click on a
message giving you this problem. Then quit the app and send us the log file. Thanks!

Comment 2

[Artem Saveliev]

Using 2001072403.
While playing around to create a good test example I noticed that it actualy
truncates the whole file sometimes. Say I attach 4 files, and it only shows me
3, also downloading only part of third file.
Attaching log file.

Comment 3

[Artem Saveliev]

Attachment: IMAP log file

Comment 4

[Artem Saveliev]

Attachment: 0.9.3 log - same story

Comment 5

[Keyser Sose]

Marking NEW.

Comment 6

[Bob Horvath]

Sounds like a similar problem that I have.  A workaround I have found is to copy
the message (usually by control-dragging) to the local Drafts folder, and then
back to the INBOX.  All of a sudden it can read the entire message again.

Comment 7

[Artem Saveliev]

Yes Bob, I can verify that, and it makes my life a little bit easier. However,
that only works in NN4 for me.
After that I am able to view attachment from the local folder and IMAP folder
using NN4. Mozilla is still not able to view that new e-mail's attachment correctly.

Comment 8

[Bob Horvath]

Hmmm, perhaps we have slightly different experiences.  The thing I have noticed
today is using Mozilla, that I can copy it to my IMAP folder named Drafts,
instead of the local version, and then copy it back, and be able to read the new
copy of it.

I wonder if there is something special about Drafts.  I also have an IMAP folder
that I created called Unreadable that I put these kinds of things to see if I
can detect a pattern.  They are still unreadable there.  I got used to using
Drafs as a staging area to do things like delete attachments with NN4.

The other thing I find curious about all of this is that if I do a View Source,
I don't get the entire message from the IMAP folder.  Is this showing me what
was actually downloaded, or what was parsed?  I have played around with imaplib
in Python, and that seems to have no problems downloading the entire thing, as
does mutt for me, and pine for others.

Any suggestions as to see what is actually coming back from the Exchange server?

Comment 9

[Artem Saveliev]

Ok, to clarify my experience
1) The same behavior is shared by Netscape Navigator (4.78) and Mozilla
2) The most obvious way to see it is by looking at attached picture, whether
inline or by clicking on it. It's truncated (note that it doesn't 
happen always).
3) That doesn't happen always.
4) In NN, dragging message from IMAP folder to any local folder fixes the
problem. Same thing doesn't work in Mozilla (tested on today's 
nightly)

To answer the last question, you can find instructions on enabling logging here
http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap
And I have a couple of logs attached to this bug already.

Comment 10

[Henrik Gemal]

I'm almost certain that what you're seeing is bug the same thing I can see in 
bug 105606.

I've had this bug for ages in Netscape 4.x and in Mozilla.

This bug is the main reason why many of my co-workers switched to Outlook since 
copying mails to local folder just isn't a solution that you can have in your 
daily work.

So to clarify: this bug is also present in Mozilla build 20011018

Comment 11

[Artem Saveliev]

I went into details of IMAP RFC and Henrik's note that Outlook uses different
command also helped.

I believe that Mozilla behaviour is OK, it's Exchange problem.
Check out http://www.faqs.org/rfcs/rfc2683.html part 3.4.5
I think it will require tweaking Mozilla specificaly to work with Exchange.

Comment 12

[Bob Horvath]

I had read about the size issue long ago in some fetchmail documentation. 
Knowing that Microsoft will most likely not change Exchange any time soon, does
this not fall into the are of "quirks" vs. evangelism? 

The one thing I don't understand is why it works if you copy it into another
IMAP folder.  Perhaps Exchange calculates the size correctly then. Time to learn
how to make IMAP logs.

Is there any chance of this being implemented in Mozilla as a quirk?

Comment 13

[Bob Horvath]

One more workaround...

File/Offline/GetSelectedMessages will download the whole thing.  Sometimes you
have to select another message and come back to see it, but it is there.

Comment 14

[Bob Horvath]

I forgot all about this bug.  I suspect it is fixed.  What is the process for
closing a bug?  Did someone have to claim to have fixed it?

Comment 15

[Keyser Sose]

Well since your not seeing it anymore I will go ahead and mark it WORKSFORME.
Thanks for getting back.

Comment 16

[Henrik Gemal]

this is not working. I'm hit on a daily basis by this bug.

Comment 17

[Bob Horvath]

ANything reproducible?  I could test it at my end.

Comment 18

[Henrik Gemal]

this normally happens when trying to open large attachments like Word documents
without first saving them to the disk. Because of this Exchange feature the
document are chopped... You can also just use view source where you can see that
some stuff (the last bytes) are missing from the mail

Comment 19

[Artem Saveliev]

Basicaly Mozilla uses martial retreival method (as described in bug 105606):
UID fetch 4519 (UID RFC822.SIZE BODY[]<0.10240>)
and Outlook Express retreives all:
UID FETCH 4519 (BODY.PEEK[] UID)

RFC I refered to (rfc 2683 part 3.4.5) does not require IMAP server to report
attachment size correctly, but Mozilla still uses that data and retreives only
part (depending on the size Exchange reported)

Comment 20

[Travis Hume]

I see this corruption almost daily.  One noticable case is image attachments
being cut off (lower part of .jpg is missing.)

I'm using Linux Build ID: 2002051409

I'm hitting against a "Microsoft Exchange IMAP4rev1 server version 5.5.2655.37"
mail server.

Comment 21

[Will Coleda]

I am having the same problem with the packaged 1.0 release.

We recently switched to Exchange as our internal mail server from iPlanet's
Messenging server... 

Now, some attachments (but not ALL!) will start downloading, and never finish.
hitting stop doesn't stop the mail client from spinning, but I never get the
download. While some attachments work, I am unable to get any to work once I try
to download one that fails..

Any kind of diagnostic that I can provide to help track this down? (My
workaround is either to copy the message to a local folder, or switch to the
outlook client for the problem messages, neither of which is really helpful)

Regards.

Comment 22

[Dustin Marquess]

Please see my comment at http://bugzilla.mozilla.org/show_bug.cgi?id=105606#c2
that explains how this bug squashes inbound S/MIME.

Comment 23

[Andrew Hagen]

Changing summary from "MS" to "Microsoft" to make this bug easier to find.

Comment 24

[alewis]

In Exchange 2000 SP3 on Win2k SP3, there is a toggle for 'Enable fast message 
retrieval'. The help menu says this about this option:

<snip>
Use this check box to send only approximate message sizes to IMAP
clients. This will speed the retrieval of MAPI messages in IMAP4
clients that do not require exact message sizes.

Important   Some IMAP clients require exact message sizes and may not
function properly if this option is enabled.
<snip>

The option is found using the Exchange System Manager > Administrative Groups > 
[Name of your Admin Group] > Servers > [Name of your Server] > Protocols > 
IMAP4 > [Virtual Server isnstance Name] <Properties

I don't know if this is enabled by default. (I think it is) or if this option 
exists in other versions of exchange.

Comment 25

[Samir Gehani]

Mail triage team: Gary, can we reproduce this easily in-house?  

Comment 26

[grylchan]

using commercial trunk
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.3b) Gecko/20021219
on XP.

I am using Microsoft Exchange IMAP4rev1 server version 5.5.1960.6.

I AM able to reproduce this problem. Imap mail acnt, in html format,
I attach 3 jpeg files in this size order (4kb, 541kb, 81kb) and send
it to myself. The last jpeg (81kb) is cut off.

Tried exact same test case on Netscape Mesg server 6.1 and no problems
displaying all 3 images.

Tried mail mesg w/6 attachments (word, jpg, excel,gif,txt,html) the total
size of mesg was only 42kb. and no problems opening or displaying inline
any of the attachments

Tried another mail mesg w/2 attachments but 3 meg word file and 1/2 meg
jpeg and no problem displaying the jpeg or opening word file via double click.

This bug was reopened on 5/14/2002. 

I can reproduce this problem with Jpegs. I have not seen the problem with
the word file.

Comment 27

[grylchan]

I have test images and the order of the attachments
in my blues directory (folder is bug92111).

Comment 28

[Dustin Marquess]

Another way to easily reproduce this bug.

Send an email to yourself, and digitally sign the email using S/MIME.  Then
retrieve the email from Exchange via IMAP, and the S/MIME attachment will not be
retrieved.

Comment 29

[Andrew Hagen]

MS Knowledge Base says they solved a seemingly related problem with this in
Exchange Server 5.5 SP 4. 

http://support.microsoft.com/default.aspx?scid=kb;en-us;271199

Can anyone say whether that applies to this problem or not? That is, are we
still having this problem with Microsoft's latest version of Exchange?

Comment 30

[Brant Gurganus]

By the definitions on <http://bugzilla.mozilla.org/bug_status.html#severity> and
<http://bugzilla.mozilla.org/enter_bug.cgi?format=guided>, crashing and dataloss
bugs are of critical or possibly higher severity.  Only changing open bugs to
minimize unnecessary spam.  Keywords to trigger this would be crash, topcrash,
topcrash+, zt4newcrash, dataloss.

Comment 31

[Ninoschka Baca]

Trunk build 2002-12-17 and the more recent 2003-02-05: WinMe, it appears to be
working for me. I used gary's attached files and some other files and in all
cases the full images appeared. 

Reporter (and anyone else), can you still reproduce this problem with a more
recent build? 

Comment 32

[Bob Horvath]

Where is the "blues" directory with the test images?

Comment 33

[grylchan]

blues dirctory is a local directory. sorry you can't
access it. but if you can still reproduce this problem
please let us know

Comment 34

[Samir Gehani]

Mail triage team: nsbeta1-

Comment 35

[Asa Dotzler [:asa]]

jacekchmiel, if you don't know how to use flags then please don't set them. Thanks.

Comment 36

[Tomasz Saniawa]

Still encountering attachment truncation on Mozilla 1.4 milestone release (Build
ID: 2003052908) connecting to MSExchange IMAP4rev1 version 5.5.2653.23. However,
it WORKED for me when message was moved to Local Folders (like in comment #9)

Comment 37

[Gustavo Chaves]

I confirm the problem using Mozilla Build 2003022516 (and some newer ones that I
don't have right now) and "Microsoft Exchange 2000 IMAP4rev1 server version
6.0.6249.0".

I've sniffed the IMAP packets using this Mozilla and Evolution to compare and
see what makes one fail and the other succeed.

The problem is exactly the one mentioned at least on comments #11, #19, and #24.
 (I disagree, however, with #19 when it says that RFC2683 doesn't require the
IMAP servers to report the message size correctly.  Section 3.4.5's last
paragraph is positive in asserting that this is a protocol violation.)

More specifically, Mozilla asks for the message size like this:

  9 UID fetch 13241:13257 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From
To Cc Subject Date Message-ID Priority X-Priority References Newsgroups
In-Reply-To)])
  <...>
  * 32 FETCH (UID 13257 RFC822.SIZE 464450 FLAGS (\Seen) BODY[HEADER.FIELDS
(From To Cc Subject Date Message-ID Priority X-Priority References Newsgroups
In-Reply-To)] {317}

The number 464450 is the size of the message components added but disregarding
any MIME headers and base64 conversions.  Believing in this information Mozilla
goes on to fetch exactly 464450 bytes from the message in a sequence of commands
like this:

  11 UID fetch 13257 (UID RFC822.SIZE BODY[]<0.10240>)
  12 UID fetch 13257 (UID RFC822.SIZE BODY[]<10240.10240>)
  <...>
  29 UID fetch 13257 (UID RFC822.SIZE BODY[]<460800.3650>)

The problem is that, in this case, the single attachment has 634006 bytes when
converted to base64.  Hence, it ends up truncated.

The problem doesn't arise with Evolution because it disregards the message size
as offered by Exchange and fetches it with these commands:

  A00030 UID FETCH 13257 BODY.PEEK[HEADER]
  A00031 UID FETCH 13257 BODY.PEEK[1.MIME]
  A00032 UID FETCH 13257 BODY.PEEK[2.MIME]
  A00033 UID FETCH 13257 BODY.PEEK[1]
  A00034 UID FETCH 13257 BODY.PEEK[2]

This way, it doesn't need to know the message size.

Actually, fetchmail isn't affected either, but I didn't sniffed it to see how it
fetches the message.

I haven't yet had the chance to check out the solution mentioned in comment #24.
 However, even if this is a confirmed Exchange bug, it's one that's hard for the
average admin to find out about and it is usually thought to be a Mozilla problem.

How hard it would be to reimplement Mozilla's algorithm for fetching messages so
that it doesn't care anymore about the message size as reported by the IMAP
server? Reading RFC2683 gives the impression that there is more than one IMAP
server with this problem, and avoiding it in the client could be the best
solution, given the context.

Comment 38

[Gustavo Chaves]

I just checked the solution proposed on comment #24 and it works.  Now I have a
bunch of happy Mozilla users thanking me.

I still think it worthwhile to evaluate the possibility of changing the fetching
algorithm.

Comment 39

[alewis]

Although comment #24 works for non-ssl IMAP, it does not help secure IMAP. My 
non-MS mail client users have set up fetchmail on solaris to remove thier 
messages from exchange, and they point their clients at the fetchmail server. 
There are some funky fetchmail tweeks to get it to properly retrieve the 
messages from exchange. I don't know what they are, but it works.

Comment 40

[Artem Saveliev]

Gustavo,
you are right about my comment #19. I was reading it couple of months after I
made it, and was like "what was i thinking?". I was waiting for someone to catch
it :)
Yes, RFC states that size has to be reported correctly, and it's an Exchange bug.

Comment 41

[Gustavo Chaves]

Regarding comment #39, I'm personally using fetchmail to fetch my email from
Exchange for a long time and so far I only had one problem which has nothing to
do with the one being discussed here.  (I looked into this problem because of
user complains, not my own.)

This is my fetchmail configuration:

  set daemon 1200
  set postmaster gustavo
  set logfile /l/disk0/Mail/fetchmail.log
  poll smaug:
        proto imap,
        timeout 2400,
        user gustavo,
        fetchall,
        ssl,
        mda "/usr/bin/procmail -Y -d gustavo";

Note that I use ssl.  The only unusual thing here (I guess) is the mda
directive.  I use it to feed procmail directly and not to feed sendmail.

I've never had any problem regarding truncated messages.

BTW, I'm using fetchmail 6.2.0-3 (RH9's rpm) right now, but I use this
configuration for two or three years I guess.

Comment 42

[Bob Horvath]

I have been seeing this again recently on Mozilla 1.5, but have also moved
Exchange servers so I might be on one without the fix.   

Some additional comments.

As for changing the fetching algorithm, I assume having the size allows the
client  to control memory allocation as it downloads the message. For clients
that don't check size, is there an issue with a giant message being transmitted?

Another thing I have done that works, but blows the size thoery out of the water
is to blow away the summary file and let it get recreated. It is a major pain as
everything has to be downloaded again, but it has fixed the symptoms for me a
number of times.  The other thing that sometimes seems to work is to move the
message to another folder on the server.

 

Comment 43

[David :Bienvenu]

taking.

Comment 44

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #43)
David, is Thunderbird Bug 243548 same problem?

Comment 45

[David :Bienvenu]

yes, it all sounds like the same problem, with exchange reporting the wrong
attachment size.

Thunderbird/Seamonkey can be configured not to fetch messages/attachments in
chunks, and that might be the easiest work-around for users with Exchange servers:

user_pref(mail.server.serverXX.fetch_by_chunks, false);

where serverXX is the corresponding imap server.

Can people try that hidden pref and see if it fixes the problem for them?
Setting this pref will make it so we fetch the whole message/attachment at once,
which will mean it's more painful to click on a large message and then change
your mind and click on a different message, but that's less of a problem than
this bug.

Comment 46

[Ernie Zapata]

I am running TB 0.7.1 under Windows 2000 Pro SP4 and have experienced the problem.

I have one of the messages on my IMAP Server with two attachments, the first is
a short README txt file and the second is a gzipped file. When I view the
message and select File -> Attachments -> Save All, only the first small text
file gets saved. The second gzipped file is about 227K and either does not get
saved or is created with a 0 byte size.

I set the preference specified and noticed no difference. An IMAP trace shows an
transmitted request "11 UID fetch 200 (BODY[3])", followed by a received "31
FETCH (UID 200 BODY[3] {229046}", followed by the entire base64 encoded stream,
ending with a received "11 OK UID FETCH completed".

Using a FILEMON (a system file monitoring utility,
http://www.sysinternals.com/ntw2k/source/filemon.shtml), I can see TB creates
the file, issues a flush, and then closes the file.

Comment 47

[Mike Cowperthwaite]

*** Bug 242062 has been marked as a duplicate of this bug. ***

Comment 48

[Rubin Simons]

I can confirm this issue on Thunderbird 1.5.0.4 with Exchange Server 2003. We are in the process of trying the workaround described in comment #24.

Comment 49

[Rubin Simons]

(In reply to comment #48)
> I can confirm this issue on Thunderbird 1.5.0.4 with Exchange Server 2003. We
> are in the process of trying the workaround described in comment #24.
> 

We have tried the workaround described in comment #24 and we can confirm that this works; Attachments are now succesfully retrieved!

Comment 50

[Ondrej]

The bug remains unsolved  in Thunderbird 2.0.0.0, where the user has no access to Exchange server configuration. The problem is obviously caused by "Fast message retrieval" option of Microsoft IMAP4 server implementation. (Exchange 2003)

Administrators of large systems have little interest in disabling this option (increases server load) and the RFC unfortunately doesn't require the server to report exact size.

I suggest duplicating Outlook's message retrieval routine.

Comment 51

[Ondrej]

As a workaround this DOES solve the problem.
Tested on Thunderbird 2.0.0.0 @ Exchange 2003

> 
> user_pref(mail.server.serverXX.fetch_by_chunks, false);
> 

Comment 52

[Sam Ritter]

I'm just entering this to show continued problems.  We have over 50 Thunderbird users, using versions from 1.5 - 2.0.0.4, using Exchange 2003.  Several have reported one of these two problems:

1.  Message sent from Outlook 2003 to Tbird, with attachment of Word doc, is blank, and no indication of attachment.  Forwarding the "blank" email reveals the contents, and the indication of attachment, but the attachment is corrupt (too small).
2.  Same as above, though message is not received as blank, but attachment is corrupt (again, too small).

I tried fix in comment 45 above, and it seemed to work, though one user reported a .pdf file was corrupt.  Unclear if that was accurate or not.

Just now turned off "Fast message retrieval" option on Exchange server.  I'll report what the results are.

I should note that we recently moved to an Exchange server from a sendmail installation.  We had never seen this issue on that platform.

Comment 53

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #19)
> RFC I refered to (rfc 2683 part 3.4.5) does not require IMAP server to report
> attachment size correctly

FYI.
It's wrong. RFC 2683 part 3.4.5 says; (http://www.faqs.org/rfcs/rfc2683.html)
> 3.4.5. RFC822.SIZE
>(Description that some servers returns estimated size,)
>(and explanation about bad affect on client when estimated size)  
>(snip)
> The protocol requires that the RFC822.SIZE value returned by the
> server be EXACT.  Estimating the size is a protocol violation, and
> server designers must be aware that, despite the performance savings
> they might realize in using an estimate, this practice will cause
> some clients to fail in various ways.  If possible, the server should
> compute the RFC822.SIZE for a particular message once, and then save
> it for later retrieval.  If that's not possible, the server must
> compute the value exactly every time.  Incorrect estimates do cause
> severe interoperability problems with some clients.

Comment 54

[Wayne Mery (:wsmwk)]

Should be bug marked OS=ALL?
Has an evangelism bug been filed to Microsoft?  :)

Comment 55

[WADA:World Anti-bad-Duping Agency]

Adding RFC822.SIZE and "RFC violation" in summary for ease of search.

Comment 56

[lohphat]

Are the following bugs dupes of this?
301541
390795

also related? 
220281
365675

Comment 57

[lohphat]

Even though it's acknowledged to be an MS bug in the spirit of enabling TB to survive in the corporate environment, could a compensating patch be rolled in TB to detect and compensate until MS gets around to addressing this?  Otherwise the unintended consequence will be to frustrate users into using another client which works.

Comment 58

[Mauro Cicognini]

I strongly support the suggestion in comment #57.
Exchange usually advertises its presence and version in the IMAP banner, so detection shouldn't be too hard, and TB's behavior should be to automagically disable chunked transfer.

I feel that instances of Exchange IMAP server whose banner gets changed so that it doesn't advertise itself as such are quite rare so that we could have at least a roughly functional workaround.

Comment 59

[Gavan Fantom]

Is there any chance of an easy way to force a reload of attachments which have already been truncated, without having to delete the files for the entire folder and force a reload of everything?

Comment 60

[David :Bienvenu]

Gavan, if you're talking about a message that has been stored for offline use, one trick you can do is to move the message to an other folder that's not configured for offline use, and then move it back to the original folder - this will make TB forget that it had an offline copy of the message.

Comment 61

[Helminthe]

I have encountered the same bug - or at least I think so - please see the forum thread at http://forums.mozillazine.org/viewtopic.php?t=617915
I noticed this behaviour sporadically on Tbird/Win XP over the years, but on OS X it is simply infuriating, so right now I'm doing exactly what comment #57 suggested: desperately looking for another IMAP client that has decent HTML message rendering / authoring. No luck so far.

Comment 62

[Xavier Caré]

I have the same problem with Thunderbird 2.0.0.9 under Windows XP Pro at work, I'de like to continue to use TB and promote it in our exchange structure, if it worked. The only solution I found was using the MS Exchange webmail to see the attached files correctly, but it's quite annoying.

Comment 63

[Helminthe]

 FYI, I have just discovered that Seamonkey does not have this problem. Just installed 1.1.8 on the exactly same system, imported everything I could from Tbird (accounts, identities, imap cache, gpg keys, extensions etc) and 
1. all the attachments are displayed for messages I am absolutely sure are wrong in Tbird
2. the entire message body is fetched as expected

Comment 64

[Mauro Cicognini]

As much as I would like this bug tackled and solved, I must say that I routinely use TB against a MS Exchange server using IMAP, and worked around this bug by disabling chunked transfer within TB's preferences.

I'd strongly suggest to the posters of comment #61 and comment #62 to try this workaround and post their experience.

Comment 65

[Mauro Cicognini]

P.S. AFAIK the pref to set still isn't available in the GUI, but can be found at "mail.server.serverXX.fetch_by_chunks" and should be set to FALSE to work correctly with Exchange 2003.

Anyone out there that can test against Exchange 2007?

Comment 66

[Helminthe]

Don't have Exch 2007 available to test, but: please note that as mentioned in the initial problem description, things worked nicely on XP with Exch 2000 and 2003. It is on Mac OS X only that everything went wrong. Also please note that Seamonkey and Tbird installed side by side, the first one does not have this problem.

Comment 67

[Xavier Caré]

Thanks Mauro, it seems to work with chunks disabled (>50KB). I'll post here if the problem happens again :)

Comment 68

[Mauro Cicognini]

In response to Comment #66: various sources (including myself with Win XP for instance) confirm that this problem is not limited to Mac OS X.

It is by all accounts (as comment #67 points out) related to a MS Exchange setting that should speed up response by the server, allowing it to report just an "approximate" size for the message, and rely on the client to send a fetch for the complete message. This is definitely a protocol violation, and an example of the contempt with which MS usually regarded communication standards (including their own, actually).

TB instead takes the reported size at face value and thus when doing chunked transfer fails miserably.
Everything works fine disabling chunked transfer (or disabling fast message retrieval on Exchange IMAP), and this regardless of OS.

Comment 69

[Wayne Mery (:wsmwk)]

summarizing status (...) of bugs mentioned in comment 56.

> Are the following bugs dupes of this?
> bug 301541  (not exchange)
> bug 390795  (dependent)
> 
> also related? 
> bug 220281  (WFM)
> bug 365675  (incomplete)

Comment 70

[Xavier Caré]

(In reply to comment #68)
I still have the problem, I think I didn't do the right thing; How do you disable chunk transfers and/or fast message retrieval?

I tried to modify the boolean in the configuration of TB using:
mail.imap.fetch_by_chunks (made it false)

Comment 71

[Mauro Cicognini]

Xavier: please read David Bienvenu's explanation of the workaround in comment #45 (if the setting isn't there, just create a new boolean and set it to false).

On my part, I may add that my setting is as follows:

user_pref(mail.server.default.fetch_by_chunks, false);

which reverses the TB's behaviour disabling chunked transfer, unless there's an explicit setting on the server.
Since I mostly use Exchange servers, this works very well for me and doesn't require me to find out the server number within prefs.js.

Comment 72

[Mauro Cicognini]

What I just wrote brings an idea to my mind: couldn't we just pull in some UI for these settings and get over with this bug?

We could possibly advertise the thing in the server settings page as "Exchange 2003 workaround" or "Set this if your downloads are corrupted most of the time" and then add some detail in the help file.

Also, it would be nice to have an explicit UI for the default setting.

If we agreed on this, we could close this bug with WONTFIX and open an enhancement bug for the UI.

Comment 73

[Helminthe]

I think there is a bit of a misunderstanding here: 
#1 The workarounds suggested (to disable chunked transfers on the client, and to report accurate attachment size on the server) do help, but they do NOT solve the problem, I am still getting only an empty file, and still not seeing all the files attached to an e-mail (only 2 out of 8 f.ex.)
#2 The problem is intermittent, and manifests a lot more often on OS X than on XP
#3 SeaMonkey does NOT have this problem, I am reliably seeing all the attached files, and can download them

I am using always the latest stable versions of TBird, SeaMonkey, Windows 2003, OS X and Exhange 2003, with all the updates installed for each.

Comment 74

[Xavier Caré]

A collegue told me to try to put the following strings to false:

mail.imap.fetch_by_chunks
mail.server.default.fetch_by_chunks

It looks like it works now, I recently had truncated attachments, and with this trick the truncated things are now ok. I'll report here if I still have the issue.

Thanks you :)

Comment 75

[Mauro Cicognini]

(In reply to comment #73)

Helminthe: this bug _is_ definitely related to Exchange behaviour, that does violate RFC standards and can be reliably worked around with the suggestions given above.

The behaviour you're reporting seems completely different, and therefore has most likely quite different causes and eventual solutions. I'd advise you to open a new bug.

Comment 76

[Gustavo Chaves]

Better yet. Wouldn't it be nice if Thunderbird simply disabled the mail.server.default.fetch_by_chunks option when it detects that is fetching messages from an Exchange server?

It could do it by guessing from the server's greating:

    $ nc mailsrv1 imap
    * OK Microsoft Exchange 2000 IMAP4rev1 server version 6.0.6619.12 (MAILSRV4) ready.

I suppose there may be some people that changes this greating, but they would simply not be benefitting from this kludge.

In the end, I think this coulnd't have any bad side effects, and only save time and frustration for many people.

Comment 77

[Mauro Cicognini]

(In reply to comment #76)
> Better yet. Wouldn't it be nice if Thunderbird simply disabled the
> mail.server.default.fetch_by_chunks option when it detects that is fetching
> messages from an Exchange server?

This workaround has been proposed in comment #57 and #58 but nothing happened :-(

The UI solution is less brilliant, but IMHO more future-proof and, arguably, more respectful of user choice (by not using hidden assumptions).

Comment 78

[Andrew McNaughton]

This problem is not specific to MS Exchange.  It's widely reported by gmail IMAP users, and I'm experiencing it with a dovecot IMAP server.

Is there some way that Thunderbird can test the server's behaviour and choose to use chunking or not accordingly?

This is a 7 year old bug with severity=critical!

Comment 79

[Mauro Cicognini]

BTW, why does this bug still have a status of NEW??

Multiple reports confirm it - at least, if this boils down to a philosophical issue, do close it as WONTFIX!

Comment 80

[Mauro Cicognini]

I forward a radical motion: let's just do away with this fetch_with_chunks madness.

Let's comment away all the code and the preferences, bandwidth and cpu power are increasing faster than attachment size so who cares about being bandwidth friendly, and if that's needed maybe the CPU stack will take care of it.

It would be better if we lived in a nicer world but we don't.

Comment 82

[Alexander Kohr]

We are having the same problems her with multiple users where images get cut off part way though when displayed inline, and images and even some other kinds of documents can't be saved when the user is interfacing with our Exchange server over Imap. 
I have found that either changes as suggested in <a href="show_bug.cgi?id=92111#c45">comment #45</a> or mail.server.default.fetch_by_chunks change to false from <a href="show_bug.cgi?id=92111#c71">comment #71</a> and <a href="show_bug.cgi?id=92111#c74">#74</a>
 worked for solving the problem on a user by user basis. 

I think either dumping the preferences as suggested in <a href="show_bug.cgi?id=92111#c80">comment #80</a> or doing a test like suggested in <a href="show_bug.cgi?id=92111#c76">comment #76</a> would be the easiest from an end users point of view, because they could be blissfully ignorant of the situation. 

However I don’t agree with changing mail.server.default.fetch_by_chunks, as suggested in  <a href="show_bug.cgi?id=92111#c76">comment #76</a>, as I believe it is a global change in mail.server.default.fetch_by_chunks and might potentially mess up another of the users email accounts. (If chunking is even necessary anymore)
I would instead suggest that changing settings of 
user_pref(mail.server.serverXX.fetch_by_chunks, false); as suggested in <a href="show_bug.cgi?id=92111#c45">comment #45</a>  as it is specific to a single mail server would be better.

I have also submitted a new feature request <A href =”https://bugzilla.mozilla.org/show_bug.cgi?id=449416” >bug 449416</a> which suggests that they change the email account set up & the preferences such as there is an option for exchange servers and it sets the setting as in in <a href="show_bug.cgi?id=92111#c45">comment #45</a>.

Also as a Note I testing only changing the mail.imap.fetch_by_chunks set to false from <a href="show_bug.cgi?id=92111#c74">comment #74</a> and it did nothing. Since I thought it was more specific than the default settings I submitted a new bug <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=449407">Bug 449407</a>

Comment 84

[Marek Vitek]

There are several reasons and also few possible solutions for solving this issue.
- MS Exchange violates RFC and provides only approximate size of the message for "performance reasons"
- I saw also other mail servers providing incorrect RFC822.SIZE value

So sticking with this value is not a good idea. It is also discouraged by IETF author in RFC 2683 http://tools.ietf.org/html/rfc2683 section 3.4.5.

Solutions:
- in Exchange disable "Fast message retrieval" function/RFC violation. http://support.microsoft.com/kb/191504 But it will have performance implications.
- follow RFC 2683 recommendations and in Thunderbird fix the code and use message size provided by "FETCH RFC822.SIZE" command only for informational purposes. For message size validation and possibly truncation use e.g. size reported by "FETCH RFC822" as it seems to be correct value all the time.

Comment 85

[Luke Driscoll]

I've found a useful addon that gives a workaround to this issue, attachment extractor seems to download the attachments fine: https://addons.mozilla.org/en-US/thunderbird/addon/556

Comment 86

[WADA:World Anti-bad-Duping Agency]

(In addition to Comment #84)

Pine/Alpine looks suffering from Exchange too.
> http://lumbgaps.blogspot.com/2009/04/exchange-2007-imap-rfc822size-and.html
Following looks the question to MS.
> http://social.technet.microsoft.com/Forums/en/exchangesvrgeneral/thread/21adbe96-e21b-458a-8242-2c3894b9d7cf
> http://www.office-outlook.com/outlook-forum/index.php/m/509237/
Related Exchange setting seems;
> You can enable the exact size for everyone with:
>   Set-ImapSettings -EnableExactRFC822Size:$true
> Or only for a specific user:
>   Set-CASMailbox "IMAP User" -ImapUseProtocolDefaults:$false
>   -ImapEnableExactRFC822Size:$true
Exchange's Help for above setting.
> Exchange 2007. 
> http://technet.microsoft.com/en-us/library/aa998252(EXCHG.80).aspx
> Exchange 2010.
> http://technet.microsoft.com/en-us/library/aa998252.aspx
> http://technet.microsoft.com/en-us/library/bb125264.aspx

Comment 87

[WADA:World Anti-bad-Duping Agency]

Linkify pointer to workaround in White Board.
  Workaround:
    See Comment #24 for Exchange setting. (Comment #86, if Exchange 2007/2010)
    See Comment #45 for Tb side setting.
        user_pref(mail.server.serverXX.fetch_by_chunks, false);

Comment 89

[WADA:World Anti-bad-Duping Agency]

FYI. UI for mail.server.serverXX.fetch_by_chunks is requested by Bug 449416. xref.

Comment 91

[Stan]

I see no resolution of this bug or target for development.  University of Maryland changed to Exchange back server from a Unix system, and my Thunderbird constantly has this problem, particularly on thin network links.  I have up to date Thunderbird, and can volunteer testing to get this resolved.  The 'fetch-by-chunks' has no effect on this.  Any attachment of substantial size(>100K) hangs indefinitely on thin links (WiFi remote connections) -- appears to eventually recover on fat network links, such as in office on ethernet.  Using a docking system, I can characterize this bug quite well.

Any recent work on this?

Comment 92

[Nikolay Shopik]

Stan, don't get wrong but I think you should ask Microsoft same question, University of Maryland already pay for product so why no just get support from Microsoft and get fix in first place. I really doubt any core developer of Thunderbird will work to get some workaround of this issue, because its not Thunderbird itself problem but Exchange. But I believe if anybody contribute patch to "fix" that issue, developers will accept it for review gladly.
It's just too common see users come to open source product and ask to "fix" other vendor product.

Comment 93

[Stan]

Nikolay,
Three thoughts on what you submitted - somewhat conflicting. 
(1) I like Thunderbird as my preferred client - and choose to use it.  I have no choice in my Universitiy's primary email system.  I am not an IT manager or in the field any longer, just a somewhat knowledgeable EE.  To survive as a viable alternative client to Outlook or others, Thunderbird must somehow work well with Exchange server.  No doubt, its not a fun task.  I have multiple email accounts, and a heavy user.
(2) How do I bring such a bug to the attention of MS as I am not in the university IT department, but a simple user.  Is there an equivalent to BUG page for MS users?  I will pursue if knowledgeable.  I was under the impression, maybe somewhat mistakenly, that the service request would need to come from the IT department of the Univsersity - and their response is 'nobody else has reported this issue, why not try Outlook.'
(3) My preferred work around would be to find a good IMAP host and forward all mail to it.  Any recommendations?  I need to research this.

Comment 94

[David :Bienvenu]

Stan, my understanding is that there is a setting on the Exchange server to make it behave correctly, so it's more a matter of contacting your IT dept. You can also tell Thunderbird to stop using mime parts on demand, which should ameliorate the issue, at the cost of downloading the full message whenever you click on it (tools, options, config editor, type mime_parts and toggle mail.server.default.mime_parts_on_demand to false by right clicking on it and picking toggle). There's a lot of information in this bug, e.g., the comment that links to http://support.microsoft.com/kb/191504

Gmail can host imap and I believe retrieve mail from other imap accounts.

Comment 95

[David :Bienvenu]

though maybe gmail just does other pop3 accounts - not sure. It definitely supports IMAP for normal gmail accounts.

Comment 96

[Stan]

(In reply to comment #95)
> though maybe gmail just does other pop3 accounts - not sure. It definitely
> supports IMAP for normal gmail accounts.

David, I will look into the other setting in Thunderbird and read up on the link you provided - maybe it will give me the info I need to work with my IT department.

Note on GMAIL - it does support an IMAP interface, but beware if you use many folders in IMAP and then look into GMAIL through a web interface.  Not very pretty.

Comment 97

[Alexander Kohr]

Does it still happen at Fox Chase? I am not sure I have moved into another sub-department of IT & don't deal with either the exchange servers or the Mac users on a Day to day basis.  I believe at this point in time most of our users are using OWA, apple mail, entourage or Outlook. I have put a call into Jim our senior macintosh support person to find out if anyone is still using Firebird & if so if they are still experiencing the problems. I will provide an update if he gets back to me.

Comment 98

[Mauro Cicognini]

(In reply to comment #94)
> Gmail can host imap and I believe retrieve mail from other imap accounts.

Gmail definitely has a functional IMAP interface: I use it as my primary email server (actually, the only one right now) and has no interoperability issues with Thunderbird whatsoever.

Gmail can download mail from other accounts only through POP3, which Exchange does support, unless it's been explicitly disabled - but it's much less common than disabling IMAP, since Exchange disk space is at a premium and sysadmins would much more have their users offload the servers than the other way around, so POP3 is good and IMAP is not.

I reckon, though, that if Gmail is the primary email server it's much better and less confusing to have it actually collect the messages so one only has to deal with one folder hierarchy (or, in Gmail-speak, only one tag set).

If all else fails, Exchange can easily be configured to automatically forward all messages to another address, through a server-side rule that can be enabled via OWA, too. Exchange 2007 and 2010 do it almost seamlessly, 2003 gets a bit more in the way but it works although it's not as pretty.

Comment 100

[lohphat]

Is there any special award for bugs over a decade old?

Comment 101

[shattered]

Another question is, can we install a workaround automatically if an Exchange server is detected?

Comment 102

[Alexander Kohr]

While I think it should really be properly fixed because it is my understanding that other mail servers return estimated sizes instead of actual sizes and as such the size should should not be relied on, I like the idea from comment 101.

Comment 103

[Stan]

I like the comment from 101 as well, but I am not an active programmer, so just a request.  Exchange has been forced upon me by the university, and Outlook is the only client that handles it well.  Outlook's IMAP interface is functional - but lacking.  Gmail's IMAP interface is fine, but it's native tag system drive's me nuts.  For what its worth.

Comment 104

[David Woodhouse]

This shouldn't be hard to work around. For Evolution we just "notice" that the server actually returns more data than it claimed was available in RFC822.SIZE. We expect the final 'chunk' to be truncated to the stated size, but in fact it isn't. At that point we *know* the server lied to us, so Evolution falls back to just asking for more and more chunks until the server really *does* stop giving us data.

In the common case of a non-broken server and an email which isn't a precise multiple of our chunk size, there's no performance hit.

When the claimed RFC822.SIZE happens to be a multiple of the chunk size, we do need to do one extra speculative fetch, which will return zero bytes on a non-broken server.

https://bugzilla.gnome.org/show_bug.cgi?id=621262 covers this.

I have to say, there is a lot to be said for simply declaring that Microsoft Exchange IS NOT an email server, and anyone trying to use it as one deserves all the breakage they get ☺

Comment 105

[Alexander Kohr]

I like David's idea better than idea 101 as idea 101 assumes that only Microsoft exchange violates RFC 3501 and I have read about other IMAP servers doing the same thing and this would follow the previously mentioned recommendations of RFC 2683 section 3.4.5 "RFC822.SIZE" that clients can't rely on the RFC822.SIZE returned as there are a number of non compliant servers out there.   http://www.ietf.org/rfc/rfc2683.txt .

Comment 107

[Sam]

The same issue with the Seamonkey 2.7.2  mail and the google's IMAP servers.
• Almost all MS file types (XLS, PPT, etc.) are corrupted (most of the time) when received as an attachment on a x86-64 ubuntu PC. 
• Another PC (x86-64, Linux Mint)  with the same exact version, Seamonkey 2.7.2, receives the same attachments correctly.  Also, using the google's own web-mail client, the attachments can be extracted correctly.
• This issue is more apparent with the last few versions of the Seamonkey mail-client on my system.

Comment 108

[:Irving Reid (No longer working on Firefox)]

Attachment: WIP, read message chunks until the server stops sending more

First cut of patch to read IMAP bodies by chunks until the server stops sending more, to compensate for bad servers (Exchange, Google Mail) that lie about the message size in response to the RFC822.SIZE request.

Still needs:

 - test cases:
   - validation that it works for messages that end exactly on a chunk boundary
   - test that it works on chunks where \r\n splits across chunk boundary
   - works on messages that don't end in \r\n?

 - Correctly store "size of message as downloaded", or some similar change, to make sure that we can re-use the downloaded message from the necko cache

Comment 109

[David :Bienvenu]

Comment on attachment 618681 [details] [diff] [review]
WIP, read message chunks until the server stops sending more

thx, looks reasonable - as you say, we'll need tests! I'm happy to help with that. Testing imap from xpcshell can be really really tricky.

The XXX comments will need to be addressed or at least unXXX'd before review...

Comment 110

[:Irving Reid (No longer working on Firefox)]

Attachment: Handle chunked messages, this time also correctly handling non-chunked messages

First version of this patch wasn't correctly handling non-chunked messages. Still need tests, and still need to deal with the necko cache message size issues.

Comment 112

[:Irving Reid (No longer working on Firefox)]

Attachment: Download all of chunked messages, with test cases

Added test cases, discovered that we weren't correctly handling CR/LF split across chunk boundary, fixed that too.

Also some clang warning cleanup; please double-check the added parentheses at nsImapProtocol.cpp:6524 (copied below); I changed the logic slightly, but I think it's better than it was:

@@ -6544,18 +6518,19 @@ bool nsImapProtocol::FolderIsSelected(co
 void nsImapProtocol::OnStatusForFolder(const char *mailboxName)
 {
 
   if (FolderIsSelected(mailboxName))
   {
     PRInt32 prevNumMessages = GetServerStateParser().NumberOfMessages();
     Noop();
     // OnNewIdleMessages will cause the ui thread to update the folder
-    if (m_imapMailFolderSink && GetServerStateParser().NumberOfRecentMessages()
-          || prevNumMessages != GetServerStateParser().NumberOfMessages())
+    if (m_imapMailFolderSink &&
+         (GetServerStateParser().NumberOfRecentMessages()
+          || prevNumMessages != GetServerStateParser().NumberOfMessages()))
       m_imapMailFolderSink->OnNewIdleMessages();
     return;
   }


This patch also includes the change to canonical line endings proposed by David for bug 740453, in order to make test_chunkLastLF.js pass on Mac/Linux; alternative if we don't want that change would be to change the test to disregard line endings or convert the test to use the same mechanism as test_imapChunks.js.

Comment 113

[David :Bienvenu]

Comment on attachment 625173 [details] [diff] [review]
Download all of chunked messages, with test cases

This breaks handling of non-chunking case when server returns an invalid size, right, and doesn't replace it with the correct counting of the download size, right? So it probably doesn't belong in this patch. The unrelated parentheses changes look OK, but it's probably better to have those in a separate patch. in case of regressions, it's better not to land patches with a bunch of unrelated changes...

other than that, this looks OK.

-          // if we are in the process of fetching everything RFC822 then we should
-          // turn around and force the total download size to be set to this value.
-          // this helps if the server gaves us a bogus size for the message in response to the 
-          // envelope command.
-          if (fFetchEverythingRFC822)
-            SetTotalDownloadSize(fSizeOfMostRecentMessage);
-

Comment 114

[:Irving Reid (No longer working on Firefox)]

Attachment: Correctly handle chunked messages, with Bienvenu's feedback

Removed unrelated warning cleanup changes. As discussed on IRC, the removed call to SetTotalDownloadSize() in nsImapServerResponseParser::msg_fetch() had no effect on correct handling of mis-stated message sizes.

Comment 115

[David :Bienvenu]

Comment on attachment 627991 [details] [diff] [review]
Correctly handle chunked messages, with Bienvenu's feedback

this looks good, in general.

one nit - brace below else here:
+  else {
+    numberOfCharsInThisChunk = 0;
+  }

unfortunately, test_chunkLastLF.js is failing on my debug windows build. 

The failure is here:

WARNING: NS_ENSURE_TRUE(thread) failed: file c:/builds/tbirdhq/objdir-tb/mozilla/netwerk/base/src/../../../../../mozilla/netwerk/base/src/nsSocketTransportService2.cpp, line 115
WARNING: unable to post continuation event: file c:/builds/tbirdhq/objdir-tb/mozilla/xpcom/io/../../../../mozilla/xpcom/io/nsStreamUtils.cpp, line 438
WARNING: nsExceptionService ignoring thread destruction after shutdown: file c:/builds/tbirdhq/objdir-tb/mozilla/xpcom/base/../../../../mozilla/xpcom/base/nsExceptionService.cpp, line 166
###!!! ASSERTION: Using observer service off the main thread!: 'Error', file c:/builds/tbirdhq/objdir-tb/mozilla/xpcom/ds/../../../../mozilla/xpcom/ds/nsObserverService.cpp, line 110
<<<<<<<
PROCESS-CRASH | c:\builds\tbirdhq\objdir-tb\mozilla\_tests\xpcshell\mailnews\imap\test\unit\test_chunkLastLF.js | application crashed (minidump found)
Crash dump filename: c:\builds\tbirdhq\objdir-tb\mozilla\_tests\xpcshell\mailnews\imap\test\unit\64888d62-4d6e-4c08-a79d-70acd1a50688.dmp

I suspect this means that test is either not cleaning up after itself, or finishes before we've stopped talking to the server. Have you tried this on a debug windows build? If you can't recreate the error, I can poke around a bit.

Comment 116

[David :Bienvenu]

Comment on attachment 627991 [details] [diff] [review]
Correctly handle chunked messages, with Bienvenu's feedback

clearing review request because of test failure, which, at least for me, can be fixed by disabling autosync in the test.

Comment 117

[:Irving Reid (No longer working on Firefox)]

Attachment: Correctly download chunked email, Windows tests fixed

Fixed test as recommended by David, cleaned his nit, tested on OS X and Windows.

Comment 118

[David :Bienvenu]

Comment on attachment 628593 [details] [diff] [review]
Correctly download chunked email, Windows tests fixed

looks good, thx!

Comment 119

[:Irving Reid (No longer working on Firefox)]

Attachment: Handle chunked messages, including 'peek'

Added an additional fix to handle the case of fetching BODY.PEEK[] as well as BODY[]; earlier version of this patch was sensitive to an existing difference between those two code paths.

Also removed two spurious assertions that were checking for conditions that occur (and are handled correctly) in normal operation.

Comment 121

[David :Bienvenu]

fixed on trunk - http://hg.mozilla.org/comm-central/rev/5a6bee217340

Comment 123

[lohphat]

Well, that took ELEVEN YEARS to fix.

Comment 124

[David Woodhouse]

No, it took eleven years to work around. The server is *still* broken.

Comment 125

[Neil Parks]

I would not say this bug is "fixed".  

I am using Tbird 16.0.1 on Win XP Pro SP3.  I received two large attachments--one approx 2 MB and one over 600 KB--from a Postfix server which I access via IMAP.

Tbird truncated both messages to approx 414,000 bytes.  Searching the forums for a solution, I found this bug and the suggestion to turn off fetching by chunks.

I made the appropriate change in my configuration, and had the messages re-transmitted to me.  This time, Tbird was able to receive them in full.

So I believe that this bug should either be reopened, or reclassified as "Won't Fix" because the suggested workaround still works.

Comment 126

[WADA:World Anti-bad-Duping Agency]

(In reply to Neil Parks from comment #125)
> I would not say this bug is "fixed".  

Writing to offline-store(offline-use=On, mail.imap.mime_parts_on_demand is irrelevant) and writing to Memory/Disk Cache(offline-use=Off, mail.imap.mime_parts_on_demand is relevant) has different mechanism.
Offline-use=On folder? or Offline-use=Off folder?

By patch for this bug, new IMAP log such as following was introduced.  
> +  PR_LOG(IMAP, PR_LOG_DEBUG, ("FetchTryChunking: curFetchSize %u", downloadSize));
Can you get IMAP log and check IMAP level flow?

If you can obtain IMAP log, open separate bug for ease of problem analysys by developers, please
If log analysis by IMAP people is needed, before attach log file to bug, remove/replace private data and remove many log lines for mail data except mail header, please. Remove also log lines for access of irrelevant Mbox, access of irrelevant server's Mbox, please.

Comment 127

[Zach Sadecki]

(In reply to Neil Parks from comment #125)
My parents are having with I think is the same problem using TB 16.0.1 on Win7.  I do not see it on Ubuntu (for the same message).  My server is Postfix with Dovecot for IMAP.

Comment 128

[Magnus Melin [:mkmelin]]

This bug is fixed (it fixed at least one case). There may be other issues of course, but file new bugs for those if you still see problems.