IMAP: Attachments not fully downloaded (due to incorrect/smaller RFC822.SIZE by MS Exchange, gmail, and possibly others)

RESOLVED FIXED in Thunderbird 16.0

Status

MailNews Core
Networking: IMAP
--
critical
RESOLVED FIXED
16 years ago
a year ago

People

(Reporter: Artem Saveliev, Assigned: Irving)

Tracking

(Blocks: 1 bug, {dataloss, imap-interop, testcase})

Trunk
Thunderbird 16.0
dataloss, imap-interop, testcase
Dependency tree / graph
Bug Flags:
blocking1.3 -
in-testsuite +

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: workaround see comment 24/45/94)

Attachments

(3 attachments, 5 obsolete attachments)

(Reporter)

Description

16 years ago
Using _both_ Netscape Communicator 4.76 and Mozilla 0.9.2 I have last attachment
in the list truncated (if it's only one attachment than it happens only
sometimes). That happens to all types of attachments (pictures - you can see
them truncated, Word documents). Outlook Express views/downloads thouse files
normaly. Same happened in older Communicators (4.7 etc) and Mozillas (M18 I think).
Using Microsoft Exchange 5.5 Build 2653.23 SP4 with default Internet Mail
Service settings.

Comment 1

16 years ago
.9.3 is due out in about a week. Can you test against that? We fixed some
attachment discovery bugs between .9.2 and .9.3. 

 At: 
http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap

you'll find instructions for generating an imap log which will show us
information about why we may not be showing your attachment. Just click on a
message giving you this problem. Then quit the app and send us the log file. Thanks!
(Reporter)

Comment 2

16 years ago
Using 2001072403.
While playing around to create a good test example I noticed that it actualy
truncates the whole file sometimes. Say I attach 4 files, and it only shows me
3, also downloading only part of third file.
Attaching log file.
(Reporter)

Comment 3

16 years ago
Created attachment 43517 [details]
IMAP log file
(Reporter)

Comment 4

16 years ago
Created attachment 44544 [details]
0.9.3 log - same story

Comment 5

16 years ago
Marking NEW.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Keywords: dataloss, testcase
Summary: Attachments not fully downloaded from Exchange → Attachments not fully downloaded from Exchange Server

Comment 6

16 years ago
Sounds like a similar problem that I have.  A workaround I have found is to copy
the message (usually by control-dragging) to the local Drafts folder, and then
back to the INBOX.  All of a sudden it can read the entire message again.
(Reporter)

Comment 7

16 years ago
Yes Bob, I can verify that, and it makes my life a little bit easier. However,
that only works in NN4 for me.
After that I am able to view attachment from the local folder and IMAP folder
using NN4. Mozilla is still not able to view that new e-mail's attachment correctly.

Comment 8

16 years ago
Hmmm, perhaps we have slightly different experiences.  The thing I have noticed
today is using Mozilla, that I can copy it to my IMAP folder named Drafts,
instead of the local version, and then copy it back, and be able to read the new
copy of it.

I wonder if there is something special about Drafts.  I also have an IMAP folder
that I created called Unreadable that I put these kinds of things to see if I
can detect a pattern.  They are still unreadable there.  I got used to using
Drafs as a staging area to do things like delete attachments with NN4.

The other thing I find curious about all of this is that if I do a View Source,
I don't get the entire message from the IMAP folder.  Is this showing me what
was actually downloaded, or what was parsed?  I have played around with imaplib
in Python, and that seems to have no problems downloading the entire thing, as
does mutt for me, and pine for others.

Any suggestions as to see what is actually coming back from the Exchange server?

Updated

16 years ago
Keywords: 4xp
(Reporter)

Comment 9

16 years ago
Ok, to clarify my experience
1) The same behavior is shared by Netscape Navigator (4.78) and Mozilla
2) The most obvious way to see it is by looking at attached picture, whether
inline or by clicking on it. It's truncated (note that it doesn't 
happen always).
3) That doesn't happen always.
4) In NN, dragging message from IMAP folder to any local folder fixes the
problem. Same thing doesn't work in Mozilla (tested on today's 
nightly)

To answer the last question, you can find instructions on enabling logging here
http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap
And I have a couple of logs attached to this bug already.

Comment 10

16 years ago
I'm almost certain that what you're seeing is bug the same thing I can see in 
bug 105606.

I've had this bug for ages in Netscape 4.x and in Mozilla.

This bug is the main reason why many of my co-workers switched to Outlook since 
copying mails to local folder just isn't a solution that you can have in your 
daily work.

So to clarify: this bug is also present in Mozilla build 20011018
(Reporter)

Comment 11

16 years ago
I went into details of IMAP RFC and Henrik's note that Outlook uses different
command also helped.

I believe that Mozilla behaviour is OK, it's Exchange problem.
Check out http://www.faqs.org/rfcs/rfc2683.html part 3.4.5
I think it will require tweaking Mozilla specificaly to work with Exchange.

Comment 12

16 years ago
I had read about the size issue long ago in some fetchmail documentation. 
Knowing that Microsoft will most likely not change Exchange any time soon, does
this not fall into the are of "quirks" vs. evangelism? 

The one thing I don't understand is why it works if you copy it into another
IMAP folder.  Perhaps Exchange calculates the size correctly then. Time to learn
how to make IMAP logs.

Is there any chance of this being implemented in Mozilla as a quirk?

Comment 13

16 years ago
One more workaround...

File/Offline/GetSelectedMessages will download the whole thing.  Sometimes you
have to select another message and come back to see it, but it is there.

Comment 14

15 years ago
I forgot all about this bug.  I suspect it is fixed.  What is the process for
closing a bug?  Did someone have to claim to have fixed it?

Comment 15

15 years ago
Well since your not seeing it anymore I will go ahead and mark it WORKSFORME.
Thanks for getting back.
Status: NEW → RESOLVED
Last Resolved: 15 years ago
Resolution: --- → WORKSFORME

Comment 16

15 years ago
this is not working. I'm hit on a daily basis by this bug.
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---

Comment 17

15 years ago
ANything reproducible?  I could test it at my end.

Comment 18

15 years ago
this normally happens when trying to open large attachments like Word documents
without first saving them to the disk. Because of this Exchange feature the
document are chopped... You can also just use view source where you can see that
some stuff (the last bytes) are missing from the mail
(Reporter)

Comment 19

15 years ago
Basicaly Mozilla uses martial retreival method (as described in bug 105606):
UID fetch 4519 (UID RFC822.SIZE BODY[]<0.10240>)
and Outlook Express retreives all:
UID FETCH 4519 (BODY.PEEK[] UID)

RFC I refered to (rfc 2683 part 3.4.5) does not require IMAP server to report
attachment size correctly, but Mozilla still uses that data and retreives only
part (depending on the size Exchange reported)

Comment 20

15 years ago
I see this corruption almost daily.  One noticable case is image attachments
being cut off (lower part of .jpg is missing.)

I'm using Linux Build ID: 2002051409

I'm hitting against a "Microsoft Exchange IMAP4rev1 server version 5.5.2655.37"
mail server.

Comment 21

15 years ago
I am having the same problem with the packaged 1.0 release.

We recently switched to Exchange as our internal mail server from iPlanet's
Messenging server... 

Now, some attachments (but not ALL!) will start downloading, and never finish.
hitting stop doesn't stop the mail client from spinning, but I never get the
download. While some attachments work, I am unable to get any to work once I try
to download one that fails..

Any kind of diagnostic that I can provide to help track this down? (My
workaround is either to copy the message to a local folder, or switch to the
outlook client for the problem messages, neither of which is really helpful)

Regards.

Updated

15 years ago
Keywords: interop
QA Contact: huang → meehansqa
Summary: Attachments not fully downloaded from Exchange Server → MS Exchange IMAP: Attachments not fully downloaded

Comment 22

15 years ago
Please see my comment at http://bugzilla.mozilla.org/show_bug.cgi?id=105606#c2
that explains how this bug squashes inbound S/MIME.

Comment 23

15 years ago
Changing summary from "MS" to "Microsoft" to make this bug easier to find.
Keywords: mozilla1.1, mozilla1.2, nsbeta1
Summary: MS Exchange IMAP: Attachments not fully downloaded → Microsoft Exchange IMAP: Attachments not fully downloaded

Comment 24

15 years ago
In Exchange 2000 SP3 on Win2k SP3, there is a toggle for 'Enable fast message 
retrieval'. The help menu says this about this option:

<snip>
Use this check box to send only approximate message sizes to IMAP
clients. This will speed the retrieval of MAPI messages in IMAP4
clients that do not require exact message sizes.

Important   Some IMAP clients require exact message sizes and may not
function properly if this option is enabled.
<snip>

The option is found using the Exchange System Manager > Administrative Groups > 
[Name of your Admin Group] > Servers > [Name of your Server] > Protocols > 
IMAP4 > [Virtual Server isnstance Name] <Properties

I don't know if this is enabled by default. (I think it is) or if this option 
exists in other versions of exchange.

Updated

15 years ago
QA Contact: meehansqa → gchan

Comment 25

15 years ago
Mail triage team: Gary, can we reproduce this easily in-house?  
Whiteboard: [need info]

Comment 26

15 years ago
using commercial trunk
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.3b) Gecko/20021219
on XP.

I am using Microsoft Exchange IMAP4rev1 server version 5.5.1960.6.

I AM able to reproduce this problem. Imap mail acnt, in html format,
I attach 3 jpeg files in this size order (4kb, 541kb, 81kb) and send
it to myself. The last jpeg (81kb) is cut off.

Tried exact same test case on Netscape Mesg server 6.1 and no problems
displaying all 3 images.

Tried mail mesg w/6 attachments (word, jpg, excel,gif,txt,html) the total
size of mesg was only 42kb. and no problems opening or displaying inline
any of the attachments

Tried another mail mesg w/2 attachments but 3 meg word file and 1/2 meg
jpeg and no problem displaying the jpeg or opening word file via double click.

This bug was reopened on 5/14/2002. 

I can reproduce this problem with Jpegs. I have not seen the problem with
the word file.

Comment 27

15 years ago
I have test images and the order of the attachments
in my blues directory (folder is bug92111).

Comment 28

15 years ago
Another way to easily reproduce this bug.

Send an email to yourself, and digitally sign the email using S/MIME.  Then
retrieve the email from Exchange via IMAP, and the S/MIME attachment will not be
retrieved.

Comment 29

15 years ago
MS Knowledge Base says they solved a seemingly related problem with this in
Exchange Server 5.5 SP 4. 

http://support.microsoft.com/default.aspx?scid=kb;en-us;271199

Can anyone say whether that applies to this problem or not? That is, are we
still having this problem with Microsoft's latest version of Exchange?
Keywords: mozilla1.1, mozilla1.2 → mozilla1.3

Comment 30

15 years ago
By the definitions on <http://bugzilla.mozilla.org/bug_status.html#severity> and
<http://bugzilla.mozilla.org/enter_bug.cgi?format=guided>, crashing and dataloss
bugs are of critical or possibly higher severity.  Only changing open bugs to
minimize unnecessary spam.  Keywords to trigger this would be crash, topcrash,
topcrash+, zt4newcrash, dataloss.
Severity: normal → critical

Comment 31

14 years ago
Trunk build 2002-12-17 and the more recent 2003-02-05: WinMe, it appears to be
working for me. I used gary's attached files and some other files and in all
cases the full images appeared. 

Reporter (and anyone else), can you still reproduce this problem with a more
recent build? 

Comment 32

14 years ago
Where is the "blues" directory with the test images?

Comment 33

14 years ago
blues dirctory is a local directory. sorry you can't
access it. but if you can still reproduce this problem
please let us know

Comment 34

14 years ago
Mail triage team: nsbeta1-
Keywords: nsbeta1 → nsbeta1-
Whiteboard: [need info]

Updated

14 years ago
Flags: blocking1.3+

Comment 35

14 years ago
jacekchmiel, if you don't know how to use flags then please don't set them. Thanks.
Flags: blocking1.3+ → blocking1.3?

Updated

14 years ago
Flags: blocking1.3? → blocking1.3-

Updated

14 years ago
Keywords: mozilla1.3

Comment 36

14 years ago
Still encountering attachment truncation on Mozilla 1.4 milestone release (Build
ID: 2003052908) connecting to MSExchange IMAP4rev1 version 5.5.2653.23. However,
it WORKED for me when message was moved to Local Folders (like in comment #9)

Comment 37

14 years ago
I confirm the problem using Mozilla Build 2003022516 (and some newer ones that I
don't have right now) and "Microsoft Exchange 2000 IMAP4rev1 server version
6.0.6249.0".

I've sniffed the IMAP packets using this Mozilla and Evolution to compare and
see what makes one fail and the other succeed.

The problem is exactly the one mentioned at least on comments #11, #19, and #24.
 (I disagree, however, with #19 when it says that RFC2683 doesn't require the
IMAP servers to report the message size correctly.  Section 3.4.5's last
paragraph is positive in asserting that this is a protocol violation.)

More specifically, Mozilla asks for the message size like this:

  9 UID fetch 13241:13257 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From
To Cc Subject Date Message-ID Priority X-Priority References Newsgroups
In-Reply-To)])
  <...>
  * 32 FETCH (UID 13257 RFC822.SIZE 464450 FLAGS (\Seen) BODY[HEADER.FIELDS
(From To Cc Subject Date Message-ID Priority X-Priority References Newsgroups
In-Reply-To)] {317}

The number 464450 is the size of the message components added but disregarding
any MIME headers and base64 conversions.  Believing in this information Mozilla
goes on to fetch exactly 464450 bytes from the message in a sequence of commands
like this:

  11 UID fetch 13257 (UID RFC822.SIZE BODY[]<0.10240>)
  12 UID fetch 13257 (UID RFC822.SIZE BODY[]<10240.10240>)
  <...>
  29 UID fetch 13257 (UID RFC822.SIZE BODY[]<460800.3650>)

The problem is that, in this case, the single attachment has 634006 bytes when
converted to base64.  Hence, it ends up truncated.

The problem doesn't arise with Evolution because it disregards the message size
as offered by Exchange and fetches it with these commands:

  A00030 UID FETCH 13257 BODY.PEEK[HEADER]
  A00031 UID FETCH 13257 BODY.PEEK[1.MIME]
  A00032 UID FETCH 13257 BODY.PEEK[2.MIME]
  A00033 UID FETCH 13257 BODY.PEEK[1]
  A00034 UID FETCH 13257 BODY.PEEK[2]

This way, it doesn't need to know the message size.

Actually, fetchmail isn't affected either, but I didn't sniffed it to see how it
fetches the message.

I haven't yet had the chance to check out the solution mentioned in comment #24.
 However, even if this is a confirmed Exchange bug, it's one that's hard for the
average admin to find out about and it is usually thought to be a Mozilla problem.

How hard it would be to reimplement Mozilla's algorithm for fetching messages so
that it doesn't care anymore about the message size as reported by the IMAP
server? Reading RFC2683 gives the impression that there is more than one IMAP
server with this problem, and avoiding it in the client could be the best
solution, given the context.

Comment 38

14 years ago
I just checked the solution proposed on comment #24 and it works.  Now I have a
bunch of happy Mozilla users thanking me.

I still think it worthwhile to evaluate the possibility of changing the fetching
algorithm.

Comment 39

14 years ago
Although comment #24 works for non-ssl IMAP, it does not help secure IMAP. My 
non-MS mail client users have set up fetchmail on solaris to remove thier 
messages from exchange, and they point their clients at the fetchmail server. 
There are some funky fetchmail tweeks to get it to properly retrieve the 
messages from exchange. I don't know what they are, but it works.
(Reporter)

Comment 40

14 years ago
Gustavo,
you are right about my comment #19. I was reading it couple of months after I
made it, and was like "what was i thinking?". I was waiting for someone to catch
it :)
Yes, RFC states that size has to be reported correctly, and it's an Exchange bug.

Comment 41

14 years ago
Regarding comment #39, I'm personally using fetchmail to fetch my email from
Exchange for a long time and so far I only had one problem which has nothing to
do with the one being discussed here.  (I looked into this problem because of
user complains, not my own.)

This is my fetchmail configuration:

  set daemon 1200
  set postmaster gustavo
  set logfile /l/disk0/Mail/fetchmail.log
  poll smaug:
        proto imap,
        timeout 2400,
        user gustavo,
        fetchall,
        ssl,
        mda "/usr/bin/procmail -Y -d gustavo";

Note that I use ssl.  The only unusual thing here (I guess) is the mda
directive.  I use it to feed procmail directly and not to feed sendmail.

I've never had any problem regarding truncated messages.

BTW, I'm using fetchmail 6.2.0-3 (RH9's rpm) right now, but I use this
configuration for two or three years I guess.

Comment 42

14 years ago
I have been seeing this again recently on Mozilla 1.5, but have also moved
Exchange servers so I might be on one without the fix.   

Some additional comments.

As for changing the fetching algorithm, I assume having the size allows the
client  to control memory allocation as it downloads the message. For clients
that don't check size, is there an issue with a giant message being transmitted?

Another thing I have done that works, but blows the size thoery out of the water
is to blow away the summary file and let it get recreated. It is a major pain as
everything has to be downloaded again, but it has fixed the symptoms for me a
number of times.  The other thing that sometimes seems to work is to move the
message to another folder on the server.

 

Comment 43

13 years ago
taking.
Assignee: mscott → bienvenu
Status: REOPENED → NEW
(In reply to comment #43)
David, is Thunderbird Bug 243548 same problem?

Comment 45

13 years ago
yes, it all sounds like the same problem, with exchange reporting the wrong
attachment size.

Thunderbird/Seamonkey can be configured not to fetch messages/attachments in
chunks, and that might be the easiest work-around for users with Exchange servers:

user_pref(mail.server.serverXX.fetch_by_chunks, false);

where serverXX is the corresponding imap server.

Can people try that hidden pref and see if it fixes the problem for them?
Setting this pref will make it so we fetch the whole message/attachment at once,
which will mean it's more painful to click on a large message and then change
your mind and click on a different message, but that's less of a problem than
this bug.

Comment 46

13 years ago
I am running TB 0.7.1 under Windows 2000 Pro SP4 and have experienced the problem.

I have one of the messages on my IMAP Server with two attachments, the first is
a short README txt file and the second is a gzipped file. When I view the
message and select File -> Attachments -> Save All, only the first small text
file gets saved. The second gzipped file is about 227K and either does not get
saved or is created with a 0 byte size.

I set the preference specified and noticed no difference. An IMAP trace shows an
transmitted request "11 UID fetch 200 (BODY[3])", followed by a received "31
FETCH (UID 200 BODY[3] {229046}", followed by the entire base64 encoded stream,
ending with a received "11 OK UID FETCH completed".

Using a FILEMON (a system file monitoring utility,
http://www.sysinternals.com/ntw2k/source/filemon.shtml), I can see TB creates
the file, issues a flush, and then closes the file.
Product: MailNews → Core

Comment 47

12 years ago
*** Bug 242062 has been marked as a duplicate of this bug. ***

Comment 48

11 years ago
I can confirm this issue on Thunderbird 1.5.0.4 with Exchange Server 2003. We are in the process of trying the workaround described in comment #24.

Comment 49

11 years ago
(In reply to comment #48)
> I can confirm this issue on Thunderbird 1.5.0.4 with Exchange Server 2003. We
> are in the process of trying the workaround described in comment #24.
> 

We have tried the workaround described in comment #24 and we can confirm that this works; Attachments are now succesfully retrieved!

Comment 50

10 years ago
The bug remains unsolved  in Thunderbird 2.0.0.0, where the user has no access to Exchange server configuration. The problem is obviously caused by "Fast message retrieval" option of Microsoft IMAP4 server implementation. (Exchange 2003)

Administrators of large systems have little interest in disabling this option (increases server load) and the RFC unfortunately doesn't require the server to report exact size.

I suggest duplicating Outlook's message retrieval routine.

Comment 51

10 years ago
As a workaround this DOES solve the problem.
Tested on Thunderbird 2.0.0.0 @ Exchange 2003

> 
> user_pref(mail.server.serverXX.fetch_by_chunks, false);
> 

Comment 52

10 years ago
I'm just entering this to show continued problems.  We have over 50 Thunderbird users, using versions from 1.5 - 2.0.0.4, using Exchange 2003.  Several have reported one of these two problems:

1.  Message sent from Outlook 2003 to Tbird, with attachment of Word doc, is blank, and no indication of attachment.  Forwarding the "blank" email reveals the contents, and the indication of attachment, but the attachment is corrupt (too small).
2.  Same as above, though message is not received as blank, but attachment is corrupt (again, too small).

I tried fix in comment 45 above, and it seemed to work, though one user reported a .pdf file was corrupt.  Unclear if that was accurate or not.

Just now turned off "Fast message retrieval" option on Exchange server.  I'll report what the results are.

I should note that we recently moved to an Exchange server from a sendmail installation.  We had never seen this issue on that platform.
(In reply to comment #19)
> RFC I refered to (rfc 2683 part 3.4.5) does not require IMAP server to report
> attachment size correctly

FYI.
It's wrong. RFC 2683 part 3.4.5 says; (http://www.faqs.org/rfcs/rfc2683.html)
> 3.4.5. RFC822.SIZE
>(Description that some servers returns estimated size,)
>(and explanation about bad affect on client when estimated size)  
>(snip)
> The protocol requires that the RFC822.SIZE value returned by the
> server be EXACT.  Estimating the size is a protocol violation, and
> server designers must be aware that, despite the performance savings
> they might realize in using an estimate, this practice will cause
> some clients to fail in various ways.  If possible, the server should
> compute the RFC822.SIZE for a particular message once, and then save
> it for later retrieval.  If that's not possible, the server must
> compute the value exactly every time.  Incorrect estimates do cause
> severe interoperability problems with some clients.
Should be bug marked OS=ALL?
Has an evangelism bug been filed to Microsoft?  :)

Whiteboard: workaround see comment 24/45
Adding RFC822.SIZE and "RFC violation" in summary for ease of search.
Summary: Microsoft Exchange IMAP: Attachments not fully downloaded → Microsoft Exchange IMAP: Attachments not fully downloaded (due to incorrect/smaller RFC822.SIZE by MS Exchange, which is RFC violation)

Comment 56

10 years ago
Are the following bugs dupes of this?
301541
390795

also related? 
220281
365675

Comment 57

10 years ago
Even though it's acknowledged to be an MS bug in the spirit of enabling TB to survive in the corporate environment, could a compensating patch be rolled in TB to detect and compensate until MS gets around to addressing this?  Otherwise the unintended consequence will be to frustrate users into using another client which works.

Comment 58

10 years ago
I strongly support the suggestion in comment #57.
Exchange usually advertises its presence and version in the IMAP banner, so detection shouldn't be too hard, and TB's behavior should be to automagically disable chunked transfer.

I feel that instances of Exchange IMAP server whose banner gets changed so that it doesn't advertise itself as such are quite rare so that we could have at least a roughly functional workaround.
Blocks: 390795

Comment 59

10 years ago
Is there any chance of an easy way to force a reload of attachments which have already been truncated, without having to delete the files for the entire folder and force a reload of everything?

Comment 60

10 years ago
Gavan, if you're talking about a message that has been stored for offline use, one trick you can do is to move the message to an other folder that's not configured for offline use, and then move it back to the original folder - this will make TB forget that it had an offline copy of the message.

Comment 61

9 years ago
I have encountered the same bug - or at least I think so - please see the forum thread at http://forums.mozillazine.org/viewtopic.php?t=617915
I noticed this behaviour sporadically on Tbird/Win XP over the years, but on OS X it is simply infuriating, so right now I'm doing exactly what comment #57 suggested: desperately looking for another IMAP client that has decent HTML message rendering / authoring. No luck so far.

Comment 62

9 years ago
I have the same problem with Thunderbird 2.0.0.9 under Windows XP Pro at work, I'de like to continue to use TB and promote it in our exchange structure, if it worked. The only solution I found was using the MS Exchange webmail to see the attached files correctly, but it's quite annoying.

Comment 63

9 years ago
 FYI, I have just discovered that Seamonkey does not have this problem. Just installed 1.1.8 on the exactly same system, imported everything I could from Tbird (accounts, identities, imap cache, gpg keys, extensions etc) and 
1. all the attachments are displayed for messages I am absolutely sure are wrong in Tbird
2. the entire message body is fetched as expected

Comment 64

9 years ago
As much as I would like this bug tackled and solved, I must say that I routinely use TB against a MS Exchange server using IMAP, and worked around this bug by disabling chunked transfer within TB's preferences.

I'd strongly suggest to the posters of comment #61 and comment #62 to try this workaround and post their experience.

Comment 65

9 years ago
P.S. AFAIK the pref to set still isn't available in the GUI, but can be found at "mail.server.serverXX.fetch_by_chunks" and should be set to FALSE to work correctly with Exchange 2003.

Anyone out there that can test against Exchange 2007?

Comment 66

9 years ago
Don't have Exch 2007 available to test, but: please note that as mentioned in the initial problem description, things worked nicely on XP with Exch 2000 and 2003. It is on Mac OS X only that everything went wrong. Also please note that Seamonkey and Tbird installed side by side, the first one does not have this problem.

Comment 67

9 years ago
Thanks Mauro, it seems to work with chunks disabled (>50KB). I'll post here if the problem happens again :)

Comment 68

9 years ago
In response to Comment #66: various sources (including myself with Win XP for instance) confirm that this problem is not limited to Mac OS X.

It is by all accounts (as comment #67 points out) related to a MS Exchange setting that should speed up response by the server, allowing it to report just an "approximate" size for the message, and rely on the client to send a fetch for the complete message. This is definitely a protocol violation, and an example of the contempt with which MS usually regarded communication standards (including their own, actually).

TB instead takes the reported size at face value and thus when doing chunked transfer fails miserably.
Everything works fine disabling chunked transfer (or disabling fast message retrieval on Exchange IMAP), and this regardless of OS.
summarizing status (...) of bugs mentioned in comment 56.

> Are the following bugs dupes of this?
> bug 301541  (not exchange)
> bug 390795  (dependent)
> 
> also related? 
> bug 220281  (WFM)
> bug 365675  (incomplete)
OS: Windows 2000 → All
Hardware: PC → All

Comment 70

9 years ago
(In reply to comment #68)
I still have the problem, I think I didn't do the right thing; How do you disable chunk transfers and/or fast message retrieval?

I tried to modify the boolean in the configuration of TB using:
mail.imap.fetch_by_chunks (made it false)

Comment 71

9 years ago
Xavier: please read David Bienvenu's explanation of the workaround in comment #45 (if the setting isn't there, just create a new boolean and set it to false).

On my part, I may add that my setting is as follows:

user_pref(mail.server.default.fetch_by_chunks, false);

which reverses the TB's behaviour disabling chunked transfer, unless there's an explicit setting on the server.
Since I mostly use Exchange servers, this works very well for me and doesn't require me to find out the server number within prefs.js.

Comment 72

9 years ago
What I just wrote brings an idea to my mind: couldn't we just pull in some UI for these settings and get over with this bug?

We could possibly advertise the thing in the server settings page as "Exchange 2003 workaround" or "Set this if your downloads are corrupted most of the time" and then add some detail in the help file.

Also, it would be nice to have an explicit UI for the default setting.

If we agreed on this, we could close this bug with WONTFIX and open an enhancement bug for the UI.

Comment 73

9 years ago
I think there is a bit of a misunderstanding here: 
#1 The workarounds suggested (to disable chunked transfers on the client, and to report accurate attachment size on the server) do help, but they do NOT solve the problem, I am still getting only an empty file, and still not seeing all the files attached to an e-mail (only 2 out of 8 f.ex.)
#2 The problem is intermittent, and manifests a lot more often on OS X than on XP
#3 SeaMonkey does NOT have this problem, I am reliably seeing all the attached files, and can download them

I am using always the latest stable versions of TBird, SeaMonkey, Windows 2003, OS X and Exhange 2003, with all the updates installed for each.

Comment 74

9 years ago
A collegue told me to try to put the following strings to false:

mail.imap.fetch_by_chunks
mail.server.default.fetch_by_chunks

It looks like it works now, I recently had truncated attachments, and with this trick the truncated things are now ok. I'll report here if I still have the issue.

Thanks you :)

Comment 75

9 years ago
(In reply to comment #73)

Helminthe: this bug _is_ definitely related to Exchange behaviour, that does violate RFC standards and can be reliably worked around with the suggestions given above.

The behaviour you're reporting seems completely different, and therefore has most likely quite different causes and eventual solutions. I'd advise you to open a new bug.

Comment 76

9 years ago
Better yet. Wouldn't it be nice if Thunderbird simply disabled the mail.server.default.fetch_by_chunks option when it detects that is fetching messages from an Exchange server?

It could do it by guessing from the server's greating:

    $ nc mailsrv1 imap
    * OK Microsoft Exchange 2000 IMAP4rev1 server version 6.0.6619.12 (MAILSRV4) ready.

I suppose there may be some people that changes this greating, but they would simply not be benefitting from this kludge.

In the end, I think this coulnd't have any bad side effects, and only save time and frustration for many people.

Comment 77

9 years ago
(In reply to comment #76)
> Better yet. Wouldn't it be nice if Thunderbird simply disabled the
> mail.server.default.fetch_by_chunks option when it detects that is fetching
> messages from an Exchange server?

This workaround has been proposed in comment #57 and #58 but nothing happened :-(

The UI solution is less brilliant, but IMHO more future-proof and, arguably, more respectful of user choice (by not using hidden assumptions).

Comment 78

9 years ago
This problem is not specific to MS Exchange.  It's widely reported by gmail IMAP users, and I'm experiencing it with a dovecot IMAP server.

Is there some way that Thunderbird can test the server's behaviour and choose to use chunking or not accordingly?

This is a 7 year old bug with severity=critical!

Comment 79

9 years ago
BTW, why does this bug still have a status of NEW??

Multiple reports confirm it - at least, if this boils down to a philosophical issue, do close it as WONTFIX!

Comment 80

9 years ago
I forward a radical motion: let's just do away with this fetch_with_chunks madness.

Let's comment away all the code and the preferences, bandwidth and cpu power are increasing faster than attachment size so who cares about being bandwidth friendly, and if that's needed maybe the CPU stack will take care of it.

It would be better if we lived in a nicer world but we don't.

Updated

9 years ago
Duplicate of this bug: 355711

Comment 82

9 years ago
We are having the same problems her with multiple users where images get cut off part way though when displayed inline, and images and even some other kinds of documents can't be saved when the user is interfacing with our Exchange server over Imap. 
I have found that either changes as suggested in <a href="show_bug.cgi?id=92111#c45">comment #45</a> or mail.server.default.fetch_by_chunks change to false from <a href="show_bug.cgi?id=92111#c71">comment #71</a> and <a href="show_bug.cgi?id=92111#c74">#74</a>
 worked for solving the problem on a user by user basis. 

I think either dumping the preferences as suggested in <a href="show_bug.cgi?id=92111#c80">comment #80</a> or doing a test like suggested in <a href="show_bug.cgi?id=92111#c76">comment #76</a> would be the easiest from an end users point of view, because they could be blissfully ignorant of the situation. 

However I don’t agree with changing mail.server.default.fetch_by_chunks, as suggested in  <a href="show_bug.cgi?id=92111#c76">comment #76</a>, as I believe it is a global change in mail.server.default.fetch_by_chunks and might potentially mess up another of the users email accounts. (If chunking is even necessary anymore)
I would instead suggest that changing settings of 
user_pref(mail.server.serverXX.fetch_by_chunks, false); as suggested in <a href="show_bug.cgi?id=92111#c45">comment #45</a>  as it is specific to a single mail server would be better.

I have also submitted a new feature request <A href =”https://bugzilla.mozilla.org/show_bug.cgi?id=449416” >bug 449416</a> which suggests that they change the email account set up & the preferences such as there is an option for exchange servers and it sets the setting as in in <a href="show_bug.cgi?id=92111#c45">comment #45</a>.

Also as a Note I testing only changing the mail.imap.fetch_by_chunks set to false from <a href="show_bug.cgi?id=92111#c74">comment #74</a> and it did nothing. Since I thought it was more specific than the default settings I submitted a new bug <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=449407">Bug 449407</a>
Product: Core → MailNews Core
QA Contact: grylchan → networking.imap
Duplicate of this bug: 401085

Comment 84

7 years ago
There are several reasons and also few possible solutions for solving this issue.
- MS Exchange violates RFC and provides only approximate size of the message for "performance reasons"
- I saw also other mail servers providing incorrect RFC822.SIZE value

So sticking with this value is not a good idea. It is also discouraged by IETF author in RFC 2683 http://tools.ietf.org/html/rfc2683 section 3.4.5.

Solutions:
- in Exchange disable "Fast message retrieval" function/RFC violation. http://support.microsoft.com/kb/191504 But it will have performance implications.
- follow RFC 2683 recommendations and in Thunderbird fix the code and use message size provided by "FETCH RFC822.SIZE" command only for informational purposes. For message size validation and possibly truncation use e.g. size reported by "FETCH RFC822" as it seems to be correct value all the time.

Comment 85

7 years ago
I've found a useful addon that gives a workaround to this issue, attachment extractor seems to download the attachments fine: https://addons.mozilla.org/en-US/thunderbird/addon/556
(In addition to Comment #84)

Pine/Alpine looks suffering from Exchange too.
> http://lumbgaps.blogspot.com/2009/04/exchange-2007-imap-rfc822size-and.html
Following looks the question to MS.
> http://social.technet.microsoft.com/Forums/en/exchangesvrgeneral/thread/21adbe96-e21b-458a-8242-2c3894b9d7cf
> http://www.office-outlook.com/outlook-forum/index.php/m/509237/
Related Exchange setting seems;
> You can enable the exact size for everyone with:
>   Set-ImapSettings -EnableExactRFC822Size:$true
> Or only for a specific user:
>   Set-CASMailbox "IMAP User" -ImapUseProtocolDefaults:$false
>   -ImapEnableExactRFC822Size:$true
Exchange's Help for above setting.
> Exchange 2007. 
> http://technet.microsoft.com/en-us/library/aa998252(EXCHG.80).aspx
> Exchange 2010.
> http://technet.microsoft.com/en-us/library/aa998252.aspx
> http://technet.microsoft.com/en-us/library/bb125264.aspx
Linkify pointer to workaround in White Board.
  Workaround:
    See Comment #24 for Exchange setting. (Comment #86, if Exchange 2007/2010)
    See Comment #45 for Tb side setting.
        user_pref(mail.server.serverXX.fetch_by_chunks, false);
Duplicate of this bug: 564755
FYI. UI for mail.server.serverXX.fetch_by_chunks is requested by Bug 449416. xref.
Depends on: 449416
Summary: Microsoft Exchange IMAP: Attachments not fully downloaded (due to incorrect/smaller RFC822.SIZE by MS Exchange, which is RFC violation) → Microsoft Exchange IMAP: Attachments not fully downloaded (due to incorrect/smaller RFC822.SIZE by MS Exchange, which is RFC 3501 violation. mail.server.serverX.fetch_by_chunks can be a workaround. See comment #45)

Updated

6 years ago
Duplicate of this bug: 643240

Comment 91

6 years ago
I see no resolution of this bug or target for development.  University of Maryland changed to Exchange back server from a Unix system, and my Thunderbird constantly has this problem, particularly on thin network links.  I have up to date Thunderbird, and can volunteer testing to get this resolved.  The 'fetch-by-chunks' has no effect on this.  Any attachment of substantial size(>100K) hangs indefinitely on thin links (WiFi remote connections) -- appears to eventually recover on fat network links, such as in office on ethernet.  Using a docking system, I can characterize this bug quite well.

Any recent work on this?

Comment 92

6 years ago
Stan, don't get wrong but I think you should ask Microsoft same question, University of Maryland already pay for product so why no just get support from Microsoft and get fix in first place. I really doubt any core developer of Thunderbird will work to get some workaround of this issue, because its not Thunderbird itself problem but Exchange. But I believe if anybody contribute patch to "fix" that issue, developers will accept it for review gladly.
It's just too common see users come to open source product and ask to "fix" other vendor product.

Comment 93

6 years ago
Nikolay,
Three thoughts on what you submitted - somewhat conflicting. 
(1) I like Thunderbird as my preferred client - and choose to use it.  I have no choice in my Universitiy's primary email system.  I am not an IT manager or in the field any longer, just a somewhat knowledgeable EE.  To survive as a viable alternative client to Outlook or others, Thunderbird must somehow work well with Exchange server.  No doubt, its not a fun task.  I have multiple email accounts, and a heavy user.
(2) How do I bring such a bug to the attention of MS as I am not in the university IT department, but a simple user.  Is there an equivalent to BUG page for MS users?  I will pursue if knowledgeable.  I was under the impression, maybe somewhat mistakenly, that the service request would need to come from the IT department of the Univsersity - and their response is 'nobody else has reported this issue, why not try Outlook.'
(3) My preferred work around would be to find a good IMAP host and forward all mail to it.  Any recommendations?  I need to research this.

Comment 94

6 years ago
Stan, my understanding is that there is a setting on the Exchange server to make it behave correctly, so it's more a matter of contacting your IT dept. You can also tell Thunderbird to stop using mime parts on demand, which should ameliorate the issue, at the cost of downloading the full message whenever you click on it (tools, options, config editor, type mime_parts and toggle mail.server.default.mime_parts_on_demand to false by right clicking on it and picking toggle). There's a lot of information in this bug, e.g., the comment that links to http://support.microsoft.com/kb/191504

Gmail can host imap and I believe retrieve mail from other imap accounts.

Comment 95

6 years ago
though maybe gmail just does other pop3 accounts - not sure. It definitely supports IMAP for normal gmail accounts.

Comment 96

6 years ago
(In reply to comment #95)
> though maybe gmail just does other pop3 accounts - not sure. It definitely
> supports IMAP for normal gmail accounts.

David, I will look into the other setting in Thunderbird and read up on the link you provided - maybe it will give me the info I need to work with my IT department.

Note on GMAIL - it does support an IMAP interface, but beware if you use many folders in IMAP and then look into GMAIL through a web interface.  Not very pretty.

Comment 97

6 years ago
Does it still happen at Fox Chase? I am not sure I have moved into another sub-department of IT & don't deal with either the exchange servers or the Mac users on a Day to day basis.  I believe at this point in time most of our users are using OWA, apple mail, entourage or Outlook. I have put a call into Jim our senior macintosh support person to find out if anyone is still using Firebird & if so if they are still experiencing the problems. I will provide an update if he gets back to me.
Whiteboard: workaround see comment 24/45 → workaround see comment 24/45/94

Comment 98

6 years ago
(In reply to comment #94)
> Gmail can host imap and I believe retrieve mail from other imap accounts.

Gmail definitely has a functional IMAP interface: I use it as my primary email server (actually, the only one right now) and has no interoperability issues with Thunderbird whatsoever.

Gmail can download mail from other accounts only through POP3, which Exchange does support, unless it's been explicitly disabled - but it's much less common than disabling IMAP, since Exchange disk space is at a premium and sysadmins would much more have their users offload the servers than the other way around, so POP3 is good and IMAP is not.

I reckon, though, that if Gmail is the primary email server it's much better and less confusing to have it actually collect the messages so one only has to deal with one folder hierarchy (or, in Gmail-speak, only one tag set).

If all else fails, Exchange can easily be configured to automatically forward all messages to another address, through a server-side rule that can be enabled via OWA, too. Exchange 2007 and 2010 do it almost seamlessly, 2003 gets a bit more in the way but it works although it's not as pretty.
Summary: Microsoft Exchange IMAP: Attachments not fully downloaded (due to incorrect/smaller RFC822.SIZE by MS Exchange, which is RFC 3501 violation. mail.server.serverX.fetch_by_chunks can be a workaround. See comment #45) → Microsoft Exchange IMAP: Attachments not fully downloaded (due to incorrect/smaller RFC822.SIZE by MS Exchange, which is RFC 3501 violation. See comment #45 & comment #94 for Tb side workaround)

Updated

6 years ago
Duplicate of this bug: 695827

Comment 100

6 years ago
Is there any special award for bugs over a decade old?

Comment 101

6 years ago
Another question is, can we install a workaround automatically if an Exchange server is detected?

Comment 102

6 years ago
While I think it should really be properly fixed because it is my understanding that other mail servers return estimated sizes instead of actual sizes and as such the size should should not be relied on, I like the idea from comment 101.

Comment 103

6 years ago
I like the comment from 101 as well, but I am not an active programmer, so just a request.  Exchange has been forced upon me by the university, and Outlook is the only client that handles it well.  Outlook's IMAP interface is functional - but lacking.  Gmail's IMAP interface is fine, but it's native tag system drive's me nuts.  For what its worth.

Comment 104

5 years ago
This shouldn't be hard to work around. For Evolution we just "notice" that the server actually returns more data than it claimed was available in RFC822.SIZE. We expect the final 'chunk' to be truncated to the stated size, but in fact it isn't. At that point we *know* the server lied to us, so Evolution falls back to just asking for more and more chunks until the server really *does* stop giving us data.

In the common case of a non-broken server and an email which isn't a precise multiple of our chunk size, there's no performance hit.

When the claimed RFC822.SIZE happens to be a multiple of the chunk size, we do need to do one extra speculative fetch, which will return zero bytes on a non-broken server.

https://bugzilla.gnome.org/show_bug.cgi?id=621262 covers this.

I have to say, there is a lot to be said for simply declaring that Microsoft Exchange IS NOT an email server, and anyone trying to use it as one deserves all the breakage they get ☺

Comment 105

5 years ago
I like David's idea better than idea 101 as idea 101 assumes that only Microsoft exchange violates RFC 3501 and I have read about other IMAP servers doing the same thing and this would follow the previously mentioned recommendations of RFC 2683 section 3.4.5 "RFC822.SIZE" that clients can't rely on the RFC822.SIZE returned as there are a number of non compliant servers out there.   http://www.ietf.org/rfc/rfc2683.txt .
Duplicate of this bug: 730231

Comment 107

5 years ago
The same issue with the Seamonkey 2.7.2  mail and the google's IMAP servers.
• Almost all MS file types (XLS, PPT, etc.) are corrupted (most of the time) when received as an attachment on a x86-64 ubuntu PC. 
• Another PC (x86-64, Linux Mint)  with the same exact version, Seamonkey 2.7.2, receives the same attachments correctly.  Also, using the google's own web-mail client, the attachments can be extracted correctly.
• This issue is more apparent with the last few versions of the Seamonkey mail-client on my system.
Assignee: dbienvenu → irving
Status: NEW → ASSIGNED
Created attachment 618681 [details] [diff] [review]
WIP, read message chunks until the server stops sending more

First cut of patch to read IMAP bodies by chunks until the server stops sending more, to compensate for bad servers (Exchange, Google Mail) that lie about the message size in response to the RFC822.SIZE request.

Still needs:

 - test cases:
   - validation that it works for messages that end exactly on a chunk boundary
   - test that it works on chunks where \r\n splits across chunk boundary
   - works on messages that don't end in \r\n?

 - Correctly store "size of message as downloaded", or some similar change, to make sure that we can re-use the downloaded message from the necko cache
Attachment #618681 - Flags: feedback?(dbienvenu)

Comment 109

5 years ago
Comment on attachment 618681 [details] [diff] [review]
WIP, read message chunks until the server stops sending more

thx, looks reasonable - as you say, we'll need tests! I'm happy to help with that. Testing imap from xpcshell can be really really tricky.

The XXX comments will need to be addressed or at least unXXX'd before review...
Attachment #618681 - Flags: feedback?(dbienvenu) → feedback+
Created attachment 619239 [details] [diff] [review]
Handle chunked messages, this time also correctly handling non-chunked messages

First version of this patch wasn't correctly handling non-chunked messages. Still need tests, and still need to deal with the necko cache message size issues.
Attachment #618681 - Attachment is obsolete: true
Duplicate of this bug: 711644
Created attachment 625173 [details] [diff] [review]
Download all of chunked messages, with test cases

Added test cases, discovered that we weren't correctly handling CR/LF split across chunk boundary, fixed that too.

Also some clang warning cleanup; please double-check the added parentheses at nsImapProtocol.cpp:6524 (copied below); I changed the logic slightly, but I think it's better than it was:

@@ -6544,18 +6518,19 @@ bool nsImapProtocol::FolderIsSelected(co
 void nsImapProtocol::OnStatusForFolder(const char *mailboxName)
 {
 
   if (FolderIsSelected(mailboxName))
   {
     PRInt32 prevNumMessages = GetServerStateParser().NumberOfMessages();
     Noop();
     // OnNewIdleMessages will cause the ui thread to update the folder
-    if (m_imapMailFolderSink && GetServerStateParser().NumberOfRecentMessages()
-          || prevNumMessages != GetServerStateParser().NumberOfMessages())
+    if (m_imapMailFolderSink &&
+         (GetServerStateParser().NumberOfRecentMessages()
+          || prevNumMessages != GetServerStateParser().NumberOfMessages()))
       m_imapMailFolderSink->OnNewIdleMessages();
     return;
   }


This patch also includes the change to canonical line endings proposed by David for bug 740453, in order to make test_chunkLastLF.js pass on Mac/Linux; alternative if we don't want that change would be to change the test to disregard line endings or convert the test to use the same mechanism as test_imapChunks.js.
Attachment #619239 - Attachment is obsolete: true
Attachment #625173 - Flags: feedback?(dbienvenu)

Comment 113

5 years ago
Comment on attachment 625173 [details] [diff] [review]
Download all of chunked messages, with test cases

This breaks handling of non-chunking case when server returns an invalid size, right, and doesn't replace it with the correct counting of the download size, right? So it probably doesn't belong in this patch. The unrelated parentheses changes look OK, but it's probably better to have those in a separate patch. in case of regressions, it's better not to land patches with a bunch of unrelated changes...

other than that, this looks OK.

-          // if we are in the process of fetching everything RFC822 then we should
-          // turn around and force the total download size to be set to this value.
-          // this helps if the server gaves us a bogus size for the message in response to the 
-          // envelope command.
-          if (fFetchEverythingRFC822)
-            SetTotalDownloadSize(fSizeOfMostRecentMessage);
-
Attachment #625173 - Flags: feedback?(dbienvenu) → feedback+
Blocks: 740453
Created attachment 627991 [details] [diff] [review]
Correctly handle chunked messages, with Bienvenu's feedback

Removed unrelated warning cleanup changes. As discussed on IRC, the removed call to SetTotalDownloadSize() in nsImapServerResponseParser::msg_fetch() had no effect on correct handling of mis-stated message sizes.
Attachment #625173 - Attachment is obsolete: true
Attachment #627991 - Flags: review?(dbienvenu)

Comment 115

5 years ago
Comment on attachment 627991 [details] [diff] [review]
Correctly handle chunked messages, with Bienvenu's feedback

this looks good, in general.

one nit - brace below else here:
+  else {
+    numberOfCharsInThisChunk = 0;
+  }

unfortunately, test_chunkLastLF.js is failing on my debug windows build. 

The failure is here:

WARNING: NS_ENSURE_TRUE(thread) failed: file c:/builds/tbirdhq/objdir-tb/mozilla/netwerk/base/src/../../../../../mozilla/netwerk/base/src/nsSocketTransportService2.cpp, line 115
WARNING: unable to post continuation event: file c:/builds/tbirdhq/objdir-tb/mozilla/xpcom/io/../../../../mozilla/xpcom/io/nsStreamUtils.cpp, line 438
WARNING: nsExceptionService ignoring thread destruction after shutdown: file c:/builds/tbirdhq/objdir-tb/mozilla/xpcom/base/../../../../mozilla/xpcom/base/nsExceptionService.cpp, line 166
###!!! ASSERTION: Using observer service off the main thread!: 'Error', file c:/builds/tbirdhq/objdir-tb/mozilla/xpcom/ds/../../../../mozilla/xpcom/ds/nsObserverService.cpp, line 110
<<<<<<<
PROCESS-CRASH | c:\builds\tbirdhq\objdir-tb\mozilla\_tests\xpcshell\mailnews\imap\test\unit\test_chunkLastLF.js | application crashed (minidump found)
Crash dump filename: c:\builds\tbirdhq\objdir-tb\mozilla\_tests\xpcshell\mailnews\imap\test\unit\64888d62-4d6e-4c08-a79d-70acd1a50688.dmp

I suspect this means that test is either not cleaning up after itself, or finishes before we've stopped talking to the server. Have you tried this on a debug windows build? If you can't recreate the error, I can poke around a bit.

Comment 116

5 years ago
Comment on attachment 627991 [details] [diff] [review]
Correctly handle chunked messages, with Bienvenu's feedback

clearing review request because of test failure, which, at least for me, can be fixed by disabling autosync in the test.
Attachment #627991 - Flags: review?(dbienvenu)
Created attachment 628593 [details] [diff] [review]
Correctly download chunked email, Windows tests fixed

Fixed test as recommended by David, cleaned his nit, tested on OS X and Windows.
Attachment #627991 - Attachment is obsolete: true
Attachment #628593 - Flags: review?(dbienvenu)

Comment 118

5 years ago
Comment on attachment 628593 [details] [diff] [review]
Correctly download chunked email, Windows tests fixed

looks good, thx!
Attachment #628593 - Flags: review?(dbienvenu) → review+
Created attachment 631065 [details] [diff] [review]
Handle chunked messages, including 'peek'

Added an additional fix to handle the case of fetching BODY.PEEK[] as well as BODY[]; earlier version of this patch was sensitive to an existing difference between those two code paths.

Also removed two spurious assertions that were checking for conditions that occur (and are handled correctly) in normal operation.
Attachment #628593 - Attachment is obsolete: true
Attachment #631065 - Flags: review?(dbienvenu)

Updated

5 years ago
Attachment #631065 - Flags: review?(dbienvenu) → review+

Updated

5 years ago
Duplicate of this bug: 105606

Comment 121

5 years ago
fixed on trunk - http://hg.mozilla.org/comm-central/rev/5a6bee217340
Status: ASSIGNED → RESOLVED
Last Resolved: 15 years ago5 years ago
Flags: in-testsuite+
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 16.0
Duplicate of this bug: 390795
Summary: Microsoft Exchange IMAP: Attachments not fully downloaded (due to incorrect/smaller RFC822.SIZE by MS Exchange, which is RFC 3501 violation. See comment #45 & comment #94 for Tb side workaround) → IMAP: Attachments not fully downloaded (due to incorrect/smaller RFC822.SIZE by MS Exchange, gmail, and possibly others)

Comment 123

5 years ago
Well, that took ELEVEN YEARS to fix.

Comment 124

5 years ago
No, it took eleven years to work around. The server is *still* broken.
Blocks: 771442

Comment 125

5 years ago
I would not say this bug is "fixed".  

I am using Tbird 16.0.1 on Win XP Pro SP3.  I received two large attachments--one approx 2 MB and one over 600 KB--from a Postfix server which I access via IMAP.

Tbird truncated both messages to approx 414,000 bytes.  Searching the forums for a solution, I found this bug and the suggestion to turn off fetching by chunks.

I made the appropriate change in my configuration, and had the messages re-transmitted to me.  This time, Tbird was able to receive them in full.

So I believe that this bug should either be reopened, or reclassified as "Won't Fix" because the suggested workaround still works.
(In reply to Neil Parks from comment #125)
> I would not say this bug is "fixed".  

Writing to offline-store(offline-use=On, mail.imap.mime_parts_on_demand is irrelevant) and writing to Memory/Disk Cache(offline-use=Off, mail.imap.mime_parts_on_demand is relevant) has different mechanism.
Offline-use=On folder? or Offline-use=Off folder?

By patch for this bug, new IMAP log such as following was introduced.  
> +  PR_LOG(IMAP, PR_LOG_DEBUG, ("FetchTryChunking: curFetchSize %u", downloadSize));
Can you get IMAP log and check IMAP level flow?

If you can obtain IMAP log, open separate bug for ease of problem analysys by developers, please
If log analysis by IMAP people is needed, before attach log file to bug, remove/replace private data and remove many log lines for mail data except mail header, please. Remove also log lines for access of irrelevant Mbox, access of irrelevant server's Mbox, please.

Comment 127

5 years ago
(In reply to Neil Parks from comment #125)
My parents are having with I think is the same problem using TB 16.0.1 on Win7.  I do not see it on Ubuntu (for the same message).  My server is Postfix with Dovecot for IMAP.

Comment 128

5 years ago
This bug is fixed (it fixed at least one case). There may be other issues of course, but file new bugs for those if you still see problems.
See Also: → bug 577533
Depends on: 803843
You need to log in before you can comment on or make changes to this bug.