Closed Bug 345832 Opened 18 years ago Closed 14 years ago

download binary IMAP attachments on demand ONLY!

Categories

(MailNews Core :: Networking: IMAP, enhancement)

Product:
MailNews Core
Component:
Networking: IMAP
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED INVALID

People

(Reporter: deeno.privat, Assigned: Bienvenu)

References

(
URL
)

Details

(Keywords: perf) 

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4
Build Identifier: Mozilla Thunderbird Version 1.5.0.4 (20060516)

TB starts (down)loading the attachment as soon as i mark/select the mail, and when i select saveto it starts the same procedure again!!!
i receive almost every 2nd time SERVER TIMEOUT, or download is really slow etc... something is going wrong here

tried those prefs in MozillaTB1.5.0.4 about:config and it doesnt work...
maybe the wrong place to set them???

MIME Parts On Demand preferences are also in:

C:\Programs\Mozilla Thunderbird\defaults\pref\mailnews.js

[1] ...so which one should i use? about:config or mailnews.js

[2] are those the right values?

mail.imap.mime_parts_on_demand = true
mail.imap.mime_parts_on_demand_max_depth = 15
mail.imap.mime_parts_on_demand_threshold = 30000

if yes, then it is not working...

[3] is the 30000 in threshhold bytes or kb???

[4] does the server have to support that somehow?
what would i need to setup?

Reproducible: Always

Steps to Reproduce:
1. select an email with a large attachment
2 [review]. its loading something for ages (i think the attachment)
3. save attachment after finished loading
4. its downloading again for ages...
Actual Results:  
sometimes i get timouts from our server

Expected Results:  
just start (down)loading when i select "save attachment"

when i use my webinterface instead everything works fine!!! no timeouts no errors, downloading fast etc.

installed OutlookExpress tried it, it works... download ok, fast
(though no Donwload attachment on demand) but it just took only the
time for selecting the email (loaded a bit), click on attachment - was instantly saved!!!
Confirmed on Mac OS X as well.
Confirmed - reproducible as above 
Thunderbird 1.5.0.10 (20070221) with Mercury MTS 4.01b on Win XP SP2, No extensions except Talkback.

Definitely downloads attachments as soon as the mail is accessed, and won't display anything until the whole thing has been downloaded. Doesn't make any difference whether the message is viewed in the preview pane or opened in a new window. Monitoring network traffic confirms that there is no new traffic when opening the attachments, ie, they have been downloaded already.

Very large or multiple attachments are impossible to access or detach - for example, with 8 x 1Mb jpegs it times out after the first one.

Comments on the questions from konstantinos - mailnews.js sets the defaults for all profiles on the client machine - prefs.js over-rides those defaults.

Tried setting mail.imap.mime_parts_on_demand to false (using Config Ed.), closing Thunderbird, restarting and setting back to true - no difference 
are the attachments attachments that we display inline, e.g., jpeg? If you don't want us to display & download attachments inline, you need to turn off view | display attachments inline.

mime parts on demand only applies to attachments that we're not displaying inline.
Display of attachments inline is set to false, and this is how Thunderbird behaves. Attachments are not displayed inline.

I have looked at the IMAP server logs when accessing a message with an attached jpeg:

11 UID fetch 4253 (UID RFC822.SIZE BODY.PEEK[])
* 227 FETCH (UID 4253 RFC822.SIZE 155140 BODY[] {155140}

I don't understand that fully, but the empty section specification [], according to RFC2060, denotes the entire message.

The bug therefore appears to be that Thunderbird is not picking up that mime_parts_on_demand is set to true, and is asking for the whole message.


Confirm this behavior also on 2.0.0.0 (20070326) on Win XP SP2, on various independent IMAP server implementations.

A fetch of (UID RFC822.SIZE BODY.PEEK[]) is just an IMAP idiom to get everything. Asking for BODY.PEEK (instead of BODY) just tells the server not to clear the "unread" flag.

This behavior definitely comes from the bayesian antispam implementation, since turning that off changes heavily the network traffic shape.
yes, that's correct - we're trying to fetch message text to do bayesian analysis on it. You have to turn that off to prevent this.
The "real problem" here, I feel, is not that TB's trying to do its job and do bayesian classification: it's the lack of feedback on the UI, and the impossibility to turn this off easily.

I mean: usually, I'll have my laptop connected to my office LAN, or to my home ADSL, and everything's good.
Sometimes, however, I'll be at the airport, with a *very* expensive WiFi or UMTS connection, and I'd like to have something better than having to start TB offline, disable Adaptive Spam Filter, go online, etc.

Maybe something like a "vulcan nerve pinch" to stop the bayesian filter downloads in emergency situations? Something better than just killing TB outright?
Disabling the Adaptive Spam filter doesn't change the behavior on my system. Thunderbird continues to ask for and download the whole message, irrespective of message size, and irrespective of whether MPOD is true or false. I have logs (too big to post here) from both the server and Thunderbird if that is any use for debugging; the server logs also record time.

Surely the Bayesian analysis should be done on the Inbox, not stored messages? 
I would not expect anything with \Seen flag to be analyzed. Also, what is the point of Bayesian analysis of a mime-converted binary attachment; there will be nothing recognizable as spam? Thunderbird should recognise that MPOD has been asked for and, where Bayesian analysis is called for, do it on the parts as demanded, excluding any non-text parts, which will only dilute the ratings.
finally something happening here!!!

you all seem to be very into it, i almost haven't got a clou what you guys talking about, i just need to get this up+running... its been a year now living with this bug and its fine when i'm at the office or at home (DSL) but as soon as i am on the road it gets really annoying + expensive. last month i was almost twice a week and from may on i will be almost every day... is there no way to switch-off this misbehavior?

(i have to admit i was "flirting" with outlook, but i don't want to!!!)

also i tried to switch off junkmail filtering / bayesian analysis from tools-junk mail... but this option does not exist with my TB... i am using V2
yes, that ui does exist in 2.0 - it's per account, in tools | account settings | junk settings.

You may also want to turn off displaying attachments inline, if your large attachments are getting displayed inline.

view | display attachments inline
I'd like to stress the point that there are 2 things amiss here:

1. Bayesian analysis should NOT download the whole message, just text parts (good point Chris, there'll be no information in a MIME-encoded binary attachment)

2. If we really need to download the whole message, at least we should keep it somewhere, instead of throwing it all away and downloading stuff again if one needs to save an attachment.

And still, at the very minimum, there should be some way to easily disable Bayesian filtering when on slow connections, and UI feedback that there's something going on in the background when TB's downloading stuff to do Bayesian classification.

I don't think the Bayesian analysis is the root of the problem; as I say above, turning off the Bayesian filter does not fix the problem for me.
This is very strange.

Have you tried starting from a fresh new install of a recent version of TB, using a new profile?
strange think happens lately... problem does not persist anymore!
i dont know why? nothing changed, same laptop, same software, it works!
please confirm if problem has been fixed in an TB update...

konstantinos: Can you provide the exact version and build ID of the TB binary you're using (i.e. the numbers displayed in Help->About)?
version 2.0.0.4 (20070604)
i just received an email from a customer with 12 jpgs attached, each around 350kb small, we are using at the office a highspeed DSL6000 and it takes ages to download the attachments!!! itsbeen 20 min now, 4 remaining it seems it is hanging now... its my 5th try now!!! it is so annoying...
(it used to happen before as well sometimes)

it is definitely not a server problem because when i go into my webmail-interface and download the whole thing from there it takes only 20 seconds!!!

...i am so disappointed and fed up with all this bugs...
...why cant it just work?
...its been over a year now and still problems,
   i dont want to switch back to bloody MS
   but if problems still keep persisting then i cant work efficiently!
   (see also https://bugzilla.mozilla.org/show_bug.cgi?id=380275)

now its speeding up download,
i think its caching or preloading all the attachments and then downloading them again each one... almost the same error as before? i dont know anymore...
That's exactly the same behavior I see all the time.
But... didn't you write that you were not experiencing the undesirable behavior anymore?

Konstantinos, could you please vote for the bug, so that we may get it confirmed sooner or later?
David: do you think there's any point in doing Bayesian analysis on non-textual attachments?

Shouldn't it only work on the text (plain and/or HTML) parts?
i just voted!

yes, it does not preload the attachments on viewing the mail by just selecting it on the pane i guess... but again, maybe it has preloaded before! or my connection is so fast that i do not see it happening...
I too can confirm this bug.  I have some large pdf attachments in e-mails within an IMAP e-mail account that are doing the same thing, tried countermeasures listed here to no avail.  Since one of them is 24 Mb, it is...distressing...to be unable to view the text from the e-mail without waiting a long time, and then have to wait a long time again in order to download the pdf.  Never had this problem in Outlook, just changed over to Thunderbird.

Thunderbird version 2.0.0.4 (20070604)
OS: Vista
Display attachments inline: off
Default settings for IMAP
To save disk space, do not download for offline use messages larger than 50 kb
Adaptive Spam filter on or off: no effect
Extension: lightning
Always reproducible
Assignee: mscott → bienvenu
Severity: major → normal
Component: General → Networking: IMAP
OS: Windows XP → All
Product: Thunderbird → Core
QA Contact: general → networking.imap
Hardware: PC → All
Version: unspecified → 1.8 Branch
This is a 1.8 branch bug only. I cannot reproduce it with a current trunk build. Switching of display inline, shows me directly the message body. Using instead a 2.0.0.5pre build on Linux stills shows me this issue, even with deselected display inline option.
Status: UNCONFIRMED → NEW
Ever confirmed: true
what is a 1.8 brunch bug?

(In reply to comment #24)
> what is a 1.8 brunch bug?

The main development happens on trunk which will be Thunderbird 3.0 in the future. But all releases of Thunderbird 2.0.0.x are build from a separate branch which is 1.8 now. You can test with a Tb 3.0 alpha release where it should work. But create another profile therefor.
when can we expect a 3.0 version? (i know, stupid question, but it might be vital for not switching back to Outlook, because this bug is really annoying)

David, do you know when or by which patch this was fixed on trunk? Is it worth to backport it for 1.8? Or won't it get approved because its not a security bug?

Konstantinos, AFAICT the release of Thunderbird 3 will be happen after Firefox 3. 
Henrik, no, I don't know what the fix was. I was going to ask you if you were sure that you tested this on the trunk and 2.0 on the same profile with the exact same conditions because I can't think of anything that happened on the trunk but not the 2.0 branch in this area.
Yay! 

I'm not the only one being driven crazy by this behavior. Until I found this I was blaming our mail server, tough.

(tb version 2.0.0.6 (20070728), winxp sp2, dovecot(?) server on our LAN).

David, it's really suspicious. I can confirm that behavior with the latest trunk and 1.8 branch builds within my normal profile. After creating a new one the issue is solved. Do we have a pref which handles the pre-loading of a message? Or are special IMAP settings responsible for? Before I start to test I just want to ask.
Ok, there was a preference I set some time ago for testing purposes I think. mail.server.default.mime_parts_on_demand was set to false. But why this preference has higher priority as mail.imap.mime_parts_on_demand?

After setting this pref to false and leave the values from comment 0 all is working fine for me in Thunderbird 2.0.0.8pre and Thunderbird 3 nightly build.
Henrik, do you have Bayesian classification on or off?
Adaptive junk mail controls are enabled, even the checkbox for scam and anti-virus.
I'm really annoyed by this behaviour now.

I tried turning off bayesian filtering, and that helped a bit, but TB is still always downloading the entire message behind my back.

And, whenever I ask it for an attachment, it goes ahead and downloads it AGAIN!

Anyone wanting to really look into this? Please?
(in reply to comment #34)
Mauro, i was able to disable this behaviour by setting mail.imap.mime_parts_on_demand=true . See the comments in bug 399200, specifically comment #5 and comment #6. 
i did that ages ago... no change at all...
this is a really annoying bug!
we just opened a new branch in athens, and on all new PCs we installed OUTLOOK2003... broken-hearted...
but, its been over a year now living with this bug and no solutions yet...
i am even considering switching to outlook on my laptop + pcs in office thessaloniki + munich... its really a shame!
Konstantinos, does that also appear for newly created profiles? Or can you solve it per my comment 31? Can you see this with different IMAP servers e.g. Exchange, Courier. And could you try to run a test with the latest Thunderbird 3 nightly build? Please make a backup of your profile first!
thank you for the reply and suggestions... as soon as i find the time (that i do not really have) i will try them! different IMAP servers is not possible. i will give nightly build 3 a chance...
tried NB3... hmmm not sure if this worked... troubles me a bit because it takes ages again to download a 4MB attachment (while not preloading i think, as i switched to a 16.000 DSL) and from my webmailinterface its been downloaded almost imediatelly... there is still something happening in the background, but i think it works a bit better
Henrik Skupin on 2007-09-27 14:38:36 PST said:
> Ok, there was a preference I set some time ago for testing purposes I think.
> mail.server.default.mime_parts_on_demand was set to false. ...

YES! I found that my mail.server.default.mime_parts_on_demand was also set to false. So I toggled it to true and problem is resolved.

So this is true resolution to problem and I think others did not pay attention to what you said.

> After setting this pref to false and leave the values from comment 0 all is
> working fine ...

I assume this is typo and that you mean to say "After setting this pref to TRUE".

I have set both mail.imap.mime_parts_on_demand and mail.server.default.mime_parts_on_demand to "true" and still reproduce konstantinos's error. Confirmed on TB 2.0.0.9 (20071031), with Bayesian analysis turned off.


Long, are these parts we can display inline (e.g., text, images?) if so, you have to toggle off view | attachments inline if you don't want us to fetch the parts and display them.
I forgot to mention - I did disable viewing inline attachments.
Just tested with a nightly build of TB3 (3.0a1pre (2007122803)) - confirmed same behavior.
There are some more preconditions how a message gets downloaded. The mentioned pref is only a part as you can see here:

http://mxr.mozilla.org/seamonkey/source/mailnews/imap/src/nsImapService.cpp#508

I will run a debug session the next days to hopefully get the information why it happens anyway that the attachments get loaded not on demand.

David, it's interesting that mxr doesn't show up a function definition for aImapServer->GetMimePartsOnDemand(). Where is it located? Lately I'll see it when running my debug build.
http://mxr.mozilla.org/mailnews/source/mailnews/imap/src/nsImapIncomingServer.cpp#401

is where the getter and setter are defined - they just use a per-server pref, e.g., from prefs.js, which is why there's no caller of the setter.

An other precondition is that the message be larger than the mime parts on demand threshold, which is controlled by the pref "mail.imap.mime_parts_on_demand_threshold", which has a default of 30K.
As I've seen now the pref mail.imap.mime_parts_on_demand never gets evaluated. It can be set to true or false and nothing changes. Only mail.server.default.mime_parts_on_demand changes the behavior.

After searching there is only one place returned where this pref is only set:
http://mxr.mozilla.org/seamonkey/search?string=mail.server.default.mime_part&find=&findi=&filter=&tree=seamonkey

So the question is where does it get evaluated? Is there a string formatting function which builds the pref string? If yes, it could only be located at the URL mentioned in the last comment.
Ok, so mail.server.default.mime_parts_on_demand is a fallback if mail.server.serverX.mime_parts_on_demand doesn't exist for the imap account serverX.

Has anyone of you where it doesn't work set such a pref? It's still working fine for me with Thunderbird 3 and Thunderbird 2.0.0.9.
My Gmail IMAP server corresponds with server5 - I've set mail.server.server5.mime_parts_on_demand to true, and it still downloads the attachments (the attachment is 20MB, and I have the demand thresold at the default value of 30k). Tested in TB 2.0.0.9.
As comment 44 says it also appears for trunk builds. Adjusting version flag.

Long, it would be great if you could send me a fresh created profile where it happens. Please use a current trunk build therefor. Don't store any passwords or other private data. I would try to change the prefs to fetch mail from my own IMAP server to hopefully be able to confirm and debug it later.
Version: 1.8 Branch → Trunk
I think that bug 235942 is probably a duplicate of this one.
Thanks Long. I created a new IMAP account for Gmail and the issue can be reproduced with the prefs set to true. I'll dive in to hopefully find the reason why we immediately load the attachments.
Thanks Henrik, it would be great to know.
Wish you luck :-)
David, I need some assistance. How the ImapURL should be look like? Is there a difference between fetching the whole message or only the body?

Strangely I cannot see that anything is added behind the 'UID>/' part within the debugger. What I have to do to see it?

http://bonsai.mozilla.org/cvsblame.cgi?file=/mozilla/mailnews/imap/src/nsImapService.cpp&rev=1.332&mark=1019-1032#1017


the url doesn't change - it's the protocol that we issue that changes. You should see us fetching bodystructure, and then selected parts: the main body text, and any inline displayable parts.

 UID fetch 2 (BODYSTRUCTURE)

4652[62ca138]: ReadNextLine [stream=62cb0b0 nb=353 needmore=0]
4652[62ca138]: 62c02b8:mail.davidbienvenu.org:S-INBOX.Personal:CreateNewLineFromSocket: * 2 FETCH (UID 2 BODYSTRUCTURE (("text" "plain" ("charset" "iso-8859-1" "format" "flowed") NIL NIL "7bit" 273 8 NIL NIL NIL)("application" "msword" ("name" "David and Scott (rev 12 13).DOC") NIL NIL "base64" 157866 NIL ("inline" ("filename" "David and Scott (rev 12 13).DOC")) NIL) "mixed" ("boundary" "------------020004020909030809000107") NIL NIL))

4652[62ca138]: ReadNextLine [stream=62cb0b0 nb=23 needmore=0]
4652[62ca138]: 62c02b8:mail.davidbienvenu.org:S-INBOX.Personal:CreateNewLineFromSocket: 7 OK FETCH completed.

4652[62ca138]: 62c02b8:mail.davidbienvenu.org:S-INBOX.Personal:SendData: 8 UID fetch 2 (BODY.PEEK[HEADER] BODY.PEEK[1.MIME] BODY.PEEK[2.MIME])

4652[62ca138]: ReadNextLine [stream=62cb0b0 nb=38 needmore=0]
4652[62ca138]: 62c02b8:mail.davidbienvenu.org:S-INBOX.Personal:CreateNewLineFromSocket: * 2 FETCH (UID 2 BODY[HEADER] {2314}

guys, sorry about it:
I HAAAATE IT!!!!!!!!!!!!!!!!!!!!!!
this is really ****, i am going to use outlook sorry...
i just received several mails with huge attachments from customers and everything is freezing because the TB is preloading the whole sh-load... 45 MB...
funny thing: even with a 16.000DSL it takes ages!!! (its been 15 min preloading now, with a white screen). with my webinterface of my mailaccount its downloaded at lightspeed! this cant be it... i cannot work productive anymore... this issue has been around for at least 18 months (i reported this bug in july06), still no progress on this matter... i know  it is freeware etc... and it is a great peace of SW, but this bug is too annoying, it drives me crazy lately... my heart is bleeding... again: sorry for this posting, i just got crazy a couple of minutes ago... take care
When I installed Google desktop search, this problem occurred.
And, the problem dissolved when I removed the following files from a thunderbird components directory.
 GoogleDesktopMozilla.dll
 GoogleDesktopMozillaStub.js
 GoogleDesktopMozillaStub.xpt

I think that GDS fetches an attached file in defiance of setting such as mime_parts_on_demand.
I hope this problem is solved.
Tested in TB 2.0.0.14 & GDS 5.7.0805.16405-ja-pb.

I can confirm the Google Desktop problem/fix for TB version 2.0.0.14 (20080421) -- Thanks Dan! 

With inline attachments disabled, the cache size set to auto, and all mime_multipart settings set to true, the preview pane did the right thing but opening the message in a separate window still took forever. 

Removing the GDS files from the components directory fixed all. Does Google have a way to report bugs in GDS?
I just sent a bug report to Google, with pointers to this bug.

Let's see if and when they follow up.
Guys:
I too am extremely annoyed with this persistent bug.
I have tried all of the workarounds described here (except setting cache to "auto" as referred in comment #61 as I cannot see where in about config to do this) and none work.  I can ALWAYS reproduce the bug, no matter whether the preferences referred to within this whole bug report thread are set to true or false, whether junk mail filtering is active or not, and whether the display attachments inline is checked or not.
I get emails from clients (PDF's) above 1MB and it just takes sooo long to display the text of the message.  The attachments should just NOT be downloaded at all until a request is made for it--look at Sylpheed email client (IMAP) which works beautifully with regards to this issue.  You can just click on the pane to read the text (body) of the message; and decide whether you want to download or open to the attachment, etc.
I am using TB as my email client and want to stick with it of course, but this bug really needs to be addressed please.

WinXP Pro SP2
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.16) Gecko/20080708 Thunderbird/2.0.0.16 Mnenhy/0.7.5.0
Build ID: 2008070808
To stef_204: you don't say whether you have Google Desktop or not. Is this the case?

I can confirm WFM if I set all prefs as advised _unless_ I have GD installed, in which case there's basically no way to avoid the local download (extremely annoying, yes, but otherwise how can GD index your messages?)
Mauro, I do NOT have Google Desktop.

GD has nothing to do with this bug.  This is a bug in TB code/functionality; and needs to be addressed as a bug independent of anything else, GD, etc.
This bugs also annoys me and a lot of people around here who use TB with IMAP servers. Was going to report it as a new bug, but found Bug 432612, which in turn was marked as a duplicate of this one.

Are there any plans to implement IMAP disk caches in TB, just like the browser cache? This would solve everything except the slow downloading of attachments. Downloading large attachments multiple times is both a traffic and time killer.
Ditto to Andrew - apart from the GDS explanation, there's no particularly good reason TB users should have to wait for TB to download the same large message twice, no matter how they've (mis)configured their TB settings.  

Ultimately, I'd love to see a TB persistent disk cache for IMAP messages (why this couldn't be the same as the "go offline" message store, I don't know).

But in the meanwhile, I'd be happy to just avoid downloading the same message twice (with all the apparently massive IMAP overhead - I concur with folks who are frustrated that their webmail is so much faster) just to save an attachment to disk.

If this is ever definitively diagnosed, please consider a backport to TB 2.x.
Ditto to all above.

1. I recently added IMAP accounts to TB, latest, & find similar activity for large attachments.  

2. I do not run GD & only recently have added Google mail accounts due to change in my ISP's way of doing things.  

3. Before Google was in use at all I found getting the mail a very lengthy, time wise, process as I frequently recived 1M+ attachments from clients.  

4. Is there a way to suppress TB from downloading an attachment until asked to do so?  In this manner the headers would be received very quickly & those messages with attachments could be opened if time allowed at that time.
FYI: https://bugs.launchpad.net/thunderbird/+bug/275471

Is it reasonable to say that mail.server.default.mime_parts_on_demand should perhaps be set to True on a default Thunderbird installation, rather than False?

Is it also fair to say that sometimes Thunderbird is downloading attachments in order to display the message when it is first opened, but then when the user is requesting to download the attachments, they are AGAIN being downloaded, a second time? It looks like this is the case, but can someone confirm that?
David and Mark, which improvements on this bug do we have with the already checked-in fixes for bug 436615 and its dependencies?

We should make this bug more important. Asking for blocking Thunderbird3.
Blocks: 436615
Severity: normal → major
Flags: blocking-thunderbird3?
Keywords: perf
we already preload the whole message for offline use, so I'd say this bug is much less important now.

mime_parts_on_demand is already true by default - but we download inline attachments automatically, where inline == attachments we can display inline.
I am strongly of the view that email attachments are being downloaded twice by Thunderbird. I don't really care much *when* they are downloaded (although I would prefer it be on demand) by I really don't like that they download twice, especially when I'm on a limited bandwidth or metered connection.

Secondly, mail.server.mime_parts_on_demands was most definitely defaulting to False on my system. There was another imap.mime_parts_on_demand that seems to be defaulting to True, but I'm not clear whether or not the former was overriding the latter.

How is an 'inline' attachment detected? Does it have to be an image embedded within the HTML of the displayed MIME message part? Or can it just be any attachment capable of being rendered within the message window (eg a JPEG attached to a plaintext email)? In the latter case, I would rather the plaintext message display as soon as it is downloaded, and for the JPEG to appear later once it's downloaded.
As a user with both POP & IMAP accounts I can state that, out of the box, TB still downloads the attachments when the message is downloaded and then downloads the attachment again when the user calls for it.  This should be an obvious user setting choice whether defaulted on or off.  In my case I would definitely prefer the default to be off but I can only speak for my self on this one.  I see the reference to mail.server.mime_parts_on_demands and will be looking for its location this weekend but if anyone already knows where it is I sure would appreciate the direction.  BTW this is not a gripe to the developers as I know the time they dedicate to these projects.  It is a request.

JamesSoCal
James: It's at Edit->Preferences->Advanced->Config Editor.
I have located mail.server.default.mime_parts_on_demand which is set to true.  I was not able to locate an IMAP equivalent but it seems true still causes the attachments to be downloaded with the mail headers and again when I go to read the attachment.  At this time I am only running IMAP on 8 accounts from 3 different servers with the same performance on each.

JamesSoCal
John/James: "already" as in using latest trunk... 
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-trunk/

The default for mime_parts_on_demands is true.
Confirming on Shredder 3.0b2pre and SeaMonkey 2.0a3pre (Windows/Linux) that
this is no longer an issue when offline storage is activated, all messages and attachments are accessed through the locally stored copy without consulting the IMAP server again (new profile with default settings). However, not everybody may consider the overhead of keeping offline copies an acceptable option.

Based on some observations I've made while testing for bug 439731, caching may indeed be a contributing factor to this issue. It appears that when the message is downloaded as a whole, it will be cached as a single item; when downloaded
in parts, each attachment gets it own cache entry (after decoding) or is not cached at all. Thus, it depends on the download scheme and available space in the cache whether or not an item is available. Even if present in the cache,
it may not be used for a second access if the cache logic assumes that a revalidation is necessary.

Unlike HTTP entries, cached IMAP messages or parts thereof don't have any explicit expiration dates. One could reasonably assume that IMAP identifiers stay the same for considerable time, even if other messages are deleted. Modified messages (detached/deleted attachments) are rewritten and receive a new IMAP identifier. Thus, a reasonable default expiration for IMAP parts to avoid revalidation may be a way to reduce the problem of repeated downloads.
Just in case this contributes anything: I'm using the latest TB2 version and this problem is still there, out of the box as well as with all the configuration options above (which should not be necessary anyway). 

I second (third, etc) everyone above: this behavior is extremely annoying, i.e.:

1. When a new message arrives with an attachment (larger than a few hundreds of KBs), the application refuses to do anything in the corresponding account until the attachment is finally downloaded. I can't view the mail or any other mail in the same folder (both for GMail IMAP as well as our University's Exchange IMAP).

2. Whenever one then wants to save the attachment, it is downloaded again, although less often with the Offline setting (I think).

I've seen multiple friends migrate back to dreaded Microsoft email clients because of this issue, which indeed has been present in TB rather ridiculously long.

Please solve it, also in TB2. I don't want to wait for TB3 for this...

Steven
This wouldn't block tb3 esp. based on comment #78.
Flags: blocking-thunderbird3? → blocking-thunderbird3-
Product: Core → MailNews Core
(In reply to comment #80)
> This wouldn't block tb3 esp. based on comment #78.

In my opinion, it is important enough. I don't want to use an offline cache for all my messages (which does not help in the latest TB2 build by the way) and/or change about 5 advanced settings.

What is reasonable to expect is for Thunderbird to show the text of any of the messages quickly (and, in case of an inline attachment and the option being enabled, to also download the attachment, once). If there is a non-inline attachment, it should download the attachment (once) when I decide to open or save it.

This behavior would also be most consistent with Firefox, or any web browser for that matter, and also with any other email program I know.
guys, look at the date i started this thread! its been almost 3 years now! still nothing happened... this is really sad... i think i am not the only one believing this is a main issue and should have been solved by now...! during the last 2 years i know of 10 people swithing back to MSoutlook, including me and the rest of our company... this is so sad... i was hoping to siwtch back soon but as i can see, nothing done yet
Before commenting please note ...

(In reply to comment #78)
> Confirming on Shredder 3.0b2pre and SeaMonkey 2.0a3pre (Windows/Linux) that
> this is no longer an issue when offline storage is activated, all messages and
> attachments are accessed through the locally stored copy without consulting the
> IMAP server again (new profile with default settings). However, not everybody
> may consider the overhead of keeping offline copies an acceptable option.

and try http://www.mozillamessaging.com/en-US/thunderbird/early_releases/

and if offline store does not resolve your issue, or offline store is not a good option for you, please explain *why*.
(In reply to comment #80 and comment #83)
There are a couple of good reasons listed in bug 439731 why offline storage is not an appropriate option in all cases, mainly due to limited bandwidth and/or limited disk space. This is currently an all-or-nothing option per individual folder, and the arguments why not to apply it are certainly valid. Many users may not want (or simply cannot afford) to keep redundant copies of their IMAP folders locally that tie up a substantial portion of their disk space.

If you continue reading in my comment #78, which appears not quite correctly being interpreted here, enabling the disk cache per bug 439731 would reduce but not entirely fix the issue either. The combination of properly loading MIME parts as needed and caching them with an appropriate expiration scheme would be a solution for the issue addressed here. Switching on offline storage is rather a workaround, not satisfactory for everybody, thus I'd still consider this bug here quite relevant and worth doing it right.
(In reply to comment #83)
> if offline store does not resolve your issue, or offline 
> store is not a good option for you, please explain *why*.

OK. I'll bite:

1. Offline store assumes I actually plan to open the attachment some day. My
father-in-law likes to send 10MB emails loaded with photos which I usually
don't look at (he CC's my wife and gmail has a very nice thumbnail preview we
use instead). It's extremely frustrating to wait 5 minutes to read two
paragraphs of text. Ditto for large pdfs from work. That, and the latest
autoupdated version of TB (2.0.0.19) regressed, ignores the on-demand setting,
*and* re-downloads the attachments if I do ask for them. Removing Google
Desktop's .dll had fixed it for a long time, though I lost the ability to
search my email (which is bad because TB really has **** email search
functionality).

2. Offline storage (w/o load on demand) assumes I want to download the
attachment the first time I open the email. Once while attending a conference
my hotel (which also hosted the conference) gave a 10MB *total* download limit
per day, no exceptions or exemptions. I had to use webmail, do without, or find
an internet cafe because Thunderbird just couldn't wait to see my attachments. 

3. Even assuming I want the attachment now and/or multiple times, offline
storage assumes I'm allowed/willing to keep local copies of my email, which may
not be an option for corporate/government email and isn't that great an idea
for anyone toting a laptop through airports and other theft-prone places.

4. Offline storage doesn't help at all when I reopen that email from my second
(or third) machine.

5. Early releases aren't a realistic option for folks who aren't computer savvy
(and aren't complaining here for the same reason), don't control the machine
they read email from, or who prefer stable software.

Honestly, the backpedaling on this issue makes no sense to me. It's almost like
it would take a complete rewrite of the SMTP engine and the devs are scared to
take it on or something...

BTW, if some of the above points doesn't apply, please feel free to shoot them
down -- there's been enough comments and versions that I could have easily
missed something.
I have to agree with comment 85 by Ryan Johnson. Not the wording though.

I don't want a local copy of my mail. If I did, then I would use POP3.
What's the difference between POP3 with Leave on Server and IMAP with offline storage? (rhetorical)

And seeing it from an administrators point, the offline storage isn't a solution, since it hammers the server when new clients connect and leaves extra storage on the clients.
People who use IMAP, usually connects from a lot of different places. Different computers, webmail etc.

The offline storage might be a great idea for one-laptop-users, who's on the run. I can see the huge advantage here and I'm already setting this up for people!
just my bit

offline messages are unusable in my case, i am accessing shared imap folders which are several gigabytes big (company email) and messages in it can be several megabytes long. if i would try to sync them to disk it would take ages

what happens now for me is
- i click on email in list, i have to wait minute or so until text appears in message window
- i want to open attachment, it gets downloaded again
- i want to save attachment, it gets downloaded again
- i want to reply, whole message is downloaded again

etc. etc. imap like this is quite unusable, for some actions i am forced to use web interface which is far from comfortable - but works

i don't mind if content will be redownloaded when i change message to another and return back. but its not aceptable when it happens on every action

i know my comment is not very constructive, sorry for that, take it as one more reason why this bug should be solved (for me its actually blocker when it comes to company email which i use most of the time)
I agree completely with commenter #85. 

Why consider an issue as not really important or even solved, if the solution proposed is not a solution, but just a workaround -- and one that, mind you, in my version of Thunderbird does not even work, even though I downloaded 2 GB of email in an attempt to at least get things working a little bit. Still, any attachment bigger than 500KB makes me burst into sweat. And I'm actually using uncapped broadband here, unlike the less lucky ones who want to check their email at airports, for example.

Once again: this problem is major, *very* major, and it seems that nobody is really willing to do anything about it -- after 2.5 years of talking about it, it is still there.
my words... thumbs up steven!!!
at least after almost 3 years of opening this thread i dont feel alone anymore...
get your grandma's, your friends, your family to vote for this BUG!!!
24 votes is a bit "armselig" = poor
Hopefully the bug count will rise as I have more of my students moving to IMAP from POP.  For anyone out there with an audience they could be an excellent resource for problems of this nature what ever the problem may be.  Get them involved as it is the only way things may change.

BTW running 2.0.0.19 does not reflect any changes in behavior.

The comment regarding offline storage above is quite correct.  Offline with IMAP or just using POP seems to be the same thing with the exception of using the search option which needs to be in offline mode in any case.

Please just give us the choice so we can use it to the best of our needs as users.  The comments about regressing back to M$Outlook is truly sad.  The only way to stop the bleeding is to fix the causes & if we want to seriously challenge M$ fixing problems causing users to go back would seem to gain a higher priority.  Remember also that most users do not express themselves in the bug arena they just leave.  For every comment received there are likely dozens, or more, that just leave.  Maybe they don't want to "hurt someones feelings."  This is business and that should not be a consideration when trying to improve a product.  I commend the developers spending dozens or hundreds of hours on these projects but also say to them this is not a game.  I once was in your shoes and know the commitment it takes but if you seriously want the movement to move forward and become a real challenge to M$ even the smallest reason users have to leave needs to receive attention.

Thanks for the work you have done but lets not rest until the job is done.  I will continue to recruit users as I am out of the programming game now so don't let use down, please.

BTW again.  As for size TB is also included in Portableapps for a flash drive and there, again, speed & size is a real consideration when using an app.

JamesSoCal
1. Can anyone confirm that the issue is really solved in TB3? We saw this statement before on this page, and it wasn't true the previous times.

2. If it is not an issue anymore in TB3 indeed, when can I expect to get a stable version of TB3? It was mentioned in the text above around mid 2007, which is almost 2 years ago. Is it coming anytime soon? And if not, will the issue be fixed in an update for TB2, please?

Once again, let me also restate that I am very happy with the fact that people are so dedicatedly building a great email client for free. I don't want to come across as someone that only criticizes.
The schedule for a final tb3 is not set in stone yet, but it's a question of a few months. Won't get backported for tb2.
Magnus, your comment #91 concerns me...

Are you saying that this bug will not be fixed properly, but merely a workaround implemented (offline mode)?

Enabling offline mode is NOT a satisfactory solution to this bug - and yes, it is a bug - it is merely a very bad workaround that is UNUSABLE to me.

Please take my words for what they are intended to be - passionate, because with all of its warts, I love my TBird! This IMAP bug is one that has been at the constant forefront as THE bug that makes me consider moving away from TBird.

Will someone elaborate on whether or not this issue will be properly addressed, without having to enable offline mode?

Thanks, and thanks to all of you (you too Magnus!) for all your hard work!
Wow. Mozilla loves such old bugs. There are bugs in firefox - one has 11 years and second has almost 9 years, but this one is different: it makes tb almost impossible to use with imap and there is no idea how to fix it. Now, i'm impressed.
So you're saying this doesn't work for you on a nigthly?
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-comm-central/

As I wrote before, what part is left here? Especially after bug 439731.
No, Shredder b3 after clicking on message starts downloading atachments and i cant read it until entire message is downloaded so i can say: it doesn't work for me.
Actually, I am happy to say that so far, on Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1b5pre) Gecko/20090511 Shredder/3.0b3pre - Build ID: 20090511030814, the bug seems to be no longer present.

I have only been using this build occasionally for a few days so I'd like o reserve my judgment until I make more intensive use of it, but again, so far so good.
Ok, so, what behavior was finally enabled/added/patched?

Meaning, how is it supposed to behave now? I will be happy to d/l a nightly and test, but I'd like to know what the expected/intended behavior is first.

I'm glad this bug is being worked on, I just hope that Ryans comment #85 is given due attention, because his concerns mirror mine. Offline mode is not a solution to this problem for anyone who has many Gigabytes of email, and likes to wipe/reload Operating Systems on a regular basis, change computers, etc... thats one reason I love IMAP (I run my own server and use getmail for all of my non-personal accounts to get them all in one place that's under my control), Offline 'on-demand' seems like it would work for me, but I guess that depends on how well it was implemented.

All I do know is that the current 2.0.x behavior just sucks ass. The only reason I'm still using TBird is, well, almost everything else about it rocks (the only other major complaint is the lack of a proper signature manager, but the Quicktext extension goes a long way to remedying that, or you could use Signature Switch, but I like Quicktext better because it does more than just signatures.
If offline mode is enabled, obviously the mail (incl. attachments) will get read from the offline store, assuming the message sync from server has had time to complete of course.

Even without offline mode bug 439731 made messages get cached to disk when you read them. Then opening the mail later will get read it from cache.
I know, but again, offline mode is simply not an option for me and many others like me.

So, I am only concerned with changes that do not require offline mode to be enabled.

The main problem that annoys me is the fact that TBird downloads ATTACHMENTS if I click on a message. It should only d/l the attachment if I ask it to. I work with LOTS of attachments, and this just drives me and many others in our company nuts when working with email from home. It isn't quite as big of a problem here in the office, because we're on a gigabit LAN (mail server is located here), but remote access is almost unbearable.

Clicking on a message with a large attachment should only d/l the headers and the body text, NOT any attachments, unless/until they are requested (dbl-click > opened, or right-click > saved).
But attachment info is not inside the headers. You have to download the complete mail to parse out the basic knowledge that there are attachments in the first place, and what filenames to show. What you're asking for sounds impossible to me.

Anyway, for office use i don't see why the offline mode wouldn't be an option in general.
There is also bug 482476 now to introduce space and time policies for the offline storage. Not sure if this will make it in, but the combination of a most recently received limit for the offline store with the most recently accessed scheme of the disk cache would accommodate both aspects of looking at new and old messages. Nevertheless, making the expiration policy for IMAP parts more solid is still an item on the todo list to avoid unnecessary revalidations.
(In reply to comment #103)
> Anyway, for office use i don't see why the offline mode wouldn't be an option
> in general.

Offline is stored in %AppData%, which is the roaming profile.

Other mail clients doesn't seem to preload attachments - if they do it's really fast and uses cache in %TEMP%, but I don't think so, since they're fast on slow connections as well.

Does this bug need Wireshark'ing logs on a bunch of different mail clients?
> But attachment info is not inside the headers. You have to download the
> complete mail to parse out the basic knowledge that there are attachments in
> the first place, and what filenames to show. What you're asking for sounds
> impossible to me.

Webmail clients do it, don't they? I thought other clients do too, but I've only used TBird for so long, I can't remember...

> Offline is stored in %AppData%, which is the roaming profile.

Well, this particular issue can be handled using redirected folders, but let me be more precise about what I'd like to see...

I have posted the following as a question on the dovecot mail list - if anyone knows whether or not this is possible, Timo will, and I'll post his response here.

Here is the behavior I would like to see when offline mode is disabled:

Using the IMAP protocol, *without* downloading the entire message (specifically, binary attachments):

1. Download *only* the message headers and the text/html/body contents
of the body of the message but *not* any binary attachments,

2. Detect that a message *has* one or more binary attachments, *without*
downloading/decoding them,

3. Display for the user the email/email headers and text/html body parts,

4. Display the existence of the attachments in whatever manner the
client normally shows them (in TBird, they show at the bottom of the
message just above the status bar),

5. Download the actual attachment *on demand* - ie, if/when it is
dbl-click > opened, right-click > saved, or the message is forwarded?

This behavior, combined with the new on demand cache, will make, for me at least, a perfect IMAP reading environment.
(In reply to comment #106)
> > But attachment info is not inside the headers. You have to download the
> > complete mail to parse out the basic knowledge that there are attachments in
> > the first place, and what filenames to show. What you're asking for sounds
> > impossible to me.
> 
> Webmail clients do it, don't they? I thought other clients do too, but I've
> only used TBird for so long, I can't remember...

Webmail is different, the server can parse the email and separate out the attachments from the email and extend the protocols to allow separate downloading.

The main question I see in this bug at the moment is: "Is there an IMAP capability (RFC) that will allow downloading of the message header and body without downloading the attachments?".
Ok, as expected, Timo was both quick to respond, and most importantly, this should be *very easy to do*...

Here is his response, just posted to the dovecot users mail list, verbatim:

******** Begin my question and Timos answer ***********

On May 17, 2009, at 11:18 AM, Charles Marcus wrote:
> My question is:
>
> Using the IMAP protocol, is there a way, *without* downloading the
> entire message (specifically, binary attachments), that an IMAP client
> can do the following:

Yes. There are even several different ways to do it. Everything starts by doing FETCH BODYSTRUCTURE command, which gives the client knowledge of what MIME parts the message contains and also all the MIME headers. After that it can fetch only those parts it's interested in.

> 1. Download *only* the message headers and the text/html/body contents
> of the body of the message but *not* any binary attachments,

BODYSTRUCTURE shows Content-Type of each MIME part. After that client can do e.g. FETCH (BODY[1]) to fetch only the first MIME part.

> 2. Detect that a message *has* one or more binary attachments, *without*
> downloading them,
>
> 3. Grab the names of all of the attachments,

BODYSTRUCTURE should have all that information. If there's something missing, it's still possible to do FETCH BODY[n.MIME] to get MIME part headers.

> 4. Display for the user the email/email headers and text/html body parts,

I'm not sure what you mean by this.

*** What I mean by this is I use the preview pane, so I'd like to be able to
*** see the contents of the text/html parts of the body in the preview pane,
*** without having to d/l the attachments.

> 5. Display the existence of the attachments in whatever manner the
> client normally shows them (in TBird, they show at the bottom of the
> message just above the status bar),

Yeah. The only issue I can think of here is if TB wants to detect the attachment type by e.g. feeding the file to "file" command or something. But even then it would be possible to download for example first and last 1 kB of the attachment and hopefully file would recognize the type based on only those.

> 6. Download the actual attachment on demand - ie, if/when it is
> dbl-click > opened, right-click > saved, or the message is forwarded?

Yes.

> Pointers to appropriate IMAP protocol docs would be happily passed on to
> them the Mozilla dev guys.

It's all in RFC 3501 in FETCH command docs.

BTW. There are many clients that do this on-demand downloading. I use Apple Mail and Evolution and they both do it.

************** End my question and Timos answer ***********

Yay! Hopefully this will be enough info to at least prove that it can be done... now, it only remains to be seen if getting TBird to behave accordingly will be trivial, or for reasons of how it works now, be a major rewrite effort because of original code behavior/design...

Thanks guys, for working on this and considering our input!
And yet another response:

On May 17, 2009, at 12:26 PM, Timo Sirainen wrote:
>> Pointers to appropriate IMAP protocol docs would be happily passed on to
>> them the Mozilla dev guys.

> It's all in RFC 3501 in FETCH command docs.

Also I once wrote something to:

http://imapwiki.org/ClientImplementation/OpenMessage
Thx, I'm changing the bug summary...
Severity: major → enhancement
Summary: IMAP attachments preloading (not on demand) → download binary IMAP attachments on demand ONLY!
Here is something that could help.

Sylpheed <http://sylpheed.sraoss.jp/en/> is a simple, lightweight but featureful, and easy-to-use e-mail client (mailer, MUA).

It handles attachment (with IMAP accounts) perfectly--no matter how large the attachment is, it never downloads it.  You can instantly read the actual message and message body immediately, even if the message has a 10MB's attachment (or any size.)

Sylpheed only uses plain text, and the reason why I prefer TB is that TB can do html (and can do it very well) and has many many excellent features.

This attachment bug is indeed a huge drawback--especially for individuals very frequently receive large files (pdf's, etc.).

Sylpheed is OPEN SOURCE.  So why not download the source and see how Sylpheed (and its able developer Hiroyuki Yamamoto) does it?

That should give one and all a clue of HOW to replicate it in TB and settle the matter....
Thanks Magnus... I sincerely hope this is doable... it would/will turn an OK/bad IMAP experience into an absolutely wonderful one! :)
> Sylpheed is OPEN SOURCE.  So why not download the source and see how Sylpheed
> (and its able developer Hiroyuki Yamamoto) does it?

So is Evolution - but the source shouldn't be necessary... simply sniffing the IMAP session of a client that works correctly should be enough to see the commands being issued, which should be enough to figure out how best to implement it in TBird... and now that we have a list of 3 clients that are able to do this, hopefully this will now be doable...

Again, thanks Magnus and David and all the rest for listening and for all your hard work on this... I can only imagine how thankless working on OSS can be when you have a bunch of illiterate (as far as coding goes) users (like me) making demands on your time...
if you turn off view | display attachments inline, we won't try to fetch attachments that we know how to display inline, like jpegs.

We already use the imap body structure command, and don't fetch binary attachments that we don't know how to display inline, like .doc files.
Hmmm... does 'we know how to display inline' equate to rendering somehow without downloading the entire thing (I don't see how)?

A lot of what we deal with are large .jpg, .gif, etc, and I don't want those displaying inline either. Are you saying I have no control over this? Is this new behavior, or is this how it has always been? I would really like to have full control...

Maybe if it was possible to only display inline attachments that are smaller than some small/max size setting? I would want to set mine small, like 128K or something...
Wait... that doesn't make sense...

Setting View > Display attachments inline should *not* cause TBird to download attachments that it is incapable of displaying inline... are you saying it does?

Turning it off, I would expect it to *not* display any attachments inline... since thats what it says...
Never mind... its been a long day... thats what you said...

Ok... so, when you say 'We already use the imap body structure command, and don't fetch binary attachments that we don't know how to display inline, like .doc files.', you are saying this is the new behavior? Because it certainly wasn't the last time I checked Shredder (haven't checked the latest beta), and it certainly isn't in 2.0.x...
It's not new behavior - it was in Netscape 4.0, for example. However, there are some clients that will give binary attachments an inline disposition (Outlook, in particular), and that tricks us into fetching the part because the sender has said to display the attachment inline.
Ok, hmmm, wonder if there is any way to detect this (binary attachments given an inline disposition)...

Well, anyway, since you guys are working to finally fix this bugger once and for all, I want to start over testing the fixes...

Does the b2 release have all of these fixes? Or should I wait for the b3 release? Or, is there a specific reasonably stable nightly you can point me to that contains finished versions of all of the fixes discussed here?

Thanks David...
In response to #118

<<there are some clients that will give binary attachments an inline disposition (Outlook, in particular), and that tricks us into fetching the part>>

I have tested this by sending myself a big attachment using TB and then opening it on another computer in TB as well. I'm using the latest stable build.

As always, the attachment needed to be fully downloaded (taking a large amount of time) before I was allowed to read the text of the message, or do anything else in the Inbox. 

As has been said quite some times before, there are many email programs that do not have this behavior, even Outlook Express (a notable exception being Outlook, in which IMAP support is quite a lot worse than in TB).

It seems the cycle within this thread continues -- a rather concentrated number of messages stating that the problem persists, a number of promising responses from developers, followed by a message denying that it is a problem, or stating that it has been fixed already long ago.

To get back to an earlier message here as well: fortunately the bug is now marked as important enough to be solved for TB3. But meanwhile, the mainstream user is still using TB2. I've seen quite some people go back to M$ products as soon as the first big attachment in an IMAP account arrived. Personally I haven't seen a bug as discouraging as this one in the entire TB.
Steven, are you displaying attachments inline or not?
(In reply to comment #121)

> Steven, are you displaying attachments inline or not?

David, thanks for a quick response! Once again, let me restate that I love TB. If I didn't, I would not be participating actively here.

To be perfectly sure: 
I disabled offline storage and changed the inline attachments setting (not displaying them inline). I sent myself (from a different account within TB -- note that this is a POP mail account, I don't have two IMAP accounts available on this computer at the moment) a 7.5 MB RAR archive. 

Result: 
The message text was displayed almost immediately after I clicked the email.

Then:
I re-enabled inline attachments.

Result:
The message text was displayed almost immediately after I clicked the email. I am not sure whether it could have been in the cache (I exited TB and came back again).

Then:
I enabled offline storage.

Result:
After restarting TB, it displayed an hourglass for quite some time, until I clicked the message with the big attachment, which got displayed promptly.

Conclusion:
If you receive an email with an attachment that was sent using TB (a GMail POP account to be precise), the attachment is *not* downloaded before you can see the text of the email. David is completely right here.

But:
In my opinion, true attachments should be fetched only on demand, independent of who sent them or whether I display attachments inline. With offline mode on, they of course need to be fetched, but only once, and in a way that allows me to read the text of the message before the download starts. I realise it may be a difficult problem, but many other clients don't seem to have it.
In addition to what I just wrote (and sorry for spamming you all): it seems we are onto something regarding this bug. 

Can anyone confirm that the problem
a) Happens when you display attachments inline in TB and receive a mail with a large attachment, sent from Outlook;
b) Does NOT happen when you do not display attachments inline in TB and receive a mail with a large attachment, sent from Outlook?

If so, the actual cause of the problem has been correctly identified: TB sees a normal attachment as an inline one due to the faulty way in which the mail was constructed by the sender's mail program.
TB never marks attachments with content disposition inline when sending, I don't think, so a RAR archive sent from TB won't be fetched when you click on a message, independent of the display attachments inline setting.  I think most peoples' complaint has to do with image attachments, which we do know how to display inline. Turning off view | display attachments inline *should* make it so we don't try to fetch those images when you click on a message.

The offine auto sync issue is somewhat different. Autosync tries not to download large messages (> 50K) unless you're idle. But it has an aggressive definition of idle (no keyboard activity for 10 seconds, or an other window is in front). We could try to fetch large messages for offline use using chunking, to allow the process to be interrupted by the user, but it can slow down download drastically, which is extra bad for large messages. We deliberately don't cache partial messages in the offline store, so that we don't accidentally think we have the complete message when it's just partial.

You may have also run into a bug where we go idle on startup, if startup takes too long.
(In reply to comment #124)
> TB never marks attachments with content disposition inline when sending, ...

Default for mail.content_disposition_type is "inline" in TB2 and "attachment" for TB3, thus attachments may be send out with either disposition depending on that setting. Basically, the reverse of bug 452092 would be needed here, having a positive list of MIME types for which inline disposition makes sense to start with and only download those (preemptively or not) if they are inline and such attachments are to be displayed with the message.
I think this question may have gotten lost:

What version of TB3 should I be testing to see the latest/current bahevior with the patches created so far?

Also - any chance of being able to define a max_size that should be automatically downloaded if 'View > Display Attachments Inline' is enabled?

This way, tiny graphics (that are in many peoples signatures), and Pictures that have been resized to be small (ie < 100K) can be displayed, but not really big ones.
Most recent developments are included in the nightly builds for testing:
> http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-trunk/
Thanks... I know about nightlies, been burned more than once...

Is there a known relatively stable nightly that hass all of the fixes that you can recommend? If not, I'll just wait for b3...
And I'll ask this separately again...

Any chance at all of having a max_size setting for 'View > Dislay attachments inline', so that only attachments smaller than a certain size will be downloaded?
(In reply to comment #128)
> Is there a known relatively stable nightly that hass all of the fixes

Thunderbird 3.0 beta 2 has all changes up to late February, which includes
bug 436615 but potentially not all follow-up patches. It won't have the disk cache enabled, for example; I don't know which relevant patches may be missing. Nightlies are "per definition" unstable, but you could use a test profile and some free IMAP account where you do not mind the (unlikely but possible) case of loosing data.

> (comment #129) max_size setting for 'View > Display attachments inline'

That's not here but filed as bug 294763.
So Reading this bug just confuses the hell out of me and makes it quite unusable for anything except for discussion on how Thunderbird should work. There's nothing much that can be done with this bug.

Reminder : discussions should go on mozilla.dev.apps.thunderbird or the Tb-planning mailing list.

Some of the things reported here are already covered by other bugs - If not please file the missing one, with clear goals (and one goal per bug).

I'm resolving this INVALID because of it's too chaotic nature.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → INVALID
Yes, its chaotic here, but there are many less chaotic bugs that were marked as a duplicate of this bug. So maybe we should reopen one of them...

I think we all agree, that thunderbird should never download a big attachment more than once within a short time. Is this bug already filed somewhere else?
wow! great solution after FOUR!!! years since i submitted this bug...
no other solution provided so far now...
this IS really sad and not really a solution to say "There's nothing much that can be done with this bug." and "I'm resolving this INVALID because of it's too chaotic nature." ... where is the solution???
This is definitely NOT INVALID and NOT RESOLVED. The bug has been confirmed by many users, has received 34 votes. At least 7 bugs have been marked as duplicates of this bugs. 44 people have subscribed to the bug.

Please before closing the bug explain clearly what we're supposed to do to prevent this constant repeat-download issue that we're all seeing.
To get it fixed, we should have a short and clear description of the issue and not this lengthy discussion thread. But if we create a new bug it should be mentioned that it is in fact already 4 years old. Or maybe we can reopen one of the older duplicates...
Much of the discussion in this bug is now out of date, however it still remains under certain conditions as it has not been directly addressed.

FYI the resolved bugs that have *partly* addressed this one are bug 439731 and bug 436615. IMO the main outstanding issue is described in bug 439731#25.

If this bug has become to difficult to follow maybe a new bug could be opened with objectives relevant to the current state of Thunderbird.
(In reply to comment #135)
> wow! great solution after FOUR!!! years since i submitted this bug...
> no other solution provided so far now...
> ... where is the solution???

To konstantinos(bug opener), what kind of real problem of your comment #0(original problem of this bug) still remains in latest Tb 3.1 or Tb 3.2pre? 
(Im' not talking about added "image/jpeg with Display Attachments Inline=Checked" case, nor added "Content-Disposition:inline for non-inline displayable part" case.)
As seen in Bug 565852, current issues around multipart-on-demand(auto-sync is not enabled for an IMAP folder) are Bug 565852 and friends of that bug. i.e. Multipart-on-demand itself works as expected.
The original problem is still present in TB 5.0. Just as is the one in https://bugzilla.mozilla.org/show_bug.cgi?id=439731 (reloading of read messages).
Not exactly the specific issue described here, but more like bug 629738 now where the cache apparently isn't keeping track of when the message parts need to be reloaded, thus there are unnecessary invalidations and recaching of content.
I can confirm this bug is still present in the latest Thunderbird 5.0 relese to date. 

1. First open a new email with 6MB attachment - 
2. Notice long download, no message body until complete. (either blank or previously selected message body often still visible).
3. Right click attachment - save as - long download, progress bar visible and the attachment is re-downloaded from the IMAP server. 
4. Click on a different email. 
5. Click back on the 6MB attachment email again. 
6. Repeat 2-5

Problems / Correct Behaviour:
1. Attachments should not download before the body. 
2. Large attachments (as selected) should (optionally) only download when requested.
3. Attachments should be retained locally
4. When saving a downloaded attachment the local copy should be saved ~instantly.
5. When reselecting an email with an attachment, it should use the local attachment copy.

*In summary - an attachment should in ALL CASES be downloaded ONLY ONCE.*

However the backend is implemented, this is what the user should experience. There seems no sane reason to redownload attachments ever apart from perhaps coming up against some set limit for local cache size. 


3.
(In reply to comment #142)
> However the backend is implemented, this is what the user should experience.
> There seems no sane reason to redownload attachments ever apart from perhaps
> coming up against some set limit for local cache size. 

And morer specifically, if a folder is set to offline mode, then the emails/bodies/attachments should be stored in the offline store, NOT the cache.

I can also confirm that TB5 still redownloads attachments over and over again in folders set to offline mode - first, when you click/select the message, then again if you choose to 'Save As' the attachment, then again if you click a different message then go back to the message with the aqttachment.

Infuriating, when you have a lot of messages with large attachments on a remote/slow link to your imap store.
(In reply to comment #142)
The memory cache should be large enough now after bug 629247 removed the 4MB limit, but it's still a problem with caching of IMAP parts not working properly. I've seen 0-size attachments showing up on the 5.0 branch resembling on-demand attachments, with a "busy" indication in the status bar when viewing such a message suggesting an incomplete download of the MIME part. It's not clear to me if that's part of bug 629738 or a new problem now, somewhat hard to reproduce.

(In reply to comment #143)
Attachments should only be downloaded in offline mode if they haven't been synchronized yet. Do you see this also for old messages or after clicking on "Download Now" in the Synchronization tab of the Folder Properties?
I can confirm that further to my report in Comment 142 this particular folder *IS* set to offline mode with "Select this folder for offline use" ticked. 

Interesting development:

1. Created new TEST subfolder (also marked for offline use)
2. Move EmailA (1MB attachment)
3. Run above test on EmailA - no redownloading. 
4. Move EmailB (6MB attachment)
5. Run above test - (try saving, click off email then back on etc) - attachment redownloaded each time. 

Conclusion - is there some kind of caching/offline limit set? What doesn't make any sense is I have 'Dont download messages larger than 50KB' ticked in my 'Synchronisation and Storage' settings for this account. Presumably this is being entirely ignored as 1MB > 50KB. Unless the bug is that the code thinks 50KB is actually 5000KB...trying this now with settings set to 5KB...

Nope - still wont' work for 6MB attachment but fine for 1MB one. 

Also - when I click 'go offline' all the 1MB emails (body and attachments) are available offline. The 6MB email has no content at all - "The body of this message has not been downloaded from the server for reading offline...".

For some reason a 6MB attachment is being entirely left on the server - no use of offline file space. 

SECOND TEST - Created new subfolder with 'use offline' switched off. Testing will TB use cache? 

Result  - identical behaviour. The 1MB file is cached. The 6MB file is redownloaded each time. Offline storage vs Cached makes no difference. 

Hope this is useful feedback.
rsx11m - ah a 4MB limit! Well that explains my behaviour but my findings suggest this limit is NOT removed in TB 5.0.
There's no limit to the offline file store cache, so if the message is in the offline file store, saving attachments should just use the offline file store.

You can tell if the message is in the offline store by going offline (file menu, offline, work offline) and then clicking on the message. If we tell you the message body has not been downloaded, then it's not in the offline store.

The disk and memory cache complicate this quite a bit. Also, simply clicking on a message with a large external attachment should not end up with it getting put in the offline store. You'd need the imap auto sync to do that, or the aforementioned download now button.
(In reply to comment #146)
> rsx11m - ah a 4MB limit! Well that explains my behaviour but my findings

That's the previous limit of the *memory* cache, which caused issues in the past. The disk cache has a default limit of 50MB, and as David just stated, the offline store doesn't have any pre-set limit. It only goes into effect though if and when the message and/or its attachments are actually synchronized, until then you have to rely on the disk and/or memory cache (which, as stated, is somewhat broken).
(In reply to comment #147)
> The disk and memory cache complicate this quite a bit. Also, simply clicking
> on a message with a large external attachment should not end up with it
> getting put in the offline store. You'd need the imap auto sync to do that,
> or the aforementioned download now button.

I would love a simple way to completely and utterly disable disk/memory caching, and use only offline storage.
More info:

I set up an entirely empty IMAP account in thunderbird and moved the 1MB and 6MB files to it. I set this up for offline mode and ran the 'Download Now' to force the offline sync. 

THIS WORKED!

The 6MB file is now loaded instantly and available when offline. I can save the attachment to my local disk and it doesn't download off the server.

So we can break down the problem into:

1. Normal cache not working - consensus is this seems to be broken generally (separate bug?)

2. Offline files sync broken for large mailboxes - seems to work on small mailboxes. Doing a 'sync now' on a large mailbox (100,000 emails, 100 folders, selected for offline use 10 folders, 10,000 emails) doesn't seem to actually work - perhaps some timeout in effect?

3. Cache -> Offline files - why on earth does TB not put out to offline files as it downloads email? Why is Offline files sync a separate process? Surely it makes sense to update the offline files storage from Cache so there is still a single download. 

Are there open bugs for the above already? If not, and if people can replicate, we should open them. I get the feeling putting all this effort into a 'RESOLVED INVALID' bug may be falling of deaf ears.
Five (5!) years later, in Thunderbird version 5.0, it is still having this most annoying bug. Mail with attachments takes forever to open, then, when you click on the attachment to open it, apparently the download starts afresh. Thunderbird has always done this, on all 4 computers in our house, for ALL IMAP accounts.
If you think about it, this bug and the way it is dealt with by the developers is exemplary for the product. It's "major updates" are minor patches. Who notices the heralded new account wizard as an exising user? Nobody. Who has problems with inline attachments? Everybody, since version 0.01 of the product. So let's develop a new account wizard and mark the inline attachments bug as "RESOLVED INVALID". 

Similarly, all the other issues and feature requests I invested time and effort in on this forum have not been seen as relevant by the developers. (E.g. the fact that entering and especially copy-pasting or cutting multiple email addresses from e.g. the To: field is so ridiculously cumbersome; the fact that the search engine introduced in TB3 is incredibly slow; the fact that properly layouting HTML emails is really a pain; the ugly and inconvenient placement of buttons in the preview pane header; the fact that only new email composition does not happen in a tab; ...) 

I've started contributing bugs and feature requests around version 1.x. We're now at version 5, five years later at least, and most major opportunities for improvement have not been addressed. Thunderbird as a consequence still is an immature product, and given how the developers prioritize their coding efforts, I'm sure version 8 will give us the 5th major GUI overhaul and a spectacular New Account Wizard... and it will still reload IMAP attachments every time you select an email.

I hate to say it as a definite non-fanboy of Apple, but if the TB developers would take a short look at Apple Mail (even the one in OS X 10.6), they'd notice everything that is wrong with TB. If Apple can do it, then it must not be too difficult to write a simple yet functional email program.
(In reply to Joe Smith from comment #140)
> The original problem is still present in TB 5.0. Just as is the one in
> https://bugzilla.mozilla.org/show_bug.cgi?id=439731 (reloading of read
> messages).

Actually, it seems to have been rather a user error in my case. :) The problem occured when I had limited the size of e-mails to be downloaded to 100kB. That was the case ever since I started using TB almost a year ago. So that those large e-mails (and also their attachments) were downloaded each time the e-mail was opened or the attachments saved.

After disabling of the mail size limit in account options the problem seems to be gone. TB then downloaded all the previously too large e-mails. It now still shows "loading" for a short time when a large e-mail is opened, but that seems to be loading from hard disk (which goes very busy during that time) while the network connection is idle. I gueass that rather short loading time now is probably a result of the compression I have activated in TB for the e-mail files, and the subsequent need to decompress when opening an e-mail.

So, there seems to be no issue for me after all.
Joe - interesting. Though what you're doing is solving the problem by downloading ALL attachments. The bug here is that attachments should not be downloaded unless requested (and then only once!).
Why is the status for this bug set to 'RESOLVED INVALID'? This bug is still very much alive and well.

Joe - do you have 'Tools' > 'Synchronization & Storage' 'Keep messages for this account on this computer' checked/enabled? If so, then that is not what this bug is about...
(In reply to Charles from comment #155)
> Why is the status for this bug set to 'RESOLVED INVALID'? This bug is still
> very much alive and well.
> 
> Joe - do you have 'Tools' > 'Synchronization & Storage' 'Keep messages for
> this account on this computer' checked/enabled? If so, then that is not what
> this bug is about...

Yes, I have it enabled. Sorry and thanks for the info.
See Also: → 1454542
You need to log in before you can comment on or make changes to this bug.