235942 - When viewing a mail on an IMAP server, the attachment is always downloaded (and again on Save As, if size > memory cache)

Status: RESOLVED DUPLICATE of bug 345832

(MailNews Core :: Networking: IMAP, defect)

Description

[Enrico Horn]

User-Agent:       
Build Identifier: 

I dont know if this is a bug or a feature.
When I view a mail on an IMAP server, the attachement is always downloaded.


Reproducible: Always
Steps to Reproduce:
1. Click on a mail with a large attachement 



Actual Results:  
Attachement is downloaded


Expected Results:  
I would expect that the attachement is only downloaded when I click on the
attachement.

Comment 1

[Tobias]

confirming this, plus if you actually want to open the attachment, it is being
downloaded again. One of the downloads are obviously not required, I would
prefer to just show the attachment first, and when I want to open it it should
be downloaded.

Happen also with the latest 0.7 version.

Comment 2

[kael]

(In reply to comment #1)
> Happen also with the latest 0.7 version.

And it still happens with version 0.9. It is only possible to stop the download
of attachments by clicking on the 'Stop' button, and hence to read the mail
without downloading all the attached files.

But it seems that there is no way to _download attachments_ and, more generally,
_parts on demand_ like it is possible to do with Mulberry http://cyrusoft.com ;
please, see http://www.cyrusoft.com/mulberry/mulbfeatures.html#mime. [1]

It is getting out of topic but there is the same problem with
'multipart/alternative' MIME mails.

Thunderbird downloads the whole mail even if it dislpays only the 'text/plain'
part - by clicking on the 'Toggle view between HMTL and plain text'.

What would be great is a *MIME button for IMAP* extension that would allow us to
download all parts on demand, attachments included.

I wish Thunderbird could behave comformable to RFC 3501 Section 2.4:
--------------------------------------------------------------------
2.4.    Message Texts

   In addition to being able to fetch the full [RFC-2822] text of a
   message, IMAP4rev1 permits the _fetching of portions_ of the full
   message text.  Specifically, it is possible to fetch the
   [RFC-2822] message header, [RFC-2822] message body, a [MIME-IMB]
   body part, or a [MIME-IMB] header.
--------------------------------------------------------------------

[1] It is currently possible to download attached files on demand but only to
save them on the hard-drive, no way to display them in the mail on demand.

Comment 3

[Atahualpa Jones]

Bug 281841 seems related to this:
https://bugzilla.mozilla.org/show_bug.cgi?id=281841

Comment 4

[Gervase Markham [:gerv]]

This is an automated message, with ID "auto-resolve01".

This bug has had no comments for a long time. Statistically, we have found that
bug reports that have not been confirmed by a second user after three months are
highly unlikely to be the source of a fix to the code.

While your input is very important to us, our resources are limited and so we
are asking for your help in focussing our efforts. If you can still reproduce
this problem in the latest version of the product (see below for how to obtain a
copy) or, for feature requests, if it's not present in the latest version and
you still believe we should implement it, please visit the URL of this bug
(given at the top of this mail) and add a comment to that effect, giving more
reproduction information if you have it.

If it is not a problem any longer, you need take no action. If this bug is not
changed in any way in the next two weeks, it will be automatically resolved.
Thank you for your help in this matter.

The latest beta releases can be obtained from:
Firefox:     http://www.mozilla.org/projects/firefox/
Thunderbird: http://www.mozilla.org/products/thunderbird/releases/1.5beta1.html
Seamonkey:   http://www.mozilla.org/projects/seamonkey/

Comment 5

[Gilles Mocellin]

seems good with Thunderbird 1.5b1.

But it always download the attachment the first time. Further reads of the mail
don't download again.
It would be good if at the first read, the attachment were not downloaded, just
on demand.

Comment 6

[guru5]

This looks to be identical to 315729. Both bugs mention 2 relating, but seperate issues.

1) When using IMAP, attachments should not be downloaded until they are selected. Only the message body should be downloaded initially when a message is opened...unless "view inline attachments" is checked in which case the attachments are being selected and should be downloaded as well (although I would limit this to only image attachments).

2) Once an attachement _has_ been downloaded (whether from being selected manually by the user or being auto-downloaded when displaying inline attachements), it should not be downloaded again when a user attempts to save that attachment or on subsequent views of that e-mail (this may be limited by cache size vs. attachment size).


Issue #1 is still a problem in Thunderbird 1.5rc1 (confirmed and reporoducible on Linux as well as Thunderbird 1.0.7 on Solaris).

Issue #2 appears to have been fixed in 1.5b1 according to comments and I can confirm using Thunderbird 1.5rc1 on Linux that attachements are not downloaded a second time if a user chooses to save the attachement or make subsequent views of a message within the same session...provided the attachment is small enough to remain in the cache. Large attachements (or attachements no longer in the cache) will still be downloaded on subsequent views which is fine. Users should save large attachments locally to avoid having to retrieve them multiple times from the IMAP server.

One of these bugs (235942 or 315729) can likely be closed. Although issue #2 appears to be fixed, issue #1 is still present in Thunderbird 1.5rc1 and causes considerable performance issues with messages containing large attachments.

Comment 7

[Nick Lott]

*** Bug 315729 has been marked as a duplicate of this bug. ***

Comment 8

[Nick Lott]

I've just tried using a fresh profile with RC2 and it doesn't look like either issue one or two has been resolved to me. 

Experimental Steps:

1. Open TB with fresh profile, set the settings, connect to IMAP server (over SSL).
2. Click on mail with large attachments - Attachments are dled before I see the message
3. Click onto a different mail
4. Click back onto first mail - Message is displayed but the attachments are downloaded again.
5. Close TB
6. Open TB - the last mail I had selected is automatically selected again; message is displayed, attachments downloaded again, "downloading attachments" continues to be shown even after the attachments appear.
7. Click onto a different mail
8. TB crashes. Talback Id: TB13224401K

I hope the crash is unrelated, but maybe the TB incident contains some useful info.

Comment 9

[Nick Lott]

Sorry, I didn't read Issue #2 properly. These attachments would have been too big to stay in the cache, so it my steps didn't test #2. 

BTW, which cache are we talking about here? Is this a system cache or a TB cache? If it's a TB cache id there any way to increase the size manually?

Comment 10

[mozilla.org]

Whatever the case, TB has major issues when dealing with large attachments over IMAP.  Even on a LAN with the mail server it is annoying.

Comment 11

[Nick Lott]

(In reply to comment #9)
> BTW, which cache are we talking about here? Is this a system cache or a TB
> cache? If it's a TB cache id there any way to increase the size manually?

To answer my own question, increasing browser.cache.memory.capacity from 4mb to 10mb seems to work.

Comment 12

[Nick Lott]

(In reply to comment #10)
> Whatever the case, TB has major issues when dealing with large attachments over
> IMAP.  Even on a LAN with the mail server it is annoying.

Now I have upped browser.cache.memory.capacity and "disabled view attachments inline" I get the behaviour I would expect. I no longer get either of the 2 issues explained by [e-mail]. 

So, *a* solution would be to just the raise the memory.capacity or set it to automatic (Using the figures given in http://kb.mozillazine.org/Browser.cache.memory.capacity, my cache would be set to 32mb, which is plenty). I guess there is reason why this is not set to automatic by default?. I don't know if this cache is cumulative or per email however.

Anyway, this will make me happy until I get an email with an attachment bigger than the new cache size, in which case the problem will come back worse than ever (although at 32mb most mailservers wouldn't let it through, and I don't think I'd want to download it). The "proper" solution, given this, would be to cache any attachment, no matter how big, either to memory or to disk. 

Judging from the name of the pref you are bound by the browser's caching mechanism? Perhaps it needs to have it's own mech? or perhaps the caching should be done to disk instead of to memory?

Comment 13

[Nick Lott]

Can someone correct the spelling mistake in the summary? It will be preventing anyone from finding this bug. (attachement -> attachment)

Comment 14

[timeless]

Incident ID: 13224401
Stack Signature	SetSecurityCallbacksFromChannel a090b94d
Product ID	Thunderbird15
Build ID	2005120115
Trigger Time	2005-12-24 00:46:34.0
Platform	Win32
Operating System	Windows NT 5.1 build 2600
Module	thunderbird.exe + (004bb324)
URL visited	
User Comments	
Since Last Crash	13843 sec
Total Uptime	13843 sec
Trigger Reason	Access violation
Source File, Line No.	e:/builds/tinderbox/Tb-Mozilla1.8/WINNT_5.0_Depend/mozilla/mailnews/imap/src/nsImapProtocol.cpp, line 666
Stack Trace 	
SetSecurityCallbacksFromChannel  [e:/builds/tinderbox/Tb-Mozilla1.8/WINNT_5.0_Depend/mozilla/mailnews/imap/src/nsImapProtocol.cpp, line 666]
nsImapProtocol::SetupWithUrl  [e:/builds/tinderbox/Tb-Mozilla1.8/WINNT_5.0_Depend/mozilla/mailnews/imap/src/nsImapProtocol.cpp, line 803]
nsImapProtocol::LoadImapUrl  [e:/builds/tinderbox/Tb-Mozilla1.8/WINNT_5.0_Depend/mozilla/mailnews/imap/src/nsImapProtocol.cpp, line 1608]
nsImapIncomingServer::GetImapConnectionAndLoadUrl  [e:/builds/tinderbox/Tb-Mozilla1.8/WINNT_5.0_Depend/mozilla/mailnews/imap/src/nsImapIncomingServer.cpp, line 450]
nsImapMockChannel::ReadFromImapConnection  [e:/builds/tinderbox/Tb-Mozilla1.8/WINNT_5.0_Depend/mozilla/mailnews/imap/src/nsImapProtocol.cpp, line 8265]
nsImapMockChannel::OnCacheEntryAvailable  [e:/builds/tinderbox/Tb-Mozilla1.8/WINNT_5.0_Depend/mozilla/mailnews/imap/src/nsImapProtocol.cpp, line 8091]
XPTC_InvokeByIndex  [e:/builds/tinderbox/Tb-Mozilla1.8/WINNT_5.0_Depend/mozilla/xpcom/reflect/xptcall/src/md/win32/xptcinvoke.cpp, line 102]
EventHandler  [e:/builds/tinderbox/Tb-Mozilla1.8/WINNT_5.0_Depend/mozilla/xpcom/proxy/src/nsProxyEvent.cpp, line 562]
shdocvw.dll + 0x150c24 (0x778b0c24)
nsWebBrowserPersist::OnProgress  [e:/builds/tinderbox/Tb-Mozilla1.8/WINNT_5.0_Depend/mozilla/embedding/components/webbrowserpersist/src/nsWebBrowserPersist.cpp, line 940]
0x1974c085

Comment 15

[timeless]

http://bonsai.mozilla.org/cvsblame.cgi?file=/mozilla/mailnews/imap/src/nsImapProtocol.cpp&mark=700,701,736,755,797,800,666,814,823&rev=MOZILLA_1_8_BRANCH#666

only one path checks mockchannel before calling into a function that expects it not to be null.

Comment 16

[timeless]

Attachment: possible patch for crash

i'm not sure if this is the behavior we want. i know i don't want to crash, and that the other callsite checks...

Comment 17

[David :Bienvenu]

Comment on attachment 206845 [details] [diff] [review]
possible patch for crash

it's worth a try, thx.

Comment 18

[Mike Cowperthwaite]

Regarding the initial report: I've tried reproducing the basic issue of 
opening a message and getting an immediate download of an attachment, and I'm not seeing it -- 1.5RC2, Win2K.  (Any reason IMAP behavior would be different 
b/t Linux and Windows systems?)  This is from the fastmail.fm server, if that makes a difference.  The message opens quickly; when I double-click the attachment, there's a wait (and a "downloading" progress dialog) while the attachment is transmitted, before it opens.

The attachment in question was a large PNG, so I made sure to turn off Display Attachment Inline.  Does that setting have any bearing on reporter's problem, 
or are the problem attachments things that normally wouldn't display inline?

Comment 19

[mozilla.org]

I am usually having this happen on attachments that don't display inline. . .like zip's, etc.

Comment 20

[David :Bienvenu]

What can happen is that the sending program sets the disposition to inline (I'm looking at you, MS Outlook!) even though the attachment should not be rendered inline. So the sending program can be at fault. (setting view | attachments inine to false will prevent that, of course)

Comment 21

[timeless]

Comment on attachment 206845 [details] [diff] [review]
possible patch for crash

mozilla/mailnews/imap/src/nsImapProtocol.cpp 	1.619

Comment 22

[Mark Johnson]

I think is a real bug, as opposed to "unconfirmed". It happens to me on XP Pro and Thunderbird 1.5 (20051201)

Comment 23

[Magnus Melin [:mkmelin]]

-> new using Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8) Gecko/20060130 Thunderbird/1.5 ID:2006013005

I'm definitely seeing this: comment 6 issues 1) and 2). Both with display attachments inline and when not. I tested with TB, large mp3 attachment (and the Content-Disposition was "attachment".)

If I increase memory cache to over the attachment size, it does not re-download on Save As any more = issue 2).

Comment 24

[Gilles Mocellin]

This cache must survive between sessions.
We need a cache on disk.
Why download again a document in an attachment every day because we have closed the mail client ?
Other mail clients do it.

Comment 25

[David :Bienvenu]

if you want a persistent cache, you need to configure the folder(s) for offline use. You can do this either from the folder properties dialog (offline tab) or from the file | offline | download and sync now | choose folders dialog.

Comment 26

[Magnus Melin [:mkmelin]]

While on the subject - is there a good reason thunderbird does not use the auto value for browser.cache.memory.capacity? (xref firefox bug 296538) 

The current default (manually set) value is 4096, which is not a lot on modern hardware. An auto value would be ~10-18M for most people.

Comment 27

[Matt Heinrichs]

FYI - these Thunderbird defects all appear to be dupes of this one:

337380 Downloaded imap attachments are re-downloaded when accessed whilst online.
345832 	IMAP attachments preloading (not on demand)
354044 	Caching for attachments (IMAP)
281841  attachments not cached when working in online mode

Comment 28

[Mike Cowperthwaite]

*** Bug 337380 has been marked as a duplicate of this bug. ***

Comment 29

[Mike Cowperthwaite]

*** Bug 354044 has been marked as a duplicate of this bug. ***

Comment 30

[Brandon Long]

It appears that thunderbird is caching messages/attachments in memory.  It appears that that cache is set by default to 4MB (browser.cache.memory.capacity).

This causes a bunch of very unhappy behavior with large messages with attachments.  For example, an 8MB attachment with 4 2MB images, Thunderbird will download the full message from the server 4 times, each time you view the message.  This is with the images displayed in-line, I don't know if not being displayed inline makes a difference.

Worse, it starts with a 10k window size for the downloads, growing it over time.  The window size generally reaches 400k or more by the time its downloading the message 4 times.  This means over the course of downloading the message 4 times, Thunderbird makes more than 150 partial BODY[] calls.

If its expensive for the server to fetch/assemble the raw message, Thunderbird makes the server do this more than 150 times.  Even without this, if the server can only handle the requests at about 5 per second, that would still take over 30s to download to the client.  Many users are probably on connections where the 32MB of download is going to take more than 30s also.

If you set the browser.cache.memory.capacity setting to more than the size of the message, it will be downloaded only once, but the small initial window size still takes 50 or more requests to download the whole thing.

Given that Thunderbird knows the size of the message (and asks for it every time its downloading the partial), it should raise the window size much faster.

As for the multiple downloads, it sounds like the simple fix is just to set browser.cache.memory.capacity based on an auto value.  Most ISPs seem to limit messages to the 10MB range, though it looks like gmail/hotmail/ymail all advertise limits in the 30MB range.

The real solution would be to cache to disk instead of memory.

Comment 32

[Mauro Cicognini]

I see this all the time using TB 2.0.0.6 (20070728) against MS Exchange 2003, regardless of view_attachments_inline setting.

TB's IMAP code apparently needs fixing, since I also notice heavy network traffic when moving large messages from a folder to another, when it should just be an IMAP move command and the bits should only travel on the server, if at all.

Comment 33

[Wayne Mery (:wsmwk)]

os=ALL per comment 28

Comment 34

[Mauro Cicognini]

As Matt pointed out in comment 27, this issue has been reported a number of times. 
Meanwhile, bug 337380, bug 354044 and bug 281841 have been duped or expired.

The only one still open is bug 345832 "IMAP attachments preloading (not on demand)".

Please dupe this bug or the other one, it's really the same issue that needs to be fixed.

If necessary, I'm available to work on it, but I'd need some pointers first on where to begin hacking (i.e. which is the relevant code section).

Comment 35

[Wayne Mery (:wsmwk)]

(In reply to comment #34)
> ...
> If necessary, I'm available to work on it, but I'd need some pointers first on
> where to begin hacking (i.e. which is the relevant code section).

bug 345832 seems to be more active and already has some of the info your talking about. you might suggest the dupe issue over there.

Comment 36

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So only issue 1 from comment 6 need to be fixed? Did I read it correctly? If yes, we should dupe this bug to 345832.

Comment 37

[Mauro Cicognini]

I can confirm issue 2 in comment 6 appears to have been resolved in all builds a while ago, provided the in-memory cache is large enough. A disk-based cache would be great but it's out of scope here.

I concur we should dupe this bug to bug 345832.