273032 - IMAP attachments are downloaded twice

Status: RESOLVED FIXED

(MailNews Core :: Attachments, defect)

Description

[Gregg Berkholtz]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.3) Gecko/20041007 Debian/1.7.3-5
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.3) Gecko/20041007 Debian/1.7.3-5

The setting "mail.server.server1.download_bodies_on_get_new_mail" is set to
true, but does not behave as expected:
 When saving an email attachment, the client goes back out to the server, and
downloads the attachment a second time - despite the fact that it had already
downloaded it along with the normal message body.

Reproducible: Always
Steps to Reproduce:
1. Open folder and watch Mozilla download/display headers and message bodies.
2. Select a message, open the attachment window right-click and select save-as.

Actual Results:  
Mail/News downloaded the message plus attachments, upon trying to save the
attachment it's downloaded from the server again.

Expected Results:  
Mozilla should have used the data which had already been downloaded.

Comment 1

[Boris Zbarsky [:bzbarsky]]

All the actual data is gotten by mailnews...

Comment 2

[Rimas Kudelis [use needinfo to ping]]

It's rather annoying when you have limits for downloaded data and see Moz
redownloading a 10-meg attachment. Plus, it's slow...

My vote for this bug.

Comment 3

[Mike Cowperthwaite]

This bug was supposed to have been fixed by bug 46233.

Testing under Windows 2000, it seems to me that it's working OK with TB 1.0, but 
*not* working with TB 1.0+0309; and working OK with Moz 1.7.5 but *not* with 
Moz 1.8b2-0309.  I'm curious that reporter claims it was broken in 1.7.3.

A workaround is to turn off   View | Display Attachment Inline
This prevents the initial download, which apparently is not getting cached for 
when the user wants to save/open/drag the attachment.

Comment 4

[Magnus Melin [:mkmelin]]

The + flag is only for drivers. Others should only set the ? flag, to request
blocking. 

Comment 5

[David :Bienvenu]

Is this bug saying that saving an attachment from a message that has been
downloaded for offline use still goes to the server?

Comment 6

[David :Bienvenu]

I've checked that if a message is downloaded for offline use, saving an
attachment from that message does not go out to the imap server, with my trunk
build.

What test did you do exactly, Mike?

Comment 7

[jsnod]

The problem is when in ONLINE mode, attachements get downloaded multiple times.

1) Make sure you have "Display Attachment Inline" enabled.
2) Open a folder containing an email with a large attachment (notice it takes a
long time to load - because it's downloading the headers, including the large
attachment).
3) Now right-click the attachment and Save As.  The attachment is downloaded AGAIN.

So attachments get downloaded twice instead of once.  This is murder with a
large attachment.

Expected behavior: Thunderbird should cache the original download so that it
doesn't have to download it again when you decide to save it to disk.

Comment 8

[David :Bienvenu]

that works for me...Do you want to try to generate an imap protocol log that
shows us downloading the attachment twice?

Follow these instructions, replacing "protocol" with IMAP

http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap

In your case, is the attachment inline (e.g., .txt or .html) or external, e.g.,
.xls or .doc?

Comment 9

[jsnod]

It doesn't appear that the logging works with Thunderbird.  Is there a way to
log with TB?

Comment 10

[David :Bienvenu]

logging works fine with thunderbird.

Comment 11

[jsnod]

Ok, I was able to get logging in TB to work (it doesn't work if you also have
Firefox open.. maybe that should be added to those instructions).

The result is an 8+ MB logfile that clearly shows the attachment being
downloaded twice (once when loading the message, again when saving attachment to
disk).  Would you like me to attach it?  It's a littlle over 1MB compressed. 
I'll also trim out some privacy stuff in the logs..

Comment 12

[David :Bienvenu]

I have Firefox open all the time, and logging still works in Thunderbird...there
must be some other variable involved.
 
You could just e-mail me the log if you want...you can also strip out the
beginning of the log, where we discover folders and fetch flags for all the
messages.

Comment 13

[Rimas Kudelis [use needinfo to ping]]

(In reply to comment #6)
> I've checked that if a message is downloaded for offline use, saving an
> attachment from that message does not go out to the imap server, with my trunk
> build.
> 
> What test did you do exactly, Mike?

If the message is not downloaded for offline use, the attachment gets downloaded
twice - just like jsnod has noted. I'm not sure if "attachments are displayed
inline" in my case (where do i check it?), but even if I have a message with an
8MiB mp3 file which I want to save, I have to wait twice - once for the message
to load, and once for the attachment to get redownloaded. That takes quite a bit
of time on a slow link. :/

Comment 14

[Mike Cowperthwaite]

Attachment: Log of attachment re-fetch

(In reply to comment #6)
> What test did you do exactly, Mike?

I have a sample message in my IMAP inbox with a small text/plain attachment.
I view the message with Inline Display of Attachments, then a moment later
select Save As on the attachment, and I can see the network traffic start up as
a result.

Here's a log file that I believe shows the issue.  Note that I only selected
the message once, and only asked to save the attachment once -- it appears to
my untutored eye that the entire message gets downloaded twice, which is pretty
bad.

Comment 15

[Mike Cowperthwaite]

Incidentally, all of that test was in online mode, I've never even tried using 
offline for IMAP.

A very similar problem for news at bug 211294.

Comment 16

[David :Bienvenu]

do you have thunderbird set to delay the marking read of a message after reading it?

Comment 17

[Eugene Savitsky]

Yes. I have.

Comment 18

[David :Bienvenu]

Attachment: proposed fix

the code we were using to remove the ? term from the url wasn't working
anymore, so the key were were looking for in the memory cache wasn't right - I
couldn't tell this from the debugger, but reworking the string code seems to
have fixed the problem. I also added header=src to the allowed ? terms, since
GDS uses that.

Comment 19

[David :Bienvenu]

FWIW, this only affects IMAP, not news.

Comment 20

[Scott MacGregor]

Comment on attachment 186682 [details] [diff] [review]
proposed fix

I vote for this piece of code as "the code most often broken"

 :)

I wish there was a cleaner way.

Comment 21

[David :Bienvenu]

Comment on attachment 186682 [details] [diff] [review]
proposed fix

with this change, it should be less fragile :-)

Comment 22

[Wolfgang Sailer]

As for comments #16 and #17: I also have this problem and I also set a 2 secs
delay when marking a read message as read.

Comment 23

[Wolfgang Sailer]

That bug is alive again in TB 1.5 RC1 (20051025):

I just started up TB and I found that I had received a 5200 kB email on an IMAP account.
1) TB downloaded the attachment
2 [review]) I chose to open the video by double-clicking it. TB downloaded from server again
3) I chose to save the video to HD. TB downloaded it a third time

indeed, the piece of code most often broken!!! Arggggggg!!!

SOMEONE PLEASE OPEN THIS BUG AGAIN!!!

Comment 24

[Ronny Adsetts]

This does appear to be broken in TBird 1.5. 5MB attachment, click on the attachment icon in the email, drag over desktop, up pops a saving dialogue (see atachment on next comment). If you drag and then drop without waiting for the dialogue, you get a 'sharing violation' error too.

Comment 25

[Ronny Adsetts]

Attachment: Save dialogue showing download

Attachment as previous comment.

Comment 26

[Magnus Melin [:mkmelin]]

I think the problems you have is for attachments over cache size - browser.cache.memory.capacity is 4096KB by default. See bug 235942.

Comment 27

[Etienne Le Sueur]

This still seems to be happening on Thunderbird 13.0.1 on Mac OS. In fact it's worse.

It seems like Thunderbird downloads attachments up to 3 (THREE) times. My cycle seems to go like this:

1) Email with attachment arrives, header is downloaded from IMAP server.
2) Double click on email, tab opens, but email is not shown, progress bar at bottom slowly creeps to 100% (It's clearly downloading the full email at this point).
3) Progress bar reaches 100%, email text loads, but if images are attached, they are downloaded a second time before they are displayed.
4) I now want to save those images, so I click on the filename at the bottom of the screen, and select Save As, and the download dialog pops up and the image starts downloading for a THIRD time.

This is so broken it's not funny. Maybe it's a cache-size thing. Regardless, something is broken.

If someone can tell me how to fix this, I would be most appreciative. This is so bad that I am close to giving up on Thunderbird and going to something else, maybe Mutt.