779717 - IMAP attachment handling is horribly inefficient (downloads whole message for each attachment even after bug 740453)

Status: RESOLVED INCOMPLETE

(MailNews Core :: Networking: IMAP, defect)

Description

[Adam Nielsen]

When opening an e-mail on an IMAP server with a number of attached pictures (not embedded but attached), this seems to happen:

 1. The *entire* e-mail is downloaded, pictures, attachments and all, then the pictures and attachments are discarded and the text content displayed.

 2. For each attached picture, the *entire* e-mail is downloaded again, and the one picture is extracted and displayed under the message text.  If there are 10 pictures, the entire e-mail (all 10 pictures) is downloaded 10 times over (e.g. 100 pictures are downloaded)

 3. Double-clicking on the e-mail to open it in a new tab repeats the entire procedure.

 4. Opening a different message in a new tab wipes the first e-mail from its tab, so switching back to the original tab repeats the whole process again, downloading the entire e-mail 10 times over.

 5. Double-clicking on an attachment (e.g. a PDF) downloads the entire e-mail again for no reason (perhaps to obtain the attachment size), then opens up the download manager which downloads the entire e-mail yet another time but showing a progress bar instead.

This is horribly inefficient and extremely slow, causing a 10MB e-mail with 10 photos to use up to ~110MB of download quota each time it is viewed (~550MB with the above example of changing tabs and saving an attachment!)

My IMAP server supports downloading messages without attachments, but even if it didn't, the message should be downloaded once only and cached for a little while to avoid this excessive waste of bandwidth.  It is completely unnecessary to download an e-mail 55 times in the process of viewing it once.

At this rate, with my limited Internet quota, I will only be able to check my inbox with Thunderbird once a month before I get cut off until the next month.

Comment 1

[WADA:World Anti-bad-Duping Agency]

Dup of bug 740453?
See also bug 774217 for problem which is perhaps caused by bug 740453.

Offline-use=On? Offline-use=Off? (Folder Properties/Synchronization)
Do you see your problem with Offline-use=On?
If yes, see also bug 92111(problem when incorrect RFC822.SIZE is returned from server).

Comment 2

[:aceman]

Adam, bug 740453 that WADA mentions was fixed in TB16. Can you at least temporarily try it (so that we know if your issue is the same)? You can download it here: http://www.mozilla.org/en-US/thunderbird/channel/ and it is the Earlybird version.

Comment 3

[Adam Nielsen]

Thanks for the quick replies!

@WADA: I have offline-use=off, because I don't want TB to download every message in my remote inbox - only the ones I choose to open.  I don't believe I would see the problem if I enabled offline use, but then it would use up my Internet quota by downloading many messages I have viewed elsewhere.

@aceman: I downloaded Earlybird (16.0a2) and there's no change.  Because it's so slow downloading the large e-mails, if I open a message in a tab (in Earlybird), then go back to my inbox and select a different message, when I go back to the tab I see the message from my inbox for 2-3 minutes until the correct/original message has re-downloaded.  It's not just a visual issue either - I can scroll around and select text in the "wrong" message before the correct message suddenly appears when the download has completed.

As a side note, in Earlybird 16.0a2 none of the menus work.  Right-clicking does nothing, and clicking File, Edit, etc. has no effect.  Thunderbird 12 works fine in this regard.

Comment 4

[:aceman]

Irving, Bienvenu, what log would you like from the user here to see if the fix in bug 740453 doesn't work for him?

Comment 5

[Kent James (:rkent)]

(In reply to :aceman from comment #4)
> Irving, Bienvenu, what log would you like from the user here to see if the
> fix in bug 740453 doesn't work for him?

The normal log is I assume IMAP NSPR logging described here:

https://wiki.mozilla.org/MailNews:Logging

Comment 6

[:Irving Reid (No longer working on Firefox)]

Adam, can you run Thunderbird 16 (or better yet, Daily) with the environment variable

NSPR_LOG_MODULES=imap;5

and then do the reproduction steps (display a message with a few attachments, switch view to another message, switch view back to the message with attachments) and capture the standard output / standard error from Thunderbird?

This log will contain contents of the message, so i'd suggest using a message with non-sensitive information.

Also, if you could let us know which platform you're running on - looks like Linux? Do you know what IMAP server you're downloading from?

Comment 7

[WADA:World Anti-bad-Duping Agency]

(In reply to Adam Nielsen from comment #3)
> @aceman: I downloaded Earlybird (16.0a2) and there's no change.

Actually "No change"?
I couldn't observe problem of bug 774217(persistently occurred in Tb 14) in latest Earlybird (16.0a2) on MS Win. This indicates that bug 740453 is surely fixed in Earlybird (16.0a2).
Do you still see next problem in your comment #0(same as problem of bug 740453) in in Earlybird (16.0a2)?
> 2. For each attached picture, the *entire* e-mail is downloaded again,  

> Because it's so slow downloading the large e-mails, if I open a message in a tab (in Earlybird),
> then go back to my inbox and select a different message,
> when I go back to the tab I see the message from my inbox for 2-3 
> until the correct/original message has re-downloaded.

"Swiching of Tab" has at least problem like bug 487386, bug 541876. Please rule out such issues by not using "Swiching Tab" in your test for this bug.
And, Tb surely has problem of bug 629738. Problem with what kind of mail?
(a) Large/simple text/plain or text/html.
(b) HTML in multipart/related with many Embed images.
(c) multipart/mixed with many inline-displayable attachment parts,
    for example, text/plain, text/html, image/jpeg,
    with View/Display Attachments Inline = Checked/Unchecked.
(d) multipart/mixed with both inline-displayable and not-inline-displayable part.
(e) not-well-formed mail, for example, cid: URL and Content-Id: in multipart/mixed, non-referred parts under multipart/rlated, nin-displayable parts such as PDF under multipart/alternative. 
If (d), bug 629738 is applicable, and if (e), bug 629738 is perhaps applicable.
 
> It's not just a visual issue either - I can scroll around and select text in the "wrong" message before
> the correct message suddenly appears when the download has completed.

Can you see it even with "Display Attachments Inline=Unchecked" with mail of (d)?

If big mail with many inline-displayable subparts under multipart/mixed with Display Attachments Inline=Checked, of if multipart/related(HTML mail) with many embed images, download of all inline-displayable subparts is required before rendering of mail and it takes long. So, problem like bug 533499, bug 538803, bug 714489, bug 612279, bug 719751 is exposed.

> As a side note, in Earlybird 16.0a2 none of the menus work.  Right-clicking
> does nothing, and clicking File, Edit, etc. has no effect. Thunderbird 12
> works fine in this regard.

Does it occur even with -safe-mode of Tb?

Anyway, get IMAP log with access of a new IMAP folder only, with a few small test mails in the folder, and check IMAP level flow.
Do you still see problem of bug 740453 in log?
Is phenomenon of bug 629738 involved in log?
How long does it take to download of all rquired parts for message rendering? (get log with timestamp parameter)

Comment 8

[:aceman]

(In reply to Irving Reid (:irving) from comment #6)
> Adam, can you run Thunderbird 16 (or better yet, Daily) with the environment
> variable
> 
> NSPR_LOG_MODULES=imap;5
> 
I think it is "NSPR_LOG_MODULES=imap:5" (a double colon).

Comment 9

[Wayne Mery (:wsmwk)]

Adam, can you supply a log?

Comment 10

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk) from comment #9)
> Adam, can you supply a log?

Comment 11

[Wayne Mery (:wsmwk)]

still needs more info, so => incomplete

Comment 12

[Wayne Mery (:wsmwk)]

Adam wrote me
According to my e-mail history, this was the last e-mail I sent using
TB:
  Date: Sun, 23 Jun 2013 11:26:08 +1000
  User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-GB; rv:1.8.1.23)
    Gecko/20091130 Thunderbird/2.0.0.23 Mnenhy/0.7.5.0
I'm not sure if the bug was still present in that version as I was only
aware of it when accessing e-mail remotely, and I was avoiding doing
that because I was assuming the bug was still there.

I suggest we keep this bug closed because we don't know which of the bugs cited by wada to dup it to