519979 - move/deletes of large number of imap messages slow without caching of input and output streams during the synchronous batch copy

Status: VERIFIED FIXED

(MailNews Core :: Backend, defect)

Description

[David :Bienvenu]

Attachment: cache the input and output streams during the synchronous batch copy

If I delete to trash 100 imap messages, with a folder configured for offline use, it's quite slow. Besides the issues raised in bug 508978, it turns out that we're opening and closing both the input and output streams for each message, instead of leaving them open, which causes a massive slowdown. In the case of deleting a lot of small messages at once (e.g., deleting bugzilla messages, or archiving a bunch of messages), the speedup is quite dramatic, roughly 2-4x.

Comment 1

[David :Bienvenu]

this may have broken test_imapUndo.js - I'm looking into why.

Comment 2

[David :Bienvenu]

Attachment: fix failing test

this fixes the failing test case by handling the case where there's no offline store for the source folder - it's not an error, so don't treat it as such.

Comment 3

[WADA:World Anti-bad-Duping Agency]

Is this bug applicable to "Copy" too? (no store flag \Deleted, no remove from msf)
Severe slowness of copy(IMAP to IMAP) I saw in Bug 519226 Commnet #2 is this bug?

Comment 4

[Wayne Mery (:wsmwk)]

is being offline required to reproduce? 
I couldn't reproduce while online (bug 506809)

Comment 5

[Wayne Mery (:wsmwk)]

oh, and is this a regression?

Comment 6

[David :Bienvenu]

It's true of copy as well - all that's required is move/copying imap folders that are stored offline, in imap folders configured for offline use. No, it's not a regression. Delete immediately is not affected since there's no move/copy involved.

Comment 7

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #6)
> It's true of copy as well - all that's required is move/copying imap folders
> that are stored offline, in imap folders configured for offline use.

When I tested "copy of small 4096 mails" and saw phenomenon of Bug 519226 Commnet #2, I disabled Gloda, and I disabled auto-sync ("Message Synchronization" option is unchecked, and all IMAP folders is set offline use = off).
Is this bug relevant to "slowness of copy of may mails", even when "copy from IMAP folder to IMAP folder" case with such setting(not offline-use=On)?
(Huge virtual memory is probably relates to four "uid copy" or "uid store flage" command for each 1000 mails when copy/move of 4096 mails.)

Comment 8

[David :Bienvenu]

this bug is strictly about the copying of the offline copies of messages between folders. If the messages are stored for offline use in the source folder, we'll copy them to the destination folder, and that copy is much slower than it needs to be.

Comment 9

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #8)
> this bug is strictly about the copying of the offline copies of messages between folders.

I see. I'll open separate bug for Bug 519226 Commnet #2, when phenomenon of my Bug 519226 Commnet #2 is found to be irrelevant to original problem of Bug 519226, or my Bug 519226 Commnet #2 is found to be independent problem from original problem of Bug 519226.

Comment 10

[David :Bienvenu]

fix checked in.

Comment 11

[Wayne Mery (:wsmwk)]

v.fixed