218075 - Compact of IMAP folder with many deleted message slow with IMAP delete model (Mark as Deleted) because of expunge notification from the server for every deletion

Status: RESOLVED WORKSFORME

(MailNews Core :: Networking: IMAP, defect)

Description

[David :Bienvenu]

As reported by Neil, if you're using the imap delete model, and have an imap
folder with lots of deleted messages, when you compact the folder, the command
takes a long time to finish (in the reported case, for 4000 messages, it took 10
minutes).  This is always going to be somewhat slow because we're going to get
4000 expunge notifications from the server, but we can speed up the UI part of
this by batching updates to the outliner.

The code in question is in nsImapMailFolder::UpdateImapMailboxInfo:
    if (keysToDelete.GetSize())
    {
      PRUint32 total;
      
      // It would be nice to notify RDF or whoever of a mass delete here.
      if (mDatabase) 
      {
        mDatabase->DeleteMessages(&keysToDelete, nsnull);
        total = keysToDelete.GetSize();
      }
    }

Unfortunately, this code knows nothing about the view - we would either need to
write some hacky code to get the view and do batching, or add some sort of batch
notification between the db and the view (or whatever listeners were open on the
db). The latter is much more general.

Comment 1

[mahmoud.1]

This is my first time submitting a bug... not sure if this is the right place. 
I've noticed "Compact Folders" is very slow on win2k.  My INBOX is about 8 Mb. 
I just deleted 1 message and compacted the folder.  It took about 3 minutes.  My
incoming server is imap and I have "When I delete a message" set to "Mark it as
deleted".  I've noticed this on multiple machines with different configurations.
 Also, I've talked to other people with the same issue.  I've noticed this since
Thunderbird ~0.9.  I am now using 1.0.  I do not get this problem with Linux.

Comment 2

[Wayne Mery (:wsmwk)]

does the general theory also apply to Bug 296453?

Comment 3

[David :Bienvenu]

maybe - I think for most operations we do batch the updates - but maybe not for the imap delete model.

Comment 4

[Wayne Mery (:wsmwk)]

comment #2
> does the general theory also apply to Bug 296453?
(In reply to comment #3)
> maybe - I think for most operations we do batch the updates - but maybe not for
> the imap delete model.

bienvenu, do we still need someone to investigate this? 

(bug 296453 is dealing with front end)

Comment 5

[Peter Lairo]

Thunderbird[1] "freezes" for (seemingly endless) seconds when I manually compact my primary account's IMAP inbox - even if there is only one marked-as-deleted message there. The whole UI is frozen until the compact is done. Oddly, this doesn't occur on the other IMAP sub-folders nor (IIRC) on my other IMAP inboxes.

BTW: Ever since Tb went to a different storage format, there are all kinds of circumstances (seemingly all message interactions) where Tb becomes unresponsive to to messages indexing/hard-drive-thrashing/etc.

[1] Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.4pre) Gecko/20090829 Lightning/1.0pre Shredder/3.0b4pre

Comment 6

[David :Bienvenu]

Tb hasn't gone to a different storage format.

But Gloda might be listening for imap compaction events...

Comment 8

[Wayne Mery (:wsmwk)]

bienvenu, given other recent batching successes, is there something we can easily do here?

Comment 10

[gene smith]

I've never noticed a problem with compact (right-click) or even auto-compact on multiple accounts and folders take a long time.

Comment 11

[Wayne Mery (:wsmwk)]

Is bug 536447 instructive?

Comment 12

[Wayne Mery (:wsmwk)]

Nevermind, I see you already commented there.

If it's not reproducible and no one is currently reporting this, then perhaps we close them?

Comment 13

[gene smith]

(In reply to Wayne Mery (:wsmwk) from comment #12)

Nevermind, I see you already commented there.

If it's not reproducible and no one is currently reporting this, then perhaps we close them?

I agree, OK to close them.