748807 - imap offline Messages not moved from "tmp" to "cur" with maildir

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[weliot]

Attachment: imap.log

User Agent: Mozilla/5.0 (X11; Linux i686; rv:14.0) Gecko/20120424 Firefox/14.0a1
Build ID: 20120424075151

Steps to reproduce:

In about:config set mail.serverDefaultStoreContractID = "@mozilla.org/msgstore/maildirstore;1" and then logged into new gmail account. 


Actual results:

In all cases only "cur" and "tmp" directories are created (for Inbox, Sent, All Mail, Trash etc); in no case is there a "new" directory, even though there are unread messages. Also, in each case the "tmp" directory contains all the message files except for one which is in the "cur" directory (this is a message with an attached file). In thunderbird itself, the messages appeared as expected.


Expected results:

Judging by what happens with kmail, I would have expected to find the unread messages in "new", all the read messages in "cur", and probably nothing in "tmp".

Comment 1

[Hiroyuki Ikezoe (:hiro)]

From mailnews/local/src/nsMsgMaildirStore.cpp:

202 /**
203  *Create a Maildir-style folder with "tmp", " and "cur" subfolders
204  * but no "new" subfolder, because it's not sensical in the mail client context.
205  ("new" directory is for messages on the server that haven't been seen by a
206 *  mail client).
207  * aFolderName is already "safe" - it has been through NS_MsgHashIfNecessary
208  */
209 nsresult nsMsgMaildirStore::CreateMaildir(nsILocalFile *path)

Comment 2

[David :Bienvenu]

we have no need for a "new" directory. It wouldn't ever have messages in it. "new" in the context of maildir doesn't mean unread.

Comment 3

[weliot]

This is most curious. Having reported this matter in Bug 58308 I was invited by Mozilla to file a new bug here. I spent some time following the instructions as to how to create a imap.log file but it was all to no avail since now this bug has been closed as invalid.

I shall continue watching my messages accumulate in thunderbird tmp, more and more of them being redundant. Fortunately I have kmail, even thought it does have the wrong "new" directory.

Comment 4

[weliot]

I have now closed and restarted thunderbird several times. It shows the following, and this is what my gmail account actually contains:

Inbox: 1 unread message
Sent: 6 messages one with attached file
All Mail: 6 Messages, one unread and one with attached file
Bin: 1 file

However, the thunderbird directories contain:

Inbox: cur - 1 message with attached file
       tmp - 14 messages

Sent: cur - 1 message with attached file
      tmp - 20 messages

All Mail: cur - 1 message with attached file
          tmp - 39 messages

Bin: tmp - 8 messages (there is no cur directory)

Comment 5

[David :Bienvenu]

oh, sorry, the summary of the bug and the argument about the lack of a "new" directory made me think this bug is about the lack of a "new" directory, not the failure of messages to get cleared from "tmp" and moved to "cur".

Comment 6

[weliot]

The new title is perhaps too restrictive. Not only do most messages (but not all) remain in tmp but they are repeatedly downloaded.

Comment 7

[Jeff Grossman]

Yes, very true.  Messages not moving from tmp to cur is not the only problem.  I have seen the same messages continue to download into tmp with a new filename and new file date.

Comment 8

[David :Bienvenu]

(In reply to Jeff Grossman from comment #7)
> Yes, very true.  Messages not moving from tmp to cur is not the only
> problem.  I have seen the same messages continue to download into tmp with a
> new filename and new file date.

The reason that's happening is because the messages aren't getting moved to cur, so we don't know we have the message, so we try to download it again.

Comment 9

[Jeff Grossman]

(In reply to David :Bienvenu from comment #8)
> (In reply to Jeff Grossman from comment #7)
> > Yes, very true.  Messages not moving from tmp to cur is not the only
> > problem.  I have seen the same messages continue to download into tmp with a
> > new filename and new file date.
> 
> The reason that's happening is because the messages aren't getting moved to
> cur, so we don't know we have the message, so we try to download it again.

Makes sense.  Thanks for the explanation.

Comment 10

[Mike Kaganski]

This problem is present on Win7x64, too.

Comment 11

[Mike Auty]

Hi there,

I just ran into the same problem on 12.0.1 on Linux AMD64 with three IMAP mailboxes, two extremely large (some with folders of 10000+ messages) and one relatively small.  I found that in combination with compact folders and offline mode I was using up disk space at a frightening rate.  Every time a compact was carried out all the messages for each folder were seemingly re-downloaded in their entirety.

A quick check showed that there was both a www.mailhost.tld and www.mailhost.tld.sbd directory, both of which appeared to contain messages (although I'd thought .sbd directories were just used to provide folder hierarchy?).  Although they did not take the same space on disk, on one account the difference was huge, and on another they appeared to practically be copies of each other.

I also found that in the cur directory of one folder with 10000+ messages I had 30000+ files (some of which were empty folders).  Doing a sorted md5 across the files seemed to reveal that they were all different, but I wasn't able to identify two files for the same message in the time where I could test this.  I've since reverted to using berkleystore for the time being, but this should be relatively easy to recreate if there are more tests required.

The servers in use were fastmail.fm, a dovecot server and an unknown proprietary server.

If you'd like any further information please let me know.

Comment 12

[David :Bienvenu]

Attachment: proposed fix

the issue is that maildir wasn't always initializing the offset of a msg as 0, which meant the code that checked if the msg size was right thought the size was wrong, which led to us leaving the offline imap message in the tmp dir instead of moving it into the cur dir. The initializing of the offset fixes the main issue; the calling of DiscardNewMessage fixes the message getting left in the tmp dir. I also cleaned up some formatting.

This should fix some of the xpcshell maildir test failures; I'm still working through those.

Comment 13

[David :Bienvenu]

Attachment: fix move/delete notification

this adds a couple lines to fix the move/delete notification that nsMsgDBView relies on to know when move/delete is done. Other than that, it's the same as the previous patch.

Comment 14

[Mark Banner (:standard8)]

Comment on attachment 622173 [details] [diff] [review]
fix move/delete notification

Review of attachment 622173 [details] [diff] [review]:
-----------------------------------------------------------------

::: mailnews/local/src/nsLocalMailFolder.cpp
@@ +1558,5 @@
>        if (txnMgr)
>          txnMgr->DoTransaction(undoTxn);
>      }
> +    if (isMove)
> +        srcFolder->NotifyFolderEvent(NS_SUCCEEDED(rv) ? mDeleteOrMoveMsgCompletedAtom : mDeleteOrMoveMsgFailedAtom);

nit: only two space indent please.

Comment 15

[David :Bienvenu]

fixed on trunk

Comment 16

[David :Bienvenu]

http://hg.mozilla.org/comm-central/rev/89bc983ef93c

Comment 17

[Benjamin Dahl]

Using Thunderbird 17.0.2 on Mac OS X 10.6 I don't think this bug is fixed. 
Looking into my ImapMail/imap.myprovider.com/INBOX/tmp folder there have been 70.000 items stored and the folder itself has 16GB. 
I've got the feeling that Thunderbird does not reindex the /cur and /tmp folder when automatic filters are applied to this mailbox.

b

Comment 18

[Mark Banner (:standard8)]

Benjamin, please file a new bug for your issue. This fix has already been out for several releases, and it is possible that a) it wasn't fixed fully (as you've suggested), or b) you are actually seeing a different issue. In any case, it is generally best to file a new bug as that will more likely get actioned rather than a comment on a bug that is closed.

Comment 19

[Bachsau]

Thunderbird still downloads same message more than once. Sad I still didn't find a better mail client to replace **** thunderbird. It's just a bunch of bugs.

Comment 20

[Kent James (:rkent)]

This fix was incomplete. See bug 856519 for followup.