Last Comment Bug 748807 - imap offline Messages not moved from "tmp" to "cur" with maildir
: imap offline Messages not moved from "tmp" to "cur" with maildir
Product: MailNews Core
Classification: Components
Component: Networking: IMAP (show other bugs)
: 14
: x86 Linux
-- normal (vote)
: Thunderbird 15.0
Assigned To: David :Bienvenu
Depends on:
  Show dependency treegraph
Reported: 2012-04-25 08:57 PDT by weliot
Modified: 2014-12-30 01:10 PST (History)
14 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---

imap.log (720.77 KB, text/plain)
2012-04-25 08:57 PDT, weliot
no flags Details
proposed fix (4.69 KB, patch)
2012-05-01 13:32 PDT, David :Bienvenu
no flags Details | Diff | Splinter Review
fix move/delete notification (5.54 KB, patch)
2012-05-08 15:25 PDT, David :Bienvenu
standard8: review+
Details | Diff | Splinter Review

Description User image weliot 2012-04-25 08:57:13 PDT
Created attachment 618299 [details]

User Agent: Mozilla/5.0 (X11; Linux i686; rv:14.0) Gecko/20120424 Firefox/14.0a1
Build ID: 20120424075151

Steps to reproduce:

In about:config set mail.serverDefaultStoreContractID = ";1" and then logged into new gmail account. 

Actual results:

In all cases only "cur" and "tmp" directories are created (for Inbox, Sent, All Mail, Trash etc); in no case is there a "new" directory, even though there are unread messages. Also, in each case the "tmp" directory contains all the message files except for one which is in the "cur" directory (this is a message with an attached file). In thunderbird itself, the messages appeared as expected.

Expected results:

Judging by what happens with kmail, I would have expected to find the unread messages in "new", all the read messages in "cur", and probably nothing in "tmp".
Comment 1 User image Hiroyuki Ikezoe (:hiro) 2012-04-25 19:52:50 PDT
From mailnews/local/src/nsMsgMaildirStore.cpp:

202 /**
203  *Create a Maildir-style folder with "tmp", " and "cur" subfolders
204  * but no "new" subfolder, because it's not sensical in the mail client context.
205  ("new" directory is for messages on the server that haven't been seen by a
206 *  mail client).
207  * aFolderName is already "safe" - it has been through NS_MsgHashIfNecessary
208  */
209 nsresult nsMsgMaildirStore::CreateMaildir(nsILocalFile *path)
Comment 2 User image David :Bienvenu 2012-04-25 20:59:26 PDT
we have no need for a "new" directory. It wouldn't ever have messages in it. "new" in the context of maildir doesn't mean unread.
Comment 3 User image weliot 2012-04-26 01:40:49 PDT
This is most curious. Having reported this matter in Bug 58308 I was invited by Mozilla to file a new bug here. I spent some time following the instructions as to how to create a imap.log file but it was all to no avail since now this bug has been closed as invalid.

I shall continue watching my messages accumulate in thunderbird tmp, more and more of them being redundant. Fortunately I have kmail, even thought it does have the wrong "new" directory.
Comment 4 User image weliot 2012-04-26 02:24:09 PDT
I have now closed and restarted thunderbird several times. It shows the following, and this is what my gmail account actually contains:

Inbox: 1 unread message
Sent: 6 messages one with attached file
All Mail: 6 Messages, one unread and one with attached file
Bin: 1 file

However, the thunderbird directories contain:

Inbox: cur - 1 message with attached file
       tmp - 14 messages

Sent: cur - 1 message with attached file
      tmp - 20 messages

All Mail: cur - 1 message with attached file
          tmp - 39 messages

Bin: tmp - 8 messages (there is no cur directory)
Comment 5 User image David :Bienvenu 2012-04-26 06:59:38 PDT
oh, sorry, the summary of the bug and the argument about the lack of a "new" directory made me think this bug is about the lack of a "new" directory, not the failure of messages to get cleared from "tmp" and moved to "cur".
Comment 6 User image weliot 2012-04-26 07:15:12 PDT
The new title is perhaps too restrictive. Not only do most messages (but not all) remain in tmp but they are repeatedly downloaded.
Comment 7 User image Jeff Grossman 2012-04-26 10:40:50 PDT
Yes, very true.  Messages not moving from tmp to cur is not the only problem.  I have seen the same messages continue to download into tmp with a new filename and new file date.
Comment 8 User image David :Bienvenu 2012-04-26 10:42:52 PDT
(In reply to Jeff Grossman from comment #7)
> Yes, very true.  Messages not moving from tmp to cur is not the only
> problem.  I have seen the same messages continue to download into tmp with a
> new filename and new file date.

The reason that's happening is because the messages aren't getting moved to cur, so we don't know we have the message, so we try to download it again.
Comment 9 User image Jeff Grossman 2012-04-26 11:08:20 PDT
(In reply to David :Bienvenu from comment #8)
> (In reply to Jeff Grossman from comment #7)
> > Yes, very true.  Messages not moving from tmp to cur is not the only
> > problem.  I have seen the same messages continue to download into tmp with a
> > new filename and new file date.
> The reason that's happening is because the messages aren't getting moved to
> cur, so we don't know we have the message, so we try to download it again.

Makes sense.  Thanks for the explanation.
Comment 10 User image Mike Kaganski 2012-04-28 20:30:17 PDT
This problem is present on Win7x64, too.
Comment 11 User image Mike Auty 2012-04-30 12:56:20 PDT
Hi there,

I just ran into the same problem on 12.0.1 on Linux AMD64 with three IMAP mailboxes, two extremely large (some with folders of 10000+ messages) and one relatively small.  I found that in combination with compact folders and offline mode I was using up disk space at a frightening rate.  Every time a compact was carried out all the messages for each folder were seemingly re-downloaded in their entirety.

A quick check showed that there was both a www.mailhost.tld and www.mailhost.tld.sbd directory, both of which appeared to contain messages (although I'd thought .sbd directories were just used to provide folder hierarchy?).  Although they did not take the same space on disk, on one account the difference was huge, and on another they appeared to practically be copies of each other.

I also found that in the cur directory of one folder with 10000+ messages I had 30000+ files (some of which were empty folders).  Doing a sorted md5 across the files seemed to reveal that they were all different, but I wasn't able to identify two files for the same message in the time where I could test this.  I've since reverted to using berkleystore for the time being, but this should be relatively easy to recreate if there are more tests required.

The servers in use were, a dovecot server and an unknown proprietary server.

If you'd like any further information please let me know.
Comment 12 User image David :Bienvenu 2012-05-01 13:32:06 PDT
Created attachment 620049 [details] [diff] [review]
proposed fix

the issue is that maildir wasn't always initializing the offset of a msg as 0, which meant the code that checked if the msg size was right thought the size was wrong, which led to us leaving the offline imap message in the tmp dir instead of moving it into the cur dir. The initializing of the offset fixes the main issue; the calling of DiscardNewMessage fixes the message getting left in the tmp dir. I also cleaned up some formatting.

This should fix some of the xpcshell maildir test failures; I'm still working through those.
Comment 13 User image David :Bienvenu 2012-05-08 15:25:58 PDT
Created attachment 622173 [details] [diff] [review]
fix move/delete notification

this adds a couple lines to fix the move/delete notification that nsMsgDBView relies on to know when move/delete is done. Other than that, it's the same as the previous patch.
Comment 14 User image Mark Banner (:standard8) 2012-05-09 06:28:39 PDT
Comment on attachment 622173 [details] [diff] [review]
fix move/delete notification

Review of attachment 622173 [details] [diff] [review]:

::: mailnews/local/src/nsLocalMailFolder.cpp
@@ +1558,5 @@
>        if (txnMgr)
>          txnMgr->DoTransaction(undoTxn);
>      }
> +    if (isMove)
> +        srcFolder->NotifyFolderEvent(NS_SUCCEEDED(rv) ? mDeleteOrMoveMsgCompletedAtom : mDeleteOrMoveMsgFailedAtom);

nit: only two space indent please.
Comment 15 User image David :Bienvenu 2012-05-09 07:52:57 PDT
fixed on trunk
Comment 16 User image David :Bienvenu 2012-05-09 11:31:13 PDT
Comment 17 User image Benjamin Dahl 2013-01-25 01:41:50 PST
Using Thunderbird 17.0.2 on Mac OS X 10.6 I don't think this bug is fixed. 
Looking into my ImapMail/ folder there have been 70.000 items stored and the folder itself has 16GB. 
I've got the feeling that Thunderbird does not reindex the /cur and /tmp folder when automatic filters are applied to this mailbox.

Comment 18 User image Mark Banner (:standard8) 2013-01-25 02:11:16 PST
Benjamin, please file a new bug for your issue. This fix has already been out for several releases, and it is possible that a) it wasn't fixed fully (as you've suggested), or b) you are actually seeing a different issue. In any case, it is generally best to file a new bug as that will more likely get actioned rather than a comment on a bug that is closed.
Comment 19 User image Bachsau 2014-06-01 01:45:17 PDT
Thunderbird still downloads same message more than once. Sad I still didn't find a better mail client to replace **** thunderbird. It's just a bunch of bugs.
Comment 20 User image Kent James (:rkent) 2014-12-30 01:10:57 PST
This fix was incomplete. See bug 856519 for followup.

Note You need to log in before you can comment on or make changes to this bug.