The default bug view has changed. See this FAQ.

imap offline Messages not moved from "tmp" to "cur" with maildir

RESOLVED FIXED in Thunderbird 15.0

Status

MailNews Core
Networking: IMAP
RESOLVED FIXED
5 years ago
2 years ago

People

(Reporter: weliot, Assigned: Bienvenu)

Tracking

Thunderbird 15.0
x86
Linux

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments, 1 obsolete attachment)

(Reporter)

Description

5 years ago
Created attachment 618299 [details]
imap.log

User Agent: Mozilla/5.0 (X11; Linux i686; rv:14.0) Gecko/20120424 Firefox/14.0a1
Build ID: 20120424075151

Steps to reproduce:

In about:config set mail.serverDefaultStoreContractID = "@mozilla.org/msgstore/maildirstore;1" and then logged into new gmail account. 


Actual results:

In all cases only "cur" and "tmp" directories are created (for Inbox, Sent, All Mail, Trash etc); in no case is there a "new" directory, even though there are unread messages. Also, in each case the "tmp" directory contains all the message files except for one which is in the "cur" directory (this is a message with an attached file). In thunderbird itself, the messages appeared as expected.


Expected results:

Judging by what happens with kmail, I would have expected to find the unread messages in "new", all the read messages in "cur", and probably nothing in "tmp".
Component: General → Networking: IMAP
Product: Thunderbird → MailNews Core
QA Contact: general → networking.imap
From mailnews/local/src/nsMsgMaildirStore.cpp:

202 /**
203  *Create a Maildir-style folder with "tmp", " and "cur" subfolders
204  * but no "new" subfolder, because it's not sensical in the mail client context.
205  ("new" directory is for messages on the server that haven't been seen by a
206 *  mail client).
207  * aFolderName is already "safe" - it has been through NS_MsgHashIfNecessary
208  */
209 nsresult nsMsgMaildirStore::CreateMaildir(nsILocalFile *path)
Status: UNCONFIRMED → NEW
Ever confirmed: true
(Assignee)

Comment 2

5 years ago
we have no need for a "new" directory. It wouldn't ever have messages in it. "new" in the context of maildir doesn't mean unread.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → INVALID
(Reporter)

Comment 3

5 years ago
This is most curious. Having reported this matter in Bug 58308 I was invited by Mozilla to file a new bug here. I spent some time following the instructions as to how to create a imap.log file but it was all to no avail since now this bug has been closed as invalid.

I shall continue watching my messages accumulate in thunderbird tmp, more and more of them being redundant. Fortunately I have kmail, even thought it does have the wrong "new" directory.
(Reporter)

Comment 4

5 years ago
I have now closed and restarted thunderbird several times. It shows the following, and this is what my gmail account actually contains:

Inbox: 1 unread message
Sent: 6 messages one with attached file
All Mail: 6 Messages, one unread and one with attached file
Bin: 1 file

However, the thunderbird directories contain:

Inbox: cur - 1 message with attached file
       tmp - 14 messages

Sent: cur - 1 message with attached file
      tmp - 20 messages

All Mail: cur - 1 message with attached file
          tmp - 39 messages

Bin: tmp - 8 messages (there is no cur directory)
(Reporter)

Updated

5 years ago
Status: RESOLVED → REOPENED
Resolution: INVALID → ---
(Assignee)

Comment 5

5 years ago
oh, sorry, the summary of the bug and the argument about the lack of a "new" directory made me think this bug is about the lack of a "new" directory, not the failure of messages to get cleared from "tmp" and moved to "cur".
(Assignee)

Updated

5 years ago
Summary: Directory Problems for Maildir → imap offline Messages not moved from "tmp" to "cur" with maildir
(Reporter)

Comment 6

5 years ago
The new title is perhaps too restrictive. Not only do most messages (but not all) remain in tmp but they are repeatedly downloaded.

Comment 7

5 years ago
Yes, very true.  Messages not moving from tmp to cur is not the only problem.  I have seen the same messages continue to download into tmp with a new filename and new file date.
(Assignee)

Comment 8

5 years ago
(In reply to Jeff Grossman from comment #7)
> Yes, very true.  Messages not moving from tmp to cur is not the only
> problem.  I have seen the same messages continue to download into tmp with a
> new filename and new file date.

The reason that's happening is because the messages aren't getting moved to cur, so we don't know we have the message, so we try to download it again.

Comment 9

5 years ago
(In reply to David :Bienvenu from comment #8)
> (In reply to Jeff Grossman from comment #7)
> > Yes, very true.  Messages not moving from tmp to cur is not the only
> > problem.  I have seen the same messages continue to download into tmp with a
> > new filename and new file date.
> 
> The reason that's happening is because the messages aren't getting moved to
> cur, so we don't know we have the message, so we try to download it again.

Makes sense.  Thanks for the explanation.

Comment 10

5 years ago
This problem is present on Win7x64, too.

Comment 11

5 years ago
Hi there,

I just ran into the same problem on 12.0.1 on Linux AMD64 with three IMAP mailboxes, two extremely large (some with folders of 10000+ messages) and one relatively small.  I found that in combination with compact folders and offline mode I was using up disk space at a frightening rate.  Every time a compact was carried out all the messages for each folder were seemingly re-downloaded in their entirety.

A quick check showed that there was both a www.mailhost.tld and www.mailhost.tld.sbd directory, both of which appeared to contain messages (although I'd thought .sbd directories were just used to provide folder hierarchy?).  Although they did not take the same space on disk, on one account the difference was huge, and on another they appeared to practically be copies of each other.

I also found that in the cur directory of one folder with 10000+ messages I had 30000+ files (some of which were empty folders).  Doing a sorted md5 across the files seemed to reveal that they were all different, but I wasn't able to identify two files for the same message in the time where I could test this.  I've since reverted to using berkleystore for the time being, but this should be relatively easy to recreate if there are more tests required.

The servers in use were fastmail.fm, a dovecot server and an unknown proprietary server.

If you'd like any further information please let me know.
(Assignee)

Comment 12

5 years ago
Created attachment 620049 [details] [diff] [review]
proposed fix

the issue is that maildir wasn't always initializing the offset of a msg as 0, which meant the code that checked if the msg size was right thought the size was wrong, which led to us leaving the offline imap message in the tmp dir instead of moving it into the cur dir. The initializing of the offset fixes the main issue; the calling of DiscardNewMessage fixes the message getting left in the tmp dir. I also cleaned up some formatting.

This should fix some of the xpcshell maildir test failures; I'm still working through those.
Assignee: nobody → dbienvenu
Attachment #620049 - Flags: review?(mbanner)
(Assignee)

Comment 13

5 years ago
Created attachment 622173 [details] [diff] [review]
fix move/delete notification

this adds a couple lines to fix the move/delete notification that nsMsgDBView relies on to know when move/delete is done. Other than that, it's the same as the previous patch.
Attachment #620049 - Attachment is obsolete: true
Attachment #620049 - Flags: review?(mbanner)
Attachment #622173 - Flags: review?(mbanner)
Comment on attachment 622173 [details] [diff] [review]
fix move/delete notification

Review of attachment 622173 [details] [diff] [review]:
-----------------------------------------------------------------

::: mailnews/local/src/nsLocalMailFolder.cpp
@@ +1558,5 @@
>        if (txnMgr)
>          txnMgr->DoTransaction(undoTxn);
>      }
> +    if (isMove)
> +        srcFolder->NotifyFolderEvent(NS_SUCCEEDED(rv) ? mDeleteOrMoveMsgCompletedAtom : mDeleteOrMoveMsgFailedAtom);

nit: only two space indent please.
Attachment #622173 - Flags: review?(mbanner) → review+
(Assignee)

Comment 15

5 years ago
fixed on trunk
Status: REOPENED → RESOLVED
Last Resolved: 5 years ago5 years ago
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 15.0
(Assignee)

Comment 16

5 years ago
http://hg.mozilla.org/comm-central/rev/89bc983ef93c

Comment 17

4 years ago
Using Thunderbird 17.0.2 on Mac OS X 10.6 I don't think this bug is fixed. 
Looking into my ImapMail/imap.myprovider.com/INBOX/tmp folder there have been 70.000 items stored and the folder itself has 16GB. 
I've got the feeling that Thunderbird does not reindex the /cur and /tmp folder when automatic filters are applied to this mailbox.

b
Benjamin, please file a new bug for your issue. This fix has already been out for several releases, and it is possible that a) it wasn't fixed fully (as you've suggested), or b) you are actually seeing a different issue. In any case, it is generally best to file a new bug as that will more likely get actioned rather than a comment on a bug that is closed.

Comment 19

3 years ago
Thunderbird still downloads same message more than once. Sad I still didn't find a better mail client to replace **** thunderbird. It's just a bunch of bugs.

Comment 20

2 years ago
This fix was incomplete. See bug 856519 for followup.
You need to log in before you can comment on or make changes to this bug.