487909 - Mails lost by Message Filter when target disk / IMAP server full

Status: RESOLVED INCOMPLETE

(MailNews Core :: Networking: IMAP, defect)

Description

[Mads Kiilerich]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3) Gecko/20090327 Fedora/3.1-0.11.beta3.fc11 Firefox/3.1b3
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3pre) Gecko/20090324 Fedora/3.0-2.1.beta2.fc11 Thunderbird/3.0b2

I use Message Filters to move messages from a remote IMAP server to a local IMAP server (dovecot).

My local system ran out of disk space, and obviuosly the filter couldn't deliver the mail. But apparently it deletes the mail from the source anyway? 



Reproducible: Didn't try

Steps to Reproduce:
Sorry. Hard and fatal to recreate the test scenario.
Actual Results:  
Comparing with the server log I can see that some mails has been lost. There is also some entries in filterlog.html from just before it ran completely full, listing some mails the local IMAP server hasn't accepted.

Expected Results:  
I would expect RFC-compliant "don't take responsibility for a mail before it has hit a disc or somebody else takes responsibility" behaviour.

Double delivery is OK, loosing messages is not.

The problem I see could also be caused by dovecot. I do however consider that less likely.

I lost data, so it could be Severity Critical. But apparently it only happens under rare conditions, so I don't expect it to be ranked that high.

thunderbird-3.0-2.1.beta2.fc11.i586
dovecot-1.2-0.rc2.1.fc11.i586

Comment 1

[Ludovic Hirlimann [:Usul]]

would be nice if you could recreate the scenario and log what imap is doing, when this occur. I understand how long this could take - but would really be nice to have imap logs (https://wiki.mozilla.org/MailNews:Logging).

Comment 2

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)

"Disk space shortage" occurred on Tb? Or move-target-IMAP-Server(a local IMAP server, dovecot, in your case)? Or both at same time during message filtering process of Tb?
If "Disk space shortage" for Tb is involved, do you use "offline-use" option for related mail folders? 

Message filter log doesn't write "status of successful move or failed move". It simply says that event of "move to a move-target folder" happened by message filter execution.
"Move to a folder" by message filter upon mail download roughly consists of;
  1. FETCH mail data from Inbox of move-source-IMAP-account
  2. Apply message filter, and condition for "move" hits
  3. Write filter log
  4. APPEND(upload) mail data to move-target-folder of move-target-IMAP-account
     on move-target-IMAP-Server
  5. OK to APPEND command from move-target-IMAP-Server
  6. At Inbox of move-source-IMAP-account, UID xxx store flag \Deleted

move-target-IMAP-Server(a local IMAP server, dovecot, in your case) returned ERR to APPEND at step 4 due to disk space shortage? Or Tb ignored ERR or BAD to APPEND and silently stopped APPEND process and issued "UID xxx store flag \Deleted" to move-source-IMAP-Server(remote IMAP server in your case)?
Or Tb continued "move by filter" process even though "disk space shortage" for TB during step 1/step 2/step3/step 5?
 
Please note that, if OK is returned at step 4, IMAP client can do nothing for your problem.

Anyway, Mads Kiilerich(bug opener), get IMAP log, and check IMAP level real flow first.

Comment 3

[Wayne Mery (:wsmwk)]

  Mads, questions for you in comment 2 and 1. please update the bug. Thanks

Comment 4

[Mads Kiilerich]

I am sorry; it isn't feasible for me to reproduce this. I had hoped to be able to at least investigate it a bit further, but that isn't realistic.

It was a local IMAP server using the same (full) disk as Tb.

"Select this folder for offline use" is checked for all involved folders, but I don't use offline (except for the local IMAP server) and don't know anything about it. I don't know if that setting is the default or if I somehow have changed something.

The algorithm you describe sounds fine, and I assume that you are sure it never continues to step 6 before step 5 has completed successfully. However, one explanation of what I saw is that it _did_ continue to step 6 after failure.

Sorry, I don't have any IMAP logs to share.

I know that this isn't very helpful, and I can understand if you either blame me or dovecot or cosmic radiation and closes this report as noise. But herhaps this report will show as a part of a pattern in the noise later on ...

Comment 5

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #4)
> But herhaps this report will show as a part of a pattern in the noise later on ...

Pattern of which do you think?
 a) Tb issues "uid store flag \Deleted" to move source folder
    even if NO/BAD for "append" to move target folder.
 b) IMAP server returns OK to "append" even though received data is not written
    in Data Base for mail data permanently, then data loss can occur
    by "disk full, power failure etc. after OK response".
I think issue like b) is not surprising with Linux on personal-use PC, if file system like JFS is not used.

If you use Cyrus, you can probably produce "ERR to append" consistently by "concecutive [LF]s". (See Bug 500479, Bug 402759, Bug 301010).
Can you test with Cyrus?

Comment 6

[Mads Kiilerich]

I don't know which pattern could show up. I am not an expert - you are ;-) But it could perhaps be something like the Cyrus pattern you mention. Perhaps there is a Dovecot pattern too. They are all fine projects, so I don't know where to look.

In my case there wasn't any power failure, so I see no reason to suspect linux or the file system.

Comment 7

[Wayne Mery (:wsmwk)]

I think without a protocol log, or test of specific known issues we don't have enough information to proceed so => incomplete.  But if others have better idea or Madshas has new information then please reopen the bug.

Comment 8

[Mads Kiilerich]

For the record: This issue might have been related to the use of the "nostalgy" add-on.