573729 - Race condition when two TB instances on different computers move messages based on filters - messages get duplicated.

Status: RESOLVED INCOMPLETE

(MailNews Core :: Filters, defect)

Description

[Jens Müller (:tessarakt)]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.4) Gecko/20100611 Firefox/3.5b4pre
Build Identifier: TB 3.1 RC 2 (Linux)

I have filter rules in place which move mailing list messages to a folder for each mailing list.

I have these filter rules set in two Thunderbird profiles on different computers. Sometimes I have both TB instances running at the same time.

I noticed that I had duplicate messages in the folder of the mailing list. The headers indicate that it is really the same message (and that it was not delivered twice by the mailing list).

I strongly suspect that there is a race condition between the two Thunderbird instances. The IMAP protocol does not have a MOVE command, so this seems like a reasonable explanation. I suppose both TBs get the notification from the IMAP server, issue a COPY command quasi-simultaneously, and then both issue a DELETE (of which only one succeeds).

Reproducible: Sometimes

Steps to Reproduce:
See above. I have not try to reproduce it explicitly, but it seems to happen for most (all?) mailing list messages that match a filter rule.

Comment 1

[WADA:World Anti-bad-Duping Agency]

> The IMAP protocol does not have a MOVE command, so this seems like a
reasonable explanation.
> I suppose both TBs get the notification from the IMAP server,
> issue a COPY command quasi-simultaneously,
> and then both issue a DELETE (of which only one succeeds).

I think your guess is right.
IMAP "Move" by message filter consists of next steps.
(a) fetch mail headers at an IMAP for newly arrived mail
(b) uid xxx copy "move target folder",
    if "Move by filter" or "Delete by filter with move to Trash model"
(c) uid xxx store flag \Deleted
(a)/(b)/(c) is executed serially at one PC, but order of (a-at-PC1)/(b-at-PC1)/(c-at-PC1)/(a-at-PC2)/(b-at-PC2)/(c-at-PC2) is unpredictable.

What is your "check for new messages every NNN minute" setting?
Do you set smallest value like NNN=1 for it?

Comment 2

[Jens Müller (:tessarakt)]

(In reply to comment #1)
> What is your "check for new messages every NNN minute" setting?
> Do you set smallest value like NNN=1 for it?

Every 10 minutes. But aren't IMAP connections kept open and the server pushes notofications about new mail?

Comment 3

[Jens Müller (:tessarakt)]

P.S.: One workaround would be to wait an _random_ amount of time before applying filter actions, and check for the message's existence again before applying the filter. OK, there might still be overlaps, but much less frequently.

Comment 4

[Jens Müller (:tessarakt)]

Is Mozilla Messaging actively involved in IMAP standardization? I cannot find a web page of a working group on IMAP, or something like that ...

Comment 5

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #2)
> Every 10 minutes. But aren't IMAP connections kept open and the server pushes
> notofications about new mail?

Yes, IMAP connection is kept and server pushes notification about new mail via IDLE.
Disable IDLE command use(Server Settings/Advanced), if you use "check every NN minutes". "Keep connection + IDLE" is for automatic mail checking of Inbox without "check new mails every NNN minutes" by IMAP client in order to reduce needless accesses to server even though no new mail.

Comment 6

[Jens Müller (:tessarakt)]

OK, I will try that.

Having filter rules on both computers which are not synced automatically is an awful hack anyway :-|

Comment 7

[Jens Müller (:tessarakt)]

OK, I guess this will at least make conflicts less likely. But is this really a solution? I don't think so ...

Comment 8

[Wayne Mery (:wsmwk)]

closing incomplete for lack of information. if you feel this change was made in error, please update the bug.