238365 - popstate.dat can't handle duplicate messages (with the same UID) with respect to the deleting messages on server

Status: REOPENED

(MailNews Core :: Database, defect)

Description

[:aceman]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7b) Gecko/20040305 Firefox/0.8.0+
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7b) Gecko/20040305 Firefox/0.8.0+

If I have 2 duplicate messages in a folder and want to delete one of them, while
keeping the other one in the mailbox AND on the server, it is not possible,
because it is marked for deletion on the server.

Reproducible: Always
Steps to Reproduce:
1.I had a folder with messages. I have keep on server enabled. And working offline.
2.My popstate.dat got corrupted so I restored it from backup.
3.It was not 100% synchronized with the folder.
4.I checked new mail - some of the messages not in the popstate.dat file got
downloaded for the second time.
5.The folder now contained duplicates of messages.
6.I anticipated Moz has this bug with popstate.dat so I backed it up now - the
new messages are marked 'k'.
7.I deleted the duplicates from the folder - they got marked 'd'.
8.I closed Moz and restored the popstate.dat - the unique messages are now in
the folder, on the server and marked 'k'. This is the case I want to achieve in
this bug report.

Actual Results:  
Messages were marked for deletion on the server, even though they still remained
in the folder (only ONE duplicate was deleted from there).

Expected Results:  
Moz should check whether there aren't any duplicates of the message to be
deleted.  It could then determine if it really should delete from the server.
But this may not be possible because the selection of 'keep on server' may not
be helpful. So it can simply ask the user.

I know this may be a hack but something like this must be implemented until the
whole problem with duplicates (as in bug 9413) is fixed cleaner.

And it shouldn't be a big headache now, beacause the other storage formats can
handle duplicates cleanly. I compacted all folders after deleting the dupes and
the correct messages remained. Mbox and .msf files can handle dupes. Just
popstate.dat is not ready.

This can lead to a dataloss (deleting messages from server) if the user is not
aware of this problem.

Comment 1

[WADA:World Anti-bad-Duping Agency]

UIDL should be unique, I believe.
Isn't this a bug of mail server?

http://getaclue.org/yoduh/outlook.html reported very old hung problem of Outlook
family due to duplicate UIDL by (q)popper (probably qpopper 2.53).
What mail transfer agent(MTA,mail server program) is used by your mail server?

Comment 2

[:aceman]

(In reply to comment #1)
> UIDL should be unique, I believe.
> Isn't this a bug of mail server?

Yeah, could you read the report again? The server was correct (at least this
time). Mozilla shouldn't have downloaded the messages twice in the first place.
But that would be much harder to do, because it then couldn't trust popstate.dat
and recreate the list of already downloaded messages at each start. I don't try
to fix this problem, even though it would be much more correct.

I accepted the fact that the messages are duplicated in the inbox and now try to
find a working way how to handle them. Actually only the case when they are
deleted but should be kept on the server.

Comment 3

[Gervase Markham [:gerv]]

This is an automated message, with ID "auto-resolve01".

This bug has had no comments for a long time. Statistically, we have found that
bug reports that have not been confirmed by a second user after three months are
highly unlikely to be the source of a fix to the code.

While your input is very important to us, our resources are limited and so we
are asking for your help in focussing our efforts. If you can still reproduce
this problem in the latest version of the product (see below for how to obtain a
copy) or, for feature requests, if it's not present in the latest version and
you still believe we should implement it, please visit the URL of this bug
(given at the top of this mail) and add a comment to that effect, giving more
reproduction information if you have it.

If it is not a problem any longer, you need take no action. If this bug is not
changed in any way in the next two weeks, it will be automatically resolved.
Thank you for your help in this matter.

The latest beta releases can be obtained from:
Firefox:     http://www.mozilla.org/projects/firefox/
Thunderbird: http://www.mozilla.org/products/thunderbird/releases/1.5beta1.html
Seamonkey:   http://www.mozilla.org/projects/seamonkey/

Comment 4

[Gervase Markham [:gerv]]

This bug has been automatically resolved after a period of inactivity (see above
comment). If anyone thinks this is incorrect, they should feel free to reopen it.

Comment 5

[:aceman]

At leat this extension can help:
https://addons.mozilla.org/en-US/thunderbird/addon/remove-duplicate-messages/

I'll see what happens with this problem in Thunderbird 7, now that bug 9413 has done something.

Comment 6

[:aceman]

The problem still happens in TB7, reopening.
TB happily downloads messages it already has in the mbox (with same UID). If one of them is deleted from the inbox, it is deleted from the server too.

Comment 7

[Wayne Mery (:wsmwk)]

(In reply to :aceman from comment #6)

TB happily downloads messages it already has in the mbox (with same UID). If
one of them is deleted from the inbox, it is deleted from the server too.

Given bug 9413, Bug 618809, and Bug 1580060, is there anything reasonable that can be done here?

Comment 8

[gene smith]

Apparently there is dup prevention/detection described here: bug 9413 comment 37. However, it defaults to disabled probably due to issues (similar to CONDSTORE maybe). Specifically, mail.server.default.dup_action (default 0).
I don't know anything about popstate so can't comment on that.
FWIW, there's a well received addon by Eyal R. to handle dupes: https://addons.thunderbird.net/en-US/thunderbird/addon/removedupes/?src=search. I've used it and it works well.

Comment 9

[Ping Chen [:rnons] (away)]

Instead of relying on popstate.dat, I think we should use msg folder as source of truth. There seems no interface yet to check if a uidl exists in a folder.