354272 - Need feature to delete & prevent duplicate messages in folders

Status: NEW

(MailNews Core :: Backend, enhancement)

Description

[Nelson Bolyard (seldom reads bugmail)]

I'm running a trunk build from Gecko/20060824

I use the mailnews feature where mail messages are kept on my POP server
for 7 days.  Consequently my POP server always has the last week of 
incoming email on it.  This is normally fine because my mail server has 
a very large inbox size limit.

Today, something happened, and my mail client downloaded the ENTIRE 
content of my POP server's mail box as if they were all new messages.
I suddenly got about 500 email messages received.  My mail filters
dutifully files them into all my mail folders, again.  Now my folders
are stuffed with hundreds of duplicated messages.  I even have 3 copies
of some messages.  

I don't know why this happened, and whether it was a fault of my ISP
or a fault of my mailnews client.  I suspect the ISP because they 
sent me an email saying they were going to be making some changes to 
my account.  But I really don't know.  But the subject of this bug
is how to deal all the duplicate messages I have now that the damage
is done.

These messages are EXACT dupliates of what they were when they were
downloaded the first time.  Same message IDs, same sets of Received-From
headers, same bodies, same everything.  

I want a feature that will go through the DB and find duplicates, 
(based on their message IDs) and delete all but one of them.   
I don't care if it saves the oldest copy or the newest.  
It would be OK if it did that one folder at a time, and I had to go
through all my folders and do it once per folder.

Even better would be a feature that detects when a dupliate message is
about to be put into a folder, and simply discards the new dupliate,
never putting it into the folder in the first place.

Comment 1

[David :Bienvenu]

Nelson, there's an excellent extension that does this: 

https://addons.mozilla.org/thunderbird/956/

I'm not sure the extension will work with the trunk, but you could always run it with a 1.5 build to clean up the dups.

Yes, it would be nice to have this part of the core code.

It might have been your server, if they reassigned uidls, but it could also be that you lost your connection in the middle of downloading, which triggers a painful bug where we redownload all the messages from the pop3 server.

Comment 2

[Nelson Bolyard (seldom reads bugmail)]

David, Thanks for the suggestion.
I'm running SM trunk, so I've got two strikes against me, WRT that extension.
What's the bug number for that bug?

Comment 3

[David :Bienvenu]

there are several dupes - here's one: https://bugzilla.mozilla.org/show_bug.cgi?id=240049

there's a better one out there somewhere, I think, with some analysis by ch.ey.  Maybe I'll have a crack at fixing the most egregious instance of this bug, where you lose your connection in the middle of downloading...

Comment 4

[Nelson Bolyard (seldom reads bugmail)]

Extending this RFE a little, I also sometimes receive dupes because a mail 
gets sent to several different mailing lists to which I subscribe.  I get a 
copy of the email for each list.  

What I really want is a feature that, as it is downloading a new message from
the server, looks to see if I already have the message (based on the message ID)
and simply and silently drops the new message if I already have it in any of
the folders for this POP email account.  (would be nice for IMAP, too).

Such a feature would also effectively solve the problem of occasional 
re-downloads of the entire inbox from the server (for people who leave mail
on the server).  The downloads might happen, but they wouldn't result in 
tons of duplicates in the mail folders, so the only issue would be wasted
bandwidth.

Comment 5

[David :Bienvenu]

Nelson, this feature exists, though in a more limited fashion. See bug 9413. It only looks at recently downloaded headers, not all headers, however, so it won't deal with the dup's due to losing the pop3 uidl state. It does help with mailing list dups, however.

Comment 7

[Nelson Bolyard (seldom reads bugmail)]

In Bug 450545, David Bienvenu explained the limits of the feature he had 
previously described in comment 5 above.  He wrote:

> it only eliminates dups received in the same session, for things like 
> where you have messages that get delivered to you and a mailing list.

That is far from what this RFE requested.

Comment 8

[WADA:World Anti-bad-Duping Agency]

FYI.
Gmail has capability of duplicated mail check.
  When I uploaded mails in local mail folder via Gmail IMAP multiple times,
    - Upload to single Gmail IMAP folder(single Gmail Label) several times
    - Upload to several Gmail IMAP folders(Gmail Labels)
  Gmail/Gmail IMAP generated only single instance in "[Gmail]/All Mail".
  (Gmail IMAP returned OK to second append, but never generated mail data.)
  (For Tb, mail by second append is deleted/expunged by other client.     )
So you can use Gmail as "duplicate mail remover" by;
  Upload all local mails to single Gmail IMAP folder(=Gmail Label)
although a free Gmail account has 6GB limit of mail data currently.
Note: Gmail has also "Mail Fetcher"(periodical download from user's POP3 server).

Comment 10

[Lionel Seydoux]

Hi,
Dups happen too when switching to Thunder from other msg clients and/or 
when restoring several backups at once not to lose anything.

They come in two forms:  duplicate messages and duplicate folders eg:
Local folders
    AAA
    BBB
        msdg1
        msg2
        msg2 (dub)
        ....
    ....
    Received from OutSee
        AAA
        BBB
        ...
The feature should address both of these dups as far as possible as the folder structure may have changed

Il should work:
- when receiving messages (fairly easy as the dates are not very differnet)
  this shouls include Transmitted msg (different header) as the same info can
  come from different sources especially if you work as a network.
- on demand

It does not answer all managing problems, especially for lazy msg managers, but would help a lot

Bye

Comment 11

[Wayne Mery (:wsmwk)]

is consolidation/linking possible between this and bug 449521 (which has a patch) and bug 513667 ?

Comment 12

[Parkhideh]

(In reply to Wayne Mery (:wsmwk) from comment #11)
> is consolidation/linking possible between this and bug 449521 (which has a
> patch) and bug 513667 ?

Wayne, I have seen the duplicates when dialup fails in the middle of e-mail downloads.  From this standpoint the three bug reports are similar.

Comment 13

[M Lopez-Ibanez]

The add-on: https://addons.thunderbird.net/en-US/thunderbird/addon/remove-duplicate-messages-alte/

will stop working for recent thunderbird. And even for TB 60, it is extremely slow and probably broken.