1779296 - Emails being duplicated multiple times when folders are compacting (POP3) - involving folder retention policy settings?

Status: RESOLVED WORKSFORME

(MailNews Core :: Database, defect, P2)

Description

[David Goadby]

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0

Steps to reproduce:

Initiate folder compaction although this problem occurs after automatic compaction.

Actual results:

After installing the update to 102.0.2 I noticed that all of the folders were being compacted. After it had finished I checked a folder and all of the emails had been duplicated once exactly. I ran "Remove duplicates" to get rid of the dups and then checked another folder. 1000's of duplicated emails were found. Usually there is one duplication but I have had as many as seven extra copies of the emails in each folder. It takes ages to clear them all. Very old emails are duplicated too so I know they are not being downloaded from my POP server as I only hold 20 days. Not all folders are duplicated with the problem being more common with large folders with more than 3,000 emails in them but some folders with just 10 emails had the problem too. Even folders with retention limits of, say 200, are duplicated. Duplicated emails can be read or unread.

I have always trusted Thunderbird but I am now taking daily snapshots to be sure.

Agent string: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.0.2

Expected results:

The folders should be compacted with no side effects on the email folders.

Comment 1

[Thomas D. (:thomas8)]

Thanks for the detailed report. I've linked some similar reports, pls have a look - any common features between those and yours?

• Accounts where duplicates occur, POP or IMAP?
• Using Norton AV?

Comment 2

[Magnus Melin [:mkmelin]]

Probably bug 1778037

Comment 3

[David Goadby]

Extra info that may help:

1. Just starting Thunderbird also causes the problem. Possibly compaction is initiated automagically?

2. I am only using POP3.

3. Sometimes the de-duplication widget says "no duplicates" when clearly there are.

4. Even emails in Drafts are being duplicated so the mailbox type is almost certainly not par of the problem.

5. Emails in folders that are no longer used (redundant projects etc) are also duplicated.

6. After a mail get emails are not always being filtered and they are left in the Inbox. If I go to Tools->Run Filters on Folder then the emails are correctly filtered so the filters are ok. All of my filters are set to "Filter When Getting Mail" so this is a new side-effect.

7. There is no evidence of any emails being lost (so far).

I hope that some of this helps.

Comment 4

[Thomas D. (:thomas8)]

(In reply to Thomas D. (:thomas8) from comment #1)

Thanks for the detailed report. I've linked some similar reports, pls have a look - any common features between those and yours?

• David, can you check if some bugs linked in See also field of this bug are similar to yours?

• Using Norton AV?

• Are you using Norton AV?
• Can you provide a list of your add-ons? Especially the "de-duplication widget" add-on
• Have you tried right-click on folder > Properties > Repair Folder?

Comment 5

[David Goadby]

I am using Avast.

The add-ons I am using are:

Compact headers
Mail Merge
Message Header Toolbar Customize
Remove Duplicate Messages
Signature Switch
Tag Popup

Other items to note....

Remove Duplicate Messages was installed after the problem first occurred.

I have not tried Repair Folder but will try to remember to use it.

Thunderbird release is 102.0.3 (up to date) build ID: 20220718182443. installed on July 23rd.

Since last startup no further duplicates have occurred. My next planned Thunderbird shutdown (PC backup time) is Friday. If I get more duplicates then I will try the Repair Folder option.

Comment 6

[Thomas D. (:thomas8)]

Thanks David for reporting back.

(In reply to David Goadby from comment #5)

I have not tried Repair Folder but will try to remember to use it.
Thunderbird release is 102.0.3 (up to date) build ID: 20220718182443. installed on July 23rd.
Since last startup no further duplicates have occurred. My next planned Thunderbird shutdown (PC backup time) is Friday. If I get more duplicates then I will try the Repair Folder option.

Now I'm curious to know, how did it go?

• More dupes?
• Have you tried Repair?

I am using Avast.

• Do you have the following setting enabled (checked)?
  `≡ > Settings > Privacy and Security > Antivirus > [ ] Allow antivirus clients to quarantine incoming messages?

The add-ons I am using are:

Can you check if the problem still occurs without active add-ons: ≡ > Help > Troubleshoot Mode…?

Comment 7

[David Goadby]

FYI I'm now on 102.1.1 and the problem still exists.

I have been checking for dups throughout the week and non were created.

I tend to stop Thunderbird once per week before taking system backups. I then reboot the system and restart Thunderbird.

The dup problem always seems to occur after program startup when, I assume, there is automatic compaction.

Last week I delayed all mailbox fetches (pop3) for 10 minutes after startup. This ensures that all startup processes are finished before adding emails. It was a long shot for sure! However, it did not change the problem.

I no longer delete the duplicates to the Deleted folder as there were over 16,000 deleted emails in there. I now delete them permanently. Again, this has not changed the problem.

The system has been up for about six hours and a quick check showed no duplicates. I requested compaction and 21Mb was saved but there were no duplicates found after the operation.

The next time I get some dups I will rebuild the folders before removing them in case there is a subtle structure problem.

The only other problem that I have is the Inbox sometimes has a few unfiltered emails in it. They are emails for which there are filters. When I run Tools->Run Filter on Folder it always clears them.

At least no emails are being lost!

Comment 8

[Wayne Mery (:wsmwk)]

David, does this still reproduce with 102.2.2 ?

Comment 9

[David Goadby]

I am still on 102.2.1 (upgrading as I type...) but can report the number and frequency of duplicates has dropped dramatically to very low numbers. On average I now get about 12 duplicates per week. Stopping and restarting Thunderbird doesn't seem to make any difference to the numbers either.

After the update to 102.2.2 I will continue to keep a check.

Comment 10

[Verner]

I du have the same error in vers. 102.3.3 abaut a week ago.
I was looking away from the screen for a short time, when I returned - 12000 new mails. Seemed to be all mails there was in all my folders
The new mail arrived in inbox, and there were now all the new mails as unread.
I use a lot af folders, and there were non new mails in these.
There were 2 to 4 copies of each mail, also deleted mail from deleted mail folder.

To day I tried to delete mail from inbox. I deleted 2 big bloks of mails from the beginning of inbox.
The result was the first block was deleted .
The second block just marked the as read.
(it seemed that the program did not ended the proces - it interupted with a msg asking me to wait until proces was finished)
and an analysis after restart shoved that only one duplicate of each mail existed now, but only until 29/9 2022 ! from there there was the 2-4 copies of mail.
The error has not occured again.

Comment 11

[Wayne Mery (:wsmwk)]

(In reply to David Goadby from comment #9)

I am still on 102.2.1 (upgrading as I type...) but can report the number and frequency of duplicates has dropped dramatically to very low numbers. On average I now get about 12 duplicates per week. Stopping and restarting Thunderbird doesn't seem to make any difference to the numbers either.

After the update to 102.2.2 I will continue to keep a check.

But it still happens occassionally?

Comment 12

[David Goadby]

Yes. Usually about 20 duplicates a week. But,I just noticed something strange; it is usually the same two folders.

I have repaired both of the folders to see if that makes a difference.

Comment 13

[Wayne Mery (:wsmwk)]

(In reply to David Goadby from comment #12)

...
I have repaired both of the folders to see if that makes a difference.

Good results?

Comment 14

[David Goadby]

No difference.

In fact I had 102 duplicates today including the usual two folders and four more. There is no easy pattern to this so far.

Comment 15

[Wayne Mery (:wsmwk)]

What is the possibility of updating Windows? Your Windows 7 is almost two years out of support.

Comment 16

[kidservice]

I am on Windows 11 and TB 102.11.2 and I also get some duplicated emails in my inbox (main folder).
I don't know if this is somehow correlated to folder compaction, but I noticed that the duplicated emails show up when I manually run the "Download emails" button and TB is already downloading emails in the background.
I have TB set to download emails every 2 minutes (POP3).
If I'm really looking for an email, and don't want to wait 2 mins, I hit the "Download emails" button. If TB is already downloading emails in the background, it pops double emails.
Maybe compacting slows down the thread that downloads emails and a new thread kicks in in the meantime.
If a second downloading thread can spawn while another one hasn't finished yet (and so it has not yet updated the list of downloaded emails), it is very likely that 2 copy of the same email will get downloaded

Comment 17

[Wayne Mery (:wsmwk)]

Does this still reproduce without antivirus software?

Comment 18

[Wayne Mery (:wsmwk)]

(I just closed bug 1778037 per reporter)

Comment 19

[Wayne Mery (:wsmwk)]

In addition, compact has been rewritten.

Can anyone still reproduce this issue, with or without antivirus?

Comment 20

[Wayne Mery (:wsmwk)]

Going to take silence as an indication the problem is gone.