1054232 - On Thunderbird 31 downloading duplicates e-mail in pop account

Status: RESOLVED WORKSFORME

(MailNews Core :: Networking: POP, defect)

Description

[Audrey]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.78.2 (KHTML, like Gecko) Version/7.0.6 Safari/537.78.2

Steps to reproduce:

Tried Safe Mode.  Deleted the index.  Shut and restarted multiple times.  Looked on the webmail version to see if the e-mails were duplicated there (they were not).


Actual results:

Nothing changed.


Expected results:

Duplicate downloads should have stopped.

Comment 1

[Wayne Mery (:wsmwk)]

(this doesn't block development or release, so not blocker serverity)

What happens when you start thunderbird in safe mode?
https://support.mozilla.org/en-US/kb/safe-mode

Comment 2

[Audrey]

I think it is extremely serious: if this continues more than one or two days, I will have to stop using Thunderbird.
I had tried safe mode yesterday, and it made no difference.

(In reply to Wayne Mery (:wsmwk) from comment #1)
> (this doesn't block development or release, so not blocker serverity)
> 
> What happens when you start thunderbird in safe mode?
> https://support.mozilla.org/en-US/kb/safe-mode

Comment 3

[Wayne Mery (:wsmwk)]

no one else is reporting this as far as I know, so there's not enough information here to address the issue.

Is the inbox pop or imap?
What is size of inbox?  (right click on folder and pick properties)
What addons are listed in Help | Troubleshooting?

Comment 4

[Audrey]

All inboxes: POP3
Here is the information from the Troubleshooter:

  Application Basics

    Name: Thunderbird
    Version: 31.0
    User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.0
    Profile Folder: Show in Finder
    Application Build ID: 20140717165725
    Enabled Plugins: about:plugins
    Build Configuration: about:buildconfig
    Memory Use: about:memory

  Mail and News Accounts
    account1:
      INCOMING: account1, , (none) Local Folders, plain, passwordCleartext

    account2:
      INCOMING: account2, , (pop3) mail.samuelsagepartners.com:110, plain, passwordCleartext
      OUTGOING: smtp.samuelsagepartners.com:587, plain, passwordCleartext, true

    account3:
      INCOMING: account3, , (pop3) pop3.live.com:995, SSL, passwordCleartext
      OUTGOING: smtp.live.com:587, alwaysSTARTTLS, passwordCleartext, true

    account5:
      INCOMING: account5, , (pop3) pop3.live.com:995, SSL, passwordCleartext
      OUTGOING: smtp.live.com:587, alwaysSTARTTLS, passwordCleartext, true

    account6:
      INCOMING: account6, , (pop3) pop3.ortrans.org:110, plain, passwordCleartext
      OUTGOING: smtp.ortrans.org:465, SSL, passwordCleartext, true

    account7:
      INCOMING: account7, , (pop3) pop3.live.com:995, SSL, passwordCleartext
      OUTGOING: smtp.live.com:587, alwaysSTARTTLS, passwordCleartext, true

    account8:
      INCOMING: account8, , (pop3) pop.googlemail.com:995, SSL, passwordCleartext
      OUTGOING: smtp.googlemail.com:587, alwaysSTARTTLS, passwordCleartext, true

  Crash Reports

  Extensions
    Dictionnaire français «Classique», 4.3, true, [e-mail]
    Gather the Senders !, 1.2.2, true, [e-mail]
    LookOut, 1.2.13, true, [e-mail]
    Mail Merge, 3.7.0, true, [e-mail]
    Personal Level Indicators, 0.7, true, [e-mail]
    Remove Duplicate Messages, 0.1.13, true, {12345678-1234-1234-1234-123456789abc}
    Saved Password Editor, 2.7.1, true, [e-mail]
    Test Pilot for Thunderbird, 1.3.9, true, [e-mail]
    ThreadVis, 2.2.0, true, {A23E4120-431F-4753-AE53-5D028C42CFDC}
    Thunderbird Captive Portal detector, 0.1, true, jid1-CuEKsiBk2Kg10A@jetpack
    ThunderPlunger, 2.14, true, {bfec0750-aa6f-11db-abbd-0800200c9a66}

  Important Modified Preferences

    Name: Value

      accessibility.typeaheadfind.flashBar: 0
      browser.cache.disk.capacity: 358400
      browser.cache.disk.smart_size_cached_value: 358400
      browser.cache.disk.smart_size.first_run: false
      browser.cache.disk.smart_size.use_old_max: false
      extensions.lastAppVersion: 31.0
      font.internaluseonly.changed: false
      font.name.monospace.x-western: Comic Sans MS
      font.name.sans-serif.x-western: Arial
      font.name.serif.x-western: Arial
      mail.openMessageBehavior.version: 1
      mail.spotlight.enable: true
      mail.spotlight.firstRunDone: true
      mail.spotlight.global_reindex_time: 1273962071
      mailnews.database.global.datastore.id: 6805f92a-c5e0-d345-b414-de1c4df6988
      network.cookie.prefsMigrated: true
      places.database.lastMaintenance: 1388755159
      places.history.expiration.transient_current_max_pages: 100664
      places.history.expiration.transient_optimal_database_size: 75161926
      plugin.importedState: true
      plugin.state.scorch: 0
      print.macosx.pagesetup-2: PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPCFET0NUWVBFIHBsaXN0IFBVQkxJQyAiLS8vQXBwbGUvL0RURCBQTElTVCAxLjAvL0VO…
      privacy.donottrackheader.enabled: true

  Graphics

      Vendor ID: 0x10de
      Device ID: 0x 863
      WebGL Renderer: false
      GPU Accelerated Windows: 1/1 OpenGL

      AzureCanvasBackend: quartz
      AzureSkiaAccelerated: 0
      AzureFallbackCanvasBackend: none
      AzureContentBackend: quartz

  JavaScript

  Incremental GC: 1

  Accessibility

    Activated: 0
    Prevent Accessibility: 0

  Library Versions

      Expected minimum version
      Version in use

      NSPR
      4.10.6
      4.10.6

      NSS
      3.16.2 Basic ECC
      3.16.2 Basic ECC

      NSS Util
      3.16.2
      3.16.2

      NSS SSL
      3.16.2 Basic ECC
      3.16.2 Basic ECC

      NSS S/MIME
      3.16.2 Basic ECC
      3.16.2 Basic ECC

Comment 5

[Audrey]

As for the size of the inboxes :
1 - 357 MB
2 - 166 MB
3 - 14.6 MB
4 - 7.9 MB
5 - 14.6 MB
6 - 357 MB

The problem is the same regardless of the size of the inbox.

Comment 6

[Wayne Mery (:wsmwk)]

Are those numbers the "size on disk" from properties?
Is the duplicate only in one Inbox? 
And always the same message?

Comment 7

[Audrey]

Yes, they are from the "size on disk" in properties.

The duplicates are in all inboxes where I have received new mail since installing the new version of Thunderbird.

There is no message other than the information that I have just received new messages from the different senders.  Thunderbird doesn't see it as an error - it's as if it doesn't recognize that it has already downloaded the message.

Comment 8

[Audrey]

I'm on my webmail, and have verified that there is only one copy of the mails showing on the web servers.

Comment 9

[Wayne Mery (:wsmwk)]

Even though it still failed in safe mode, my first inclination is this is caused by an addon. So as a first step I recommend you delete (not disable) Remove Duplicate Messages, and thunderplunger, lookout. If it doesn't help you can add them back.

Comment 10

[Audrey]

OK - how do I delete them?

Comment 11

[Wayne Mery (:wsmwk)]

Tools, addons. Remove

Comment 12

[Audrey]

OK - done.  There are a number of add ons that are waiting for installation (they tell me to restart Thunderbird in order to install - I cannot remove these).  Undo doesn't work, either, for them - how can I stop them?

Comment 13

[Audrey]

Still the same problem with downloading messages.

Comment 14

[Wayne Mery (:wsmwk)]

Thanks for the info. The good news is not much has changed in the pop code since version 24. 

So we (you) take a couple stabs to find which versions still work (and from that we find the first version that fails). Note - this is not a cop out, but the best path toward getting a solution. Since you filed the bug and can reproduce you are our best hope.  Pretty quick - maybe 30 minutes of time.

So 
- backup your thunderbird profile directory
- please install https://ftp.mozilla.org/pub/mozilla.org/thunderbird/releases/30.0b1/mac/en-US/
- if 30.0b1 fails in the same way, please install https://ftp.mozilla.org/pub/mozilla.org/thunderbird/releases/27.0b1/mac/en-US/
- if 27.0b1 fails in the same way, please install version 24
- if 24.0.6 also fails (and presuming it previously worked for you), then the problem is either an addon, server or corrupted popstate.dat file

On that thought, what is the size of your popstate.dat file?

Comment 15

[Wayne Mery (:wsmwk)]

so I don't forget, the two pop checkins of the past year are bug 86233 and bug 760289

Comment 16

[Audrey]

Is popstate.dat a mac type file?
OK - many, many thanks for all of your help!!  I will try this and let you know

Comment 17

[Audrey]

popstate.dat = 680 kb

Comment 18

[Audrey]

Hi Wayne - with 24.6.0, the download duplication has stopped.  The versions 30.0b1 and 27.0b1 had the same problem as the TB 31.
Thank you again for all of your help - please let me know if you find out what the problem was.

Comment 19

[Audrey]

Hi Wayne - with 24.6.0, the download duplication has stopped.  The versions 30.0b1 and 27.0b1 had the same problem as the TB 31.
Thank you again for all of your help - please let me know if you find out what the problem was.

Comment 20

[Wayne Mery (:wsmwk)]

Do you have 'Leave Messages on Server' enabled in your pop account settings??

I'm betting you will find that  https://ftp.mozilla.org/pub/mozilla.org/thunderbird/releases/25.0b1/mac/en-US/ works and  https://ftp.mozilla.org/pub/mozilla.org/thunderbird/releases/26.0b1/mac/en-US/ fails.  If so, then the problem is likely bug 239455, which I discounted because it was designed to avoid the type of problem you are seeing.

Can you try those two beta versions for us?

Comment 21

[Audrey]

Hi Wayne - 
As you predicted, 25.0b1 is fine, while 26.0b1 shows the same problem as in 31.
All of the e-mail accounts have "leave mail on server" checked.
I hope this helps understand - and thank you again.

Comment 22

[Wayne Mery (:wsmwk)]

Thanks Audrey - that's great detective work!  I wish every bug filer would do as well.

I'm tempted to mark this tracking esr because I'm concerned we will get many reports after we unthrottle TB31 updates. But today I find other user reports of this problem, so perhaps Audrey is hitting an edge case.

Johannes (and others) any thoughts on why this is happening?

Comment 23

[Johannes Buchner]

Audrey, is there anything particular about your file-system setup? 
How much disk-space is left on the file-system where Thunderbird could attempt to write (in particular the profile folder)? 
Have you set up any soft-links that might confuse things (If you do not understand this question, the answer is no)?

Comment 24

[Audrey]

Hi Johannes -

"is there anything particular about your file-system setup?" 
Not that I'm aware of. The file system is on SSD.  Trim is enabled.  I have Mac OS/X 10.9.4.

"How much disk-space is left on the file-system where Thunderbird could attempt to write (in particular the profile folder)?"
54 GB available (out of 480).

"Have you set up any soft-links that might confuse things (If you do not understand this question, the answer is no)?"
Not that I'm aware of.

Please let me know if you need further information.

Kind regards,
Audrey

Comment 25

[Johannes Buchner]

Thunderbird for some reason forgets which emails have already been downloaded. So much is clear. The problem could be either that the emails look different to Thunderbird, or that it was not successful in storing its memory. For the latter, I can only say that the patch I contributed was due to Thunderbird not being able to write to a file (popstate.dat in particular), due to a full disk. It does so by creating a new temporary file and tries to move it over. Perhaps this fails for you. If you want to get your hands really dirty, you could use "strace thunderbird 2>&1|grep -i 'open|error' -5 > strace.log" or so to see if it is a file-opening issue related to writing to the profile. Also be sure to check the Javascript console. Otherwise I am out of ideas.

Are your email accounts all with the same server? Is it a popular one? Some POP servers give out new, random IDs for the emails every time you log in. However, this cause seems to be ruled out since it is software-version related.

Comment 26

[:aceman]

We could use a pop3 protocol log to diagnose whether it is a problem on the server (serving unique UIDs of the messages each time) or of TB (always requesting even unchanged UIDs).
Please see https://wiki.mozilla.org/MailNews:Logging, then 
set NSPR_LOG_MODULES=pop3:5
set NSPR_LOG_FILE=/path/to/file/name

Please only fetch messages from one server when you do the test and the you can remove private data (e.g. bodies of the messages) from the log file.

Comment 27

[Audrey]

@ Johannes -
The e-mail accounts are on hotmail / live, gmail and another (hosted by OVH).
I'm not very technical, so I'm not sure what to do exactly with "strace thunderbird 2>&1|grep -i 'open|error' -5 > strace.log", although I would be happy to try if that would help.  I'm on the 25.0b1 version of TB.

@:aceman: 
Is this the correct command to use in terminal?
"#!/bin/sh
export NSPR_LOG_MODULES=pop3:5
export NSPR_LOG_FILE=/Users/audreyjolivet-habiby/Documents/pop3.log
/Applications/Thunderbird.app/Contents/MacOS/thunderbird-bin &"

I know all of this is clear to you, but it's new to me - it's my first bug report. What does it mean in plain language? 

Thanks you both of you, and Wayne, for your help.

Kind regards,

Audrey

Comment 28

[:aceman]

(In reply to Audrey from comment #27)
> @:aceman: 
> Is this the correct command to use in terminal?
> "#!/bin/sh
> export NSPR_LOG_MODULES=pop3:5
> export NSPR_LOG_FILE=/Users/audreyjolivet-habiby/Documents/pop3.log
> /Applications/Thunderbird.app/Contents/MacOS/thunderbird-bin &"
> 
> I know all of this is clear to you, but it's new to me - it's my first bug
> report. What does it mean in plain language? 

I don't know OS X much but this looks correct for a shell script. If you type it directly into the terminal, you probably don't need the "#!/bin/sh" line.

When you produce the log, please disable any mail download at startup (in account settings). Then click Get mail for the chosen account. Fetch mail twice in a row. You say you should get all messages twice. We will then see what the log looks like for these two mail fetches.

Comment 29

[Stefan]

This is a long standing problem.  It has been around since long before v24, but I didn't report it since I couldn't reproduce it.  I can verify that it is not related to disk space problems on the Thunderbird profile disk because I have seen it many times and checked to make sure there was generous disk space available many times for POP accounts (all my testing has been on Windows machines).  Recently I saw Outlook report that our server was returning an incorrect message count (server query vs actual headers counted) for an IMAP connection.  I was wondering if there was a similar query for POP and whether a mismatch could trigger a duplicate message download.  I also noted that for the cases I observed personally, it was more likely to occur on a server inbox that had a lot of messages marked as deleted (i.e. not purged yet).  I could make the problem go away for a while if I could get the server to compact the mailbox (Compact of Inbox from an IMAP connection does not accomplish compact on the server).  The duplicate download is more likely to occur in Thunderbird 31 if I have both a POP profile/account and an IMAP profile/account for the same server account and I browse both in a Thunderbird session.  I also suspect that Thunderbird is not committing deletes consistently for POP accounts.  A month ago I reconfigured my email header based junk processing rules so that messages identified as junk would be COPIED to the junk folder and then the original deleted.  Either on exit or on startup, Thunderbird should delete the messages, but it isn't keeping up.  Our server stamps X-UIDL on messages in the inbox.mbx on the server (Imail), and each pair of duplicates has the same X-UIDL.  I just now turned off the account settings checkbox for Junk Settings\Move new junk messages and added a "filter after junk classification" rule to COPY msgs identified by Thunderbird as junk to the Junk folder and delete the original downloaded message.  This is all to help avoid the duplicate message download problem, or reduce it's severity.

Comment 30

[Stefan]

Also, the duplicate download seems more likely to occur if I leave Thunderbird running overnight and then browse messages using the IMAP connection.

Comment 31

[:aceman]

Accessing the same server via POP3 and IMAP in parallel does not seem safe to me.

Comment 32

[Johannes Buchner]

Sorry, I just saw this bug is still assigned to me. I do not know what to do about the bug report.
I can not see how it was created, and how my patch from another bug could have cause this as a regression. If there is any relation at all?
The lack of other users reporting this issue may indicate that it is unrelated -- unless my patch was dropped from of the builds?

Stefan seems to be another problem, although with the same consequences. Perhaps a new bug would help (with reference here). Also, please report one issue at a time -- ideally, reproduceable :) As I said in an earlier comment, the server may do funny things with the IDs it assigns to messages, which could confuse Thunderbird as well.

Sorry to be not more helpful.

Comment 33

[Wayne Mery (:wsmwk)]

To test if bug 239455's checkin is involved the builds to be tested are...
https://archive.mozilla.org/pub/thunderbird/nightly/2013/09/2013-09-05-04-59-33-comm-central/ (should work) and https://archive.mozilla.org/pub/thunderbird/nightly/2013/09/2013-09-11-03-03-41-comm-central/ (should fail).  (the other nightly builds adjacent to 2013-09-09 are unfortunately broken)

Comment 34

[Wayne Mery (:wsmwk)]

reporter Audrey wrote privately " The problem looks like it is finally solved with the latest version."