1213021 - IMAP from GMAIL INBOX keeps downloading every message, again and again

Status: RESOLVED DUPLICATE of bug 816327

(MailNews Core :: Backend, defect)

Description

[João Carlos Mendes Luís]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36

Steps to reproduce:

- Have a Corporate GMAIL account
- Configure TB for IMAP
  - Happens with maildir and 
- Have a BIG inbox (4500 msgs, 211M bytes)
- Configure TB to offline store INBOX
- Make sure [Gmail]/All is not subscribed (workaround for another bug)
- Run "Download/Sync now"
- Some time downloading almost all messages from INBOX
- Read some email, delete some email, etc...
- Run "Download/Sync now"
- Some time downloading almost all messages from INBOX AGAIN!!!!
- Browse messages
- Note that when you click in another message, sometimes TB will show "Downloading Message" in status

Notes:

- This happens with mailbox configured for maildirstore and berkeleystore.
- While TB says my INBOX has 210MB, maildir "cur" folder has a total of 12GB occupied!!!
- There are some old files (1 month old) in maildir "tmp" folder.
- I never saw this happening in my personal gmail account, in a dovecot imap server account, nor in any other mailbox other than INBOX.  But I do not keep ALL in sync, and INBOX is my larger IMAP box.




Actual results:

See above


Expected results:

Do not download every message again
Do not forget messages in maildir store

Comment 1

[João Carlos Mendes Luís]

May be related:

Bug 534297
Bug 492687
Bug 590922

And many others that suggest workarounds and about:config options, but none solved my problem.

Comment 2

[João Carlos Mendes Luís]

Not sure if should fill another bug, or just related to this one:

I noticed that deleting messages, SOMETIMES, DO NOT delete the respective file in local maildir storage.  That's why my local maildir was so much larger.

After sending the initial report, I deleted all local storage and some big emails.  Now my INBOX is reported as occupying  126M, but indeed it uses 517M of local disk space.

Comment 3

[João Carlos Mendes Luís]

Yesterday I could see this bug happening outside of google.

It happened in a private server, running dovecot IMAP, maildir storage on server side, maildir storage on local side.

The same symptoms: downloading entire INBOX again, large mailbox, maildir email files "lost" in local folder even after they had been deleted on server side.

I could also note another symptom: some nstmp-xx files appeared around the same moment the email started to download again.  The nstmp files appear to be like berkeley mailbox format, and are never deleted by thunderbird.

Also, it appears that these files are created whenever I try to compact the folder and get a message like "The folder 'Inbox' cannot be compacted because another operation is in progress. Please try again later."

Comment 4

[Wayne Mery (:wsmwk)]

João, does it stop if you unsubscribe the All Mail folder under [Gmail] ?

Comment 5

[Wayne Mery (:wsmwk)]

(In reply to João Carlos Mendes Luís from comment #3)
> ...
> I could also note another symptom: some nstmp-xx files appeared around the
> same moment the email started to download again.  The nstmp files appear to
> be like berkeley mailbox format, and are never deleted by thunderbird.

this can happen if you get a uid roll from imap server. Although I don't understand the "lost" aspect.

> Also, it appears that these files are created whenever I try to compact the
> folder and get a message like "The folder 'Inbox' cannot be compacted
> because another operation is in progress. Please try again later."

what happens if you disable auto compact at edit | prefs | advanced | network and disk ?

Comment 6

[João Carlos Mendes Luís]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #4)
> João, does it stop if you unsubscribe the All Mail folder under [Gmail] ?

Wayne,

1 - I do not subscribe to All Mail but my Inbox and Sent are big (inbox currently with 170MB, 4500+ messages)
2 - This also happens with non-gmail accounts: My personal dovecot imap server (currently 91MB, 3500+ messages).

Comment 7

[João Carlos Mendes Luís]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #5)
> (In reply to João Carlos Mendes Luís from comment #3)
> > ...
> > I could also note another symptom: some nstmp-xx files appeared around the
> > same moment the email started to download again.  The nstmp files appear to
> > be like berkeley mailbox format, and are never deleted by thunderbird.
> 
> this can happen if you get a uid roll from imap server.

Please explain this uid roll...


>  Although I don't
> understand the "lost" aspect.

lost: individual email files in the cur folder that are not mapped by thunderbird index.

For example, once my INBOX had a size of 150MB, as said by thunderbird.  But I had more than 1TB inside the cur directory.  Many files were simply ignored, lost from the internal index, and re-downloaded form the server.

> 
> > Also, it appears that these files are created whenever I try to compact the
> > folder and get a message like "The folder 'Inbox' cannot be compacted
> > because another operation is in progress. Please try again later."
> 
> what happens if you disable auto compact at edit | prefs | advanced |
> network and disk ?

Just did this.  If this helps I'll reply back...

Comment 8

[João Carlos Mendes Luís]

(In reply to João Carlos Mendes Luís from comment #7)
> (In reply to Wayne Mery (:wsmwk, NI for questions) from comment #5)
> > what happens if you disable auto compact at edit | prefs | advanced |
> > network and disk ?
> 
> Just did this.  If this helps I'll reply back...

Did not work...    :-(

Comment 9

[Phoenix]

Resolved per whiteboard

Comment 10

[Wayne Mery (:wsmwk)]

Curiosity question, do you set Thunderbird offline for a particular reason?

Comment 11

[Wayne Mery (:wsmwk)]

James does this reproduce for you?

Comment 12

[James Rome]

I never set Thunderbird offline. I never see this problem on my Mac.

Comment 13

[João Carlos Mendes Luís]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #10)
> Curiosity question, do you set Thunderbird offline for a particular reason?

Apart from uncontrolled network outages, no.

Comment 14

[João Carlos Mendes Luís]

(In reply to João Carlos Mendes Luís from comment #8)
> (In reply to João Carlos Mendes Luís from comment #7)
> > (In reply to Wayne Mery (:wsmwk, NI for questions) from comment #5)
> > > what happens if you disable auto compact at edit | prefs | advanced |
> > > network and disk ?
> > 
> > Just did this.  If this helps I'll reply back...
> 
> Did not work...    :-(

Indeed, did not fully resolved the problem, but apparently it has reduced a lot the number of occurrences.

Comment 15

[João Carlos Mendes Luís]

I got some new info.


Comparing individual "repeated" files for maildir mode, I found that the each message is being downloaded again, and its on-disk file is almost the same, except for the first line, which appear to be some kind of ref to the downloaded timestamp:

-From - Fri Jun 17 14:46:00 2016
+From - Mon Jun 20 11:55:39 2016

IMHO, shouldn't thunderbird save some kind of unique id for each message?

From a post in stackoverflow, I got referenced to this:

https://tools.ietf.org/html/rfc3501#section-2.3.1.1

It appears that determining IMAP uniqueness is difficult, and maybe this is the core problem!

One stackoverflow answer said: "A tuple of (folder name, folder UIDVALIDITY, message UID) distinguishes a message from all others in an IMAP account. For any given message, that tuple can change between sessions, so you might want to track its Message-ID as a backup in case this happens."

Unfortunately, not every message has a Message-ID header.


At least gmail has an extension to deal with it:


https://developers.google.com/gmail/imap_extensions#access_to_the_gmail_unique_message_id_x-gm-msgid



Hope this helps...

Comment 16

[João Carlos Mendes Luís]

BTW: Note that there are TWO bugs here:
- Not detecting a message uniqueness (probably also affects bsd mailbox storage)
- Leaving untracked and repeated files in maildir (probably only affects maildir storage)

Comment 17

[Wayne Mery (:wsmwk)]

(In reply to João Carlos Mendes Luís from comment #7)
> (In reply to Wayne Mery (:wsmwk, NI for questions) from comment #5)
> > (In reply to João Carlos Mendes Luís from comment #3)
> > > ...
> > > I could also note another symptom: some nstmp-xx files appeared around the
> > > same moment the email started to download again.  The nstmp files appear to
> > > be like berkeley mailbox format, and are never deleted by thunderbird.
> > 
> > this can happen if you get a uid roll from imap server.
> 
> Please explain this uid roll...

https://github.com/k9mail/k-9/issues/1074
https://www.limilabs.com/blog/unique-id-in-imap-protocol

Comment 18

[Wayne Mery (:wsmwk)]

(this isn't unique to linux)