Closed Bug 1733966 Opened 3 years ago Closed 3 years ago

Unnecessary mail synchronization, with TB91. (Then side issue with 94 beta)

Categories

(MailNews Core :: Networking: IMAP, defect)

Product:
MailNews Core
Component:
Networking: IMAP
Version:
Thunderbird 91
Type:
defect

Tracking

(thunderbird_esr78 unaffected, thunderbird_esr91 affected)

Status:
RESOLVED WORKSFORME
Tracking Flags:
Tracking Status
thunderbird_esr78 --- unaffected
thunderbird_esr91 --- affected

People

(Reporter: evekitas, Unassigned)

References

(Regression)

Details

(Keywords: perf, regression, Whiteboard: [beta case only: regressed by 1717147, fixed by bug 1734843])

Attachments

(5 files, 1 obsolete file)

downloading_msg_while_sync_disabled.png
11.05 KB, image/png
Details
sync.png
237.29 KB, image/png
Details
imap.log.zip
4.82 MB, application/octet-stream
Details
is-that-even-possible.png
3.40 KB, image/png
Details
Bug 1733966-UnecessaryEmailSync-2.pdf
78.36 KB, application/pdf
Details

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36

Steps to reproduce:

Nothing.

Actual results:

Thunderbird tried to synchronize all my email in all accounts (many thousands of emails). Google started to throttle synchronization, leading to massive slowdown of Thunderbird, rendering it unusable.

Expected results:

By default, Thunderbird should not synchronize folders other than Inbox and Sent Mail, and folders created manually. In particular, synchronizing "synonym" folders such as Important and All Mail results in significant overuse of bandwidth.

Status: RESOLVED → REOPENED
Component: General → Untriaged
Ever confirmed: true
Product: Invalid Bugs → Thunderbird
Resolution: INVALID → ---

What version of Thunderbird, IMAP or POP3 account?

Flags: needinfo?(evekitas)

This is IMAP, Thunderbird 91.1.2 (64-bit), Gmail account.
But the problem is probably universal? It should never try to synchronize folders like All Mail, which are full of links to mail that is actually located in the Inbox or somewhere else.

I think that we should only set the synchronize flag by default on a few folders, but I'd assume it's set on all folders on all versions right now.

Flags: needinfo?(evekitas)

Start Windows' safe mode with networking enabled

Does problem go away?

Status: REOPENED → UNCONFIRMED
Component: Untriaged → General
Ever confirmed: false
Flags: needinfo?(evekitas)
Version: unspecified → Thunderbird 91

No, that didn't work. The solution is to turn off synchronization, as noted.

Flags: needinfo?(evekitas)

Does the re-syncronisation occurs by itself after a certain period of time or upon a specific event, for example when you click on a folder while browsing, upon TB startup, etc...

As a tempoary workaround to minimise bandwith usage in Account Settings > Synchronisation you can choose which folders to synchronise or not... you can also right click on a Folder (e.g All Mail), go to Properties > Synchronisation and untick the box. Thunderbird may need to be restarted for change to take effect.

On another Folder affected but with smaller number of emails you can try Right click Folder > Properties > Repair that shall delete msf file and re-download headers + msg first time (expected) then upon completion of process order emails the way you want. Then restart TB and check if that sort the issue for that folder and emails appears in the same order you wanted.

Other user story:
https://thunderbird.topicbox.com/groups/beta/Tbcb8ae01b8d4f347

@Wayne,

After setting up a Gmail account in new profile in 91.1.2 (64-bit) on Windows 10 Pro (x64 bits) and disabling synchronisation on all folders but Inbox + TB restart, it appears that TB is not respecting user wishes.

Indeed when selecting the Spam folder, TB started to download all the messages (as per status bar progress msg) in it while sync is disabled for that folder... see attached... and below spam.msf file content...

Could that be the reason TB keep re-downloading messages as per this bug report?

// <!-- <mdb:mork:z v="1.4"/> -->
< <(a=c)> // (f=iso-8859-1)
  (80=ns:msg:db:row:scope:msgs:all)(81=subject)(82=sender)(83=message-id)
  (84=references)(85=recipients)(86=date)(87=size)(88=flags)(89=priority)
  (8A=label)(8B=statusOfset)(8C=numLines)(8D=ccList)(8E=bccList)
  (8F=msgThreadId)(90=threadId)(91=threadFlags)(92=threadNewestMsgDate)
  (93=children)(94=unreadChildren)(95=threadSubject)(96=msgCharSet)
  (97=ns:msg:db:table:kind:msgs)(98=ns:msg:db:table:kind:thread)
  (99=ns:msg:db:table:kind:allthreads)
  (9A=ns:msg:db:row:scope:threads:all)(9B=threadParent)(9C=threadRoot)
  (9D=msgOffset)(9E=offlineMsgSize)
  (9F=ns:msg:db:row:scope:dbfolderinfo:all)
  (A0=ns:msg:db:table:kind:dbfolderinfo)(A1=numMsgs)(A2=numNewMsgs)
  (A3=folderSize)(A4=expungedBytes)(A5=folderDate)(A6=highWaterKey)
  (A7=mailboxName)(A8=UIDValidity)(A9=totPendingMsgs)
  (AA=unreadPendingMsgs)(AB=expiredMark)(AC=version)(AD=forceReparse)
  (AE=fixedBadRefThreading)(AF=onlineName)(B0=folderName)>
{1:^80 {(k^97:c)(s=9)} }
{FFFFFFFD:^9A {(k^99:c)(s=9)} }

<(80=1)(81=0)(82=Spam)(83=8000004)>
{1:^9F {(k^A0:c)(s=9u)} 
  [1(^AC=1)(^AD=0)(^AE=1)(^AF^82)(^88^83)]}

@$${1{@
[1:^9F(^A7^82)]
@$$}1}@

@$${2{@
<(84=8082004)>[1:^9F(^88^84)]
@$$}2}@

@$${3{@
<(85=8082014)>[1:^9F(^88^85)]
@$$}3}@

@$${4{@
@$$}4}@

@$${5{@
@$$}5}@

@$${6{@
@$$}6}@

@$${7{@
@$$}7}@

@$${9{@
<(86=82014)>[1:^9F(^88^86)]
@$$}9}@

@$${A{@
< <(a=c)> // (f=iso-8859-1)
  (B8=useServerRetention)(B1=retainBy)(B2=daysToKeepHdrs)
  (B3=numHdrsToKeep)(B4=daysToKeepBodies)(B5=useServerDefaults)
  (B6=cleanupBodies)(B7=applyToFlaggedMessages)>
[1:^9F(^B8=1)]
@$$}A}@

@$${B{@
@$$}B}@

It will download all headers, the synchronization is about the message bodies.

When downloading headers only, usually TB status shows "Downloading headers X of X" which is the first step... then if synchronisation enabled says in a second step "Downloading msg X of Y" but that is not what happened during test related to my previous comment. TB direclty went to download msg and dis-regarding settings. Putting TB offline showed msg were available offline while they shouldn't if sync is disabled on Folder, only available in online mode.

I don't now if related to this bug, but fyi someone also reported a strange increase of IMAP cache filesize in TB 94.0b1 reaching limit...
User story: https://thunderbird.topicbox.com/groups/beta/Tbcb8ae01b8d4f347-M79d98af8bf92d4b4b19114f7/93-0b5-32-bit-keeps-redownloading-emails-each-time-software-is-reopened

Not sure what's happening here and in the "topicbox" reports, but a possibility for why all messages in a folder get re-downloaded might be because of a UIDVALIDITY problem. It will cause cause a complete re-download of a folder when it occurs. There is now a new log facility that will catch the problem: IMAP_CS. So a log with IMAP:5,IMAP_CS:5 could be recorded.
I think at one point we got at least one report of Exchange servers reporting a bad UIDVALIDITY and causing bulk re-download, but my o365 test account on exchange seems to be behaving.

Also, FWIW, I never see full message bodies being downloaded for folders configured to not have offline store.

A while back there was a bug that caused a message with a bad header to get re-downloaded each time tb is started and caused the mbox file to grow. But that was fixed. But it didn't cause all the message in the folder to be re-downloaded.

Blocks: tb91found

According to user stories, repair of folder does not fix the issue.
The current workaround that works is to turn off message synchronisation (disable download for offline use).

(In reply to gene smith from comment #11)
So a log with IMAP:5,IMAP_CS:5 could be recorded.

Would that log any sensitive information?

Flags: needinfo?(gds)

(In reply to gene smith from comment #11)

Not sure what's happening here and in the "topicbox" reports, but a possibility for why all messages in a folder get re-downloaded might be because of a UIDVALIDITY problem. It will cause cause a complete re-download of a folder when it occurs. There is now a new log facility that will catch the problem: IMAP_CS. So a log with IMAP:5,IMAP_CS:5 could be recorded.

If anyone want to produce a IMAP:5,IMAP_CS:5 log as per Gene suggestion, here are instructions that may be helpful :-)
https://wiki.mozilla.org/MailNews:Logging

I tried myself but it seems the issue is not occurring at every restart of TB in my case... so a bit hard to catch/log... from my end...

After some primary testing, first thing spotted...

When I created my IMAP google account with default settings (sync message enabled), I got in my TB profile an ImapMail\imap.gmail.com\INBOX file that populated with all downloaded messages and in parallel another ImapMail\imap.gmail.com\INBOX.msf file. No folders appeared apart from Inbox. I waited full download of all the message before exiting TB... took quite a while as I have 1GB+ data in the Inbox.

After restart of TB, "Looking for folders" is triggered and all folders are populated now populated but I ended up with:

  • ImapMail\imap.gmail.com\INBOX - no longer present - It has been deleted
  • ImapMail\imap.gmail.com\INBOX-1 - newly created empty and re-populated with data as TB re-download headers and messages
  • ImapMail\imap.gmail.com\INBOX.msf - still present but non longer updated
  • ImapMail\imap.gmail.com\INBOX-1.msf - newly created

Is that a normal behaviour in the first place? Seems odd...
I notice other folder where also named with -1 suffix like Drafts, etc...

At successive restart of TB the ImapMail\imap.gmail.com\INBOX-1.msf keep being emptied while ImapMail\imap.gmail.com\INBOX-1 keep being added data as TB continue to download messages... the ImapMail\imap.gmail.com\INBOX-1.msf file being filled in with data only upon TB exit.

I don't know if that can give some starting clues... I will try to get an IMAP log... ones I figure how to properly reproduce the issue...

I've observed this behavior for the 1st time beginning with 94.0b1. Like in comment 7, I can sometimes observe it when I select the Spam folder. Most of the time I see it when a new message arrives and clicking between the Inbox /Sent Mail / Spam folders will trigger it. It eventually goes away.

(In reply to Richard Leger from comment #13)

According to user stories, repair of folder does not fix the issue.
The current workaround that works is to turn off message synchronisation (disable download for offline use).

You say that, but my strong impression (from looking for a solution) is that this is a common problem, and this workaround isn't well-known. I really think it drives a lot of people off of Thunderbird.

I stopped using TB for a while, but I couldn't find anything better, so I went back for yet another round of looking for workarounds and finally figured out what to do -- but not before I moved a couple hundred thousand emails onto my local drive, only to find that they are no longer searchable, and so might as well have been deleted :(

(In reply to Ness Blackbird from comment #18)

(In reply to Richard Leger from comment #13)

...current workaround...

This was not meant to be a fix but a quick way for people to keep going while issue is being investigated and fixed... especially if you have a 40GB mailbox that keep re-downloading at each TB startup!

You say that, but my strong impression (from looking for a solution) is that this is a common problem,

I don't believe it is a common problem to my knowledge only been raised recently (re)raised with TB 93.0b5 (a beta version is not a finished product) as per https://thunderbird.topicbox.com/groups/beta that said it does not mean issue has not appeared/happened in older version... at other occasions...

Any help welcome. I myself try to contribute as an end user toward a solution to this big problem... and help others...

First steps would be to know more about your environment and setup, and to find a way/steps for anyone to reproduce the issue...

Which version of TB you encounter the issue with?
Are you able to reproduce it at will?
If yes how?
Would you be able to provide an IMAP logs as per Comment 15 to help identify what may cause the issue? Assuming you use IMAP account...
Are you encountering the issue with Google account or else?

and this workaround isn't well-known.

It is better this way as this is not a fix, though can still help those in the pickle that may need it :-)
My comment was meant to help those...

(In reply to Ness Blackbird from comment #18)

(In reply to Richard Leger from comment #13)

According to user stories, repair of folder does not fix the issue.
The current workaround that works is to turn off message synchronisation (disable download for offline use).

You say that, but my strong impression (from looking for a solution) is that this is a common problem, and this workaround isn't well-known. I really think it drives a lot of people off of Thunderbird.

I stopped using TB for a while, but I couldn't find anything better, so I went back for yet another round of looking for workarounds and finally figured out what to do -- but not before I moved a couple hundred thousand emails onto my local drive, only to find that they are no longer searchable, and so might as well have been deleted :(

Ness,

A user and Moz volunteer here. I don't consider it a common problem as I, using TB 94 beta, only have just begun seeing it beginning with that version. I never saw it when I was on 91 but it isn't to say it isn't happening or hasn't been introduced in a recent 91.x version as a bug or coding mistake. I myself have 20-25+ years of emails totaling 110GB (that came from back in the Eudora days of the 90s). If you have many hundreds of thousands of emails as I myself do, re-indexing that all over again after coming back from another client, I would surmise, would take a long, long time.

If you didn't know, TB has seen a resuscitation after having been on life support with effectively zero development since mid-2012. Google "thunderbird life support", there's a CNET article about it. It's only just been in the past couple years that >active< development re-started again and some new blood (in the form of a couple new developer hires, if my memory serves me correctly) was injected into TB. There is a Mount Everest's worth of catching up to do from 2012 to present. Also, There was also a HUGE amount of layoffs that happen at Mozilla in 2020 which further slowed TB development.

For some, the development or bug fixing pace may seem glacial. To a large degree, there is a historical reason for that. As a user, I've seen a lot of break/fix happening in the past couple years. But I've seen a HUGE improvement in TB performance as well. For a while, TB felt like a car that was left in the driveway under a tarp and finally got started 10 years later. Whatever code was left to rot between 2012 and present is getting fixed/updated. Slowly. It'll take a lot of massaging. I understand that fully. I have ZERO expectations of any immediacy of bug fixing for a product for which I pay $0.

I am not here to apologize for things that get broken along the way but I am thankful to Mozilla for keeping TB going and being motivated to update it after so many years of neglect. I report bugs as I find them too because I know I'm not the only one it's happening to. As you rightly state "I couldn't find anything better", is, rightly, true. Couldn't agree more.

Yay for TB developers -- paid and volunteer!

Another user feedback:
https://thunderbird.topicbox.com/groups/beta/Te8c2ccf26e15a3ee-Me68976b2148ddf411a77f253/94-0beta1-and-on-thunderbird-repeats-downloading-emails-starting-again-every-2-minutes

There may be different symptoms depending on TB version and user stories:

  • 91esr - possibly no re-download issue but sync settings on folders not always respected by TB - TB download message for folders with sync disabled!
  • 93b - possibly since b5 - Re-downloaded at TB startup
  • 94b - possibly since b1-2 - Re-downloaded during TB usage - either by browsing folders or after some elapsed time - may cause search/indexing issues

I see this with Yahoo Mail also. Very annoying. Sometimes, just closing and reopening Thunderbird will fix it, but not always.

(In reply to Richard Leger from comment #14)

(In reply to gene smith from comment #11)
So a log with IMAP:5,IMAP_CS:5 could be recorded.

Would that log any sensitive information?

No more than you get with just IMAP:5. Most users edit or change anything deemed "sensitive" before attaching the log file.

Could this be windows specific? I've yet to duplicate this on linux after much trying.

Another temporary workaround to try first might be to disable auto-sync rather than offline store on offending folders. Auto sync is set with the Config Editor parameter: mail.server.default.autosync_offline_stores
Set it to false and restart TB.

If that doesn't help, set it back to true and go ahead and disable offending folder's offline store.

You seem to be able to go back to an earlier version (i.e., 91 or even 78) after running with latest beta if you run tb with --allow-downgrade parameter. Tb uses either panacea.dat or folderCache.json depending on if older version or newer version of TB is running. At least it works for me and I see no obvious problems with both files present at the profile top level while running a release or latest beta from the same profile.
Re: bug 418551

Flags: needinfo?(gds)
Attached image sync.png —

same problem here. IMAP mail synced on the left. Then app restarted and the same emails are downloaded again - on the right.
It's redownloaded every few minutes.

(In reply to Richard Leger from comment #9)

When downloading headers only, usually TB status shows "Downloading headers X of X" which is the first step... then if synchronisation enabled says in a second step "Downloading msg X of Y" but that is not what happened during test related to my previous comment. TB direclty went to download msg and dis-regarding settings. Putting TB offline showed msg were available offline while they shouldn't if sync is disabled on Folder, only available in online mode.

This may be a bit off of the main topic but when setting up a new account you have to go into "manual mode" and accept the warning. Then immediately go to Synchronization and Folders and select which folders you want to have or not have offline store. Then click Inbox and headers and appropriate bodies will be downloaded and sub-folders will be listed. If you click Inbox before going into Synch and Folders or don't do manual setup, the default setting will apply and you may get some folders with full bodies downloaded that you don't really want before you change the settings in Sync and Folders.

(In reply to freeman3 from comment #26)

Created attachment 9245981 [details]
sync.png

same problem here. IMAP mail synced on the left. Then app restarted and the same emails are downloaded again - on the right.
It's redownloaded every few minutes.

Thanks for the report. Can you provide a bit more detail? Like which version of TB are you using? Do you have offline storage enabled for the re-downloading folder, imap server type, check for new messages poll interval setting, etc.

Also, an IMAP:5,IMAP_CS:5 log would be helpful. See https://wiki.mozilla.org/MailNews:Logging

See Also: → 1734843

Can someone please add "Perf" key word? Thank you.

Keywords: perf

freeman3, The attachment shows 8 accounts. Are they all on different imap servers or the same server? That shouldn't really matter but who knows.
Do you have other accounts that don't download messages on startup and every few minutes?
Did this start happening after a tb version change?

See Also: → 418551

Gene,

(In reply to gene smith from comment #28)

Also, an IMAP:5,IMAP_CS:5 log would be helpful. See https://wiki.mozilla.org/MailNews:Logging

Someone may be able to get you some logs, would you be able to assist him...
https://thunderbird.topicbox.com/groups/beta/Te8c2ccf26e15a3ee-Mdc43f1c50916c33671605e9e/94-0beta1-and-on-thunderbird-repeats-downloading-emails-starting-again-every-2-minutes

Regards,

(In reply to Richard Leger from comment #22)

  • 93b - possibly since b5 - Re-downloaded at TB startup

I have not been able to re-produce this issue to gather the logs. I have seen it at some point, but could not re-produce it at will...
Make me wonder if may be caused by upgrade from previous version of TB like 78 or 91 esr... In my testing I did removed and re-added gmail accounts at some point so could that have fixed the issue somehow?

  • 94b - possibly since b1-2 - Re-downloaded during TB usage - either by browsing folders or after some elapsed time - may cause search/indexing issues

I have now upgraded to 94.0b2 (64-bit) and email did not re-downloaded at startup. But I noticed that:

  • When sending an email to myself (Gmail account - IMAP/SMTP - on Win 10), it took quite a long time first time for the email to be sent and the following to appear in the console:
XHRPOSThttps://www.googleapis.com/oauth2/v3/token
[HTTP/2 200 OK 60ms]
XHRPOSThttps://www.googleapis.com/oauth2/v3/token
[HTTP/2 200 OK 50ms]

Likely related to Bug 1735864

  • The mention Downloading message 1 of 119 in Message envoyes... appeared in status bar - while all Sent items shall have been downloaded already while Activity Manager is saying that all is up-to-date...

  • The following appeared in the console:

1634288119862	addons.xpi	WARN	Checking C:\Program Files\Mozilla Thunderbird\distribution\extensions for addons
Error: Can't find profile directory. XULStore.jsm:66:15
    load resource://gre/modules/XULStore.jsm:66
    XULStore resource://gre/modules/XULStore.jsm:24
Trying to load C:\Program Files\Mozilla Thunderbird\libotr.dll OTRLib.jsm:64:11
Successfully loaded OTR library C:\Program Files\Mozilla Thunderbird\libotr.dll OTRLib.jsm:72:13
Successfully loaded OpenPGP library rnp.dll version 0.15.2+git20210806.dd923a4e.MZLA from C:\Program Files\Mozilla Thunderbird\rnp.dll RNPLib.jsm:92:15
Found 0 public keys and 0 secret keys (0 protected, 0 unprotected) RNPLib.jsm:288:15
services.settings: thunderbird/hijack-blocklists has signature disabled RemoteSettingsClient.jsm:1029
[Exception... "Favicon at "https://start.thunderbird.net/favicon.ico" failed to load: Not Found."  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: resource:///modules/FaviconLoader.jsm :: onStopRequest :: line 253"  data: no] FaviconLoader.jsm:253:22
    onStopRequest resource:///modules/FaviconLoader.jsm:253
tb.account.size_on_disk - Truncating float/double number. 4
 - Unknown scalar. 3
mail.activity: Exception: [Exception... "Component is not available"  nsresult: "0x80040111 (NS_ERROR_NOT_AVAILABLE)"  location: "JS frame :: resource:///modules/ActivityManager.jsm :: getActivity :: line 135"  data: no] ActivityManager.jsm:86:16
    removeActivity resource:///modules/ActivityManager.jsm:86
    onFolderRemovedFromQ resource:///modules/activity/autosync.jsm:247
mail.activity: onFolderRemovedFromQ: [Exception... "Component is not available"  nsresult: "0x80040111 (NS_ERROR_NOT_AVAILABLE)"  location: "JS frame :: resource:///modules/ActivityManager.jsm :: getActivity :: line 135"  data: no] autosync.jsm:257:22
NS_ERROR_NOT_AVAILABLE: ActivityManager.jsm:135
    getActivity resource:///modules/ActivityManager.jsm:135
    removeActivity resource:///modules/ActivityManager.jsm:84
    onFolderRemovedFromQ resource:///modules/activity/autosync.jsm:247
The Web Console logging API (console.log, console.info, console.warn, console.error) has been disabled by a script on this page.
XHRPOSThttps://www.googleapis.com/oauth2/v3/token
[HTTP/2 200 OK 60ms]
XHRPOSThttps://www.googleapis.com/oauth2/v3/token
[HTTP/2 200 OK 50ms]
mail.activity: Exception: [Exception... "Component is not available"  nsresult: "0x80040111 (NS_ERROR_NOT_AVAILABLE)"  location: "JS frame :: resource:///modules/ActivityManager.jsm :: getActivity :: line 135"  data: no] ActivityManager.jsm:86:16
mail.activity: onFolderRemovedFromQ: [Exception... "Component is not available"  nsresult: "0x80040111 (NS_ERROR_NOT_AVAILABLE)"  location: "JS frame :: resource:///modules/ActivityManager.jsm :: getActivity :: line 135"  data: no] autosync.jsm:257:22
NS_ERROR_NOT_AVAILABLE: ActivityManager.jsm:135
mail.activity: Exception: [Exception... "Component is not available"  nsresult: "0x80040111 (NS_ERROR_NOT_AVAILABLE)"  location: "JS frame :: resource:///modules/ActivityManager.jsm :: getActivity :: line 135"  data: no] ActivityManager.jsm:86:16
mail.activity: onFolderRemovedFromQ: [Exception... "Component is not available"  nsresult: "0x80040111 (NS_ERROR_NOT_AVAILABLE)"  location: "JS frame :: resource:///modules/ActivityManager.jsm :: getActivity :: line 135"  data: no] autosync.jsm:257:22
NS_ERROR_NOT_AVAILABLE: ActivityManager.jsm:135
  • When sending another test message to myself, the mentions Downloading message 1 of 11xxx in ???... (could not read xxx and ??? because it disappear so quickly that I cannot read fully) immediately followed by Downloading message 1 of 6 in Inbox... appeared in status bar and is not disappearing or progressing - Activity manager still show that all is up-to-date - There are only two new message in Inbox my two test messages... so why trying to re-download 6 items?

It seems that sending/receiving message may be the action triggering the re-download... browsing between folder so far did not seemed to trigger a re-download...

I also use only one account for testing, people encountering issues often seem to use multiple accounts... maybe an aggravating factor? Or the mailbox size or quantity of data being synced?

As a topicbox thread of mine is mentioned above I just want to add what I'm seeing here as it differs with respect to what is being re-downloaded in an aspect not mentioned in any above.

TB fine for me up to and including 93.0b5. The problem starts when I update further, initially to 94.0b1, now if I go from 93.0b5 to 94.0b2 (as I've reverted to 93.0b5). Any time I am running 94.0b1 or b2 TB starts re-downloading all messages that have arrived since the update to 94. It does NOT re-download all the approx. 23000 messages between the 6 accounts my TB is configured with that arrived BEFORE the update to to 94. But once the first new message arrives after updating to a 94 version the re-downloading starts on any folders in any account - at the moment inbox on the two of my accounts almost all emails to me are addressed to, sent and draft in the one account I do most of my emailing from - that have messages since the update to 94. The re-downloads for each folder with post-update messages happen every time I switch to that folder, every time I open a new message from the folder (even though it re-downloaded moments before when I opened the folder), and otherwise there will be a re-download of each folder with post-update messages 2 minutes after the last re-download (and syncing is NOT set to 2 minutes). All folders of all accounts are set to offline use. The re-download each time is of the message, not the headers again. Each re-download of each message adds the message again (and again, and again ...) to the mbox file for that folder. For me, 94.0beta 1 and b2 simply does not remember that it has already downloaded and stored the messages that have arrived since the update to 94 in the mbox file, although it does still remember the message headers were downloaded and the latest message flag settings (read, star etc). So the location of the problem for me is limited as to where in TB it can be - the recording or reading by 94 of whether the whole message has already been downloaded or not. In fact it must just be the recording of that, because 94 for me is NOT re-downloading the pre-update-to-94 23000 messages, so it can clearly read that they have already been downloaded.

Come to think of it ... when I first became aware of the problem after my original update from 93.0b5 to 94.0b1, at that point it was re-downloading, every two minutes after the last re-download finished, the full messages of the 50 emails that had arrived in one particular account inbox since the update (along with some others for other folders). As part of a particularly foolish experiment I deleted that one account's INBOX.msf file. Thunderbird then spent several hours re-downloading the entire 11000 emails in that account inbox (all my emails for that account since moving to Thunderbird in 2018) rather than just the 50 since the update it had been re-downloading before, and then, two minutes after it finished, started re-downloading the entire 11000 full messages again. So I ended up restoring both the Thunderbird Program Files folder and the profile folder from an Acronis backup, restoring Thunderbird to 93.0b5 and the old INDEX.msf file. Which makes me think, alongside what I wrote above, that the problem, at least for me, is specifically a problem in 94.0b1 and b2 affecting its updating the .msf files, which I presume contain a flag for each message as to whether the whole message has been downloaded or just the headers. Either 94 isn't setting the flag to show messages have been downloaded as it downloads them, or is setting it incorrectly.

Oh, Windows 10 Home, 64-bit, all updates up to and including this month's patch Tuesday..

DavidGB,

(In reply to DavidGB from comment #33)

Any time I am running 94.0b1 or b2 TB starts re-downloading all messages

Does the same happens if you run TB in safe mode?
Would you be able to provide some logs to Gene as per his request in Comment 28 and Comment 15?

Regards,

Attached file imap.log.moz_log (obsolete) —

94.0a1.en-US.win64

Having the same issue. I'll wake up to a full hard drive. Messages seem to be completely redownloaded every time it checks for messages (10 minutes current setting).

I can reproduce this issue on every launch.

IMAP:5 caused a multi-GB file (seems to contain all email content via CreateNewLineFromSocket), IMAP_CS:5 only attached. If I can somehow give additional info let me know.

I did pull out this section from the IMAP:5,IMAP_CS:5:

[Parent 17312: IMAP]: D/IMAP SetConnectionStatus(0x0)
[Parent 17312: IMAP]: D/IMAP_CS Do full sync?: mFolderHighestUID=279, added=0, useCS=false
[Parent 17312: IMAP]: D/IMAP_CS needFullFolderSync=true, needFolderSync=false
[Parent 17312: IMAP]: I/IMAP 221ab58b000:mail.privateemail.com:S-INBOX:SendData: 14 UID fetch 1:* (FLAGS)
Attachment #9246168 - Attachment mime type: application/octet-stream → text/plain

Yowza guys. Yesterday my profile was 110GB. Today it's 116GB. No way is heck I got 6GB of mail delivered in 24 hours. Something is SERIOUSLY wrong.

(In reply to Richard Leger from comment #36)

Does the same happens if you run TB in safe mode?
Would you be able to provide some logs to Gene as per his request in Comment 28 and Comment 15?

Yes it happens in safe mode - TB just keeps on downloading all emails not already in the .msf files at the point of update to TB 94.0b1 or b2.

(Thinks) Could the odd timing I'm seeing be something to do with use of IDLE? I know that mail servers are supposed to keep connections alive for at least 30 mins of inactivity, or 15 mins for those solely for mobile devices, and the RFC recommendation for clients is just 1 IDLE followed by a NOOP every 29 minutes to keep the connection active, or 1 idle then a NOOP every 14 minutes if a server is only for mobile clients. But I also know a lot of email clients chuck out IDLEs every few minutes and NOOPs every few seconds, even though it's completely unnecessary and bandwidth and battery wasting. What is Thunderbird's use of IDLE? If THAT is every two minutes after a connection and fetch, that would explain why for me I'm getting a re-download every two minutes when my 'check every' is set to 5 minutes.

However while that would explain the particulars of what I'm seeing, it's not actually relevant to the problem - the problem is TB not remembering what messages it has downloaded since the update to 94.

May try to do the log later. Updating to 94 again and getting enough traffic for the problem to be logged, then the whole lengthy palaver of restoring the profile to a 93 state will take time. I'm disabled and have a limited capacity to focus on things, and most of today's stock has been used up already. I'm about to lapse into a pained unfocused fog.

(In reply to Arthur K. [He/Him] from comment #38)

Yowza guys. Yesterday my profile was 110GB. Today it's 116GB. No way is heck I got 6GB of mail delivered in 24 hours. Something is SERIOUSLY wrong.

In just the few minutes since I posted, profile size jumped to 126GB+. Going to roll back to 93.0b. This is nuts.

(Thinks) Could the odd timing I'm seeing be something to do with use of IDLE?

I haven't changed anything recently regarding imap IDLE so I don't think its the problem. But to be sure, you can turn it off by unchecking "Allow immediate server notification when new messages arrive" under Server Settings and restart TB.

So, I downgraded back to 93.0b5 and ran a Properties > Repair Folder on both my Inboxes. TB's presently churning and doing it's thing. My disk space usage went from 13GB free to 69GB free. So that did something to correct whatever the problem was. CPU is currently pegged and TB is not responding so I'll report later what happens when all settles.

(In reply to gene smith from comment #41)

(Thinks) Could the odd timing I'm seeing be something to do with use of IDLE?

I haven't changed anything recently regarding imap IDLE so I don't think its the problem. But to be sure, you can turn it off by unchecking "Allow immediate server notification when new messages arrive" under Server Settings and restart TB.

I'm not saying it's the problem. I was just wondering if it is set to re-issuing an IDLE every two minutes (which is not wrong, but would be extravagantly unnecessary if it is) which could cause the actual problem to appear every two minutes after a fetch. The problem is TB 94 not remembering it has already downloaded messages recorded in the .msf files after updating to 94.0b.

It does exist IDLE on the 2 minute (or whatever you've set it to) interval to explicitly check for new mail and then go back into IDLE. For that reason I usually recommend disabling IDLE if your timed new mail check interval is very short (1 or 2 min).

Re comment 37: The "need full folder sync" log lines only mean all the imap flags are being fetched. It doesn't mean all the messages bodies are to be downloaded and stored. The IMAP_CS:5 was mainly needed to see if there was a UIDVALIDITY problem, but I don't see it in the attached log.

(In reply to gene smith from comment #44)

It does exist IDLE on the 2 minute (or whatever you've set it to) interval to explicitly check for new mail and then go back into IDLE. For that reason I usually recommend disabling IDLE if your timed new mail check interval is very short (1 or 2 min).

Re comment 37: The "need full folder sync" log lines only mean all the imap flags are being fetched. It doesn't mean all the messages bodies are to be downloaded and stored. The IMAP_CS:5 was mainly needed to see if there was a UIDVALIDITY problem, but I don't see it in the attached log.

Got it, if there is any other type of logs I can run to help track the issue please let me know.

DaganHoover,
Not sure if you have multiple accounts having this problem but if you can "disable" all but one offending account and record an IMAP:5 log it might be informative and produce a smaller log.
To disable accounts all you have to do in Server setting is uncheck "Check new mail at startup", "Check new mail every X min" and "Immediately notifiy". Then select the Inbox of the one remaining enabled account and shutdown. Then run TB to record the log and see if already stored messages are downloaded for the remaining account. If so, shutdown tb and attach the log.
Thanks.

DaganHoover,
You can also include IMAP_CS:5 in the MOZ_LOG variable since it doesn't add a lot to the file size and might also reveal something. Also, be sure to include timestamps, i.e., MOZ_LOG=IMAP:5,IMAP_CS:5,timestamp,sync

Attached file imap.log.zip —

(In reply to gene smith from comment #47)

DaganHoover,
You can also include IMAP_CS:5 in the MOZ_LOG variable since it doesn't add a lot to the file size and might also reveal something. Also, be sure to include timestamps, i.e., MOZ_LOG=IMAP:5,IMAP_CS:5,timestamp,sync

Sure.

(As a note, the previously attached log was the entirety of a solo IMAP_CS:5 log)

The log was still very large in size (it seems to have included all the emails and content from the account), but workable after redacting the email content and zipping.

My process was to:

  1. Disable all but one confirmed offending account. I made sure the account was up-to-date with emails. I confirmed that the account had synchronization turned back on.
  2. Restart Thunderbird with log env vars.
  3. The status bar displayed the message: Downloading name@email.com: Downloading message NNNN of 1034 in Inbox.... I waited until it finished and displayed that the account was up to date.
  4. I confirmed the increase of the INBOX file by about 500mb.
Attachment #9246168 - Attachment is obsolete: true

(In reply to gene smith from comment #44)

It does exist IDLE on the 2 minute (or whatever you've set it to) interval to explicitly check for new mail and then go back into IDLE. For that reason I usually recommend disabling IDLE if your timed new mail check interval is very short (1 or 2 min).

I have IDLE turned on, but I am not aware of any way the user can specify how often it is set to re-issue IDLEs (or see what it's set to), which is why I'm asking if it happens to be configured to two minutes by default. This is purely because I have no other idea why my accounts' folders that have messages since the update to 94 re-download every two minutes after their last re-download. It's not the 'check every' because that's set to the default 5 minutes that was set when I created the accounts in TB back in 2018. Now, we could argue about how often an IDLE should be issued, but that's not at all the point here. The point is that TB 94 - at least for me - is never remembering which messages that have arrived in a folder since the update to 94 (inbox, draft, sent, junk or even trash) have already been downloaded, so anything that could cause it to download a full message that hasn't been downloaded, like just opening a folder that's had message since the update, or opening a message that's arrived since the update, or perhaps the response to an IDLE, is causing TB 94 to re-download the messages again each time and adding them again and again and again to the INBOX or whichever mbox file.

I'm curious as to why my re-download of all messages that arrived since the update to 94 is two minutes since the last re-download as opposed to any other time interval. If IDLEs are re-issued every two minutes by default as I don't know how to change that, then that's that particular trigger. But it's not the fault of the IDLEs. That's just one amongst the number of things that trigger a re-download and a growing mbox file from all the multiple copies of the since-update-to-94. The focus is why 94, when set to keep all messages on the client for use offline, can't remember it's already downloaded any messages since the update to 94 so keeps re-downloading them, but can remember it's already downloaded, so does not re-download, all the - many thousand in my case - messages that were downloaded before the update to 94. I submit to you that there's a problem with 94 with respect to TB recording in the .msf files that it has downloaded and stored messages that have arrived since the update.

Hello, I have the same issue on my side. It also appeared when I upgraded from TB93b05 to TB94.0b2. Logs are too big to upload here, here is the link to download : https://www.getigroup.com/imap.log.moz_log.7z

Please let me know what else I can do. I think this bug is also reported here (duplicate?) : https://bugzilla.mozilla.org/show_bug.cgi?id=1734843

DanganHoover-
Thanks, but that looks really bad. Definitely all of the emails in inbox are getting re-downloaded and I can't really tell why. It does a full flag fetch on startup and then then fetches 5 message bodies (UID 57, 1033-1036) and then fetches what looks like all the rest to end of the log. Then you shutdown.
I can't tell the UIDs of the full flag fetch since you have redacted the responses but I'm sure they just correspond to the UIDs of the messages that got fetched.
One thing missing from the file is "capability" request and responses. I'm not sure if you have deleted them or they just didn't occur. But I've never seen that happen and don't think tb imap will work without them.
Maybe the messages are not getting stored properly in the mbox file called INBOX. (If there were problems with the INBOX file you might also see files named INBOX-1, INBOX-2 etc but the one with the highest suffix is the one in use and others can be deleted.) Could you take a look at the active INBOX file and see if your messages are in there. The oldest one should be on top I think. The messages should be separated with"From - " delimiters between them, e.g.

:
:
From - Thu Oct 14 21:42:05 2021
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
Return-Path: <0101017c819cf6c9-9bb04733-b209-4a7c-9406-5ba0ada10622-000000@us-west-2.amazonses.com>
Received: from charter.net ([47.43.20.24])
          by mtain005.msg.chrl.nc.charter.net
          (InterMail vM.9.01.00.036 201-2473-137-122-162) with ESMTP
          id <20211015014202.VYSO32423.mtain005.msg.chrl.nc.charter.net@charter.net>
          for <gds@chartertn.net>; Fri, 15 Oct 2021 01:42:02 +0000
Received: from a58-224.smtp-out.us-west-2.amazonses.com ([54.240.58.224])
	by cmsmtp with ESMTP
	id bCETmzBTA6C3TbCETmCKRi; Fri, 15 Oct 2021 01:42:02 +0000
:
:
From - Thu Oct 14 21:43:02 2021
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
Return-Path: <0101017c819d098f-10f3f9b3-b1c1-4cc8-8ae4-ce85ad441b01-000000@us-west-2.amazonses.com>
Received: from charter.net ([47.43.20.38])
          by mtain013.msg.chrl.nc.charter.net
          (InterMail vM.9.01.00.036 201-2473-137-122-162) with ESMTP
          id <20211015014206.CKJI1194.mtain013.msg.chrl.nc.charter.net@charter.net>
          for <gds@chartertn.net>; Fri, 15 Oct 2021 01:42:06 +0000
Received: from a58-224.smtp-out.us-west-2.amazonses.com ([54.240.58.224])
	by cmsmtp with ESMTP
	id bCEXmclNifvKDbCEYmdLE2; Fri, 15 Oct 2021 01:42:06 +0000
X-Authority-Analysis: v=2.4 cv=UqWmi88B c=1 sm=1 tr=0 ts=6168dc6e b=1
:
:

It's possible that if the messages are corrupted somehow in this file so TB will request that they be re-downloaded. That seems to be what is happening since TB is happy with headers and doesn't try to re-download them. But even after re-downloading there is still a problem and happens again the next time.

I don't know if you have deleted the INBOX (and INBOX-x) and INBOX-x.msf files but I think others have with no benefit. This will cause the files to be rebuilt from scratch on tb startup and, of course, another full body download will occur.

DavidGB, I think the log supplied by DanganHoover pretty much confirms what you see too. However, I'm not sure why you are seeing everything download every 2 minutes. Also, FWIW, going into IDLE also only occurs on the timed interval (5m in your case, I think).

Camille, Thanks for the log! I'll take a look at it right now.

gene smith, hey thank you buddy! I disabled all but one account as you explained. But this still produced a 244Mb file... Anything else you need I'm here. Thanks again for the work. To me, all the people working on TB are heroes. TB is my main tool for work, it is a superb piece of software and way ahead of anything else (I have Outlook on my PC, I never use it).

(In reply to gene smith from comment #51)

DanganHoover-
Thanks, but that looks really bad. Definitely all of the emails in inbox are getting re-downloaded and I can't really tell why. It does a full flag fetch on startup and then then fetches 5 message bodies (UID 57, 1033-1036) and then fetches what looks like all the rest to end of the log. Then you shutdown.
I can't tell the UIDs of the full flag fetch since you have redacted the responses but I'm sure they just correspond to the UIDs of the messages that got fetched.
One thing missing from the file is "capability" request and responses. I'm not sure if you have deleted them or they just didn't occur. But I've never seen that happen and don't think tb imap will work without them.

The only wildcard parts I redacted was /CreateNewLineFromSocket: .*$/, so only those end of lines. If the capability req/resp are logged on separate lines, then I don't believe I removed them.

Maybe the messages are not getting stored properly in the mbox file called INBOX. (If there were problems with the INBOX file you might also see files named INBOX-1, INBOX-2 etc but the one with the highest suffix is the one in use and others can be deleted.) Could you take a look at the active INBOX file and see if your messages are in there. The oldest one should be on top I think. The messages should be separated with"From - " delimiters between them, e.g.

If I do a count of From -, I get 1034 results which matches the original download. I turned synchronization back on and let it run through another download attempt and I get 2069 copies of From - (1034 + 1035). Just typing that previous sentence and thunderbird downloaded another copy. Looking for counts of a single Message-ID, I see three complete copies of the email in the INBOX file.

I can tell you in my case:

With all 4 of my ISP mailboxes, and the gmail.com and outlook.com, the servers all correctly respond with capability (including IDLE for all, and NOTIFY with my ISP's servers).

The mbox files (after a very long load time into my text editor of choice for an 11000+ email inbox) are fine apart from the repeated copies of the emails from after the update to 94 due to the re-downloaing. Those copies and separators individually are fine. This is not an mbox problem.

The .msf files clearly aren't fine (and this is NOT limited to inboxes - ANY folder that has received a message since the update to 94 immediately then starts re-downloading the messages since the update to 94 in that folder, e.g. after sending my first email after updating to 94 from one account, that Sent folder also started re-downloading its since-94-update messages, as did the Drafts folder after a draft was saved).

I say the .msf files are NOT fine because:

When I first noticed and started looking into it, on one particular account that had had 50 emails arrive in the inbox since the update to 94 - all of which I read when they arrived - TB was re-downloading the 50 emails since the update-to-94 every two minutes or on other provocations like switching to that inbox from the unified inbox. But TB was NOT re-downloading the 11000 emails that were already in that inbox before the update to 94.

I then - foolishly - deleted the INBOX.msf file. TB then downloaded all 11000 emails, reconstructing an INBOX.msf file as it went ... then two minutes after it finished it re-downloaded ALL 11000 emails again. Then two minutes after that finished it started all 11000 AGAIN ... (during which I stopped it and restored the TB Program Files folder and the profile from a backup to 93.0b5).

So ... with an .msf file created by all versions of TB since 2018 up to and including 93.0b5, TB 94 knew that all the messages had been downloaded and didn't re-download anything. Once 94 itself added to the .msf on receiving new messages, it still knew all the pre-94-updates had already been downloaded and did not re-download them, but all the messages it had received itself it couldn't see it had already downloaded them and started the cycle of re-downloading them. And when the .msf file created by all the previous versions of Thunderbird was deleted and replaced by a new one created by 94 itself for all 11000 emails, it then couldn't remember it already had downloaded ANY of them and started re-downloading and re-downloading all 11000. So - 94 can see from entries made in an .msf file by all those earlier versions that a message has already been downloaded, but it CANNOT see from its OWN entries that messages have already been downloaded so re-downloads, then re-downloads, then re... That's the problem.

At the moment I haven't figured out how to parse .msf file entries for messages, but I'm figuring it out in the expectation I'll find 94 isn't creating correct .msf entries.

(In reply to gene smith from comment #52)

DavidGB, I think the log supplied by DanganHoover pretty much confirms what you see too. However, I'm not sure why you are seeing everything download every 2 minutes. Also, FWIW, going into IDLE also only occurs on the timed interval (5m in your case, I think).

(My #55 was a reply to your #51, which wasn,'t addressed to me but I'd looked at several of the things you mentioned re. capability and the mbox and msf files)

I also don't know why they download every two minutes (note that's two minutes from the end of the last re-download for that folder - when the re-downloads were of 11000 messages, the next re-download started 2 minutes after the previous one finished).

Thunderbird SHOULD issue a new IDLE as it completes a 'check every ...' check and fetch (presuming the setting to use IDLE is checked). Not on some timer, but just as part of finishing the check-and fetch, however long that takes. But - and I'm not sure in TB, which is why I'm asking - many mail clients issue repeated IDLEs and NOOPS on the same connection between timed fetches, which technically speaking they shouldn't, but do. For instance, the mail client I was using for many years with POP3, when I finally switched to IMAP I found it was issuing an IDLE after a timed fetch, them issuing a NOOP every 5 seconds and another IDLE every 25 seconds when the timed fetches were every 10 minutes. Perhaps I'd get the answer I'm looking for if I ask this instead: In TB, if 'Check for new messages every ...' is turned OFF, but 'Allow immediate server notifications when new messages arrive' is ON, what timed pattern of IDLEs and NOOPs does TB use?

Problem unchanged after updating from 93.0b5 to 94.0b3. Now I've got to recover the TB Program Files and Profile folders from a backup to 93.0b5 again ....

status-thunderbird_esr91: --- → affected
Component: General → Networking: IMAP
Product: Thunderbird → MailNews Core

DavidGB wrote:

Perhaps I'd get the answer I'm looking for if I ask this instead: In TB, if 'Check for new messages every ...' is turned OFF, but 'Allow immediate server notifications when new messages arrive' is ON, what timed pattern of IDLEs and NOOPs does TB use?

In that case IDLE will be on until the server drops the connection. During that time nothing is sent by tb at all (since in idle state). There is an issue with doing this since most servers disconnect after 30m and tb doesn't reconnect so you won't receive new mail notification after 30m unless you click "get mail". So I recommend that if you want to mostly rely on IDLE, still poll for mail every 29 minutes or so.
Also, there is an issue that IDLE is only guarantee to apply to INBOX and not to other folders that may have mail delivered to them (e.g, by server side filters). For those folder you must poll.

To test your theory on why messages are re-downloaded I created a new profile for my charter account using release v78 and set up using all default settings. Of course, no problem with this. Then I shutdown v78 and started 94b2 with the same new profile. Then with another client I sent 9 new messages to my charter account Inbox and they appeared in the tb 94b2 Inbox and auto-sync'd to the INBOX file. I set the 94b2 new mail check interval to 1m and never see any spurious downloads from the server and INBOX file never grows even after tb 94b2 restarts.

I'll try deleting just the INBOX.msf file from the charter profile and see if that causes anything bad to happen. (Before I have deleted both INBOX and INBOX.msf and saw nothing bad happen after restarting 94b2.)

You mention "unified" inbox. I wonder if that has something to do with this and why I don't see it since I never use that feature in my testing.

When you look at a folder's properties, what metric is used for Size on disk?

I notice that the property page shows a reasonable looking size that definitely doesn't match the actual size on disk.

It would be useful to track it down more exactly, either using nightly builds or betas (mozregression can help)
Here are changes from 93b5->94b2: https://hg.mozilla.org/releases/comm-beta/pushloghtml?fromchange=THUNDERBIRD_93_0b5_RELEASE&tochange=THUNDERBIRD_94_0b2_RELEASE

So after the revert to 93.0b5 and a Repair Folder operation on my work PC Inboxes, I did reclaim all the lost disk space but it took a good 4-5hrs for it to do it's thing (re-download and re-index I am assuming) before finally settling. I didn't see it happening any more after that and disk use stayed more or less the same. Did the same operation on my home PC with the same results as my office PC.

That being said, I would think the disk consumption issue that seems to be a by-product of whatever 94.0b is doing would be more pressing here.

Also attaching something I noticed in the Status Bar. It's showing it downloading 48k+ messages of only 47,200. How can it be downloading more than what it claims is present?

(In reply to gene smith from comment #58)

DavidGB wrote:

Perhaps I'd get the answer I'm looking for if I ask this instead: In TB, if 'Check for new messages every ...' is turned OFF, but 'Allow immediate server notifications when new messages arrive' is ON, what timed pattern of IDLEs and NOOPs does TB use?

In that case IDLE will be on until the server drops the connection. During that time nothing is sent by tb at all (since in idle state). There is an issue with doing this since most servers disconnect after 30m and tb doesn't reconnect so you won't receive new mail notification after 30m unless you click "get mail". So I recommend that if you want to mostly rely on IDLE, still poll for mail every 29 minutes or so.

That's what the NOOP is for. After starting an IDLE, the recommendation is to send a NOOP every 29 minutes, as that re-sets the server's no-activity timer to zero before the 30 minute cut-off, allowing the original IDLE to go on and on providing the connection doesn't drop for some other reason. (Or a NOOP every 14 minutes if a mail server dedicated to mobile clients, which often cut-off after 15 minutes rather than 30.)

Also, there is an issue that IDLE is only guarantee to apply to INBOX and not to other folders that may have mail delivered to them (e.g, by server side filters). For those folder you must poll.

That's what NOTIFY is for, as it causes the server to report changes in every specified folder, not just Inbox. I know GMail and outlook.com servers don't support NOTIFY, but my ISP's mail servers and many others do (for a start with any using Dovecot), and I found it disappointing that Thunderbird doesn't at least have the option to use NOTIFY where the server supports it (unlike the Android client I use, which does). Then again, the saving in energy and data transfer by using NOTIFY rather than frequent fetches has a real benefit with a mobile phone in a way it doesn't with a desktop or laptop that's permanently plugged in like mine. But TB not supporting NOTIFY does make it feel a bit ... backward.

To test your theory on why messages are re-downloaded I created a new profile for my charter account using release v78 and set up using all default settings. Of course, no problem with this. Then I shutdown v78 and started 94b2 with the same new profile. Then with another client I sent 9 new messages to my charter account Inbox and they appeared in the tb 94b2 Inbox and auto-sync'd to the INBOX file. I set the 94b2 new mail check interval to 1m and never see any spurious downloads from the server and INBOX file never grows even after tb 94b2 restarts.

I really think I unintentionally introduced a red herring here. My every-two-minutes-after-the-last re-download is one, but only one thing triggering re-downloads. Just clicking on any folder (that syncs) causes a re-download of all messages in that folder that have arrived since the update to 94. Clicking on a message in the list causes a re-download of the message as it opens (I have messages opening in a separate window) even though the message was downloaded in the fetch after it arrived, and downloaded again when I then selected the folder (if it wasn't open when the message arrived). I was curious about the 2-minute trigger's timing, as nothing user configurable is set to that interval I'm aware of, so wondered about how IDLE is being implemented in TB as I've seen other mail clients emitting wholly unnecessary IDLEs every few seconds (it's really not implemented properly in a surprising number of clients), but it's clearly not the IDLEs triggering that re-download from what you say, and even if it was that's not the actual problem. The problem is that anything that would correctly trigger a download of messages (that arrived since the update to 94) if those messages had never been downloaded before (because of a setting to download headers only on fetch) is wrongly triggering a download of all messages since the update to 94 every time even though TB is NOT set to download headers only, and the messages HAVE been downloaded before and stored in the mbox file time after time.

(Two minutes couldn't be the time on my laptop TB takes to check through 11000 emails to see which need re-downloading could it? Two minutes of checking with nothing in the status bar or activity manager before actual downloading with the Downloadng X of Y? It just so happens both inboxes I mostly get mail to so am seeing the problem in have 11000messages in them, give or take? No ... even if it is the reason for the 2 minutes, again that's not the problem - the problem is repeatedly re-downloading things it should know it already has downloaded and stored.)

I'll try deleting just the INBOX.msf file from the charter profile and see if that causes anything bad to happen. (Before I have deleted both INBOX and INBOX.msf and saw nothing bad happen after restarting 94b2.)

My experience is that after updating to 94 and leaving the mbox file and matching .msf alone (again, this is NOT just INBOX - after sending a couple of emails after the 94 update I started getting the re-downloading and re-downloading of all the since-98-update messages in the Sent and Draft folders too. It's ANY syncing folder, not just inboxes), all the messages in any folder that arrived after the update to 94 start re-downloading, 2 minutes after last, or opening the folder, or opening the message again. But all the messages that were in the folder do NOT re-download. However, when I deleted an msf file (only) all the emails in that folder re-downloaded as would be expected as a new ,msf file was generated, but then ALL of them - not just the ones since the 94 update - re-downloaded again, and again, and again until I stopped it and recovered the original .msf file from a backup.

This is why I am saying the problem is with the .msf file. For me the issue is that 94 can see that messages downloaded by earlier versions of TB up to and including 93.0b5 have been downloaded so doesn't re-download them, only the ones since the update; but if you delete the .msf file so a new one is written by 94, 94 then can't see that ANY were downloaded before and starts re- and re-downloading the lot. It appears that TB 94 can read entries in .msf files signifying a downloaded status that were written by any earlier version of TB (at least back as far as whatever the version of TB was in July 2018 when I started using TB), but it CANNOT read the entries signifying a downloaded status written by itself (94.0b1 2 and 3). Where in the code the problem is I can't tell you - the last time I wrote a program was in the early 90s. But in terms of the process the problem HAS to be in is how 94 is recording the download status in the appropriate .msf file (it can read the other flag statuses fine, like read or starred). It's the only thing that makes sense of what i see with the original .msf file (re- and re-downloading only the ones that arrives after 94 install) versus what I see after making no change apart from deleting the original .msf file (re- and re-downloading EVERY message that .msf file refers too). It's just not reading its own record of downloaded status.

Anyone reading this, do NOT delete an .msf file testing this if it applies to a folder with more than a few messages unless you either (a) make a copy, or (b) have a backup you can recover a recent version from. When I did it to an .msf file for a folder with 11000 messages in from before the 94 update, and then the repeated re-downloading started now redownloading all 11000 each time rather than the 20 since-TB94-update it had been doing and realised I had forgotten to make a copy, I would have been in a serious mess if i hadn't been able tom recover it from a recent c-and-d backup.

You mention "unified" inbox. I wonder if that has something to do with this and why I don't see it since I never use that feature in my testing.

I normally have the unified inbox, set to show only unread and starred emails, open all the time. It was only yesterday I thought to click on the actual folders and noticed that every opening of a folder triggered a re-download of all messages that had arrived in that folder since the update to 94 every time I re-opened the folder. Switching unified inbox on or off made no difference.

I can't see how any would be relevant, but I'm trying to think of any way I have TB set up that might not be common. I usually have unified inbox, unread and starred only, open. I read messages in separate windows and have the message pane switched off. My profile is on the D drive, not in appdata on the C drive. Can't think of anything else.

DavidGB, I tried a unified folder setup and can't duplicate the issue with that either.

Anyhow, I seem to be doing the same thing as you do and don't see a problem with 94b2, even when running a profile originally produced 2 versions back, v78.

Let me ask a simple question: What happens if you just delete all the mbox and msf files on an account running 94b? I don't see a problem doing this after everything re-downloads. Do you?

Off topic addendum:
Regard comments above about tb supporting imap NOTIFY (https://datatracker.ietf.org/doc/html/rfc5465), I just checked the CAPABILITY response of my 12 or more popular imap servers I use for tb testing and development and I only see one that advertises support for NOTIFY. So even though it sounds like a good idea, seeming lack of support for it probably disqualifies it for addition to tb. FWIW about 14 years ago it was suggested to be added but nothing happened but one user attempt an add-on to support it. See bug 479133.

(In reply to Arthur K. [He/Him] from comment #61)

Created attachment 9246278 [details]
is-that-even-possible.png

So after the revert to 93.0b5 and a Repair Folder operation on my work PC Inboxes, I did reclaim all the lost disk space but it took a good 4-5hrs for it to do it's thing (re-download and re-index I am assuming) before finally settling. I didn't see it happening any more after that and disk use stayed more or less the same. Did the same operation on my home PC with the same results as my office PC.

That being said, I would think the disk consumption issue that seems to be a by-product of whatever 94.0b is doing would be more pressing here.

Also attaching something I noticed in the Status Bar. It's showing it downloading 48k+ messages of only 47,200. How can it be downloading more than what it claims is present?

I tend to think the excess disk space consumption is due to the continuous downloading and increasing .msf and mbox file sizes due to whatever this bug is.

Yes, I've also seen that number discrepancy and I think there have been attempts to fix it but not sure which specific bug.

DavidBG,
Re: https://bugzilla.mozilla.org/show_bug.cgi?id=1734843#c44

Another thing to try is what happens if you setup one of your accounts in a brand new profile using 94b3? I would be curious if that works for you too like the above user NoJoyInDelay says.

But to repeat what I told reporter NoJoy, there is still a bug if a new profile is required.

Paul,
Thanks again for the logs you sent me via email. They show basically the same info as I see in the logs form Camille and NoJoyInDelay.
I'm wondering if you would be willing to start a new profile for one of your accounts while running 94b2 or later?
NoJoyInDelay reported that the continuous downloads stopped when a new profiles is setup as referenced in comment 65 above and in bug 1734843 comment 44.

Flags: needinfo?(paul)

gene, I created a new profile using "thunderbird -p" and the issue is not here anymore with the new profile. When I switch back to the old profile the issue resurfaces. Also to note that bug 1734825 also disappears when a new profile is created (emails are no more set as read when opened - contrary to settings).
Could those two issues be related ?
Here are the logs : https://www.getigroup.com/imap.log.moz_log

I meant as per settings. In my setting the "set emails as read when opened" is unchecked.

I will recreate all my accounts with the new profile and keep an eye on it over the coming days to see if the issue reappears with the new profile. I still have my old profile if any test is needed.

Another similarity with bug 1734825 is that the behaviour only happens with emails received after the upgrade. No issue with emails that were received before the upgrade.

Thanks Camille. So we have two data points indicating 94b* doesn't like the earlier profile.
Looking at your log log link, I assume it is with the new profile since the only fetches (downloads) I see is TB asking for imap flags with no spurious body or header fetches.
And, yes, it does sound like bug 1734825 is somehow related to the profile problem and tb is not respecting the "don't store the \seen flag" parameter (Pref: mailnews.mark_message_read.auto) for newly received emails. (FWIW, I've never ever changed this parameter from its default until now.)

But I still can't duplicate the problem. :(

If anyone seeing the problem could try Magnus's suggestion in comment 60 to narrow down the cause it would be most helpful.

(In reply to gene smith from comment #71)

If anyone seeing the problem could try Magnus's suggestion in comment 60 to narrow down the cause it would be most helpful.

Is there a repository for the various beta version installations?

Is there a repository for the various beta version installations?

The releases and betas are all here: https://archive.mozilla.org/pub/thunderbird/releases/
The daily builds are here: https://archive.mozilla.org/pub/thunderbird/nightly/2021/
I'm not really familiar with the mozregression tool he suggests to use.

Note: This user "oggvorf" claimed to see "re-downloading all message on each tb start-up" at 93.0b5 on Oct 3:
https://thunderbird.topicbox.com/groups/beta/Tbcb8ae01b8d4f347
Then the user "oggvorf" went back to 92.0b5 and the problem still happened.
But may this is a special case since other report the problem (continuous downloads, not just at startup) starting with 94.0b1.

See Also: → 163964

Hello gene,

I tested the daily builds but I first looked at when the issue of emails marked as read appeared (bug 1734825). I found out that the issue appeared in the daily build of 27 September (https://archive.mozilla.org/pub/thunderbird/nightly/2021/09/2021-09-27-09-53-41-comm-central/). In the previous daily things are fine.

I will need a few days of running the daily 2021-09-27 to see if the Bug 1733966 (this bug) is also present in this version because I need a lot of emails to be able to see if this is indeed downloading all emails since installation (with a few emails I can't see the difference). Unless you have another way of knowing?

So assuming there is a correlation between the two bugs, the commits from 26 September might have broken something.

I will need a few days of running the daily 2021-09-27

Camille, The change Magnus points out would be in the 9-27 daily. Looks to me like you need to test with the 9-26 daily, if I'm looking at this right?

Camille, The change Magnus points out would be in the 9-27 daily. Looks to me like you need to test with the 9-26 daily, if I'm looking at this right?

Ok, I see. You want to run the 9-27 daily to see if this bug happens too. Where this bug is continuous downloads, not just the mark as read issue.

... Unless you have another way of knowing?

DavidGB says deleting the INBOX.msf file for an account causes the problem where all emails get re-downloaded on tb restart and then continuously re-download. (I would also delete the corresponding INBOX (mbox) file but I don't think DavidGB did.)

Thanks for the suggestions Magnus!
gene, yes I am running 9-27 daily to see if the continuous download bug happens too as I need a lot of emails to see the behavior (sending myself a few emails won't be enough).
I forgot about DavidGB's comment, I just tried it and I confirm that if I delete INBOX.msf, this triggers the following behavior:

  • In 9-26 : download of all the messages once
  • In 9-27 : download of all the messages again and again...

PS: both 9-26 and 9-27 were using the same profile so doesn't this mean that the issue is not with the profile data but with the way TB interprets the profile data ?

I just downgraded to 9-26 and launched it on the profile from 9-27.
9-26 started by downloading again all the emails so forget what I said in Comment #81, something in 9-27 might be setting some values on the profile that is triggering the full download.
I'm going to keep 9-26 running for a few hours to make sure nothing is triggered anymore.

(In reply to Camille from comment #82)

I just downgraded to 9-26 and launched it on the profile from 9-27.
9-26 started by downloading again all the emails so forget what I said in Comment #81, something in 9-27 might be setting some values on the profile that is triggering the full download.
I'm going to keep 9-26 running for a few hours to make sure nothing is triggered anymore.

So you're saying 9-26 seems to re-download only on startup? But you are letting it run a while to see if more spurious downloads occur?
I guess it's possible that something gets latched or locked into the profile after running 9-27 that also affects earlier versions like 9-26.

Yes I was waiting to see if more spurious download occurs with 9-26. Nothing happened so I guess 9-26 is fine. Yes I confirm that running 9-27 and then running 9-26 on the same profile kicks a full download of emails but only once. After that 9-26 behaves correctly.
So I guess 9-27 is setting some flags that make TB download emails over and over.

running 9-26 on the same profile kicks a full download of emails but only once. After that 9-26 behaves correctly.

So are you saying you can restart 9-26 now and you don't get re-downloads? Or are you saying that on startup, 9-26 still downloads but just once per startup?

If I restart 9-26 now it doesn't trigger any re-downloads.
If I run 9-27 and then I run 9-26, 9-26 downloads everything again but only once. Any subsequent run of 9-26 does not trigger re-downloads.

Excellent! So I guess you have definitely isolated the point where the bug appears!

Thanks gene! I hope I didn't do anything incorrect. Please let me know if that was it. It would be the first time I contribute, I could brag about it :p
And thanks again for the fantastic work you guys are doing.
Don't hesitate if there is any other test I can do to try the resolutions. I'll keep my old profile for this.

I hope I didn't do anything incorrect.

You did what I would do if I could duplicate the problem.

Don't hesitate if there is any other test I can do to try the resolutions.

Not sure if it will be fixed here or in another bug. Debugging the patches on 9-27 are beyond by pay grade (which is $0.00). :)

In re-reading comment 0 by reporter Ness Blackbird, it appears that it was reported for release 91 and does not really describe what we have been seeing and discussing in most of the comments above (continuous re-downloading after upgrade to 94.0b1). So rather than confirming this bug, I think the bug where you and user breezymozilla originally reported this is actually confirmed: bug 1734843.

Oops yes we continued the work here. What should we do? Do you think we should make a comment on bug 1734843 with the summary of the findings ? This is definitely for release 94.

(In reply to Camille from comment #90)

Oops yes we continued the work here. What should we do? Do you think we should make a comment on bug 1734843 with the summary of the findings ? This is definitely for release 94.

I already entered a comment/reference to this bug in bug 1734843. So it should be OK. Also marked bug 1735317 as a dupe.

See Also: → 1735317

I am seeing:
"The operation failed because another operation is using the folder. Please wait for that operation to finish and then try again."
when I go to Inbox, properties, "Repair folder".
Yet, when I check in "Activity Manager", there is no activity shown. Seeing no current activity, I go back and try again, and just see the same message. Is it this bug?

Worcester12345,
I've seen what you describe when I have messed something up in the tb code. So with things somewhat confused with 94b, I wouldn't be concerned at this time. It will probably clear up after a tb restart. But if you see this consistently on a more stable release it might be a real bug that got put in. (Just my opinion.)

No problem. I just wanted to mention it, because it might be relevant. If they are indeed related, maybe the fix will fix both, or maybe my info could provide a clue towards the fixing of this bug. The knee bone is connected to the shin bone.

FYI, it seems a fix was published Bug 1734843 Comment 55.

Some explanations on the possible root cause available in Bug 1734843 Comment 53.
"...when message quarantining is enabled (mailnews.downloadToTempFile=true), checking messages on IMAP causes re-download and duplication in the mbox file..."

(In reply to Worcester12345 from comment #92)

Yet, when I check in "Activity Manager", there is no activity shown.

Activity Manager may have some issues on its own... and may need some love and attention... I have filed a new Bug 1736526 for dealing with this separate issue...

Confirming that the chewing up of disk space seems to exist in 93.0b5 as well. I was down to 53GB free but after doing a repair on Inboxes it went up to 63GB free. 5 days ago I was at 69GB free.

Gene,

Don't know if helpful but I think I managed to re-create the issue...

Setup environment:

  • creating a new profile in 68.x ESR and adding one gmail account into it... keeping all TB default settings.
  • waiting to download all emails in all folders...
  • restart TB make sure you can send/receive emails
  • upgrade to 78.x ESR
  • click all the folders to make sure they are all up-to-date
  • upgrade to 91.x ESR
  • click all the folders to make sure they are all up-to-date
  • upgrade to 93.0b5
  • click all the folders to make sure they are all up-to-date
  • upgrade to 94.0b2
  • click all the folders to make sure they are all up-to-date
  • close TB

Trigger:

  • Wait 24H prior opening TB (maybe less time required but I tried everyday after work :-)
  • Receive new message in Inbox
  • Reply to a message
  • As soon as sent TB is trying to re-download email 1 of 111 or more in Message Envoyes (Sent folder) - Download does not seems to progress on screen... till at some point it disappears...

But does not happen all the time after that till the next day... same trigger...

Some more details attached for details (console errors, av option disabled)...

Regards,

Forgot to precise in previous comment that I also have Favorites and Unified Folders view enabled in addition of All Folders view... in the testing profile I use...

See Also: → 1736526

Arthur K.

(In reply to Arthur K. [He/Him] from comment #97)

Confirming that the chewing up of disk space seems to exist in 93.0b5 as well. I was down to 53GB free but after doing a repair on Inboxes it went up to 63GB free. 5 days ago I was at 69GB free.

In your Preferences does the option "Allow antivirus to quarantine individual incoming messages" appears ticked or unticked at your end?

If ticked, try to untick, restart TB then Compact folder you are seeing issue with, then Repair it... does it fix?

Source: Bug 1734843 Comment 53, Bug 1734843 Comment 67

Cheers,

(In reply to Richard Leger from comment #100)

Arthur K.

(In reply to Arthur K. [He/Him] from comment #97)

Confirming that the chewing up of disk space seems to exist in 93.0b5 as well. I was down to 53GB free but after doing a repair on Inboxes it went up to 63GB free. 5 days ago I was at 69GB free.

In your Preferences does the option "Allow antivirus to quarantine individual incoming messages" appears ticked or unticked at your end?
Yup.

If ticked, try to untick, restart TB then Compact folder you are seeing issue with, then Repair it... does it fix?

Source: Bug 1734843 Comment 53, Bug 1734843 Comment 67

I'll give it a shot. Thanks for this.

Richard,
Yesterday I saw something very similar. I had a profile unused for a few days while working on the "Unnecessary downloads" bug. In that profile I have a "Trash" mailbox that contained thousands of old emails that I never flushed and was using for testing another bug. When I went into that trash folder, tb started downloading all the headers for no apparent reason. (I only store headers.) This is running with a daily I built myself and is marked 92.0a1. I haven't pull in any code updates since about end of July (other than my own changes made locally).

(In reply to Richard Leger from comment #100)

Arthur K.

(In reply to Arthur K. [He/Him] from comment #97)

Confirming that the chewing up of disk space seems to exist in 93.0b5 as well. I was down to 53GB free but after doing a repair on Inboxes it went up to 63GB free. 5 days ago I was at 69GB free.

In your Preferences does the option "Allow antivirus to quarantine individual incoming messages" appears ticked or unticked at your end?

If ticked, try to untick, restart TB then Compact folder you are seeing issue with, then Repair it... does it fix?

Source: Bug 1734843 Comment 53, Bug 1734843 Comment 67

Well, I don't know if un-ticking the box is helping or not but I went down the line and ran a Compact and Repair action on every folder one at a time. One Inbox is still downloading messages but I went from 63GB free to 90.5 GB free. That's more than I got back a few days ago when I only did it to the Sent and Inbox folders. That is a HUGE recapture of disk space!

Severity: -- → S2
Status: UNCONFIRMED → NEW
status-thunderbird_esr78: --- → unaffected
Ever confirmed: true
Summary: Unnecessary mail synchronization → Unnecessary mail synchronization with TB91
Whiteboard: [beta see bug 1734843]

(In reply to gene smith from comment #28)

(In reply to freeman3 from comment #26)

Created attachment 9245981 [details]
sync.png

same problem here. IMAP mail synced on the left. Then app restarted and the same emails are downloaded again - on the right.
It's redownloaded every few minutes.

Thanks for the report. Can you provide a bit more detail? Like which version of TB are you using? Do you have offline storage enabled for the re-downloading folder, imap server type, check for new messages poll interval setting, etc.

Also, an IMAP:5,IMAP_CS:5 log would be helpful. See https://wiki.mozilla.org/MailNews:Logging

Enabled the log, still downloading now. The log has 2 GB after 5 minutes.
I think it's eating the HDD space too because I ran out of space on Windows.
Yes I have offline storage enabled.
It's IMAP with SSL/TLS. The settings are default: check for messages on startup enabled, check every 10 minutes, allow immediate server notifications.

(In reply to gene smith from comment #30)

freeman3, The attachment shows 8 accounts. Are they all on different imap servers or the same server? That shouldn't really matter but who knows.
Do you have other accounts that don't download messages on startup and every few minutes?
Did this start happening after a tb version change?

Yes, almost all the accounts are on the same server.
Yes, I did happen probably after version change recently when I noticed that it's always downloading.

94.0b2 on Linux, b3 on Windows, same problem

Arthur,

(In reply to Arthur K. [He/Him] from comment #103)

(In reply to Richard Leger from comment #100)

In your Preferences does the option "Allow antivirus to quarantine individual incoming messages" appears ticked or unticked at your end?
If ticked, try to untick, restart TB then Compact folder you are seeing issue with, then Repair it... does it fix?
Source: Bug 1734843 Comment 53, Bug 1734843 Comment 67

Well, I don't know if un-ticking the box is helping or not but I went down the line and ran a Compact and Repair action on every folder one at a time.

How was the option set when you first checked at it: checked or unchecked? Important to know...

If checked, did you unchecked it? Important to know...

After you unchecked it, did you restarted TB? Important to know...

Did you run Compact on the folder after the restart?

Did you wait for compact process to complete? Status bar or Activity Manager (not sure it shows in it)...
While compacting, did you checked if cached files were reducing in size?

Did you run repair after the Compacting had completed?

Someone confirmed to me by email that unticking option and restart was sufficient to fix the issue for them...
Arthur in your case may be due to another issue perhaps...

An alternative for you if above does not work, may be to exit TB and manually delete the INBOX, INBOX-1, etc... and INBOX.msf file in your profile cache... this would immediately restore disk space... and of course cause re-download of emails... You may have tried already but not without the option above unticked yet... just a though/suggestion...

Otherwise new TB beta shall pop up at the end of this week... let see if that may help... or not... in your case...

Regards,

I did not restart it yet but I think just unchecking it helped. I think I can test it on another computer later.

I meant with stopping downloading, not with reclaiming space.

(In reply to Richard Leger from comment #106)

Arthur,

(In reply to Arthur K. [He/Him] from comment #103)

(In reply to Richard Leger from comment #100)

In your Preferences does the option "Allow antivirus to quarantine individual incoming messages" appears ticked or unticked at your end?
If ticked, try to untick, restart TB then Compact folder you are seeing issue with, then Repair it... does it fix?
Source: Bug 1734843 Comment 53, Bug 1734843 Comment 67

Well, I don't know if un-ticking the box is helping or not but I went down the line and ran a Compact and Repair action on every folder one at a time.

How was the option set when you first checked at it: checked or unchecked? Important to know...

If checked, did you unchecked it? Important to know...

After you unchecked it, did you restarted TB? Important to know...

Did you run Compact on the folder after the restart?

Did you wait for compact process to complete? Status bar or Activity Manager (not sure it shows in it)...
While compacting, did you checked if cached files were reducing in size?

Did you run repair after the Compacting had completed?

Someone confirmed to me by email that unticking option and restart was sufficient to fix the issue for them...
Arthur in your case may be due to another issue perhaps...

An alternative for you if above does not work, may be to exit TB and manually delete the INBOX, INBOX-1, etc... and INBOX.msf file in your profile cache... this would immediately restore disk space... and of course cause re-download of emails... You may have tried already but not without the option above unticked yet... just a though/suggestion...

Otherwise new TB beta shall pop up at the end of this week... let see if that may help... or not... in your case...

Regards,

Originally it was checked on and I unchecked and restarted TB. TB took a good couple hours re-downloading messages and doing background tasks but finally settled and disk use is at 72GB free. Still must be something amiss to go back down to 72GB free after all is said and done. I'll keep an eye on it.

I also installed the 94.0b4 build and, indeed, I'm not seeing it re-downloading messages as with previous betas so that issue seems to be gone. Will keep an eye on it the rest of the day.

Glad to hear it may be fixed at your end...

(In reply to Arthur K. [He/Him] from comment #109)

(In reply to Richard Leger from comment #106)
Originally it was checked on and I unchecked and restarted TB. TB took a good couple hours re-downloading messages and doing background tasks but finally settled and disk use is at 72GB free. Still must be something amiss to go back down to 72GB free after all is said and done. I'll keep an eye on it.

It is likely that you set TB to automatically Compact the folder (manually triggered via Folder > right click > Compact) which may have automatically reduce the size of the cache file.

I also installed the 94.0b4 build and, indeed, I'm not seeing it re-downloading messages as with previous betas so that issue seems to be gone. Will keep an eye on it the rest of the day.

Your feedback seems to confirm once more that Bug 1734843 and its fix are likely to fix this bug.
As per Bug 1734843 Comment 70, fix was pushed in 94.0b4...

So anyone encountering re-download issue shall try first to upgrade to 94.0b4 see if that help sort the problem by itself.

(In reply to Richard Leger from comment #110)

Your feedback seems to confirm once more that Bug 1734843 and its fix are likely to fix this bug.
As per Bug 1734843 Comment 70, fix was pushed in 94.0b4...

So anyone encountering re-download issue shall try first to upgrade to 94.0b4 see if that help sort the problem by itself.

To a point. I have re-enabled the Antivirus checking feature again to make sure I am at the same config as I was when things were going south. Thus far things are holding steady, no decrease in disk size, etc. Seems quiet, which is good thus far. I think the Antivirus thing might be a red herring. I do have it set to auto compact folders but only when it saves over 1GB (1000MB) so probably not related to that.

(In reply to Arthur K. [He/Him] from comment #111)

I think the Antivirus thing might be a red herring.

I agree it could (or not) that is why I am asking you to check as you encounter the issue first hand and you are eager to test and provide feedback.

(In reply to freeman3 from comment #105)

Yes, I did happen probably after version change recently when I noticed that it's always downloading.

94.0b2 on Linux, b3 on Windows, same problem

Sorry for the late reply. I also see the problem on linux with 94.0b2 with "antivirus quarantine" selected. Un-selecting it or moving to 94.0b4 should fix the continuous re-downloads.

Depends on: 1737330

(In reply to gene smith from comment #113)

... Un-selecting it or moving to 94.0b4 should fix the continuous re-downloads.

I agree.

Does anyone disagree, so that we can close this out?

Hello Wayne, everyone,

While running TB94.0b3 I created a new profile, moved my offline (local) folders to the new profile and ran a repair on those local folders. Things are running smoothly since then.

I kept a copy of my old profile (the one I got the issues with in the beginning with TB94.0b1) for testing purposes. I just launched TB94.0b5 with that old profile and I don't see the issue of repeated downloads anymore. There might be another issue though. I have a form that sends emails twice. So instead of receiving one email I receive two with the same content and same date/time. Those emails are merged (their content is displayed twice in the email body) when running TB94.0b5 with the old profile but they appear as separate emails in the new profile. This is another issue so I believe the current bug can be closed.

Hey Ness Blackbird, is this working for you?

Flags: needinfo?(evekitas)
Summary: Unnecessary mail synchronization with TB91 → Unnecessary gmail synchronization with TB91

(In reply to Worcester12345 from comment #23)

I see this with Yahoo Mail also. Very annoying. Sometimes, just closing and reopening Thunderbird will fix it, but not always.

(In reply to:)
Summary: Unnecessary mail synchronization with TB91 → Unnecessary gmail synchronization with TB91

???

(In reply to Worcester12345 from comment #116)

Hey Ness Blackbird, is this working for you?

Ness wrote me today, "I have turned synchronization back on for several folders, and I'm not observing the problem. It didn't seem to happen constantly, but I'd have expected to see it by now. I'll let you know if it comes back."

Flags: needinfo?(evekitas)

We seem to have lost Paul, but Ness declares WFM (at least tentatively), so closing WFM. And adding bug dependency notations for the injected beta issue caused by bug 1717147, which is also now gone.

Removing bug 1737330 because the crash signature exists in other versions, and indeed in Firefox, so I think it's occurrence during this bug is circumstantial.

Thank you all for your tenacity.

Status: NEW → RESOLVED
Closed: 3 years ago → 3 years ago
Depends on: 1734843
No longer depends on: 1737330
Flags: needinfo?(paul)
Regressed by: 1717147
Resolution: --- → WORKSFORME
See Also: 1734843 →
Summary: Unnecessary gmail synchronization with TB91 → Unnecessary mail synchronization, with TB91. (Then side issue with 94 beta)
Whiteboard: [beta see bug 1734843] → [beta case only: regressed by 1717147, fixed by bug 1734843]

Hey, it's pretty cool that you all fixed this so fast! It bugged me for a long time.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: