Closed Bug 1734843 Opened 3 years ago Closed 3 years ago

downloading messages all the time when "Allow antivirus clients to quarantine" is enabled (mailnews.downloadToTempFile=true)

Categories

(MailNews Core :: Networking: IMAP, defect, P1)

Product:
MailNews Core
Component:
Networking: IMAP
Version:
Thunderbird 94
Platform:
Unspecified
All
Type:
defect

Tracking

(thunderbird_esr91 unaffected, thunderbird93 unaffected, thunderbird94+ fixed)

Status:
RESOLVED FIXED
Milestone:
95 Branch
Tracking Flags:
Tracking Status
thunderbird_esr91 --- unaffected
thunderbird93 --- unaffected
thunderbird94 + fixed

People

(Reporter: alliancemillsoft, Assigned: benc)

References

(Regression)

Details

(Keywords: regression)

Attachments

(1 file, 1 obsolete file)

Bug 1734843 - Mark offline-stored messages correctly when using quarantining (mailnews.downloadToTempFile=true). r=mkmelin
48 bytes, text/x-phabricator-request
rjl
: approval-comm-beta+
Details | Review

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0

Steps to reproduce:

The beta 93.0b3 auto updated to beta 94.0b1 and since it takes minutes to load even a simple text email so I can read or work with it. Also don't seem to be able save attachments. I literally cannot use this like it is. When I try to uninstall and reinstall version 93.0b3 I am told that my profile has been changed due to the new release of Thunderbird so I cannot revert back without damaging my profile.

Actual results:

it takes minutes to load even a simple text email so I can read or work with it. Also don't seem to be able save attachments. It seems to be perpetually downloading emails over and over again.

Expected results:

normal operation in clicking on, reading, and replying to emails.

Do you have any non-gmail accounts, and is it happening to them?

Flags: needinfo?(alliancemillsoft)

Sure you don't have disk full or other disk problems?
It's possible to use the old profile with --allow-downgrade (as long as the data is compatible, it is not fully so back to 91 anymore, at least foldercache has changed)

Sorry, I've been trying to report this to discourse ( https://discourse.mozilla.org/c/thunderbird/beta/257 ) over the last few days. I never was able to setup an account to report there, despite numerous attempts. Really frustrating. The problem is worse than what's above, which was my initial thoughts on the problem. A truer explanation is this ..

Beta 94.0b1 is corrupting my INBOX files within minutes
I am having problems with a series of the latest betas for Thunderbird, 94.0b1. This problem seems was present in 93.ob2 and 3 too, I believe. Both releases seem to perpetually download emails so if I were to try to repair a folder while online I cannot.

I am finding that my INBOX file, not inbox.msf is being corrupted. I have fought with this issue for days. Fortunately I have a "spare" copy of my gmail account that I keep in TB as well and I can use the INBOX file from that account to restore things momentarily. Only then am I able to view an actual email and interact with it.

Then to make matters worse, when I try to revert to an earlier TB install I get a message telling me that my profile account has been changed by the latest betas and my profile will be corrupt if I install the earlier release. I do not continue with the install of the earlier release at that point. That means that I am stuck.

Please advise on how to get out of this loop or on how to correct issues with the 94.0b1 beta. My drive has plenty of space, as do my email accounts.

My profile is QUITE old. I cannot tell how old, exactly butI've used Thunderbird for over a decade and probably 2 decades.

Same issue on my side. Thunderbird is continuously downloading emails from server. This is the case for both my gmail and non-gmail accounts. I have IMAP email accounts hosted at OVH.
Anything I can do to let you investigate further?

PS : my disk is not full I still have 18gb free

Anything I can do to let you investigate further?

FWIW I don't think profile age is a factor for this problem.

I installed release 93.0b5 and the issue with setting emails as read is not occuring anymore. But Bug 1734843 is stille here, I will update the other ticket. Attaching the screencast to the ticket.

Attached file Downgrading to TB93.0b5 removes the issue (obsolete) (deleted) —

I downgraded to TB93.0b5 and the issue remained. This issue wasn't here before I upgrade from TB93.0b5 to TB94.0b2. You can watch the video I uploaded on another ticket and it shows the continuous polling of emails : https://bug1734843.bmoattachments.org/attachment.cgi?id=9245553

Sorry I mixed up my comments. Can someone erase my comment which is supposed to be on another ticket : "I installed release 93.0b5 and the issue with setting emails as read is not occuring anymore. But Bug 1734843 is stille here, I will update the other ticket. Attaching the screencast to the ticket."
Sorry

For this bug downgrading to 93.0b5 doesn't solve the issue

So is there a way I can use 91.1.1 and not destroy my profile? That was a stable beta for me, by my memory. If I do it what part of the profile is destroyed?

I want to add this as a clue .. IF a folder is not opened it seems that the email container file is not corrupted. It seems to happen when the email container file is being appended, seemingly added to. I'm not sure if deleting emails negatively affects the email container file.

I am back at my PC and still running 93.0b5 and it seems the continuous download of messages stopped.

Camile

Have you experienced trying to open an email and only getting a white screen and the email not being marked as read only? Sometimes I can wait 5 minutes or so and an email will eventually appear but then eventually emails never appear. Do you have ANY of this with 93.0b5?

Breezymozilla,
Nope I haven't experienced this behavior.

One more hint .. while TB is closed if I originate an email from another program it is composed in a TB compose window and when I send it it DOES get appended into the INBOX file, as I have the program setup to do with sent emails. It does this without hesitation.

If I have TB open though it site there trying for a LONG time to add that email to the INBOX file and I think it fails at that.

It might be valuable to compare the email file appends of these two separate modules.

Just downloaded and installed 94.0b2 (latest beta) and running with a profile containing o365 and gmail accounts. Don't see a problem. If this is imap related, an IMAP:5 log needs to be recorded and attached above: https://wiki.mozilla.org/MailNews:Logging

Please also include info on exactly what you were doing in TB and what you saw while recording the log.

Not sure that I completely follow what you were saying in comment 19 above. I sent an email from another client to the gmail account running in tb 94.0b2 and received the email OK into Inbox of gmail. Gmail also has lots of message in Inbox (almost 50k) and I can open and save attachments OK.

In reference to comment 19 .. Since this issue, along with my inability to revert back to a known good beta, I've been handling my emails via the Gmail webpage for the last few days. I do not want to chance corrupting my folder email files. Before that, while using TB beta releases 94.0b1 and the 93 series of betas any sent email would sit on screen while TB attempted to add them to the INBOX, which is where I put all sent emails in order to organize email conversations. I would often have to cancel the copying to INBOX process to get it off screen.

Comment 19 is directed towards this long pause/inability for the betas to copy the sent emails to the INBOX. However, I have other applications that use the default email client which is still TB and when I compose an email from one of those apps WHILE TB is closed, there seems to be no problem appending these sent emails to the INBOX file. I can tell because they are located in my inbox in the Gmail webpages after the process.

I was hoping that explaining this might someone to understand where this appending of the INBOX file is going wrong. I hope this explanation is clear enough. If not I'm happy to do more.

I did just download 94.0b2 and am testing that over the next few hours. It is downloading a lot of messages at the moment. Hopefully this isue is fixed in this new beta release.

I'm still not following this:

However, I have other applications that use the default email client which is still TB and when I compose an email from one of those apps WHILE TB is closed, there seems to be no problem appending these sent emails to the INBOX file.

What do you mean by "other applications" and "default email client which is still TB" and "while tb is closed"? Do you somehow have TB built into another app?

How have you configured TB to save sent messages to INBOX? I just configured 94.0b2 to save sent mail to inbox and it worked fine. This was on o365. Also tried it on gmail and it worked OK there too with 94.0b2.

I just saw that someone had better results directly downloading and installed 94.0b2 rather than via the auto-update. You can get it here: https://archive.mozilla.org/pub/thunderbird/releases/94.0b2/
But maybe that's where you just downloaded it from...

Is the compacting of folders supposed to take all resources? I agreed to the compacting of folders, and also making it an automatic process. Is that what would initiate a situation where I click on an email, even a simple one, and it takes maybe 2 minutes for it to display?

Sorry Gene, I'll try to explain it better. So imagine that Thunderbird is closed. You're browsing and a web page has an email link. When you click that link a TB compose window comes up even though Thunderbird is still closed. When you send that email it is sent and then it is written to the specified sent email location for the email account. I set mine to the INBOX of that account so that my conversations are in one place (for the most part) and threaded. When I did that thru Thunderbird while open (94.0b1) the compose widow would just sit there after sending the email because it couldn't write to the INBOX file, presumably. It would eventually give me a warning that the email had not been saved to the sent emails location.

However if TB was closed it would send the email from the compose window and write to the INBOX without issue, and promptly.

Does that help? BTW, I did download the new beta, and updated that way.

I have no special implementation of Thunderbird with other apps BTW.

BTW AFTER compacting, and having the process appear complete there is a delay in displaying emails from time to time, as much as maybe a minute or two.

Okay, here it is, many hours after installing 94.0b2 and the program is STILL downloading emails, over and over again. Not good.

I see what you mean looking at the log. Every time tb checks for new mail it downloads a set of messages for some reason. I haven't tried to run the beta on the old profile yet but with a new profile it works fine for me with my charter account (same as yours).

One thing I notice in your log is that on every check for new main (every minute) tb does a select. At one time about 5 years ago charter had a bug where they didn't detect new mail with an IMAP noop like other servers. A special feature was added so that a select would occur to detect new mail for charter. However, FWICT, charter has fixed this (or Intermail their server supplier actually fixed it). So you might try this:

Go into Preference/General at at the bottom there is "Advanced Preferences". Type in force_select and it change it to no. (If the fix is in effect, this will be set to yes-auto or yes.) Setting to no will prevent the old charter fix from happening and new mail check will again be done with just noop. Be sure to restart TB after making this change.

I'm not sure this will affect anything but worth a try. I'll keep trying to duplicate what you see.

Re: comment 24. Ok, now I understand what you mean but I haven't really tried what you describe (opening a compose window via a web link and sending a message with or without tb initially running). I guess that's a secondary problem so I'll try that when we find out what the primary problem is (i.e., re-downloading messages).

Re: comment 23,25: I don't know of a problem with compacting. For me it only usually occurs after I have mass deleted a lot of email and seems to occur almost instantly. Again, I guess this is a secondary issue. Also, I haven't tried setting to automatic so I'm always prompted if it's OK to compact.

Re: comment 26: I'm not sure if the "many hours" is the time it took for the tb beta to download or the running time required for it to start re-downloading. For me, the download via charter (now spectrum) only takes a few seconds.

Go into Preference/General at at the bottom there is "Advanced Preferences"

Sorry, it's actually called Config Editor... (But the tab that appears is called Advanced Preferences.)

One more thing. There are actually 3 items that match force_select. The ones to change looks like this: mail.server.serverX.force_select where X is a number index 1, 2, 3.... for the account. Also, since you have several accounts on charter, you probably should change all of them to no since I can't tell easily from the log which account is re-downloading constantly (but it appears to just be one of them).

For me, the force_select thing doesn't seem to cause a problem for charter account no matter how it's set, so probably a red-herring.
The account that is re-downloading on every message poll is this: imap://firstname%2Esurname%40charter%2Enet@mobile.charter.net. (Maybe you have disguised the names in the log, not sure.)
So a thing to try is to disable polling for that account and see if the downloading stops after tb restart. To disable polling go to the account firstname%2Esurname and uncheck "check for new messages every X min" and restart.
Also, you might try disabling polling for all the charter accounts and then enable just one at a time to see which account causes problems.

If you can isolate the problem to a single charter account, you might need to delete the mbox and .msf files for that account (probably just for Inbox since that's the folder with messages being re-downloaded) and when tb starts up it will recreate the files while downloading all the messages when you open the folders.

Sorry, I really messed up. My comment 27 and others later is actually referring to a similar bug for a charter user bug 1735317 and the log he recorded. So ignore the comments about "force_select" since it pertains to that bug and should not apply to gmail accounts.
At this point, about all I can do is ask you to record an IMAP:5 log as described here and previously requested in comment 20 above: https://wiki.mozilla.org/MailNews:Logging

Okay, it's a new morning and upon waking my computer, I click on a simple email and it still hasn't displayed some 12 minutes later. Is there a way to copy the items in the activity manager so I can post them here? After reading about logs with the link Gene provided, setting up that log file to record sounds real complex and it doesn't work if you're running both TB and FF, which I need to do.

I'm really concerned now that my INBOX type files are being corrupted. For anyone else in my situation I would say that you should NOT choose to compact folders. Actually I wish I hadn't done this 94.0b2 release.

What is actually in my profile and if I install an earlier stable release like 91.xx will my old profile be left intact and a new profile created or will it just destroy my existing profile?

Upon program restart it's downloading all the time and over 20 plus minutes it still cannot display an email from my inbox, other than the first email in the list, which was not the email selected, and was also not the original first email in the inbox.

When I install TB or TB updates betas does it replace ALL files or only certain ones, if they exist? This is really getting beyond terrible ..

I have noticed that there are no message threads anymore in my inbox as well. INBOX seems to be truly corrupted. 4000 plus emails. That INBOX file that is in use/displayed/selected in TB is 6,836,705 in size. My backup INBOX file is 3,605,134

Copying my backup INBOX and inbox.msf file to the active email account, and then reopening TB doesn't help the problem. Maybe my backup is corrupted too? Both INBOX files (active account and backup) receive emails and should be the same size.

Also I renamed the defective INBOX files to INBOXBAD and upon opening TB, it deleted them. Normal behavior I guess but it might make troubleshooting that much more complicated.

Previously this corrupt behavior was found only on the actively displayed account .. the account selected to be displayed in TB. I'm not sure about that now .. maybe both are corrupted or the program corrupts the file as soon as TB is opened? Not sure.

I sure wish I could get some guidance on what happens to a profile when I revert to an earlier version.

I tried to find my bug report in bugzilla but I guess it doesn't show unconfirmed bugs in the lists, maybe????? Does anyone else have this issue or similar issues? Seems like it but I cannot find the bug reports to tell.

I wouldn't worry too much about the INBOX and INBOX.msf files getting corrupted since whatever they contain is still on the gmail server and tb just mirrors the content locally. If you shutdown tb and delete both of them (but make a backup before deleting them if you are concerned) and then restart tb they will be recreated when tb tries to check new mail in Inbox. Of course, this will take some time for the download to occur depending on how much is in Inbox.

The alternative to this the the "repair" function but I find this manual method of just deleting the files to be more effective. But then I've only had to do this rarely when I am tweaking the tb code and I've messed something up myself.

Do you have tb setup to check other folders for new mail? If not, then Inbox should be the only folder that is accessed on tb startup if you shutdown tb with Inbox selected.

Note: If you see other files with number suffixes like Inbox-2 or Inbox-2.msf you should delete them too. (I think the highest suffixed files is the one actually in use so you might want to back them up too.)

Is there a way to copy the items in the activity manager so I can post them here?

I don't know but I've never been able to get useful info from activity msg so probably wouldn't help.

After reading about logs with the link Gene provided, setting up that log file to record sounds real complex and it doesn't work if you're running both TB and FF, which I need to do.

I run TB and FF together all the time and do TB logging with no problem. There was one corner issue but it was a very special case that shouldn't apply to a short term IMAP:5 logging session. You just create a bat file and run it.

I sure wish I could get some guidance on what happens to a profile when I revert to an earlier version.

If you start an earlier tb version with the --allow-downgrade option it works fine for me. I don't see a problem. You may need to add this to the "shortcut" parameters that you click on to start tb. However, let try to fix the problem you are seeing using the latest beta before reverting to the earlier version.

Thanks Gene. Regarding the --allow--downgrade switch, do you have to do that every time or just the first time of the older version?

I'm not sure, maybe just one time is OK. I always run tb with that parameter. You can just set it where the link is configured to run tb in windows. I think called a shortcut. I'm on linux so not sure what it's called or how to set it.

I tried to find my bug report in bugzilla but I guess it doesn't show unconfirmed bugs in the lists, maybe?????

It show them all, even if unconfirmed.

Does anyone else have this issue or similar issues? Seems like it but I cannot find the bug reports to tell.

I think your issue is mostly the same as bug 1733966 if you are also seeing re-downloading. Not sure about the slow-down.
Also maybe same as this: bug 1735317

See Also: → 1733966, 1735317

(In reply to breezymozilla from comment #13)

So is there a way I can use 91.1.1 and not destroy my profile? That was a stable beta for me, by my memory. If I do it what part of the profile is destroyed?

see comment 2. prior releases beta and version releases can be found at http://archive.mozilla.org/pub/thunderbird/releases/

Component: Untriaged → Networking: IMAP
Product: Thunderbird → MailNews Core
The content of attachment 9245553 [details] has been deleted for the following reason:

Attachment deleted for privacy concerns

Thanks David!

Repost from bug 1735317 -

Problem solved – Created a new profile and the issue was fixed.

Thanks everyone for your ideas. I tried most of the ideas from Bug 1734843 (except rolling back to 93.0b5) without success.

Steps to fix the problem:

  1. From cmd prompt – "C:\Program Files\Mozilla Thunderbird\thunderbird.exe" -p
  2. Create new profile and start TB
  3. Add my four (4) charter.net mail accounts and TB automatically syncs Inbox and Sent folders with the charter server – success.

To bring over the address book and calendar from the old profile:

  1. Copy file abook.sqlite from old profile to new profile
  2. Copy entire folder calendar-data from old profile to new profile
  3. Extract from file prefs.js found in the old profile folder calendar-data, all entries starting with “user_pref("calendar.” (there are 15 lines) and edit file prefs.js found in new profile folder calendar-data, replacing all of the “user_pref("calendar.” entries with those from the old profile prefs.js file.

These entries look like this:

user_pref("calendar.list.sortOrder", "77f3ca3c-133e-492b-a63b-47c9149f7a6f 4011805e-6fb7-4bba-a710-4f17bffb3ff6");
user_pref("calendar.registry.4011805e-6fb7-4bba-a710-4f17bffb3ff6.calendar-main-in-composite", true);
user_pref("calendar.registry.4011805e-6fb7-4bba-a710-4f17bffb3ff6.color", "#a8c2e1");
user_pref("calendar.registry.4011805e-6fb7-4bba-a710-4f17bffb3ff6.imip.identity.key", "id1");
user_pref("calendar.registry.4011805e-6fb7-4bba-a710-4f17bffb3ff6.name", "My Calendar");
user_pref("calendar.registry.4011805e-6fb7-4bba-a710-4f17bffb3ff6.type", "storage");
user_pref("calendar.registry.4011805e-6fb7-4bba-a710-4f17bffb3ff6.uri", "moz-storage-calendar://");
user_pref("calendar.registry.77f3ca3c-133e-492b-a63b-47c9149f7a6f.calendar-main-default", true);
user_pref("calendar.registry.77f3ca3c-133e-492b-a63b-47c9149f7a6f.calendar-main-in-composite", true);
user_pref("calendar.registry.77f3ca3c-133e-492b-a63b-47c9149f7a6f.disabled", false);
user_pref("calendar.registry.77f3ca3c-133e-492b-a63b-47c9149f7a6f.name", "Home");
user_pref("calendar.registry.77f3ca3c-133e-492b-a63b-47c9149f7a6f.type", "storage");
user_pref("calendar.registry.77f3ca3c-133e-492b-a63b-47c9149f7a6f.uri", "moz-storage-calendar://");
user_pref("calendar.timezone.local", "America/New_York");
user_pref("calendar.ui.version", 3);

Now running 94.0b3. Again, thank you for you suggestions and help.

NoJoy,
Glad you fixed it but I guess having to make a new profile is still a bug. So far I still haven't been able to duplicate the issue.

It was mentioned in bug 1733966 that one user seeing the problem has unified inbox. I haven't tried that yet to duplicate the problem but curious if that might be common thread?

This is now confirmed and verified to occur with the push to comm-central on 9-27-2021.
See bug 1733966 comment 76 and later for details.
Note: bug 1733966 actually describes a different issue in the early comments so it's not a duplicate.

Status: UNCONFIRMED → NEW
Ever confirmed: true

Regression from bug 1717147 - https://hg.mozilla.org/comm-central/rev/35e064ada8debd021ce507fd0e8b20be363a9046 - Ben, please have a look ASAP!

Assignee: nobody → benc
status-thunderbird93: --- → unaffected
status-thunderbird94: --- → affected
status-thunderbird_esr91: --- → unaffected
Flags: needinfo?(alliancemillsoft)
Keywords: regression
Priority: -- → P1
Regressed by: 1717147

I am seeing:
"The operation failed because another operation is using the folder. Please wait for that operation to finish and then try again."
when I go to Inbox, properties, "Repair folder".
Yet, when I check in "Activity Manager", there is no activity shown. Seeing no current activity, I go back and try again, and just see the same message. Is it this bug?
Thunderbird 91.2.0 (64-bit)

I saw the same issue and it looks like changing mailnews.downloadToTempFile to false fixed it.
Maybe that's what's different in a new profile?

Quick steps to reproduce:

  1. have an IMAP account (I've just got a local IMAP server, with a single test message in the INBOX)
  2. set "mailnews.downloadToTempFile" to true in the config file
  3. run TB
  4. keep an eye on your INBOX file (ImapMail/imap.blah.com/INBOX)
  5. click "Get Messages"
  6. see that your INBOX file has doubled in size. If you open it in a text editor, you can see the messages have been duplicated.

So when message quarantining is enabled (mailnews.downloadToTempFile=true), checking messages on IMAP causes re-download and duplication in the mbox file. And massive slowdowns if you've got a lot of messages :-(

If you do steps 3-5 with "mailnews.downloadToTempFile" set to false, then it all seems to work fine.

So, quick mitigation in the meantime is to turn off "mailnews.downloadToTempFile" (but be aware that if your OS viruschecker decides it doesn't like an incoming message, it might lock your entire mbox file!)

So almost certainly related to what I did in bug 1717147. Looking into how to fix it now.

None of my tb profiles on linux even have this pref. I guess that explains why I couldn't duplicate what everyone else reported.
Edit: Ok, I see that if I set antivirus setting "Allow antivirus to quarantine individual incoming messages" it creates the pref as true. Then un-setting it sets it back to false. So effectively the pref is defaulting to false it seems.
Also, it was suggest a while back that the pref be true by default but only for windows, but there were issues. Re: Bug 507090

That patch should fix things up for now.
The cause was another chunk of code which assumed it was always writing directly into an mbox.
There'll be followup cleanup of code in the vicinity of this patch, but mostly under my ongoing move-all-the-mbox-code-into-the-mbox-mailstore quest, so I'll open new bugs as needed.

In TB 94.0b3 (64-bit) on Windows 10, I do not see any mailnews.downloadToTempFile option in Preferences > Config Editor.
Does it mean the feature is not present/enabled) or it is and has a default value mailnews.downloadToTempFile=true in the code, that cannot be changed in that version of TB?

In 94.0b2 (64-bit) with an IMAP account created in 68.x and uprated to 78.x, 91.x, 93.0b5 and 94.0b2 (64-bit)... I could see occasionally re-download of messages... but is seems very hard to reproduce at will... at one point enabling the unified view seemed to have triggered it but it could also have been that an elapsed time had passed... or some Get Message auto-triggered... not sure...

Also noticed like Worcester12345 in Comment 51, that while the status bar may show "Download message 1 of 111 in Message Envoyes" while Activity Manager shows no activity... or it does but seems stuck... error console appears as well... so I have filed a new Bug 1736526 for dealing with this separate issue...

How should users go about fixing the duplication issue after getting a build with this patch?

Status: NEW → ASSIGNED
tracking-thunderbird94: --- → +
Keywords: checkin-needed-tb
Target Milestone: --- → 95 Branch

Pushed by mkmelin@iki.fi:
https://hg.mozilla.org/comm-central/rev/5b79e208d056
Mark offline-stored messages correctly when using quarantining (mailnews.downloadToTempFile=true). r=mkmelin

Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Keywords: checkin-needed-tb
Resolution: --- → FIXED

(In reply to mozbugz from comment #58)

How should users go about fixing the duplication issue after getting a build with this patch?

Maybe https://addons.thunderbird.net/en-US/thunderbird/addon/removedupes/

Severity: -- → S2
OS: Unspecified → All
Summary: downloading messages all the time → downloading messages all the time when "Allow antivirus clients to quarantine" is enabled (mailnews.downloadToTempFile=true)

(In reply to Wayne Mery (:wsmwk) from comment #60)

(In reply to mozbugz from comment #58)

How should users go about fixing the duplication issue after getting a build with this patch?

Maybe https://addons.thunderbird.net/en-US/thunderbird/addon/removedupes/

That is an addon that deals with actual messages, not mbox corruption.

If there are multiple messages listed in the folder then the addon will be sufficient.

If there are not multiple messages listed, I don't see an automated solution happening soon, or perhaps not at all. The quick method, for imap folders anyway, is delete the files in the OS and let Thunderbird rebuild them.

Someone mentioned repairing the folder solves the issue. Did you try it ? I can rebuild the case and test it if needed.

Richard wrote:

In TB 94.0b3 (64-bit) on Windows 10, I do not see any mailnews.downloadToTempFile option in Preferences > Config Editor.
Does it mean the feature is not present/enabled) or it is and has a default value mailnews.downloadToTempFile=true in the code, that cannot be changed in that version of TB?

Richard, I think I answered this in my edited comment 54 above. The downloadToTempFile is undefined until you set/check the Antivirus option "Allow antivirus to quarantine individual incoming messages".

An undefined pref option is effectively false so I would assume that users who saw this bug with 94b had this option set/checked in the past and left it set. User who have this option not selected/checked don't see a problem with 94b, e.g., me. Maybe Camille can verify this.

Camille wrote:

Someone mentioned repairing the folder solves the issue. Did you try it ? I can rebuild the case and test it if needed.

If this bug causes duplicate messages, I doubt if repair will fix it. I think Wayne's 2nd suggestion in comment 62 is best: just delete the mbox and .msf file for the problem folder(s) and restart tb and let it rebuild them. And if there are mbox and .msf that have number suffixes, delete them too.

(In reply to gene smith from comment #64)

Richard wrote:

In TB 94.0b3 (64-bit) on Windows 10, I do not see any mailnews.downloadToTempFile option in Preferences > Config Editor.
Does it mean the feature is not present/enabled) or it is and has a default value mailnews.downloadToTempFile=true in the code, that cannot be changed in that version of TB?

Richard, I think I answered this in my edited comment 54 above. The downloadToTempFile is undefined until you set/check the Antivirus option "Allow antivirus to quarantine individual incoming messages".

I have this option unchecked... so not enabled/false.

An undefined pref option is effectively false so I would assume that users who saw this bug with 94b had this option set/checked in the past and left it set. User who have this option not selected/checked don't see a problem with 94b, e.g., me. Maybe Camille can verify this.

I did though see occasional re-download at my end with one gmail imap account... but not as bad and frequent as the others... nothing that I can reproduce at will... it just sometime happens... so while the above parameter may contribute to the issue... that may not be the only cause of it... just for your info... but let see how this but is fixed first... then we can revisit if need be...

What will be the first available beta that addresses/fixes this problem and supports the new profile layout?

Compact removed all duplicate messages for me. Repair also works because it deletes the msf and mbox file.

(In reply to breezymozilla from comment #66)

What will be the first available beta that addresses/fixes this problem and supports the new profile layout?

If all goes well, Thurs-Fri this week or early next week.

Comment on attachment 9246574 [details]
Bug 1734843 - Mark offline-stored messages correctly when using quarantining (mailnews.downloadToTempFile=true). r=mkmelin

[Triage Comment]
For Thunderbird 94.0b4.

Attachment #9246574 - Flags: approval-comm-beta+

Believe the comments I have made in bug 1534119 are more relevant here

I have reported tht TB seems to be continuously downloading emails and sometimes I get dupes. Plenty of disk space 40g, profile is same been using for a decade - I do have other copies and alternative profiles, (used for testing TB releases before mainly)

I am using MacBook Retina 15 2013 macOS Catalina 10.15.7 enUK - currently back to 93.0b5 as stable, even after updating to 94 versions via FTP ( as originally thought there might have ben some issue with the "About" prompt and update), I am still getting freeze (stuck ona download of a msg somewhere) then either crashes or I kill TB

Crash reports:
bp-36ca62e1-4c2b-4131-84e1-b535f0211020.txt
bp-2360d486-90fc-4998-bc3e-dcf3b0211020.txt

from yesterday evening - 2nd one self-generated when not at the floptop

I've been working with 94.0b4 and the continual downloading seems to be continuing. What are nstmp files? I have nstmp files 1 thru 3 and one of them is a staggering 32 gigs. My inbox this morning has swelled from 4.5 gigs to 5.3 gigs in just the last hour. I also have a suspect INBOX file that 10.8 gigs. Something is still going on here.

Would anyone recommend that I delete some of these files and let them rebuild themselves? If so should I delete the ones without file extensions or the msf files?

I just got this message as well .. The folder "important" on xxxxxx cannot be compacted because another operation is in process. I got these messages with earlier betas as well.

Lastly when I ran the new beta for the first time it occupied all of my internet bandwidth .. 500 mps.

I will continue to try to work with this beta today and learn more about it.

I have found 1 email that are actually 2 emails bound together as well. Would someone like me to forward it somewhere?

Actually it's one email that contains numerous emails from a variety of sources. Those emails contained appear to come thru as individual emails as well.

Actually I have a number of these emails which are really numerous emails in a single email container/file. Funny the email size does not appear correct either. It looks like the size info is from the first email in the set. This obviously creates problems such as being able to reply to these things. While I will spend the day working in TB I will have to reply via my gmail account.

Could this be what's really happening with these large downloads? .. emails getting stuck together and being downloaded many times to fit into these email containers/files?

(In reply to breezymozilla from comment #72)

I just got this message as well .. The folder "important" on xxxxxx cannot be compacted because another operation is in process. I got these messages with earlier betas as well.

I see this one a lot. I figured it is getting mail, or filters are doing their thing or something. Could this be a filters issue?

Now TB won't display an email, seemingly unable to load the email fully. Even simple emails that I opened only an hour ago are verrrrry slow to load/display. Also some do not fully complete loading after many minutes and they are not marked as opened either.

A few minutes later, and without starting TB I can now open these emails normally. I think that compacting might be using all the resources when emails won't fully load. I'm also getting highly stylized, complex emails showing up as code.

BTW, these email containers are new emails, just loaded over the last 2 or 3 hours.

I'm having lots of trouble with my IMAP account and filters on the latest beta (94.0b4 64-bit). Sounds a lot like the stuff I'm reading in this thread. This version is almost unusable for me. I don't really want to go thru the hassle of trying the various fixes suggested and would rather just leave the beta channel and revert to the most recent stable release. Can anyone point me to what I need to do to accomplish this?

I found this https://thunderbird.topicbox.com/groups/beta/Te8c2ccf26e15a3ee/94-0beta1-and-on-thunderbird-repeats-downloading-emails-starting-again-every-2-minutes and the suggestion there to simply disable "Allow antivirus to quarantine individual incoming messages" was all that I needed to do. At least so far this seems to allow everything to work right again. Fingers crossed!

(In reply to breezymozilla from comment #72)

I've been working with 94.0b4 and the continual downloading seems to be continuing. What are nstmp files? I have nstmp files 1 thru 3 and one of them is a staggering 32 gigs. My inbox this morning has swelled from 4.5 gigs to 5.3 gigs in just the last hour. I also have a suspect INBOX file that 10.8 gigs. Something is still going on here.

I have one nstmp file (4.1G) from back on Sept 22. Looking at code, it appears to have something to do with compacting folders. However it appears bigger than any one folder in the account. I only see it on one account.

Are you seeing file nstmp-1, nstmp-2, and nstmp-3? I only see just nstmp.

Would anyone recommend that I delete some of these files and let them rebuild themselves? If so should I delete the ones without file extensions or the msf files?

I, personally, with tb shut down, would just delete everything under the problem account and let tb 94.0b4 do a clean re-download and rebuild of all the files. Sounds like everything got really messed up with the previous beta and the attempt to make it work, including some failed compacting leaving the nstmp files.
I am referring to all the files under:
<profile root>/kgmivl5o-etc/ImapMail/<account name>/
(You can make a backup first if you are paranoid but everything should still be on the server.)

Then when you restart tb, it will download everything from the server and rebuild the account files.
The alternative is to just create a new profile, but any previous setting will be lost. Just deleting the account's Imap files won't affect previous settings.

So leave the folders intact and delete the msf and non extension files?

BTW, I also lost all of my conversation threading. Hopefully that comes back after a rebuild. I'll do all this tonight.

I've decided to hang onto my profile since the layout was recently changed.

Thanks

(In reply to breezymozilla from comment #82)

So leave the folders intact and delete the msf and non extension files?

Pretty much everything, including folders (but not the account folder itself. So in windows file explorer, drill into the account folder, select everything, then unselect msgfilterrules.dat, then shift+delete.

BTW, I also lost all of my conversation threading. Hopefully that comes back after a rebuild. I'll do all this tonight.

Threading will return. However, custom folder colors and custom column order and sorts will be lost. (easily redone)

Depends on: 1737330

(In reply to Wayne Mery (:wsmwk) from comment #83)

(In reply to breezymozilla from comment #82)

So leave the folders intact and delete the msf and non extension files?

Pretty much everything, including folders (but not the account folder itself. So in windows file explorer, drill into the account folder, select everything, then unselect msgfilterrules.dat, then shift+delete.

By doing this (deleting all files under /ImapMail/<account name>/) I have now got Thunderbird into a state where it hangs indefinitely when I try to open a message.

There is an msf file with the name of <account name>.msf in the /ImapMail/ folder, above the /<account name>/ folder. Should that file also be deleted?

(In reply to mozbugz from comment #85)

There is an msf file with the name of <account name>.msf in the /ImapMail/ folder, above the /<account name>/ folder. Should that file also be deleted?

Tried that, didnt help, still hanging.

I would not have expected that. And I'm guessing <account name>.msf was created on your first attempt the deletion - something definitely wrong there.

I suppose the next, after saving off msgfilterrules.dat if you have filters, is start Thunderbird with -offline switch, delete the account in account manager, and add it back.

<account name>.msf in the top-level folder (/ImapMail/) was there already in the beginning. I dont have any filters. Guess its time to re-create the account.

I haven't thought this through, but can we construct an automated test that generally ensures a message doesn't get stored twice, i.e. that a folder doesn't grow for situations where we know the message already exists in the folder? Or if not a generalized test, at least one specific to this incident?

I'm pretty sure we've had one or more similar issue several years ago

Flags: needinfo?(benc)

I have the same behavior. Actually this beta 94. 0b4, seems to have some kind of memory useage problem, using tons of CPU and never quitting. It's so unstable that I haven't been able to use it since I deleted some of these files to rebuild my inbox. I only deleted the files without file extensions, leaving the msf files. Maybe that's where I went wrong.

In any case it hangs trying to open an email and on complex emails (stylesheets, etc) it never gets those emails to fully open.

I am gonna try again, this time also deleting my msf files too.

This has gone so wrong that it makes me question the changes to the user profiles from this beta string. Where do the new profiles begin? Is it 93x or 94x?

breezymozilla wrote in comment 90:

I have the same behavior. Actually this beta 94. 0b4, seems to have some kind of memory useage problem, using tons of CPU and never quitting.

Not sure what you're seeing. I just update from 94b3 to 94b4 and see no excess cpu or memory usage (but didn't have to re-download). One thing it might be is when you re-download all you emails it looks like new stuff and re-indexing for global search occurs which, for me, is pretty cpu intensive. (In linux "top" I see 100% cpu when re-indexing major folders and drains my laptop battery faster than usual).
You can switch indexing off at "Enable Global Search and Indexing" in General settings. But I think you have to restart tb for it to stop indexing in progress. And I think when you turn it back on, indexing starts again from the beginning.

I am gonna try again, this time also deleting my msf files too.

Yes, deleting all of both the <folderName>.msf and <folderName> files is what I would recommend.

Wayne wrote in comment 82:

Pretty much everything, including folders (but not the account folder itself. So in windows file explorer, drill into the account folder, select everything, then unselect msgfilterrules.dat, then shift+delete.

I forgot about the filter rules file which you probably don't want to delete if you have filters set up for the folder. Thanks for pointing that out Wayne!

I would also recommend after starting tb after deleting the files, just keep Inbox selected until it finishes, then move to Sent Mail let it finish then move to others. (There is an "auto-sync" process that should go through all your folders and download them even if you just stay parked on inbox.) If you try to open messages before they have completely downloaded, you may see problems when they are complex. Smaller folder will download quicker of course.

This has gone so wrong that it makes me question the changes to the user profiles from this beta string. Where do the new profiles begin? Is it 93x or 94x?

Not sure I understand the question. You are still using the same profile, I think, but just cleaning up a corrupted account due to problems with 94b1-3.

Try deleting your imap folder and enjoy tb in a broken state

(In reply to gene smith from comment #81)

I have one nstmp file (4.1G) from back on Sept 22. Looking at code, it appears to have something to do with compacting folders. However it appears bigger than any one folder in the account. I only see it on one account.

Sorry to post a comment from the sideline.
I have found an issue with compact code for 4GB or larger folder in C-C tree lately. I am not sure if this is applicable to beta, though.
Bug 1737488

(In reply to gene smith from comment #91)

This has gone so wrong that it makes me question the changes to the user profiles from this beta string. Where do the new profiles begin? Is it 93x or 94x?

Not sure I understand the question. You are still using the same profile, I think, but just cleaning up a corrupted account due to problems with 94b1-3.

My understanding is that the 93.x beta changed the profile so that you cannot go back to earlier versions of TB, though you can use a startup switch to specify running TB in a mode that will accept the modified profiles from beta 93.x and above. It sounds like a recipe for disaster to me so I wanted to stay on a beta where reverting the profile is not required. That's what I was referring to.

Yes, it appears that 93.0b1 changed the profile from using a file called "panacea.dat" to one called "FolderCache.json". I've used the same profile with pre-93.0b1 and post-93.0b1 tb versions with no problem, going back and forth with --allow-downgrade parameter. FolderCache and panacea get created when needed, depending on version. However, I didn't do extensive tests to see if everything continued working.
However, I think once you have run a profile with a newer tb version and then try to go back to the older version, you will see the "must create new profile" dialog unless you use --allow-downgrade option.

"must create new profile" .. Yes, that's the message I got when I first tried to step back off this beta cycle, due to the problems I was having.

I have spent some of the day trying to get 94.0b4 to work as designed. I deleted imap folders and let TB try to rebuild them. It has crashed/frozen numerous times, as well as affecting the entire PC on one occasion. Is there sharing of memory space with Firefox, or system files shared with Firefox. The two seem somewhat connected in memory behavior when both are running.

Lastly, I have many decades .. yep, decades .. of locally archived emails. I have looked where these emails might be stored but cannot find much. In the Mail folder I find a pop.googlemail.com folder which contains an archive folder (what I named my stored emails area) which contains only 1 year of emails. Have I lost all of my locally stored emails? There is also a "local folders" folder which contains only trash and unsent messages. If I've lost all of my archived emails can restore them via my backup program? Maybe this is why I'm having so many problems getting TB 94.0b4 to run properly???

Sorry to beat this thing to death but I need to make sure I do the right steps to get back up and running properly.

The mails will be under something named according to the account you had them in. The mails are in mbox files - which is just one big text file with all the mails after each other. The file has no extension. If the folder is named Example, the file name is Example. With Thunderbird closed, find and move that file to where ever you want it. On restart Thunderbird will find such new folders and show them in the UI.

That's exactly what I expected but the folders are NOT in the profile folders BUT they do show up in TB but right now it's still downloading 306 of 9912 emails in the All Mail box. I don;'t think the folders were ever in the profile. Is there another location that used to used ages ago? I've used TB since it's inception.

FYI, I even reinstalled 94.0b4 for the third time and I'm getting this nearly perpetual downloading of messages. This is a mess right now. Should I be disabling allowing antivirus to quarantine individual emails? It's active right now.

(In reply to Magnus Melin [:mkmelin] from comment #97)

The mails will be under something named according to the account you had them in. The mails are in mbox files - which is just one big text file with all the mails after each other. The file has no extension. If the folder is named Example, the file name is Example. With Thunderbird closed, find and move that file to where ever you want it. On restart Thunderbird will find such new folders and show them in the UI.

Does that also move them on the server? In other words, will Yahoo or other server now show them in the new location?

Now TB 94.0b4 is downloading 3761 of 15,108 "important" emails, which I'm sure I don't have. My total inbox has only 4500 or so emails in it. This release still has numerous issues with downloading emails.

(In reply to Worcester12345 from comment #99)

Does that also move them on the server? In other words, will Yahoo or other server now show them in the new location?

You can't move them that way to under an IMAP account. Just a POP account, or Local Folders.

(In reply to breezymozilla from comment #98)

Is there another location that used to used ages ago? I've used TB since it's inception.

Not really. Check Account Settings | Server Settings | Local Directory

So I have wiped out all of my folders and files for my imap emails and let TB rebuild them. It was a process I started on Friday, and now 4 days later is seemingly complete. However even now TB is downloading 1904 of 2138 emails in my sent box. I should add here that don't even use the "sent" box. I send my sent copies to the inbox. Occasionally when I reply by phone I'll have a few items in my sent box but then I drag them into my inbox to complete a thread. There is still something WAY off with the near continual downloading of emails.

Also while I've been working with this 94.0b4 release and am finding that it still occasionally groups multiple emails into a single email. I will be happy to forward one of these examples should anyone desire it. Interestingly, to me, is that there is a separate header for each email in my list but when I open an email like this I get the original headered email along with a series of email pathing code, or so it looks to me, followed by another email. This pattern can extend for multiple emails in a single email packet.

breezymozilla,
Sorry for the delay in getting back to you. From you previous comments, you are trying to fix the gmail account. You can look at the gmail.com site with a browser and see what is in you "Sent" folder (2138 messages?). You can also configure there which folders are visible via imap, which I would assume gmail's Sent folder is. So tb is trying to download 2138 messages from gmail's Sent folder. You can disable Sent (or any other folder like "Important" or "All Mail" from IMAP access at gmail.com so they are not reported to tb. You can also un-subscribe from any folder in tb so that tb basically ignores the folder. I would recommend at least setting tb so "All Mail" doesn't have offline storage and tb only stores the headers. If you don't care about "All Mail" at all in tb, you can unsubscribe from it.

I think gmail itself puts anything you send via gmail's SMTP server into Sent folder regardless of how tb is set. So that might explain why you have many unexpected messages there getting downloaded by tb.

Also while I've been working with this 94.0b4 release and am finding that it still occasionally groups multiple emails into a single email. I will be happy to forward one of these examples should anyone desire it. Interestingly, to me, is that there is a separate header for each email in my list but when I open an email like this I get the original headered email along with a series of email pathing code, or so it looks to me, followed by another email. This pattern can extend for multiple emails in a single email packet.

This sounds like a completely different bug than this bug "downloading all the time...". I assume by "pathing code" you are referring the the routing information in the email header. Sound like this might be an email with other emails as attachments. I recommend opening a new bug and attaching there the relevant .eml file you have saved for the problem message. Or you can forward the info directly to me and I'll take a look at it.

I am trying to forward one to you but the email says the total forwarded email is 28 mb so be forewarned. It's leaving here really slowly for some reason. These joined emails are very unstable. I think these kinds of emails are what's making TB really unstable here.

(In reply to Wayne Mery (:wsmwk) from comment #89)

I haven't thought this through, but can we construct an automated test that generally ensures a message doesn't get stored twice, i.e. that a folder doesn't grow for situations where we know the message already exists in the folder? Or if not a generalized test, at least one specific to this incident?

Definitely a goal of mine to add tests informed by stuff that's broken in the past.
I'm doing a whole load of surgery on message parsing and storage (moving all the mbox handling into the mbox mailstore code - there are currently mbox assumptions and extra message parsing all over the codebase). So I really want to add tests to sanity-check this kind of thing as I go along.

I'm pretty sure we've had one or more similar issue several years ago
Yes - there was an issue with badly-formed IMAP messages not being marked as stored offline, so they would be downloaded every time. I've got this in mind (i.e "don't assume servers will always send perfectly-formed messages!").

Flags: needinfo?(benc)

So I uninstalled TB and reinstalled it, and deleted the INBOX and inbox.msf files before I started TB. It immediately popped up with my most recent inbox messages but is hanging up my entire pc and has all of the same symptoms I had before .. slow loading of messages, not marking messages as read, etc. I have only been able to view a few messages. It's very unstable.

I asked before but does TB share memory space with FF?

Does anyone also see the symptoms of Bug 1734847 ? TB (beta) 94.0b1 occasionally mis-indexing mail "subject" with previous inbox msg

RE: Comment # 108

No, have not seen these symptoms.

When the duplication issue appeared in 94.0b1, I rebuild my profile on 94.0b2 and turned off allow "quarantine" message based on anti-virus.

In 94.0b3, turned on allow "quarantine" message based on anti-virus and it works.

See Bug 1735317, comment 9 at https://bugzilla.mozilla.org/show_bug.cgi?id=1735317#c9

No Joy

(In reply to breezymozilla from comment #107)

So I uninstalled TB and reinstalled it, and deleted the INBOX and inbox.msf files before I started TB. It immediately popped up with my most recent inbox messages but is hanging up my entire pc and has all of the same symptoms I had before .. slow loading of messages, not marking messages as read, etc. I have only been able to view a few messages. It's very unstable.

I asked before but does TB share memory space with FF?

No, they are separate programs. They may share some common system libraries or DLLs but they don't explicitly share any memory space.

I think the "joined" emails problem you see may be described in Bug 1734847 so I added you to the CC list there.

(In reply to Wayne Mery (:wsmwk) from comment #108)

Does anyone also see the symptoms of Bug 1734847 ? TB (beta) 94.0b1 occasionally mis-indexing mail "subject" with previous inbox msg

I've seen all sorts of mixups.

The latest is the sort order is on date, yet they are not sorting that way. Just seems to be doing its own thing.

What about message subject lines that are not related at all to the contents of the email? Something is getting mixed up somewhere, and I have a feeling it has to do with all this overlapping downloading going on.

Different bug (bug 1734847). Probably not related but still remains to be seen.

Have any of these bugs been addressed in 95.0b1????????????

This one is fixed in 95 yes (see the target milestone)

Blocks: 1733966
See Also: 1733966 →
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: