Closed Bug 1777776 Opened 2 years ago Closed 2 years ago

TB 102 overwrites message content with content from other messages (POP, may involve Compact after filtering)

Categories

(Thunderbird :: Folder and Message Lists, defect, P1)

Product:
Thunderbird
Component:
Folder and Message Lists
Version:
Thunderbird 102
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 1777454

People

(Reporter: andreshko, Assigned: benc)

References

(Blocks 2 open bugs)

Details

(Keywords: dataloss)

Attachments

(2 files)

Monster message created by Thunderbird by merging 2 messages
593.94 KB, image/jpeg
Details
Screenshot 2022-10-04 144633.png
40.39 KB, image/png
Details

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0

Steps to reproduce:

Downloaded messages from a POP server to local inbox.

Actual results:

Messages in local inbox (it is possible that the bug occurs in other or all local mailboxes too) get completely overwritten by Thunderbird 102 with parts of neighboring messages or by truncated content from the same message. The damage is irrecoverable and irreversible - original content is permanently lost.
Message subject still appears in the list of messages, but in the message view pane there is no subject and no message, or just random HTML source parts of another message.
Destroyed messages are OK after downloading and display correctly until overwritten under unclear circumstances.
In one occasion, there was a monster message created by merging 2 messages into one - with a subject line and top part of the message header from the first message and bottom part of the header and broken content of the second message.
"Folder properties"/<Repair folder> does not restore the lost content.

Expected results:

Messages should remain intact after downloading to local inbox.

Try right-clicking Inbox, select properties, and then click on Repair. If it fixes it, it might be the same problem that I saw when compacting folders.

Same problem here, and Repair doesn't fix it.

(In reply to Sam from comment #1)

Try right-clicking Inbox, select properties, and then click on Repair. If it fixes it, it might be the same problem that I saw when compacting folders.

"Folder properties"/<Repair folder> does not restore the lost content.

It may certainly be related to compacting folders, because I frequently use it after automatically filtering messages or manually moving messages to other folders.

Have you read bug 1742975 comment #78? Have you ever run a beta version between 91 ESR and 102 ESR? Have you ever used the "Allow antivirus clients to quarantine individual incoming messages" option?

Current understanding of the TB team is that if you went from 91 to 102 and skipped the various broken betas, you should not have a problem. Maybe your case (and all the other people complaining on SUMO) disproves that assumption.

This should be looked into asap. If this is happening as reported, it's seriously bad. Tentatively marking as P1/S1.
I'm not sure if I'll succeed to reproduce this (e.g. wrt factors Russian language and broken betas), so anyone who can, pls try and give us some steps!

Reporter, does this problem occur for you (ideally on clean install of 102) with ≡ > Help > Troubleshoot Mode…?
Pls make appropriate backups of your messages on server before testing, maybe a test email address.

OT: tracking-firefox103 flags were not set by me, but automatically - probably erroneously caused by S1/P1. I cannot remove them either because I don't see them.

Severity: -- → S1
tracking-firefox103: --- → ?
tracking-firefox103: ? → ---
tracking-firefox104: --- → ?
tracking-firefox104: ? → ---
Flags: needinfo?(bugzilla2007)
Priority: -- → P1
Summary: Thunderbird 102 irrecoverably destroys messages content overwriting them with random content from other messages → TB 102 irrecoverably destroys messages overwriting them with random content from other messages (POP, Russian, may involve Compact after filtering)
Flags: needinfo?(bugzilla2007)
Keywords: dataloss
Flags: needinfo?(benc)

(In reply to Thomas D. (:thomas8) from comment #5)

Reporter, does this problem occur for you (ideally on clean install of 102) with ≡ > Help > Troubleshoot Mode…?
Pls make appropriate backups of your messages on server before testing, maybe a test email address.

^^

Flags: needinfo?(bugzilla2007)
Flags: needinfo?(andreshko)

(In reply to newsfan from comment #4)

Current understanding of the TB team is that if you went from 91 to 102 and skipped the various broken betas, you should not have a problem. Maybe your case (and all the other people complaining on SUMO) disproves that assumption.

Could you pls mention a couple of those SUMO URLs?

Flags: needinfo?(bugzilla2007) → needinfo?(newsfan)

How about your paid staff take care of what they caused? Apparently 102.0 which has a host of problems is offered as update for TB 91.11 ESR when FF 91.11 ESR offers no such update. As for SUMO, I suggest you do your own research.
Hint: https://support.mozilla.org/en-US/products/thunderbird#search --> Search for "102" (without the quotes).
https://support.mozilla.org/en-US/questions/1381198
https://support.mozilla.org/en-US/questions/1381547
https://support.mozilla.org/en-US/questions/1381312

Flags: needinfo?(newsfan)

I've never run any Tbird beta - I just clicked the button to upgrade to 102 about 12 hours after Tbird auto-updated to the latest 91.x version (or whatever it was, on Win 11). I tried disabling antivirus scanning and will report back when I know something useful. It's an intermittent problem, so not easy to reproduce, unfortunately. My FAVORITE kind of bug! It almost seems like I need to have 2 e-mails in sequence with just the right combo of content, and then POOF! Mayhem.

PS-- I dunno what ya'll fixed in Quick Search, but DAYUM it's fast! 5 stars for that one!

(In reply to Thomas D. (:thomas8) from comment #6)

(In reply to Thomas D. (:thomas8) from comment #5)

Reporter, does this problem occur for you (ideally on clean install of 102) with ≡ > Help > Troubleshoot Mode…?
Pls make appropriate backups of your messages on server before testing, maybe a test email address.

^^

The problem happens on 2 different machines (2 separate independent installations of Thunderbird 102).
Both machines were upgraded manually from 91.11.0 to 102 (before the <Update to 102.0> button appeared in "Help"/"About Thunderbird" dialog). Both installations have active mail filters and folder compacting has been run at least once on each of the 2 installations.
Never run any Thunderbird beta on the 2 machines.

Flags: needinfo?(andreshko)

Thanks for the details. Please also answer (comment #4):
Have you ever used the "Allow antivirus clients to quarantine individual incoming messages" option?

Also note a similar report in bug 1777738.

Yes, always used "Allow antivirus clients to quarantine individual incoming messages", however none of the destroyed messages is a quarantined one.

bug 1777738 may look similar, but unlike it in bug 1777776 repair folder does not fix destroyed messages.

OK, but do things improve when you deselect the antivirus option? I'm asking since the option caused IMAP folder corruption which was fixed in bug 1742975 comment #126. I'm just wondering whether you're seeing the local folders/POP3 version of the issue. Please also read bug 1777454 where this option also seems to be at fault.

Okay, Inbox corruption still occurs with AV scanning turned OFF. Was always using it before.

I have mostly POP3 and 2 IMAP accounts, so I use the Inbox on Unified Folders thing. None of the IMAP messages seem to get screwed up, FWIW.

Also, one Amazon e-mail that did appear fine this morning is now corrupted (after AV scanning was off, and new mail came in).

Another note: I don't use Threaded Messages, but for some reason it keeps turning itself on again.
I have a series of 3 e-mails. Only the latest from today is supposed to still be in my Inbox. I moved the other 2 messages (from June 19) into a Local Folder back then. Earlier today, I only saw message #3, as expected. Right now, I only see Message #1 unless I enable Threading, and then I see all 3 - even though Mesg 1 and 2 are still moved in my Local Folder. Weirdness.

Maybe another clue: When viewing old e-mails with attached PDF invoices in my 'Receipts' Local Folder, I see the paperclip icon in the message list next to the e-mail subject. But when I click the individual e-mails, there is no attachment and the paperclip icon from the list view vanishes.
If I repair the folder, the paperclip icon re-appears. Rinse and repeat.

Hope something in there helps.

There are way too many moving parts here. Here they are:

Option "Allow antivirus clients to quarantine ..." - That was the causing factor for bug 1742975 and it appears to be relevant for bug 1777454. Are you 100% sure that the problem persists on a fresh setup with the option deselected? Of course deselecting the option won't fix existing corruption or adding new message to an already corrupted MSF (message summary file/database).

Mix of POP3 and IMAP accounts - Well, bug 1742975 was for IMAP, so it would be good to confirm this bug in a POP3-only setup.

Unified folders - they are another source of trouble. TB 102.0 is known to delete MSF files in large quantities under certain circumstances. That would of course affect a unified views. This will be fixed in 102.0.1.

Threaded views - TB 102.0 shipped the option enabled to create all new folders threaded, a bad choice IMHO, can be disabled again (pref mailnews.default_view_flags, set to 0 for no threading).

The paperclip issue and the originally reported issue "irrecoverably destroys messages overwriting them with random content from other messages" hints at MSF corruption. But are the messages really irrecoverably overwritten? Have you checked the mbox or maildir storage? What happens if you delete the MSF file corresponding to an mbox file?

Filtering - more added complication.

Here is what I do: I have heaps of POP3 accounts, heaps of filters, I don't use "Allow antivirus ...", no unified folder, no threading (which wouldn't make a difference), and there is no problem with version 102.0.1 which doesn't have the MSF mass deletion.

(In reply to newsfan from comment #13)

OK, but do things improve when you deselect the antivirus option? I'm asking since the option caused IMAP folder corruption which was fixed in bug 1742975 comment #126. I'm just wondering whether you're seeing the local folders/POP3 version of the issue. Please also read bug 1777454 where this option also seems to be at fault.

No, things do not improve.
With disabled filters and quarantine, I got 2 replies from 2 different correspondents to the same message - one of the replies has been overwritten by the other - I have 2 messages with identical content, but sent from 2 different correspondents at 2 different times.

Are you using mbox or maildir storage? It almost sounds like a maildir single message file is overwritten. For mbox storage, messages received via POP3 just get appended to the mbox file, there's little chance for an overwrite. If it were mbox storage, what happens when you repair or delete the MSF? Of have you opened the mbox file with a text editor to check whether there are two identical messages at the end? Or checked the maildir files? Showing the same message for two different replies can have different causes. First we need to determine whether the raw message data was lost/overwritten or whether this is more like a "display issue".

Some additional information and question. In TB 102 the POP3 implementation written in C++ was replaced by a new implementation in JavaScript. It's not entirely impossible that this new JS implementation doesn't work well with some servers. Please try switching back to the old implementation by setting pref mailnews.pop3.jsmodule to false and then restarting TB. Does that change anything?

(In reply to newsfan from comment #17)

Are you using mbox or maildir storage? It almost sounds like a maildir single message file is overwritten. For mbox storage, messages received via POP3 just get appended to the mbox file, there's little chance for an overwrite. If it were mbox storage, what happens when you repair or delete the MSF? Of have you opened the mbox file with a text editor to check whether there are two identical messages at the end? Or checked the maildir files? Showing the same message for two different replies can have different causes. First we need to determine whether the raw message data was lost/overwritten or whether this is more like a "display issue".

I am using mbox storage. After deleting the Inbox.msf file, with the new index:

  1. the monster message remained as originally reported
  2. one of the messages with identical content from comment #13 disappeared - the one, which had its original content overwritten.
  3. the Inbox mbox lost its customized column order and defaulted to thread view.

Hmm, that sounds like MSF corruption and sadly also raw message data corruption and data loss :-(

Point 3 is normal. Can you please see whether the problem persists with pref mailnews.pop3.jsmodule set to false.

I can confirm the reports above on two separate computers with 2 profiles each having several IMAPS accounts.

The repair folder doesn't have any affect when a folder starts to show messages garbled together. Manually removing the corresponding .msf file may resolve the problem if done very early. If not, loss of messages rapidly happen.

The bug is frequent but random and I don't have repeatable steps to reproduce. It happens both on Inboxes or any IMAPs server stored folder and locally stored folders. On Local Folders though, this might be the result of copying a message from the Inbox to the affected folder, even though the selected message looked fine in the Inbox at first.

The situation is very stressful as there is no clear downgrading path. The update was from the Check for update button with no warning that the new version was unstable. Every .msf removed means column settings are lost and the dreaded view (sorry threaded view) is back.

Will try to answer any question if it might be useful here.

I tried pref mailnews.pop3.jsmodule = false, then nuked all the MSF files to ensure nothing was hosed.
I'm still seeing corruption in the inbox, so it looks like the POP3 JS is NOT the problem.

OK, thank you. Switching back to the "tried and testes" old module was just to rule out that the issue is in the new one. I hope you mean "I'm still seeing [new] corruption in the inbox", since of course existing corruption won't magically be fixed in messages downloaded with the new JS module when switching to the old C++ module.

Yup, new corruption. :(

In my case, setup very close to first bug report, with some differences/specifics:

  • TB 102.0 (32-bit)

  • Windows 10 Pro 19043.1766 64-bit

  • Processor AMD Ryzen Threadripper 1950X

  • RAM 32,0 GB

  • English US OS/UI/TB, some messages in french but no correlation with message corruption

  • Had "Allow antivirus clients to quarantine..." enabled when first noticed the problem. Will update if disabling it changed something.

Status: UNCONFIRMED → NEW
Ever confirmed: true

Please, correct the bug title and remove 'Russian' (I am Bulgarian if that matters). The bug appears on different Thunderbird UI languages, OS locales and Thunderbird message encodings.

Can we clarify things a bit: All people affected here use POP3, right? There is a similar complaint for IMAP in bug 1777738.

What is the OS? Windows? Want is the file system? FAT32, NTFS? (It really doesn't matter which processor you have or how much RAM, at least not for this issue.)

What is the minimum setup? POP3 account receiving into a local Inbox folder? Are there multiple accounts at play, so perhaps a global inbox? Are there filters at play? What do those filters do? "Mark as read", move somewhere?

Comment #16 and comment #19 suggest an mbox file positioning issue: The second message, instead of being appended to the mbox overwrites an existing message at a lower offset into the file. The offset into the file is recorded in the MSF, hence two identical messages are showing. Deleting the MSF and reconstructing it from the raw message gets rid of the second message.

@sancus: Just FYI.

Flags: needinfo?(sancus)

Can we clarify things a bit: All people affected here use POP3, right?

I notice the original reporter on this bug has both POP3 and IMAP accounts. On 2 different computers, I have different profiles each with only multiple IMAPS accounts, no POP. Unified inbox not used. Filters heavily used on one profile and not the other. MSF Corruption occurs on both. Filters used to directly move messages in Local Folders subfolders on reception. No destination folder ever corrupted after a "move by filter" up to now.

What is the OS? Windows? Want is the file system? FAT32, NTFS?

W10, NTFS (Hard to have FAT32 these days, sorry I didn't know that was what mattered...)

There is a similar complaint for IMAP in bug 1777738.

Looks similar to my case if repair doesn't fix the problem. Repair never fixed the problem for me. Deleting the .msf file could, if done early.

After disabling "Allow antivirus clients to quarantine..." and deleting the MSF for all inboxes, no new corruption for a few hours now. No antivirus other than W10. Went directly from 91 to 102 and "skipped the various broken betas", too.

... I didn't know that was what mattered

So far we don't know what matters. Some people have POP3, others IMAP (like bug 1777738 or you here in this bug). Sure, FAT32 was a silly question, sorry. Your "other" profile uses filters a little, or not at all? I'm still thinking that if the filter moves a message, the original needs to be marked deleted in the offline storage, that writes to the header of the message. If it's then not positioned to the end again before appending the next message to the offline store, you get corruption.

I've just uploaded a patch on Bug 1777454, which I suspect might be the same as this one.
See https://bugzilla.mozilla.org/show_bug.cgi?id=1777454#c25
Does anyone here want to try it out and see if it helps?

@Abrena:
Thanks for the information you added, about filters, Anti-Virus, POP3 vs. IMAP. Any of these are potential factors.

@Ben Campell:

See https://bugzilla.mozilla.org/show_bug.cgi?id=1777454#c25 Does anyone here want to try it out and see if it helps?

Could you please push a try build and reference the Windows installer EXE URL here, so that non-dev users can try it out?

(In reply to newsfan from comment #30)

Sure, FAT32 was a silly question, sorry.

I'm not averse myself to contribute a little levity to dry and boring matters such as this one, from time to time.

Your "other" profile uses filters a little, or not at all?

Not at all.

(In reply to Ben Bucksch (:BenB) from comment #32)

Could you please push a try build and reference the Windows installer EXE URL here, so that non-dev users can try it out?

This could be useful to me.

So far today, no new corruption occurred. It looks like disabling "Allow antivirus clients to quarantine...", if not linked to a direct cause, could have some sort of impact.

Both profiles under observation are a little busy, so this should count for something. The one using filters receives somewhat heavy messages with attachments (from a few hundred KB to something like 20MB of size).

Also, am I right in thinking that occurrences of this bug aren't that widespread in the user base? Could it be some unusual setting?

Of note in my case (don't know if it is of any relevance or if it has been modified from default):

  Settings > General > Disk Space >

        Override automatic cache management >
        Disabled

        Compact all folders when it will save over > 
        1 MB in total

So it looks like compacting should occur pretty often.

I have my hands a little full at the moment but I'll check back here later today or tomorrow to see if this is still investigated and try and whip out a "stress test" of sort, re-enabling "Allow antivirus clients to quarantine..." and see if I can reliably reproduce the .msf corruption. And may be tweak that compacting threshold too?

occurrences of this bug aren't that widespread in the user base? Could it be some unusual setting

Bug like this are very often timing-related, e.g. the bug is triggered only if 2 things coincidentally happen at the same time or in short succession. That's also what makes them so hard to reproduce and debug.

Blocks: tb102found

(In reply to Ben Bucksch (:BenB) from comment #32)

@Ben Campell:

See https://bugzilla.mozilla.org/show_bug.cgi?id=1777454#c25 Does anyone here want to try it out and see if it helps?

Could you please push a try build and reference the Windows installer EXE URL here, so that non-dev users can try it out?

I kicked off a try build using Ben's patch. It just started so will be a while before it finishes:
https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=bdca5b36f835557853c5a655cdf21bc71beaa101
This will produce a daily build for all archs, optimized and in debug mode.
Not sure this affects this bug, but maybe.

Ok, the try build is finished. Go here
https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=bdca5b36f835557853c5a655cdf21bc71beaa101
and click the green "B" next to the appropriate architecture (unlabeled 32-bit or 64-bit) and optimization. Debug build may run a bit slower than Opt since there is more runtime error checking for asserts and warnings printed to console. Then down below select "Artifacts and Debugging Tools" and download the appropriate installer found in the list:

Windows: target.installer.exe
OSX: target.dmg
Linux: target.tar.bz (unzip and run thunderbird inside the thunderbird directory)

(In reply to Wayne Mery (:wsmwk) from comment #39)

Abrena, Scottie, Andrey, please test if you can
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/FP3Sy6IuSearFYJ9Y5rnvQ/runs/0/artifacts/public/build/install/sea/target.installer.exe
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/GIJKptnfRpWOktRtpLe5Hg/runs/0/artifacts/public/build/target.tar.bz2
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/eIhtAvMWTWOcZ6uiylZqaQ/runs/0/artifacts/public/build/target.dmg

I ran the try build offline with enabled messages filters, manually did several folder compacting to Inbox and Sent after moving numerous messages a couple of times from Inbox to other folders and didn't find new damaged messages. Let me run it online for a couple of hours in real use conditions.
mailnews.pop3.jsmodule is false
"Allow antivitus…" is disabled

Flags: needinfo?(andreshko)

I can now report that no new corruption occurred with TB 102.0 Release in almost two days since disabling "Allow antivirus clients to quarantine...".

I had to enable it back to confirm it had some sort of impact. I first created a fresh new dummy profile (should have done that sooner, sorry). No add-on of any sort. Most settings back to defaults.

Of note: "Allow antivirus clients to quarantine..." kept its disabled state at first, while "Compact all folders when it will save over" reverted to what must be the default 20MB. I just enabled back "Allow antivirus clients to quarantine..." and started to test.

It was not even hard to get corruption again. I just sent myself a dozen messages of 17 KB size, no attachments. Everything looked fine. I forwarded the received messages to myself again and had then the Inbox folder and Sent folder a little populated. Then I deleted every third message in Sent. Everything still looked fine in there. Then Right click on the Sent Folder > Compact. Folder trashed, messages corrupted.

Tried Right click on the Sent Folder > Properties > Repair folder. No effect. Closed TB, went to manually delete the .msf file. There was some unexpected -1. files there:

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---          06/07/2022    15:13           2877 Archives-1.msf
-a---          06/07/2022    14:25           1180 Archives.msf
-a---          06/07/2022    14:42         106886 Drafts-1
-a---          06/07/2022    15:13           2718 Drafts-1.msf
-a---          06/07/2022    14:25           1177 Drafts.msf
-a---          06/07/2022    14:46         459626 INBOX
-a---          06/07/2022    15:13          17457 INBOX.msf
-a---          06/07/2022    14:27           1325 Junk.msf
-a---          06/07/2022    14:25              0 msgFilterRules.dat
-a---          06/07/2022    14:48         221343 nstmp
-a---          06/07/2022    14:36           1763 Remote Store.msf
-a---          06/07/2022    15:10         578091 Sent-1
-a---          06/07/2022    15:13           9164 Sent-1.msf
-a---          06/07/2022    14:25           1175 Sent.msf
-a---          06/07/2022    14:25           1183 Templates.msf
-a---          06/07/2022    14:48         223739 Trash
-a---          06/07/2022    14:53           8659 Trash.msf

I manually deleted Sent.msf and restarted TB. As could be expected, no effect on the trashed Sent folder. Closed TB again, deleted Sent-1.msf and restarted TB. Threaded view back in Sent folder and remaining messages looking fine.

So it looks like I can reliably reproduce the problem. How can I help?

Flags: needinfo?(mozilla.org)

So it looks like I can reliably reproduce the problem. How can I help?

That's great!

Test the same scenario with the builds from comment 39, and tell us whether the bug goes away.

(In reply to Wayne Mery (:wsmwk) from comment #39)

Abrena, Scottie, Andrey, please test if you can
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/FP3Sy6IuSearFYJ9Y5rnvQ/runs/0/artifacts/public/build/install/sea/target.installer.exe

I installed that Daily. I went to the ImapMail file folder in the test profile created for my previous report. Deleted all the .msf files in that folder tree and started the Daily with the test profile.

Proceeded to do (almost, of course) exacly the same actions as in my previous report and did not get any corruption.

@Abrena: Thank you! Given that the bug is not 100% reliably reproducible, could you go back and forth a few times between the broken TB 102 release build and the try server build, and see whether the TB 102 release build reliably triggers the bug, and the try build never ever (!) triggers the bug, even though you preform the same actions? Can you try at least 3-5 times in each build, whereas TB 102 relase at all times triggers the bug, and the try build never triggers the bug? Can you please report the results for every test you make, so that we can see the consistency of the results?

Thanks for the tests. This testing is very important to verify that the proposed code change really does reliably fix the bug.

(In reply to Ben Bucksch (:BenB) from comment #44)

@Abrena: Thank you! Given that the bug is not 100% reliably reproducible, could you go back and forth a few times between the broken TB 102 release build and the try server build, and see whether the TB 102 release build reliably triggers the bug, and the try build never ever (!) triggers the bug, even though you preform the same actions? Can you try at least 3-5 times in each build, whereas TB 102 relase at all times triggers the bug, and the try build never triggers the bug? Can you please report the results for every test you make, so that we can see the consistency of the results?

I was already trying to do that for the same reasons you give. I'm stress testing the daily with more and more sending/deleting/compacting with no adverse result up to now.

It seems I can't continue to use my testing profile once I opened it with the daily though (newer version conflict). How would you do that test? (Me lazy bum looking for easy shortcuts).

Is there any "more rigorous" way to perform those tests?

BOOYAH!
I installed it and have been testing live with my full profile (15 mail accounts - 2 IMAP and the rest POP3, 23 Local Folders, 31 message filters, many 10k's of e-mails).
Also using:

  • mailnews.pop3.jsmodule = false
  • Allow antivirus = false
    After several hours of my daily mountains of e-mail coming in, NO corruption. :)
    I've even been trying to randomly Repair and Compact folders in a valiant effort to break it, but it just works!
Flags: needinfo?(scottie)

I can't continue to use my testing profile once I opened it with the daily though (newer version conflict). How would you do that test?

Note that you're not testing the real "Daily", even if it says that in the UI. You're testing a so-called "try build", a custom build.
You can download and test the real "Daily" from https://www.thunderbird.net (bottom right, and select tab "Daily Channel"). This would be the latest Thunderbird without the fixes here, whereas the "try build" from comment 39 is a similar version, but with the fixes. You can alter between those two.

Thanks for your help!

UPDATE: I just see that the fix here landed 5 hours ago, so the fix would be included in any Daily builds that are newer than 2022-07-06 10:00 UTC, so make sure to grab one before that. You can find it on the file server: https://ftp.mozilla.org/pub/thunderbird/nightly/2022/07/2022-07-05-10-17-31-comm-central/thunderbird-104.0a1.en-US.win32.installer.exe

(In reply to scottie from comment #46)

BOOYAH!
I installed it and have been testing live with my full profile (15 mail accounts - 2 IMAP and the rest POP3, 23 Local Folders, 31 message filters, many 10k's of e-mails).
Also using:

  • Allow antivirus = false
    After several hours of my daily mountains of e-mail coming in, NO corruption. :)

Can't remember exactly in your case, but in my tests, I have never get any corruption once "Allow antivirus clients to quarantine..." was disabled, whatever the version I used (release or try build).

Could it be that this is the case for you too?

To Abrena, comment 45:

It seems I can't continue to use my testing profile once I opened it with the daily though (newer version conflict). How would you do that test? (Me lazy bum looking for easy shortcuts).

You may need add the option --allow-downgrade to the thunderbird command used to start the thunderbird program. I think you can add that in the shortcut properties if you are using windows.

(In reply to Ben Bucksch (:BenB) from comment #47)

Note that you're not testing the real "Daily", even if it says that in the UI. You're testing a so-called "try build", a custom build.

Thanks for the clarification. I'm not fluent in Thunderspeak as you can see. ;-)

So... in the end, I think did the tests you asked for.

I found out that to get corruption 100% of the time in TB 102.0 release, I needed:

  • Messages of some size. Subject of just 1 digit, body just 1 digit would not work. Subject 1 digit with body rich text of 17.6 KB would.
  • Enough new messages in a remote folder. 9 wouldn't work, 12 would.

So I left 12 such messages in a remote inbox on an IMAP server. The start of the body could be easily matched to the message subject (same number in both).

Before each test, I deleted the test profile (Delete Profile > Delete files), recreated it, checked up that "Allow antivirus clients to quarantine..." was enabled before starting TB for the test. The remote server was left every time with all remote folders empty save the 12 test messages in the inbox.

From there, I followed the same script: I forwarded myself the 12 messages. Everything looked fine both in the Inbox and Sent folder. Then I went in the Sent folder and deleted every odd message (1, 3, ...11). Everything still looked fine, message subjects matching the bodies. Then Right click on the Sent Folder > Compact.

There I'm happy to confirm:

102.0 Release: Folder trashed, messages corrupted. 5/5 times
Try build: everything fine 5/5 times.

I noticed in one of the 102.0 release test (not included in my count) I messed up by just compacting the folder before deleting the odd messages. The subsequent compacting went fine. Don't know if this has any interest.

I noticed too that leaving "Allow antivirus clients to quarantine..." enabled would cause from time to time TB 102.0 release to hang for a long time when trying to access a folder. With the Try build, it would happen too but be a lot shorter.

Finally I noticed that in 102.0 release, [Folder] > Properties > Repair folder would often have no effect at all (has would be evidenced for me by the return of the threaded view). The Try build could always repair a folder.

(In reply to gene smith from comment #50)

To Abrena, comment 45:
You may need add the option --allow-downgrade to the thunderbird command used to start the thunderbird program. I think you can add that in the shortcut properties if you are using windows.

In the end I went the hard way and deleted the test profile each time... Better test that way anyway, isn't it? ;-)

(In reply to Abrena from comment #51)

102.0 Release: Folder trashed, messages corrupted. 5/5 times
Try build: everything fine 5/5 times.

Thanks, Abrena!

compacting the folder before deleting the odd messages

Yes, reproduction steps like that are very helpful, thank you!

(In reply to Abrena from comment #49)

Can't remember exactly in your case, but in my tests, I have never get any corruption once "Allow antivirus clients to quarantine..." was disabled, whatever the version I used (release or try build).

Could it be that this is the case for you too?

I got corruption with and without AV quarantine. I've left it off now simply because I really don't need it anyway.

Assignee: nobody → benc
Flags: needinfo?(sancus)
Flags: needinfo?(benc)
See Also: → 1777454
Whiteboard: [dupe of bug 1777454?]

Based on the comments above, it seems like the try build containing the patch from bug 1777454 resolves the issue for all the reporters?

Is there anybody who CAN still reproduce corruption, with OR without the AV quarantine option using the build from Comment #39 or a current Daily?

Summary: TB 102 irrecoverably destroys messages overwriting them with random content from other messages (POP, Russian, may involve Compact after filtering) → TB 102 overwrites message content with content from other messages (POP, may involve Compact after filtering)
See Also: → 1778353

Okay, I re-enabled AV scanning on the test build... The corruption does NOT occur.

The only difference with AV on is that when I repair folders (like Inbox or Junk), old filtered messages re-appear as unread even tho they were filtered and marked read in my various Local Folders (or Junk folder). Turning AV off stops this issue.
So, for me anyway, AV thing seems to be a different issue that is not causing the mail corruption I was seeing before.

I've now set mailnews.pop3.jsmodule = true, and we'll see if that makes any difference.

2 hours of mail coming in with AV OFF and pop3.jsmodule ON, and still no corruption.

I'm marking this as a dupe. Please reopen if you can reproduce this on Thunderbird 103 Beta 4 or current Daily.

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → DUPLICATE
See Also: → 1778917

(In reply to scottie from comment #46)

BOOYAH! ...
After several hours of my daily mountains of e-mail coming in, NO corruption. :)
I've even been trying to randomly Repair and Compact folders in a valiant effort to break it, but it just works!

Way to go scottie, thanks! Keep hitting it hard :-)

Blocks: msgreader-jumble
Whiteboard: [dupe of bug 1777454?]
See Also: → 1778814

Sorry to say it, but this bug appears to have returned today. I updated to 102.3.1 today, suddenly got this on a new message.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: