1777738 - Inbox corruption in version 102 (IMAP, POP) - wrong message shown (or html of other) - repair folder fixes the issue (expected to be fixed by compact bug 1777454)

Status: RESOLVED FIXED

(Thunderbird :: Folder and Message Lists, defect, P1)

Description

[Sam]

Steps to reproduce:

Problem has occurred three times in normal email usage, but I cannot reproduce it at will.

Actual results:

My inbox is short, usually less than 25 messages. Intermittently, after updating to version 102.0, I click on a message and the preview shows the wrong message or displays raw html of some other message. When I right click the Inbox / Properties / Repair, the repair process fixes the problem.

Expected results:

I have been using Thunderbird since Eudora retired, and I have never seen this problem.

Comment 1

[Sam]

Two more observations:

1. It is possible that it is the "Compact Folders" operation that is corrupting the Inbox. I did a compact and then shortly after noticed the corruption.

2. The fix, doing a "repair on the folder", doesn't always do anything. I just tried half a dozen times without the repair doing anything (if it does the repair, I know it because it changes the columns in the Inbox back to the default). After closing Thunderbird and reopening it, for the second time, the repair ran and fixed the corruption again.

Comment 2

[mauriz]

me too have this problem after updating to 102
Maurizio

Comment 3

[mauriz]

me too have this problem after updating to 102
Thunderbird 102, MacOS 12.2.1
Maurizio

Comment 4

[mauriz]

me too have this problem after updating to 102
Thunderbird 102, MacOS 12.2.1
Maurizio

Comment 5

[Kai Engert [:KaiE:]]

What protocol do you use to access your emails? POP or IMAP ?

Comment 6

[Kai Engert [:KaiE:]]

Are your emails small or big?
With complex structure, or rather simple text?
What's your typical workflow, do you use "compact" regularly?
Anything unusual to where your Thunderbird profile folder is stored? Regular local disk, or some remote drive?

Comment 7

[mauriz]

hi Kai:
IMAP folders
small emails, sinple text
nearly never use "compact"
regular local disk
[same troubles with addons running and with all addons disabled]

Comment 8

[Sam]

IMAP

Messages are typically small, none with attachments. Most have html structure,

I typically compact after I have deleted a bunch of messages. However, since my original comments, I have seen the corruption without compacting, but compacting seems to induce it much more regularly.

Profile is on my local C drive, but not in Appdata because I want my backups to process it. The profile has not changed. This problem is only with version 102.0.

Comment 9

[newsfan]

Kai, there is a POP3 variation of this bug: bug 1777776.

Comment 10

[mauriz]

my msg are small and mainly textual (HTML)
for example your message, Kai, when copied by thunderbird from the katamail.com inbox to the local folders inbox, becameblank.
but i must correct myself about compacting: i had set the compact to happen when more than 3MB could be recovered. now i disabled it.

Comment 11

[mauriz]

after completely disabling the compactation, the troubles apparently are gone

Comment 12

[Sam]

Mauriz, I think you are right. I previously reported corruption without compacting, but it was probably doing automatic compaction. Since disabling automatic compaction, I have not seen the error without compacting manually.

Comment 13

[Ben Campbell]

Sam and Mauriz, this does sound rather similar to bug 1777454 (I suspect bug 1777776 might have the same root cause too).
There is a potential fix it over there. Want to give them a go and see if it sorts it out for you? Test builds for linux/windows/mac are linked here: https://bugzilla.mozilla.org/show_bug.cgi?id=1777454#c34

Comment 14

[Sam]

Hi Ben. Thanks for working on this. It does sound like 1777454 might be the same. The reporter for 17777776 stated that doing a repair does not restore the content. I have found, in this bug, that repair does restore the inbox content. However, I'm wondering if that is because it is IMAP; perhaps POP does not have the information needed to restore. I have not dared to invoke a compact on a local folder because I can imagine that it might not be so easily recovered with the repair. Incidentally, I can now corrupt the inbox at will with a compaction. Also, be advised that I have to close Thunderbird and reopen it, at least twice and often three times, before the repair function runs. Until then, clicking on repair has no visible effect. Also, I should have said that I am running Windows 11 Pro.

I tried running:
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/FP3Sy6IuSearFYJ9Y5rnvQ/runs/0/artifacts/public/build/install/sea/target.installer.exe

It appeared to install TBird, but the install date in the control panel didn't change, so I don't know where it put it.

Comment 15

[Sam]

I found where the Daily was stored and ran it, version 104.0a1. That did fix the problem. I was able to compact the inbox without corruption. Thanks!

Comment 16

[mywildimagination]

I can confirm this issue with corrupt emails (Only local IMAP cache, server webmail shows correct emails) and also the issue with repair folder not working or only working when you first open thunderbird.

Comment 17

[mauriz]

hi ben, i am a little afraid of installing the test version over this one (102) which after disabling automatic compaction is working well

Comment 18

[Andrei Hajdukewycz [:sancus]]

The fix for bug 1777454 is now in Thunderbird 103 Beta 4 so that can be tested instead of the test builds above.

Comment 20

[Thomas D. (:thomas8)]

Same corruption reported in Bug 1779455 - Windows: TB 102.0.2; IMAP; a few msgs show wrong msg body vs header displayed
Bug 1779455 Comment 0 has very detailed information.

Comment 29

[Wayne Mery (:wsmwk)]

Better with 102.0.3 ?
https://archive.mozilla.org/pub/thunderbird/candidates/102.0.3-candidates/build1/win64/en-US/Thunderbird%20Setup%20102.0.3.exe

Comment 31

[mywildimagination]

I can confirm this bug is still an issue in 102.0.2 but not as regular as with the previous release version.

I use IMAP on all accounts. Only effecting locally downloaded data, server data unaffected so a repair fixes issue for a while. Repair option still broken, will work sometimes when you first load TB if you are lucky.

Comment 32

[Thomas D. (:thomas8)]

Per BenC's comment 13, also Magnus' Bug 1779990 Comment 5, we're expecting that all the bugs collected here are manifestions of compact bug 1777454 (fixed for 102.0.3), which also means that this bug affects both IMAP and POP3 accounts.

Bug 1777454 - After compacting, messages are no longer added to the messages file

We encourage affected users to check if the 102.0.3 candidate build fixes this issue for them. Prior data backup is advised.

(In reply to Wayne Mery (:wsmwk) from comment #29)

Better with 102.0.3 ?
https://archive.mozilla.org/pub/thunderbird/candidates/102.0.3-candidates/build1/win64/en-US/Thunderbird%20Setup%20102.0.3.exe

Comment 35

[Duane Brosius]

I have loaded 102.0.3 - I will let you know if I see anything different. BUT FOR NOW I WILL LEAVE AUTOMATIC COMPACTING OFF.
As for 102.0.3 the "Repair Folder" seems to be working better. A side note for the "Repair Folder" it looks like it goes and redownloads the emails in the folder. For my last attempt it downloaded 564 messages for 200 in the INBOX. I am not sure if this is normal and is expected to download more.

PLEASE NOTE I HAD TURNED OFF AUTOMATIC COMPACTING IN 102.0.2 AND I DID NOT DO ANY COMPACTING UNDER 102.0.2 AND THIS SEEMS TO HAVE CORRECTED THE ISSUE UNDER 102.0.2 AS LONG AS NO COMPACTING IS DONE as far as the corrupted emails goes. Repair Folder was still hit and miss.

Comment 36

[Ataro]

My contribute:

I'm using TB 102.0.2

It don't empty the trash folder on exit and not compact empty folder (the folder file don't empty).

The compact command executed on a single folder and the compact folders command executed from file menu don't work correctly, create a nstmp or nstmp-x hidden folder on the on disk account folder, and in some case corrupt the INBOX folder on a IMAP server account.
For now are'nt some issues on POP accounts.

But luckily seems that don't corrupt mail database on the imap server.

Comment 37

[Wayne Mery (:wsmwk)]

Thomas smoketested TB 102.0.3 candidate build today, and using steps from bug 1777454 Comment 55, he was not been able to reproduce the inbox display corruption.

It would be great if users affected by this bug could try the following:

1. On TB 102.0.2 (affected), execute STR of bug 1777454 comment 55, which should cause message display corruption on the last received message (at least).
2. On TB 102.0.3 candidate build, execute STR of bug 1777454 comment 55 again (try to do it in exactly the same way) - which should no longer cause message display corruption.
3. Let us know SUCCESS if you're seeing the problem in 102.0.2, but no longer seeing it in 102.0.3.

Otherwise report FAIL if you cannot reproduce with bug 1777454 comment 55 in 102.0.2 OR if you can still reproduce in 102.0.3, and say where/in which way it failed. Keep your comment as short as possible. Please comment in only one bug report.

Comment 38

[Adam Reece]

Good morning.

Thank you very much for the update. I've updated to 102.0.3 on both my personal and work computers. Immediately after updating I ran a compact of all folders in all accounts. (My work account showed approximately 148 GB saved!!) Browsing messages in folders, particularly ones that show up in the status bar during compaction, all appear to be displaying as expected.

So in short, SUCCESS, problem occurred in 102.0 through to 102.0.2, but so far appears to be resolved as of 102.0.3.

I will continue to keep lookout for this issue over the coming week and feedback further. I will also activate the automatic compacting option again as I had previously to tempt the original issue to occur again.

Comment 39

[djk]

I installed the 102.0.3 yesterday and so far with auto compacting enabled the problem has not occurred. More time is needed to confirm fully but so far this is positive. Thank you for the attention taken to this issue.

Comment 40

[wfroml]

102.0.3 works correctly so far, on two pc's.

Comment 41

[Ataro]

Not for me.
102.0.3 installed. Compaction on a IMAP server dont work. I'm using 3 servers imap.
I've seen that after compaction the messages separator (From ) are bad written. Follow some examples:

YWdlMT9lbHE9ZDQwYjcwYzEwMWYxNDFmYjg0MTI4ZGJjZTEzOGYwNGEmc2l0ZWlkPTQ5MDQw
NDczNycgYWx0PSIiIGJvcmRlcj0wIHdpZHR**From **
Received: from 127.0.0.1
by atlas-production.v2-mail-prod1-ir2.omega.yahoo.com with HTTP; Mon, 18 Jul 2022 12:44:50 +0000
Return-Path: <[e-mail]>

---

fQfi0+q/FjxaureLPiF4fuJ9E0zWdb1m81KXTrR1G+OFJpXjj3AkFlUMQSCSDiiiijj6tU/tutHm
do2SV9lZOy7LXYy4dpxeBhJrV3P/2Q==
--=_ED0736302BFD73F18486000D3ABBA5A3--
**From **
=
<table style=3D=22font-family:Arial; font-size:13px=22> =
=
=
<tr><td>Mittente: </td><td><b>=
AMAZON EU SARL</b></td></tr> =
=
=
<tr><td>Destinatario: </td><td><b>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=

---

6uXy{position:absolute;top:35%}.p13n-asin-list-plain_style_notInterestedMe=
ssage__2bIn5{position:absolute;top:50%}.p13n-asin-list-plain_style_image=
rSmhM{opacity:.1}._p13n-asin-list-plain_style_feedbackRow__23bqz{margin-rig=
ht:0!important}
._p13n-asin-list-plain_canariesFeedbackModalStyle_p13n-flex-container-heade=
r-k**From **
Received: from 127.0.0.1
by atlas-production.v2-mail-prod1-ir2.omega.yahoo.com with HTTP; Tue, 19 Jul 2022 10:50:49 +0000
Return-Path: <[e-mail]>

---

The mail file folder INBOX begin with this From:

**From **
Received: from 127.0.0.1
by atlas-production.v2-mail-prod1-ir2.omega.yahoo.com with HTTP; Mon, 18 Jul 2022 15:24:55 +0000
Return-Path: <[e-mail]>

And all misplaced/misformatted From don't include a Date and Time, how a normal one differently do:

--=_ED0736302BFD73F18486000D3ABBA5A3--
**From **- Tue Jul 19 11:20:19 2022
X-Mozilla-Status: 0001

Comment 42

[Ataro]

In previous message I'have inserted ** for write From in bold, but i've inserted a space and don't have worked correctly ...

Comment 43

[djk]

(In reply to Ataro from comment #42)
Have you deleted the local files and directories with the exception of the msgFilterRules.dat. With IMAP these will be rebuilt when Thunderbird starts up and gets mail. This has been my fix when the problem occurs. I did this with Thunderbird close prior to upgrading to 102.0.3 and so far with multiple IMAP configured the issue has not occurred. I did try the repair option but that had no impact.

C:\Users[USER]\AppData\Roaming\Thunderbird\Profiles[ACTIVEPROFILE]\ImapMail[IMAPACCOUNT]

Comment 44

[Ataro]

(In reply to djk from comment #43)

(In reply to Ataro from comment #42)
Have you deleted the local files and directories with the exception of the msgFilterRules.dat. With IMAP these will be rebuilt when Thunderbird starts up and gets mail. This has been my fix when the problem occurs. I did this with Thunderbird close prior to upgrading to 102.0.3 and so far with multiple IMAP configured the issue has not occurred. I did try the repair option but that had no impact.

C:\Users[USER]\AppData\Roaming\Thunderbird\Profiles[ACTIVEPROFILE]\ImapMail[IMAPACCOUNT]

Important is do not delete the messages into the server before ....
Yes, I also do this procedure for the recovering of my mail on imap, and then i copy the messages on a my local folder for backup.
But if after this procedure i issue the compact folder command, the problem return again.
As reported in the examples in to the previous messages.

Comment 45

[Ataro]

I forget to say that the problem now appear only on one server imap of my thee.

Comment 48

[Worcester12345]

(In reply to Sam from comment #1)

Two more observations:

1. It is possible that it is the "Compact Folders" operation that is corrupting the Inbox. I did a compact and then shortly after noticed the corruption.

2. The fix, doing a "repair on the folder", doesn't always do anything. I just tried half a dozen times without the repair doing anything (if it does the repair, I know it because it changes the columns in the Inbox back to the default). After closing Thunderbird and reopening it, for the second time, the repair ran and fixed the corruption again.

One user:
3 out of maybe 45 mails visible. Repair mailbox, then all show. Then go to different folder, or compact, or close and reopen, only 3 showing again.

Comment 49

[Sam]

The problem occurred with TBird 102.0 and 102.0.1. Since updating to 102.1.0, I have used the compaction option a lot and I have not seen the corruption. I also saw the problems with the repair function that you mentioned, but I just did a repair as a test and it worked the first time, so maybe that is fixed too.

Comment 50

[Wayne Mery (:wsmwk)]

(In reply to Ataro from comment #45)

I forget to say that the problem now appear only on one server imap of my thee.

When posting updated information, it is important to state which version is being used.
That said, if you still see a problem using a newer, current version (102.1.0 or newer) please create a new bug report

Comment 51

[Wayne Mery (:wsmwk)]

If anyone else still sees an issue, please file a new bug report