1890230 - [meta] folder corruption tracker for thunderbird 122+ - If you add a comment, please state which version you have used. - Please test the most recent 128.x version

Status: ASSIGNED

(Thunderbird :: General, defect, P1)

Description

[Wayne Mery (:wsmwk)]

from bug 1872849 comment 0, "Filing this as a tracker for folder corruption occurring with 122 and above, to at least track feedback on whether it got worse. xref bug 1719121."

Comment 1

[John Owles]

Attachment: Screenshot 2024-04-10 085749.png

The bug has returned again. Totally blank body this time
As per, Repair Folder brought it back together

Comment 2

[dquiros]

How did you repair the folder?

My TB still has blank emails.

Comment 3

[John Owles]

Right click the offending folder, and you should see the option to Repair Folder

Comment 4

[John Owles]

Sorry, I got that wrong.
Click Properties and the repair option is there, towards the bottom of the window.

Comment 5

[dquiros]

I dont know what folder the email is in.

My tab is totally blank with no hint to where it is located.

I see the option Right Click, Properties, Repair Folder.

There should be an option to Repair All Folders.

Comment 6

[John Owles]

I think that would be sensible, but I don't see anything global

Comment 7

[John Owles]

The folder has just gone blank again.
Repair sorted it again, but it doesn't seem to last long.

Comment 8

[John Owles]

Attachment: Corrupt email body

I am also getting a different style of corrupted email body, displaying this:-

From and Subject fields are blank
The Body displays thus:-

Comment 9

[John Owles]

I am also getting a different style of corrupted email body, displaying this:-

From and Subject fields are blank
The Body displays thus:-
--006_LO6P123MB66005B72769A05D7B0EE23B4B3282LO6P123MB6600GBRP
Content-Type: image/png;
name="image456233.png"
Content-Transfer-Encoding: base64
Content-ID: <image456233.png@B40ED131.D0FB59A2>
Content-Description: image456233.png
Content-Disposition: inline;
creation-date="Fri, 15 Mar 2024 08:39:30 +0000";
filename=image456233.png;
modification-date="Fri, 15 Mar 2024 08:39:30 +0000";
size=10275

iVBORw0KGgoAAAANSUhEUgAAALMAAABLCAYAAAA71baHAAAACXBIWXMAAC4jAAAuIwF4pT92AAAn
1UlEQVR4nO2dd3gc1dW435nZne2rVZdcZVu4dxsb2xgbjCmhBkiAEFqA0AIfJST0lo+EHupHS0jo
xUB+hGYCGINxA2Nw702WrS7tavtOub8/RrJla1fNNqHofZ41PNo7t8yeuXPuueecKwkhlgB9gQgg...

etc

Comment 10

[John Owles]

This morning, when the above appeared, Repair Folder did not work.
I have justried again and Repair Folder did work....?

Comment 11

[Arthur K. (he/him)]

(In reply to John Owles from comment #7)

The folder has just gone blank again.
Repair sorted it again, but it doesn't seem to last long.

Forgive me if it hasn't been asked yet but by chance is your instance of TB on something that is not a local drive?

Comment 12

[John Owles]

No, TB is on my local drive c

Comment 13

[Arthur K. (he/him)]

(In reply to John Owles from comment #12)

No, TB is on my local drive c

Anything of interest in the Error Console (ctrl-shift-J) shortly after it happens? Do you happen to know if your email storage capacity is not maxed out? You can find out by doing a right-click on your Inbox > Properties > click the Quota tab. I'm not assuming it's a GMail account but if it is you'll see a % detailing your storage usage.

Comment 14

[John Owles]

I didn't check the error console I'm afraid. When it happens again I certainly will.
My email capacity is not maxed out.
This only happens in one parent folder and it's sub-folder, and even then, it seems quite random in it's choice of emails to corrupt.

Comment 17

[Magnus Melin [:mkmelin]]

For my own tracking: had it again just now with 2024-04-17 daily.

Comment 18

[Magnus Melin [:mkmelin]]

Once again with 2024-04-22

Comment 19

[Wayne Mery (:wsmwk)]

Note:

• Bug 1890448 - Rewrite folder compaction - is fixed in nightly 2024-05-24 and will hopefully be in tomorrow's beta 2024-05-29
• nstmp specific issues have been consolidated into bug 1878541

Comment 21

[AFOC]

Since the apparently duplicate bug in which I left comments has been closed, I'm adding those same details here in order to subscribe to & track this issue. I want to receive updates on this issue until it is 100% resolved and never experienced again.

There are details I mentioned there which are not mentioned anywhere here, so I am pasting them verbatim:

Windows 11, Thunderbird v115.10.1 (64-bit)

I've also been experiencing this for an indeterminate amount of time.

I have lots of emails (2000+) sorted into non-local folders (30+). Occasionally when I go to a folder & click on an email the wrong email is displayed. The subject, correspondents, etc is all correct in the list. Usually (if not always) it's the full source code of another email in the same folder. Regarding the same folder part, I'm not certain this is true 100% of the time.

This can occur both for emails sent from & to myself as well as emails from other senders. It has never happened to me in the primary inbox, it only happens with emails in non-local folders.

I've confirmed via webmail that there is no actual data loss or issue with the emails themselves, so this is definitively something Thunderbird does locally. When this happens, the correct email is no longer anywhere to be found in Thunderbird as if the body has been completely overwritten.

Unfortunately I also don't know of any deterministic way to reproduce this. I could give a ballpark estimation of this maybe occurring to 0.25 - 0.5% of emails in folders. I've not been able to observe any distinguishing element that might cause this, it seems to be completely random as far as I can tell.

The only previous action taken prior to encountering this issue is to right click on a message in the inbox and then move it to a non-local folder. Later when viewing the folder, the issue may appear.

As another bit of data I should also add that deleting the email via Thunderbird deletes the correct email.

What @quadronom mentions about the headers missing (and therefore the subject, etc) also happens to me in the email display pane and when it is opened in a separate tab. So it appears that wherever Thunderbird pulls information from for the index / email list is not the same as when you click on the subject of an email to display it. Perhaps the problem lies somewhere around there.

Comment 22

[Ben Campbell]

(In reply to AFOC from comment #21)

Since the apparently duplicate bug in which I left comments has been closed, I'm adding those same details here in order to subscribe to & track this issue. I want to receive updates on this issue until it is 100% resolved and never experienced again.

Thanks for the details!

It sounds like a screwed-up mbox file, but it'd be nice to confirm (I've a suspicion that the IMAP code is interrupting it's own writes to the mbox).

Next time it happens, could I get you to have a look through the raw mbox file on disk corresponding to that folder?
Each message in the file is separated by a line containing "From " (including the space!).
In a broken mbox file, you'll find a "From " line beginning a new message right in the middle of another message, cutting it off in mid-flow.

I'd be interested to see if you can spot any breakages like this, or if I'm looking in the wrong place.

It's frustratingly intermittent, so I'm looking to see what logging I can add to help pinpoint what might be going wrong, so this might take a little to-ing and fro-ing to figure out...

Comment 23

[John Owles]

I will keep an eye and let you know

Comment 24

[Ben Campbell]

Oh, forgot to mention - if there is corruption/interruption as the message is written to disk, It'd be good to know if it was just the normal downloading-the-message-from-the-server routine, or if it was due to a filter rule being applied after the message had been downloaded.
Not so easy to tell after the fact, but if there are filters involved, it could be a different mechanism going wrong, rather than the normal message download path.
Thanks!

Comment 25

[John Owles]

In my case there are no filters involved

Comment 26

[quadronom]

Neither were they involved in my case.

Comment 27

[Wayne Mery (:wsmwk)]

if it was due to a filter rule being applied after the message had been downloaded.

Filter logging can be turned on, at the bottom right of the filters dialog.

Comment 28

[quadronom]

So today it happened on 6 subsequent emails – never had this before!
4 had the same minute of sending, but the others were hours apart.
All they had in common was that when looking at the source (cmd + U) all of them seemed to show an excerpt of one of the broken emails, randomly cut off (it started with a closing tag).
There were no issues in the console, at least nothing obviously related.
Until this gets resolved: Can we get a hint on how to deal with this? This is really a big issue. Could we e.g. tell TB just to download an email again? It's IMAP, so that should be possible, no?

Comment 29

[John Owles]

I have been having this intermitantly for a while now. Doing a repair folder puts everything back to rights. Until the next time.

Comment 30

[gene smith]

Until this gets resolved: Can we get a hint on how to deal with this? This is really a big issue. Could we e.g. tell TB just to download an email again? It's IMAP, so that should be possible, no?

I don't know of a way in TB to download a selected imap message again. Of course, repairing the folder will download ALL the messages again, but I don't think you mean this.

You might try copying the corrupted message to an empty folder on the same server and try opening it from there. This may effectively re-download the message.

You might avoid future problem by disabling offline storage for Inbox (or any problem folder) so when future message come in they are not stored to an mbox file but to disk cache. It will keep your existing mbox file (one for each imap folder) but new messages won't go there.

Of course, if you need to work in offline mode sometimes, it may not be possible to read all your older messages while offline if they are not stored in the mbox file. Recently accessed messages still in disk cache (default cache size 1G) will be accessible offline.

Comment 31

[Ben Campbell]

A couple of things for people to try:

1. Does the problem still occur if you reduce the maximum number of IMAP connections down to one? (I've an idea Magnus tried this, but there have been a few changes since so I'd like more data on it).
2. Does the problem still occur if you turn message quarantining on? [1]

The quarantining saves messages to a temp file before adding them to the mbox. I've been thinking for a while that we should always do this, just for the sake of simplifying the code. And I suspect doing that might help this problem...
But really, I'd like to understand what the problem is before fixing it :-) Such "fixed" problems have a nasty habit of popping up again in unexpected ways...

In the meantime, I'm refining the logging in the mbox code to better try and track what's going on. I'll get some people to run with the logging on and try and capture this bug in the act.

[1] That means setting mailnews.downloadToTempFile to true.
In the TB settings GUI, I think it's exposed under the "Privacy & Security" tab, as "Allow antivirus clients to quarantine individual incoming messages"

Comment 32

[quadronom]

(In reply to Ben Campbell from comment #31)

A couple of things for people to try:

1. Does the problem still occur if you reduce the maximum number of IMAP connections down to one? (I've an idea Magnus tried this, but there have been a few changes since so I'd like more data on it).
2. Does the problem still occur if you turn message quarantining on? [1]

Thanks for the recommendations, also to John and Gene. I repaired the folder, so at least I got my mails back for the moment.
As a first step I did number (2) and will report back here, if it happens again or not.
The issue is not occuring so frequently that it might be too soon however.

Comment 33

[Francesco]

I'm refining the logging in the mbox code to better try and track what's going on.

That happened to the idea from bug 1872849 comment #15. Would a build with that tweak help identify the cause?

Comment 34

[Ben Campbell]

(In reply to Francesco from comment #33)

I'm refining the logging in the mbox code to better try and track what's going on.

That happened to the idea from bug 1872849 comment #15. Would a build with that tweak help identify the cause?

No, that was just to check if there was any left-over pre-mbox-refactor code sneakily trying to write directly to the mbox file, but I'm pretty sure now that's not the problem.

Comment 35

[Ben Campbell]

Attachment: Bug 1890230 - Improve mbox write logging. r=#thunderbird-reviewers

Comment 36

[Ben Campbell]

That patch tweaks the mbox logging to be more useful for this kind of case.
You can grab a build with it right now from the try server:
https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=5bf5133b6caffdda750afb61b5948c2b2f41db71
(Click on the 'B' for the platform you want, then download the installer on the "Artifacts & Debugging Tools" tab...)
But hopefully it'll land into the nightly build soonish.

So I'd really like to catch this error red-handed. Best way to do this:

1. Run with MOZ_LOG="mbox:3" in effect. It's tricky because the borkage will likely happen during message download, well in advance of a user noticing it... but I live in hope. I want to see the log at the time this problem occurs, so I guess logging to a file is probably the way to go, so there's a bit of history there. Other than the folder names, there's no personal information logged with "mbox:3".
2. Ideally, I'd also like to know the byte offset in the mbox file for the borked message so I can correlate it with the log. To do this, load the mbox into a text editor and search "From " at the beginning of a line (the message separator). Borked messages are easy to spot - you'll see a message suddenly get interrupted by a "From " line, and the beginning of the headers for a new message.

I realise that's a bit of a faff, but it'll shortcut my fingertip search through the IMAP code, trying to spot potential places which might go wrong.

Thanks!

Comment 37

[quadronom]

Attachment: broken_message.png

It happened again, so Number 2 alone does not do the trick.
This time the email content is not empty, it shows part of another email, weirdly cut off.
I'll try suggested number 1 now.

Comment 38

[Yury]

Drive-by comment: You're calling m_OutstandingStreams.remove(folderURI); twice in FinishNewMessage(). That's not intentional?

Comment 39

[Ben Campbell]

(In reply to Yury from comment #38)

Drive-by comment: You're calling m_OutstandingStreams.remove(folderURI); twice in FinishNewMessage(). That's not intentional?

Oh yes - well spotted, thanks!
I've removed the redundant one (it wouldn't have caused problems - the remove() works even if the item is gone).

Comment 40

[Ben Campbell]

(In reply to quadronom from comment #37)

It happened again, so Number 2 alone does not do the trick.

Oh, that's interesting. Quarantining should make the message writes more atomic - much less chance of partial writes. So that gives me some clues where to look...

This time the email content is not empty, it shows part of another email, weirdly cut off.

Thanks for that! Next time it happens, if you get the chance, could you try and have a look at the folder's file on disk and see where the corruption looks like?
The mbox file for the folder will be in <profiledir>/ImapMail/<hostname>/<foldername>.
If you load that into a text editor and do a search for something unique in the borked message (like that `Part_3753541_...`` boundary id), you should find somewhere nearby where a message is interrupted by new "From " line, followed by a new message. So, just search for the message, then look for odd-seeming "From " lines above and below it. If you post a little snippet centered around there (with any personal info edited out, of course!), that might help.
Thanks!

Comment 41

[gene smith]

I tried to duplicate this by putting lots of new messages externally into imap folders so they look like new messages. Never could see a problem after doing this a bunch of times. When new messages are delivered to the imap server folder they are fetched serially by TB all in the same imap thread. I don't think it's possible for more than one thread to be streaming the same message to the folder's mbox file. This is especially true when autosync is enabled which causes all new messages in a given folder to be auto-fetched serially and stored, all by the same imap thread. Even if autosync (on by default) is turned off, the messages would only be downloaded and stored when accessed the first time. If a messages is partially downloaded before the next one is accessed, something called "pseudo-interrupt" is supposed to abort the download and prevent partial writes of offline store and other issues.

Comment 42

[Yury]

Another drive-by question: Are you sure that new messages are always written at the end? Not that the users reads or tags a message which writes to the middle of the mbox file (changing headers) and then writing new messages continues from there with the content of a new message.

Comment 43

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/a6a72f079851
Improve mbox write logging. r=mkmelin

Comment 44

[Ben Campbell]

I just noticed that all the screenshots in this bug show the borked message body to be the start of an attachment (or at least the start of a mime part - one looks like the top-level multipart mime container).

Now that smells odd.

Too late at night here for me to pursue this right this instant, but just wanted to post it here as something to look out for. Maybe having lots of test messages with big attachments might be a way to get a repeatable test case...

Comment 45

[Ben Campbell]

[EDIT: I'm pretty sure the mime-boundary thing here is a red herring. I think it's an artifact of how the display code handles being fed data starting part-way through the message. It assumes that there message begins with a block of headers, and attempts to parse them until it sees a blank line to denote the start of the message body. Statistically the first blank line is very likely to be because of a mime block, hence the separator being interpreted as the start of the message body.]

I'm increasingly thinking that it's not a case of the mbox writing getting screwed up, but something else (Gene's comment #44 kind of adds a bit of weight to that too).
mbox totally ignores multipart structures and treats the whole message as a header block followed by a body block (any multipart stuff is contained within the body block, and handled elsewhere).

But the problem we're seeing here seems to be very aligned to multipart parts:

1. The displayed bodies of all the reported cases here seem to start very precisely at a multipart boundary (eg Comment #9, Comment #37).
2. We can see multiple parts of the message (Comment #37)
3. The first visible part in the borked multipart message doesn't always seem to be the text/plain (or even text/html) parts. Sometimes it's image/png, which you would not expect to see at the start of a message - rather it looks like an attachment or inline (as in Comment #9).
4. I think the reported cases display blank values for To:, From:, Subject: etc...

It kind of feels like IMAP is occasionally skipping the first part (or parts) of messages - that's what 1, 2 and 3 look like.
Given that all the email headers (To/From/Subject etc) are in the top-level root part, skipping the beginning of it would leave the header parser (which builds the entry in the msg database) with blank values, which looks like 4.
I'm not saying that's what's happening, but it describes the kind of borkage we see.

Annoyingly, I still can't replicate this myself :-(

Comment 46

[gene smith]

It kind of feels like IMAP is occasionally skipping the first part (or parts) of messages - that's what 1, 2 and 3 look like.

When offline store is in use for a folder, TB's imap has always only fetched the whole message -- never intentionally fetches an individual part.
When offline store is not used, imap previously (before Bug 1805186) would try to fetch individual mime parts. But now, even without offline store, it doesn't. And, of course, no offline store means there is no mbox file.

Another possibility is maybe the server itself is returning just a part. I've looked back over the comments and no one has said what type of IMAP server is being used. It's unlikely, but maybe there is a bug in the server.

A big "faff" (had to look that up :)) but for someone who sees this bug more often than typical, running and continuously recording an IMAP:4 level log might show something. Normally running continuous can fill up an unlimited amount of disk space to hold the log. But there is a parameter "rotate" that allow you to limit the space used by the log and cycle the output between 4 files. Here an example setup using the required environment variables MOZ_LOG_FILE and MOZ_LOG:

MOZ_LOG_FILE=~/tblog
MOZ_LOG=IMAP:4,timestamp,sync,rotate:1000

This will rotate the log output between 4 files tblog.moz_log.0 .. tblog.moz_log.3 with each file reaching maximum size of 250M = 1000/4. So the total log size will never exceed 1000M bytes. More or less space can be allocated for the total log file size depending on the amount of free space available, i.e., doesn't have to be exactly 1000.

The general logging instruction (doesn't mention rotate) are dependent on OS type and are here:
https://wiki.mozilla.org/MailNews:Logging

The info about "rotate" is here:
https://searchfox.org/comm-central/rev/4b1d2b3806df696348bbab462ff9f7049b37e576/mozilla/netwerk/docs/http/logging.rst#234

So when the bug occurs, should be able to see exactly what is getting fetched from the server by looking at the most recently updated of the 4 log files.

Note: Probably should also include Ben's new mbox item in the log like this:

MOZ_LOG=IMAP:4,timestamp,sync,rotate:1000,mbox:3

Comment 47

[Ben Campbell]

(In reply to gene smith from comment #46)

Thanks for the excellent logging summary Gene!

Another possibility is maybe the server itself is returning just a part. I've looked back over the comments and no one has said what type of IMAP server is being used. It's unlikely, but maybe there is a bug in the server.

Yeah, I didn't mean to imply that I thought a server bug was the issue here - I think a server bug like this is really unlikely. It was just a handy way to describe the kind of symptoms we're getting.
That said, it would be good to know what servers people experiencing this bug are on, just to help rule it out entirely.

Comment 48

[Ben Campbell]

Magnus: have you seen this bug occur again recently?
If so, does the resulting mbox look like the old new-message-interrupting-old-one-at-a-random-point issue?
Or does it look more like the surgical starts-exactly-at-a-mime-boundary we seem to be seeing here?

Comment 49

[Worcester12345]

(In reply to Ben Campbell from comment #47)

(In reply to gene smith from comment #46)
...
That said, it would be good to know what servers people experiencing this bug are on, just to help rule it out entirely.

Courier

Comment 50

[John Owles]

In my case, I have IMAP server provided by Freeola

Comment 51

[Magnus Melin [:mkmelin]]

For better or worse, I haven't seen it since comment 18. (Earlier observations were on microsoft and google mail services, so I think we can rule out server involvement)

Comment 52

[gene smith]

I wrote in comment 46:

TB's imap has always only fetched the whole message -- never intentionally fetches an individual part.

Actually, today I remembered that the "whole message" is not always fetched, depending on the current "chunk size". By default, the chunk size is 65536 but TB is supposed to dynamically increase the chunk size to an optimum size after measuring the fetch speed of newly fetched messages. But what I noticed is if I set the "chunk" prefs back to their default values and let them be re-optimized to larger size, nothing happened. I.e., the chunk size remained at 65536 after fetching a lot of large messages (>50M)

I finally determined that the chunk size optimization process only occurs when the full message is fetched and not "peeked" (but don't know why). IMAP peeking causes the message to be fetched but not marked with \seen imap flag. So only if the settings "Automatically mark message as read" and "Immediately on display" are both selected will the optimization occur (peeking doesn't occur). Since I prefer to mark messages as read manually, no chunk optimization occurred until I made these settings.

Anyhow, I'm not saying chunking is causing this bug, but it's possible. Larger messages with more default size chunks (65536) might have more opportunity to have problems. So it might be interesting to see if the optimization has occurred by looking at these parameters and see if they are still at the default settings:
mail.imap.chunk_size
mail.imap.min_chunk_size_threshold

So, assuming "small" chunks are causing the bug, allowing optimization to larger chunks, or disabling chunking completely with mail.server.default.fetch_by_chunks set false may fix or at least work-around the problem.

Comment 53

[Francesco]

There seem to be a few related reports coming in: Bug 1911235, bug 1911067, bug 1911284, bug 1911393, maybe also bug 1911012.

Comment 54

[Wayne Mery (:wsmwk)]

pop gmail https://support.mozilla.org/en-US/questions/1455811

Severity TBD https://support.mozilla.org/en-US/questions/1456238

(perhaps unrelated) feeds https://support.mozilla.org/en-US/questions/1455756

Comment 55

[david]

These three showed up in SUMO yesterday and all seem similar (blank messages). Any tips for a workaround will be appreciated. One user downgraded to 115 and not fixed, and a folder repair wiped out whatever was left. Another user did cntl and 'u' and content was garbage. I'm at a loss in SUMO on responding.

Comment 59

[Francesco]

FWIW, https://support.mozilla.org/en-US/questions/1460772 states: Messages arriving when the email client is closed are displayed incorrectly when it is opened. Messages received while the program is running are displayed correctly.

That coincides with my own observation in bug 1872849 comment 62. Somewhat concerning warnings are issued after starting the program when new messages are downloaded.

Comment 64

[WfJjrdAr]

(In reply to Magnus Melin [:mkmelin])

As a workaround, right click the folder and chose Repair Folder.

I don't have such an item in the context menu of my mailbox folders. Only the following:

• Open in New Tab
• Open in New Window
• Search Messages…
• New Subfolder…
• Delete
• Rename
• Move To
• Copy To
• Compact
• Mark Folder Read
• Favourite Folder
• Properties

Comment 65

[WfJjrdAr]

(In reply to dquiros from comment #5)

I see the option Right Click, Properties, Repair Folder.

Yes, thank you: "Folder Properties" / "General Information" / "Repair Folder (Sometimes the folder index (.msf) file becomes damaged and messages may appear missing or deleted messages continue showing; repairing the folder may fix these issues)".

Comment 67

[Aaron]

This issue is still occurring in the Thunderbird software.

The procedure to right click on the inbox go to properties and repair folder seems to resolve it till it happens again.

I previously logged this under bug 1913425 which has been closed due to being a duplicate.

Comment 68

[Wayne Mery (:wsmwk)]

(In reply to Aaron from comment #67)

This issue is still occurring in the Thunderbird software.

Please state your full version number when providing data

Comment 70

[Francesco]

If you're looking for a reproducible case, see bug 1909111 comment 25.

Comment 72

[quadronom]

(In reply to Ben Campbell from comment #40)

Thanks for that! Next time it happens, if you get the chance, could you try and have a look at the folder's file on disk and see where the corruption looks like?

So it happened again, "sadly" with an empty Email (= not a cut-off message).
I opened the mbox file and lo and behold: This very last email, that is shown empty, is not even in there.
The email before is a HTML email, and it ends with "=0A </tbody>=0A </table>=0A </body>=0A=0A</html>" and then three empty lines. No "From" line, so even that one is missing (!)

Comment 73

[Francesco]

In the reproducible case we have in bug 1909111 comment 33, there is a discrepancy between raw data mbox file and database (.msf) file. The message is registered in the database and hence shows in the message list, however, it's not in the mbox file, so when you click on it, you get an empty message display. Appears to be related to compaction which runs while or shortly after the message is received. Do you have automatic compaction turned on or can you even tell that it's running?

That said, other reporters don't get an empty display but instead some message source is shown, usually starting at a MIME boundary, see for example bug 1920329, your own comment 37 or comment 9.

Comment 74

[José María Fernández]

Hi, I am a 128.2.x user who is also suffering these issues, and from what I have read in this thread I'm suspecting that the folder indexes could be stale after mail compaction.

In my local setup, long time ago I disabled automatic mail compaction, so I explicitly have to approve it when I receive the notification. And a few days ago, these issues appeared just after I approved mail compaction in the compacted folders. I recovered them just repairing each one of the IMAP folders (each one with thousands of e-mails).

Hope this helps!

Comment 75

[quadronom]

(In reply to Francesco from comment #73)

Do you have automatic compaction turned on or can you even tell that it's running?

Yeah, automatically after 20 mb, that's currently the setting (the default I guess, never made changes).
Not sure how I could tell if compaction is currently in progress or not. Does it still say so at the bottom? I'll keep an eye on it.

Comment 76

[Francesco]

It still says do, but due to bug 1901846 it's not so obvious any more.

If you have the problem regularly, you could switch off compaction completely and see whether that makes problem disappear. Bug 1909111 and comment 74 suggest that the issue it caused by compact that runs very soon after a message was added to a folder.

Comment 78

[Yogesh]

I am facing the same issue in 131.0beta. I have not tried repairing offending folders. But restarting TB fixes the problem temporarily for me.

Comment 79

[Ben Campbell]

I don't think Bug 1909111 is the same issue. The symptoms on that bug seem different.
The symptom that stands out to me on this bug is that the corrupted messages always seem to break at the mime boundary - that also matches the reports in Bug 1919667, Bug 1920329 and Bug 1913425.

Stuff we know:

• Filtering doesn't seem to be a factor (comment 25, comment 26).
• Turning on mbox Quarantining doesn't help (comment 37).
• Messages with attachments (or html parts) may be more susceptible, going by the mime-boundary symptom.
• There are strong hints that folder compaction is a trigger (comment 73 to comment 76).

Comment 80

[Ben Campbell]

OK, I still don't know if the mbox file itself is being corrupted, or if it's something in the DB or display making things look odd.

Ideally, I'd like an example containing a screwed-up message to study: an mbox file and the .msf file to go with it.
If anyone has such an example that doesn't contain sensitive data, they can email them to me at: [e-mail]

Failing that, I thought of a way for people to check the mbox file themselves:

1. while Thunderbird is not running, copy the borked mbox file from the <profile>/ImapFolders/<host>/ dir into the <profile>Mail/Local Folders/ directory. Make sure to give it a name that doesn't overwrite existing files in there!
2. start Thunderbird.
3. You should see the new folder in Local Folders
4. If you click on it, it should list all the messages
5. see if the corrupted message still looks corrupted in the local folder copy.

step 4 might take a while for big files - the process forces TB to scan the mbox, building up a fresh database for the folder from scratch.

If the local-folder copy has corruption, then the mbox is probably corrupted.
If the local-folder copy looks OK, then the mbox is probably OK, and the problem lies with the message data in the original DB for the imap folder (or potentially the display code, but that seems less unlikely)

Comment 81

[Ben Campbell]

(In reply to Ben Campbell from comment #79)

I don't think Bug 1909111 is the same issue. The symptoms on that bug seem different.
The symptom that stands out to me on this bug is that the corrupted messages always seem to break at the mime boundary - that also matches the reports in Bug 1919667, Bug 1920329 and Bug 1913425.

I think I was wrong on that, and that Bug 1909111 might back in play for this.
It seems that the mime-boundary display symptom is just a side effect - I think it's the display code being given a partial message and just trying to make sense of it. I'd guess the first part of the message is missing, and the display code is skipping everything until the first blank line, assuming it's the message headers.

Comment 82

[Kai Engert [:KaiE:]]

Does anyone have a copy of a corrupted folder file?

Comment 87

[Ben Campbell]

Thanks to more detective work from @darktrojan, we think we have an idea what might be happening and how to fix it.
End of the day here so I'm leaving some developer-oriented breadcrumbs to follow.

Firstly, how to replicate it (assuming it's the same thing):

1. change the folder compaction code to stop it releasing the folder lock when it's done: https://searchfox.org/comm-central/source/mailnews/base/src/FolderCompactor.cpp#110
2. run TB, connected to an IMAP account
3. delete a message, then compact the folder
4. send another message to the account, so IMAP tries to pipe it to the folder.

Because the folder is still locked, the delivery partly fails, and the message appears borked.
So we think it might be down to messages arriving while a folder compaction is in progress.

Details:

1. folder compaction starts, and locks the folder
2. the IMAP protocol code sees a new message on the server, downloads it, and tries to pipe it to the folder
3. writing the message to the mbox fails (as it should), but a msgHdr for the new message is still added to the DB. This msgHdr:
   • has the "Offline" flag set
   • has no "storeToken" or .messageOffset values (which define the location of the message in the local msgStore).
4. when displaying the message, an old workaround (entirely my fault!) meant that the .messageOffset returned a rubbish value, and that was the place in the mbox file the display code was starting from. So inevitably it'd be smack bang in the middle of another message. Statistically that message would likely be a big one with attachments, and the display code would discard everything up to the first blank line (indicating the end of the message headers and the start of the message body). Inevitably that'll be a MIME part boundary.

What we're doing about it:

We're taking the opportunity to perform the long-overdue work of entirely removing .messageOffset (Bug 1720047).
Preliminary results are that it solves the corrupted-message display issue. Because there's no local copy of the message, it downloads it from the server. However, it doesn't store that message locally - it'll have to download it again every time you display it.
We think this is because the "Offline" flag is set on the message. This flag is supposed to tell the folder that we've got it locally.
To patch things over, clearing this flag in such a case should ensure that it's downloaded again, just once, and stored locally.
Better still would be to figure out why the flag is ever set in the first place, and stop that happening (remember the folder is locked by the ongoing folder compaction in this case).

More tomorrow.

Comment 88

[Ben Campbell]

(In reply to Ben Campbell from comment #87)

1. change the folder compaction code to stop it releasing the folder lock when it's done: https://searchfox.org/comm-central/source/mailnews/base/src/FolderCompactor.cpp#110

Quick clarification - that ReleaseSemaphore() call is just a failsafe one - the real one you want to disable is:
https://searchfox.org/comm-central/source/mailnews/base/src/FolderCompactor.cpp#515

There are three altogether. The one in OnCompactionCompleted() is the only one used if compaction goes normally, but doesn't hurt to disable the other two as well.

MOZ_LOG="compact:4" will show you when compaction runs. Remember that there's an expungedBytes check, so if the folder has no messages to delete in it compaction won't run at all. So delete a message, compact, then you should be left with a nice, locked folder for replicating the problem!

Comment 89

[Antony]

i staerted to post this a couple of days ago - there have been a number of posts since then so might be redundant,

i have had this issue, intermittently, over a number of releases, going back a couple of years at least i would say.

Often it clears by a refresh for the folder or a TB restart. Sometimes its a specific email as well as a whole subfolder of emails.

In addition i now have compaction to run on approval not Auto, no filters and all my accounts are IMAP

Comment 90

[Ben Campbell]

(In reply to Antony from comment #89)

i have had this issue, intermittently, over a number of releases, going back a couple of years at least i would say.

If it really is the issue we've been chasing down over the last day or two, then there's definitely scope for it to have been happening for a long long time. But I suspect that recent changes jiggling things about a bit mean that bugs that used to occur so rarely that hardly anyone noticed started happening more frequently.

Update for today: I think the 4 or 5 patches involved (spread out over a few bugs) have landed now, so they'll be heading into daily builds and further testing now.

Comment 91

[Tanja Kluth]

Hello, i got refered to you here, i frequently get emails which seem corrupted
• Thunderbird- version: 128.3.0esr (64-Bit)
• BOS: Windows Home 23H2
• Account (POP / IMAP): IMAP
• selfhosted on our own company server + 1 mailbox on a friends server
• Bitdefender Total Security
• Firewall: the win 11 system firewall (standard, nothing changed)

So randomly when me or 2 other collegues get emails they turn out like this
"--------------mVnhv0xhjJuIvAM5f981p1iQ--"
i can be that they load after some time or when i move them into another folder.
we also use roundcube or an old win 10 PC as backup, there we can always read the "corrupted" emails.

I have an inbox, inbox.msf and inbox.sbd (because it has a subfolder) mostly empty except another email and the corrupted email if you want and it if helps. but since i m new i do not know how to upload a file / these files if you still need them

since i have that issue frequently i probally can provide files any other moment as well

(i am getting along with this issue now, tho it is alil annoying, i just want to help)

Comment 92

[Tanja Kluth]

Attachment: corrupted folder

i don't know if you still need this, but thats 2 "freshly corrupted" emails. It is important to note however that those are getting displayed in Roundcube, only under windows 11 for me they seem to be corrupted no matter what i do. later in the day they might be displayed. all other emails are displayed properly

Comment 93

[Tanja Kluth]

Attachment: corrupt.msf

2nd file, sorry this is new to me and i hope i did this right

Comment 94

[Kai Engert [:KaiE:]]

Tanja, thanks a lot for providing these files!
Did you experience this fresh corruption with a Thunderbird version 128.x ?

Comment 95

[Ben Campbell]

(In reply to Tanja Kluth from comment #92)

Created attachment 9430315 [details]
corrupted folder

Thanks for that! Quick question: was that the mbox that originally experienced the corruption, or did you copy the corrupted emails into a fresh folder to post here?

Comment 96

[Kai Engert [:KaiE:]]

Tanja, if you manually created new folder named "corrupt" and then copied over 2 messages from another folder, the files don't fully help us. We are trying to understand what the original folder file looks like.

If the original-folder.msf index lists incorrect positions into file original-folder, then it explains why the messages that are copied to folder corrupt are incomplete.

We are trying to understand whether "original-folder" contains more than the data that was copied to "corrupt".

For example, the first message you copied contains an end marker "005_e38dee8ad306479b8e874dfd275e7085personalwerkde".
Could you check whether this text can be found in "original-folder" multiple times?
Is there an email header section with "content-type: ....; boundary....005_e38dee8ad306479b8e874dfd275e7085personalwerkde", and at least two more lines that contain 005_e38dee8ad306479b8e874dfd275e7085personalwerkde ?

Comment 97

[Tanja Kluth]

Hallo,
yes, these corruptions keep appearing randomly since thunderbird update to 128.x minimum 128.2 and now 128.3 (not sure about 128.1 tho coz i was away fromeverything for a few weeks)

unfortunaly because of all the emails i have i needed to copy the corrupted emails into a different folder. sorry. i might try and do it the other way around, move everything in another folder and give you my inbox if (and for some reason i m sure there will be) a new corrupted email.

"Is there an email header section with "content-type: ....; boundary....005_e38dee8ad306479b8e874dfd275e7085personalwerkde", and at least two more lines that contain 005_e38dee8ad306479b8e874dfd275e7085personalwerkde ?" those header sections seem odd to me, i do get emails from that server but on a different account (in the same thunderbird tho" those 2 corrupted emails are supposed to have absolutly nothing to do with what ever personalwerkde.

I have like 5 different email Accounts all on 3 different selfhosted or "friend-hosted" Server. These 2 Emails were in an Email Account on my friends server, which is my personal / private mail. i never get any emails from "personalwerk" there so it should not be in any header. i wanted to check the 2 emails but they are now completly white and empty
(the overall mailbox size of this account according to the server is 202 MB, the 2 broken emails are still being properly displayed by roundcube)

at least 2 of those Account have that issue (the other ones do not get enough emails so to get that issue i guess?) the 2nd account is my companies email with my name as well and i do almost daily get emails from personalwerk there, i checked them but i found neither "005_e38dee8ad306479b8e874dfd275e7085personalwerkde" not even "005_e38dee8ad306479b8e874dfd275e7085"

sorry i couldnt help more, i will try to get inbox when it happens again for now the correpted emails go in the trash (and i m even more confused)

kind regard
Tanja

Comment 101

[John Owles]

Strange. This happened to me a few times 6 months ago but, so far, it hasn't happend again. I don't think I have changed anything.

Comment 102

[Kai Engert [:KaiE:]]

Attachment: WIP: Bug 1890230 - Backout bug 1719121 and later related patches since TB 122.

Includes the following bugs:
https://bugzilla.mozilla.org/show_bug.cgi?id=1891426
https://bugzilla.mozilla.org/show_bug.cgi?id=1719121
https://bugzilla.mozilla.org/show_bug.cgi?id=1868504
https://bugzilla.mozilla.org/show_bug.cgi?id=1868705
https://bugzilla.mozilla.org/show_bug.cgi?id=1872232
https://bugzilla.mozilla.org/show_bug.cgi?id=1873282
https://bugzilla.mozilla.org/show_bug.cgi?id=1857450
https://bugzilla.mozilla.org/show_bug.cgi?id=1880765
https://bugzilla.mozilla.org/show_bug.cgi?id=1872849
https://bugzilla.mozilla.org/show_bug.cgi?id=1882971
https://bugzilla.mozilla.org/show_bug.cgi?id=1885812
https://bugzilla.mozilla.org/show_bug.cgi?id=1880043
https://bugzilla.mozilla.org/show_bug.cgi?id=1882282
https://bugzilla.mozilla.org/show_bug.cgi?id=1887047
https://bugzilla.mozilla.org/show_bug.cgi?id=1887778
https://bugzilla.mozilla.org/show_bug.cgi?id=1890135
https://bugzilla.mozilla.org/show_bug.cgi?id=1890453
https://bugzilla.mozilla.org/show_bug.cgi?id=1889653
https://bugzilla.mozilla.org/show_bug.cgi?id=1891937
https://bugzilla.mozilla.org/show_bug.cgi?id=1857450
https://bugzilla.mozilla.org/show_bug.cgi?id=1892866
https://bugzilla.mozilla.org/show_bug.cgi?id=1890448
https://bugzilla.mozilla.org/show_bug.cgi?id=1888790
https://bugzilla.mozilla.org/show_bug.cgi?id=1900172
https://bugzilla.mozilla.org/show_bug.cgi?id=1872230
https://bugzilla.mozilla.org/show_bug.cgi?id=1906835
https://bugzilla.mozilla.org/show_bug.cgi?id=1904799

Comment 103

[Kai Engert [:KaiE:]]

Attachment: WIP: Bug 1890230 - Fix issues and test failures after the backout.

Comment 104

[Kai Engert [:KaiE:]]

Attached is an experimental backout patch.
It's not yet clear if that approach is sufficiently stable or whether it introduces other issues.
Attaching here for investigation.

Comment 105

[Ben Campbell]

For users of version 128 Geoff did a Try run with our potential fix in it.

So if anyone wants to give it a go and see if it improves the situation, download and install one of these for version 128:

• target.installer.exe (Windows 64bit)
• target.dmg (Mac)
• target.tar.bz2 (Linux)

For beta users, update to the beta 132.0b4 or newer.
drill down from https://archive.mozilla.org/pub/thunderbird/candidates/132.0b4-candidates/build2/

Also, comment 87 has details about steps that leads to failure.

Please post here to let us know how it goes!

Comment 106

[Kai Engert [:KaiE:]]

Hello, has one of you been able already to test the Thunderbird version from comment 105 and check if it prevents new corruption?
Thank you in advance for testing.

Comment 107

[Harold]

Bug showed up again after I turned off all compacting options on October 7th.
Since it is usually the odd message that shows up with this bug, it is not easy to reproduce it.

In any case, here are a few observations:

• I have verified the INBOX (mbox) file and the INBOX.msf file. From what I can decipher of the Mork format msf is that it references the correct starting point of the email in the mbox file.
• Cleared the cache - no difference, message still showing up as corrupted.
• Closed TB and deleted the gloda (global-messages-db.sqlite) to rebuild that index - no difference after restarting TB. The same message is still corrupt.
• Although the compact option is turned off in setting, TB seems to do something after all.
• I have observed several INBOX-<n> folders created after I turned off compacting
• A new a new msf was also created today (maybe related to the gloda rebuild?)
• The INBOX-<n> folders are a lot smaller than the main INBOX (15-35%)
• INBOX.msf-4.compact-backup - this seems to be a folder related to a compact process - however compacting is turned OFF.

Regards, Harald

Comment 108

[Wayne Mery (:wsmwk)]

(In reply to Harold from comment #107)

Bug showed up again after I turned off all compacting options on October 7th.

Using version 128? There are no fixes yet in 128 for this class of issues.

But if you were using a recent beta or daily build ... antivirus software do you use?

Also noting from your bug report "Several times over the last couple of years, I have noticed that the list of messages in GMAIL Inbox sometimes refer to a corrupted messages", so you have had long term issues.

Thanks

Comment 109

[Harold]

I've had this issue for several years - not just with the current 128.2.3 release. - and I never used beta releases.
Without finding any good way to troubleshoot it, I got tired of it and started looking into the msf index which I thought was the source. This was November/December 2023. According to my update history, I was running 115.5 or 115.3 at that time.
AV products I've been using were Cylance (until they closed the individual offering) and Microsoft Defender after that.
TB has always been configured to "Allow antivirus clients to quarantine individual incoming messages" and I have later whitelisted the profile folder of TB in Defender (not the best security practice, but...)

Just lately I found out how to decipher the msf (mork) format, and I was able to find where it referenced in the MBOX file (byte location).

So far in my troubleshooting, I don't know what is triggering the the issue. I thought initially it was the compact process due to multiple INBOX-<n> files and previously the nstmp files showing up in the folder, and I still suspect there is a process that causes the MBOX file to split / recreate.

It only happens for the GMAIL imap folder where I have a total of around 13K messages, but this is also the most active account.
I am now going to setup some directory monitoring of the files in there to find out when those extra INBOX-n files starts to show up.

Harold

Comment 110

[Anje]

Still getting reports in Support forum.
https://support.mozilla.org/en-US/questions/1469800

Comment 111

[Kai Engert [:KaiE:]]

While working on the best fix for this issue, we're trying an experiment in parallel.
If you're willing to test the experimental fix, please have a look at bug 1925085.
Thank you!

Comment 113

[Peter Bariska]

Yet another very strange bug of folder corruption. I tried to forward an email. The new window that opened contained the original subject but the email body contained a totally different message of an email I received 6 months ago. To be sure, the sender of that email was different, the subject was different too. It had nothing to do with the email I wanted to forward.
I closed the window and discarded the message. I tried once again to forward the email. Once again, the same email received 6 months ago appeared in the body. I could not believe my eyes. Finally I had to do a folder repair to fix the issue.

Comment 114

[Wayne Mery (:wsmwk)]

Reporters,

Fixes are now available in 128.3.3. First right+click on broken folder > properties > Repair.
Then update to 128.3.3 with Help > About.
Please post your results.

Comment 115

[James L]

(In reply to Wayne Mery (:wsmwk) from comment #114)

Reporters,

Fixes are now available in 128.3.3. First right+click on broken folder > properties > Repair.
Then update to 128.3.3 with Help > About.
Please post your results.

Wow, I just got this problem and looking for solution. This totally fixed all the corrupted emails for me. 👍🏻

Comment 116

[Tanja Kluth]

it seems to have it fixed for me, thank you everyone, awesome!!

Comment 117

[Harold]

Upgraded to 128.3.3, but bug was still present.

I noticed by monitoring my imap folder with the MBOX for INBOX that during the update and restart, TB created the odd smaller INBOX-1 and INBOX-2 files together with INBOX.msf.compact-backup.

This seems to be related to the** "EXPUNGE Inbox on Exit"** setting for the account, which has now been turned off.

My file monitoring script hasn't detected any of those INBOX-<n> files for the last week, and I didn't notice any corrupted messages until today AFTER the upgrade and restart of TB.

Is it possible that when TB does a Compact operation, it stops in the middle of the process and leaves partial files ?

Comment 118

[Kai Engert [:KaiE:]]

Harold, thanks for your feedback. Please allow me to ask for clarification. Besides the creation of the additional INBOX-* files, after repairing your folders, did 128.3.3 again corrupt at least one of your (previously repaied) folders?

Comment 119

[Harold]

A 99% yes Kai, although it is difficult to search for such corruption.
I found the recent corruption by comparing INBOX-1 with INBOX-2 and noticed where they started to differ.
Would you have a suggestion on how to find possible corrupted messages?

Another clarification: It is only the GMAIL INBOX that has this corruption issue, and I have about 9 email accounts in TB. Just a few of them were set with Expunge on - in addition to Gmail. They were small folders without many messages, and I never noticed any problems with those.

I will see if I can reproduce the issue after the upgrade and this current repair by turning the Expunge on again. As I have 13K messages in my Gmail inbox I am trying not to do this repair more than once a day since TB has to download all messages again.

Another observation:
If I shut down TB, delete the msf (index) and start TB again, TB will not download all the messages. It just rebuilds the msf.

Comment 120

[Kai Engert [:KaiE:]]

Thank you Harold, we appreciate your help very much!
We're crossing fingers and hope for the 1%, so we're very interested if the issue re-appears for you, or whether the issue might be fixed (as it seems to be fixed for Tanja).

Comment 121

[Richard Leger]

Attachment: Beta_upgrade_path.png

Fyi,

As reported here: https://thunderbird.topicbox.com/groups/beta/T0d539f174ed7b6cd-M9b226d8d27f7857e07b4b003/inbox-not-updating-after-upgrade-to-tb-131-0b5-64-bit

I started to observe folder corruption after upgrade to TB 131.0b5 (64-bit), my IMAP Inbox is no longer updating.
Running a folder Repair on Inbox completely emptied my Inbox which was then showing "No Message Found"

In my case, issue fixed itself automatically by upgrading to 132.0b4 (64-bit).

In TB 132.0b6 (64-bit) I can still see a lot of errors like the below in the console...

21:11:58.184
gloda.index_msg: Problem entering folder: 201703 VLAN Switch installation in rack, skipping. Error was: undefined:645: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIMsgFolder.msgDatabase]"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: resource:///modules/gloda/IndexMsg.sys.mjs :: _indexerEnterFolder :: line 645"  data: no] IndexMsg.sys.mjs:649:17
    _indexerEnterFolder resource:///modules/gloda/IndexMsg.sys.mjs:649
    _worker_folderIndex resource:///modules/gloda/IndexMsg.sys.mjs:1372
    workBatch resource:///modules/gloda/GlodaIndexer.sys.mjs:1069
    callbackDriver resource:///modules/gloda/GlodaIndexer.sys.mjs:853
    _timerCallbackDriver resource:///modules/gloda/GlodaIndexer.sys.mjs:779
21:11:59.858
gloda.index_msg: Problem entering folder: accounts and licenses review, skipping. Error was: undefined:645: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIMsgFolder.msgDatabase]"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: resource:///modules/gloda/IndexMsg.sys.mjs :: _indexerEnterFolder :: line 645"  data: no] IndexMsg.sys.mjs:649:17
    _indexerEnterFolder resource:///modules/gloda/IndexMsg.sys.mjs:649
    _worker_folderIndex resource:///modules/gloda/IndexMsg.sys.mjs:1372
    workBatch resource:///modules/gloda/GlodaIndexer.sys.mjs:1069
    callbackDriver resource:///modules/gloda/GlodaIndexer.sys.mjs:853
    _wrapCallbackDriver resource:///modules/gloda/GlodaIndexer.sys.mjs:786
    handleCompletion resource:///modules/gloda/GlodaDatastore.sys.mjs:61

Comment 122

[Arthur K. (he/him)]

(In reply to Richard Leger from comment #121)

Created attachment 9432945 [details]
Beta_upgrade_path.png

Fyi,

As reported here: https://thunderbird.topicbox.com/groups/beta/T0d539f174ed7b6cd-M9b226d8d27f7857e07b4b003/inbox-not-updating-after-upgrade-to-tb-131-0b5-64-bit

I started to observe folder corruption after upgrade to TB 131.0b5 (64-bit), my IMAP Inbox is no longer updating.
Running a folder Repair on Inbox completely emptied my Inbox which was then showing "No Message Found"

In my case, issue fixed itself automatically by upgrading to 132.0b4 (64-bit).

In TB 132.0b6 (64-bit) I can still see a lot of errors like the below in the console...

21:11:58.184
gloda.index_msg: Problem entering folder: 201703 VLAN Switch installation in rack, skipping. Error was: undefined:645: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIMsgFolder.msgDatabase]"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: resource:///modules/gloda/IndexMsg.sys.mjs :: _indexerEnterFolder :: line 645"  data: no] IndexMsg.sys.mjs:649:17
    _indexerEnterFolder resource:///modules/gloda/IndexMsg.sys.mjs:649
    _worker_folderIndex resource:///modules/gloda/IndexMsg.sys.mjs:1372
    workBatch resource:///modules/gloda/GlodaIndexer.sys.mjs:1069
    callbackDriver resource:///modules/gloda/GlodaIndexer.sys.mjs:853
    _timerCallbackDriver resource:///modules/gloda/GlodaIndexer.sys.mjs:779
21:11:59.858
gloda.index_msg: Problem entering folder: accounts and licenses review, skipping. Error was: undefined:645: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIMsgFolder.msgDatabase]"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: resource:///modules/gloda/IndexMsg.sys.mjs :: _indexerEnterFolder :: line 645"  data: no] IndexMsg.sys.mjs:649:17
    _indexerEnterFolder resource:///modules/gloda/IndexMsg.sys.mjs:649
    _worker_folderIndex resource:///modules/gloda/IndexMsg.sys.mjs:1372
    workBatch resource:///modules/gloda/GlodaIndexer.sys.mjs:1069
    callbackDriver resource:///modules/gloda/GlodaIndexer.sys.mjs:853
    _wrapCallbackDriver resource:///modules/gloda/GlodaIndexer.sys.mjs:786
    handleCompletion resource:///modules/gloda/GlodaDatastore.sys.mjs:61

No better with 131.0 final or 131.0.1?

Comment 123

[Wayne Mery (:wsmwk)]

The release channel, of which 131.0 is a part, are not receiving most recent patches. And so it is best not to test against them.

Note, 131.0 is equivalent to 131.0b1 which is almost two months old.

Comment 124

[Wayne Mery (:wsmwk)]

In TB 132.0b6 (64-bit) I can still see a lot of errors like the below in the console...

This suggests some corruption still exists, or that corruption resolved in such a way that gloda indexing still fails. The typical symptom will be that global search will fail to find new messages.

Comment 125

[Wayne Mery (:wsmwk)]

Discussion of Harald's issue is moving back to their Bug 1923355 - Corrupted MBOX file for GMAIL

Comment 126

[Antony]

(In reply to Ben Campbell from comment #90)

(In reply to Antony from comment #89)

i have had this issue, intermittently, over a number of releases, going back a couple of years at least i would say.

If it really is the issue we've been chasing down over the last day or two, then there's definitely scope for it to have been happening for a long long time. But I suspect that recent changes jiggling things about a bit mean that bugs that used to occur so rarely that hardly anyone noticed started happening more frequently.

Update for today: I think the 4 or 5 patches involved (spread out over a few bugs) have landed now, so they'll be heading into daily builds and further testing now.

Ben et. al

Currently running latest beta 132.0b6 and not experiencing any issues with lost emails or corruption, though for the latter, i have rarely seen this.
This is where i get details of one email appearing under another.

If this issue shows up again ill check the logs and feedback

Comment 127

[Helder Guerreiro]

I've installed the linked version in one of the users that was having daily folder corruption and since Friday until today she reports that she had no problems.

Comment 129

[Helder Guerreiro]

(In reply to Helder Guerreiro from comment #127)

I've installed the linked version in one of the users that was having daily folder corruption and since Friday until today she reports that she had no problems.

Unfortunately this user, today, once again had this problem. It was displayed the source code of the body of the message. It was fixed after repairing the folder. Note that she was using the 128.3.1esr daily made available in this bug report, the message was from today morning.

Comment 130

[Francesco]

Please upgrade to 128.4.0 which has all the fixes from the "pre-release". The developers like to know whether 128.4.0 still has the issue.

Comment 131

[Arthur K. (he/him)]

<2 Cents>
Just curious if any user experiencing the issue and a Dev have possibly thought to collab on this issue to catch it in real time or peek under the hood so to speak? I know this is a rather unorthodox statement but it seems that even after some fixes went into 128.3.x, it still seems to be happening and something must be at play here? You'll be spinning your wheels endlessly unless you can hands-on analyze the exact setup a user has.
</2 Cents>

Comment 132

[Helder Guerreiro]

Please upgrade to 128.4.0

Is this 128.4.0esr?

Comment 133

[Francesco]

Just FYI, following comment 82, some data was submitted in bug 1920329. That data was an IMAP mailbox with 53 messages, 37 downloaded offline, 16 only on the server. Corruption was reported for three of the 16 not downloaded locally.

Comment 134

[Francesco]

(In reply to Helder Guerreiro from comment #132)

Is this 128.4.0esr?

Yes, anything 128.*.* is automatically ESR.

Comment 135

[Arthur K. (he/him)]

(In reply to Helder Guerreiro from comment #132)

Please upgrade to 128.4.0

Is this 128.4.0esr?

User should be able to do a Help > About Thunderbird and be presented with the option to update to 128.4.0 ESR.

Comment 136

[Helder Guerreiro]

It's done. Will report back if something happens.

Comment 137

[Ben Campbell]

It's worth noting that upgrading to 128.4.0 won't automatically fix up damaged messages - you still need to manually perform "repair folder" to re-download them.

Comment 138

[Arthur K. (he/him)]

(In reply to Ben Campbell from comment #137)

It's worth noting that upgrading to 128.4.0 won't automatically fix up damaged messages - you still need to manually perform "repair folder" to re-download them.

Correct. Let's hope the fixes will prevent corruption post-"repair folder" though. I just got offered 128.4.1 but no relnotes made public yet.

Comment 139

[Richard Leger]

Attachment: TB not parsing correctly a Sent item.png

Seems my Sent items folder seems again corrupted in TB.132.0b6 (64-bit) on Windows 10.

After a quick search, I cannot access a message content in message preview pane. See attached I only see message source code. No recipients appear listed.

I only cache 3 days of items in my IMAP account, so this item content would have been downloaded directly from the server.

Comment 140

[gene smith]

Richard,
"Francesco" at bug 1920329 comment 18 has found a useful perl program to parse the .msf summary files, mork.pl. If you would download this and run it on your Sent.msf file it might show something, e.g., if you problem message is actually stored Offline or not.
I don't know how to run "perl" in windows but I'm sure it is possible. Here's how I run it in linux:

cd ~/.thunderbird/kgmivl5o.default-release/ImapMail/mobile.charter.net
perl ~/Downloads/mork.pl SentMail.msf > SentMail.txt

Then look at the SentMail.txt file (or whatever you want to name it) and find the block of info about the problem message. It will look something like this:

              ID = E10 (3600)
          _scope = ns:msg:db:row:scope:msgs:all
            date = 67087d85 (Thu Oct 10 21:21:09 2024)
    dateReceived = 67087d85 (Thu Oct 10 21:21:09 2024)
          sender = Gene Smith <[e-mail]>
      recipients = Wayne Mery <[e-mail]>
         subject = Bizzare Trash folder
            size = 4ba (1210)
           flags = 11 ( Read HasRe )
      message-id = [e-mail]
      msgCharSet = UTF-8
     msgThreadId = e10 (3600)
        priority = 1
ProtoThreadFlags = 0
      references = <[e-mail]>
    threadParent = ffffffff (-1)
           _refs = E10

I'm not sure about all the fields but the flags = ... is relevant to what you report. If the "Offline" flag is reported for the problem message it is getting fetched from mbox. If it is not set, the message should get initially fetched from the server and subsequently fetched from disk cache (as long as other stuff doesn't "eject" it from cache). Just to be clear, "disk cache" is not the mbox file but the mozilla internal "cache2".

Edit: I don't see that mork.pl does anything malicious (makes network connections, changes the .msf file etc) but if you don't trust it and don't want to run it directly on the in-place .msf file in your profile, you could shutdown TB and copy your .msf somewhere else outside the profile and run mork.pl there.

Comment 141

[José María Fernández]

I updated to 128.4.1esr , I have repaired the 729MB INBOX of one of my IMAP accounts (hosted at imap.gmx.com) , and now INBOX has 17 entries out of "9769+ conversations" with completely wrong dates (i.e. the timestamp of the re-fetch instead of the right one), and one entry with no subject sender, date, ... which once open seems to have the start of the e-mail misaligned.

This broken entry is a false positive, as it is an ill-formed IMAP entry (the GMX webmail is showing it in the very same way). But about the other 17 e-mails with wrong timestamps, I have been having a look at these e-mails through the GMX web site, and they are properly represented (even the correct date). As I was present when the rebuilding process was happening, I was able to realize the issue about the dates appeared on initial index creation, before the bulk of messages was fetched.

Could those wrong dates at the index be provided by the IMAP server (or not properly parsed), so they are blindly trusted by the code? And as they are trusted those timestamps are not replaced with the correct info when the whole e-mail body is fetched, maybe.

Hope this helps!

Comment 142

[Arthur K. (he/him)]

(In reply to José María Fernández from comment #141)

I updated to 128.4.1esr , I have repaired the 729MB INBOX of one of my IMAP accounts (hosted at imap.gmx.com) , and now INBOX has 17 entries out of "9769+ conversations" with completely wrong dates (i.e. the timestamp of the re-fetch instead of the right one), and one entry with no subject sender, date, ... which once open seems to have the start of the e-mail misaligned.

This broken entry is a false positive, as it is an ill-formed IMAP entry (the GMX webmail is showing it in the very same way). But about the other 17 e-mails with wrong timestamps, I have been having a look at these e-mails through the GMX web site, and they are properly represented (even the correct date). As I was present when the rebuilding process was happening, I was able to realize the issue about the dates appeared on initial index creation, before the bulk of messages was fetched.

Could those wrong dates at the index be provided by the IMAP server (or not properly parsed), so they are blindly trusted by the code? And as they are trusted those timestamps are not replaced with the correct info when the whole e-mail body is fetched, maybe.

Hope this helps!

See if 128.4.2 is any better. There is mention of "Repair folder could result in older messages showing wrong date and time" but not sure if that will help with your issue specifically.

Comment 143

[Ben Campbell]

(In reply to José María Fernández from comment #141)

I updated to 128.4.1esr , I have repaired the 729MB INBOX of one of my IMAP accounts (hosted at imap.gmx.com) , and now INBOX has 17 entries out of "9769+ conversations" with completely wrong dates (i.e. the timestamp of the re-fetch instead of the right one)

Sounds a lot like Bug 1911916... (I think that was the one Arthur was referring to, and the patches were uplifted to 128esr a couple of days ago, but I don't know which version they'll show up in EDIT: they're in 128.4.2esr ).

But about the other 17 e-mails with wrong timestamps, I have been having a look at these e-mails through the GMX web site, and they are properly represented (even the correct date).

I would check the raw headers for one of your affected messages ("view source" in TB, or "download raw" maybe in a webmail interface). Bug 1911916 was a workaround for malformed messages without a "Date:" header - it stores an additional timestamp in the mbox separator (the line just before the first message header, eg "From - Tue Dec 09 15:30:45 2014") and uses this if "Date:" is missing. The "From " timestamp is just the time the message was written into the mbox - usually time of reception, or time of folder repair.

It's likely that the GMX IMAP server does a similar trick - it keeps track of the reception time of messages and uses that to fill in if "Date:" is missing.
I would guess that most of those 17 emails are emails from an automated system, rather than sent directly by a real person.

Comment 144

[Francesco]

(In reply to Ben Campbell from comment #143)

Sounds a lot like Bug 1911916... (I think that was the one Arthur was referring to, and the patches were uplifted to 128esr a couple of days ago, but I don't know which version they'll show up in).

128.4.2, but those patches don't affect IMAP, see bug 1911916 comment 15, last paragraph. For the record, with missing Date: headers, IMAP message have always received new envelop dates when re-downloaded, that hasn't changed in 128.

The discussion in this bug should focus on one question only: Have the fixes made in 128.3.3 stopped new IMAP folder corruption issues?

Is is clear that issues caused in earlier versions need to be repaired first to then see whether new issues arise or not.

Comment 145

[José María Fernández]

(In reply to Ben Campbell from comment #143)

(In reply to José María Fernández from comment #141)
I would check the raw headers for one of your affected messages ("view source" in TB, or "download raw" maybe in a webmail interface). Bug 1911916 was a workaround for malformed messages without a "Date:" header - it stores an additional timestamp in the mbox separator (the line just before the first message header, eg "From - Tue Dec 09 15:30:45 2014") and uses this if "Date:" is missing. The "From " timestamp is just the time the message was written into the mbox - usually time of reception, or time of folder repair.

It's likely that the GMX IMAP server does a similar trick - it keeps track of the reception time of messages and uses that to fill in if "Date:" is missing.
I would guess that most of those 17 emails are emails from an automated system, rather than sent directly by a real person.

Yes, as you suspected, I can confirm both the lack of Date header on those e-mails, and that they were sent by diverse automated systems.

I can propose a better (but more expensive) heuristic for the missing dates. I have realized that Received: headers contain timestamps which resemble (or they are near to) what the Date header should have had. So, the oldest of all the appearing ones could be used.