1915166 - 128.1.1esr After compact INBOX/SENT at 4 GB exactly, message bodies are blank, after repair a year of messages are missing. Filter related?

Status: UNCONFIRMED

(Thunderbird :: Folder and Message Lists, defect, P1)

Description

[DavidW]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36

Steps to reproduce:

I have not tried to reproduce - I have simply turned off COMPACT and UPDATE on all Thunderbird users.

Actual results:

Thunderbird version 128.1.1esr Win 10 (32 bit)
Compacting left INBOX/SENT mailboxes at 4GB.
The message list is intact but attempting to load messages just produces an empty screen.

REPAIR causes message loss from 02/2023 to today.

Expected results:

COMPACT should not be dropping messages or affecting the INDEX.

I am not sure how this bug is marked as fixed when the update to 128.1.1esr happened recently and the compact / message loss problem occurred yesterday.

Comment 1

[Magnus Melin [:mkmelin]]

IMAP or POP3?
Were you running the 128.0.x versions before?

Comment 2

[Wayne Mery (:wsmwk)]

Speculatively added tracking, severity and priority.

Comment 3

[DavidW]

The update history of this user is: Update from 115 to 128.0 on Aug 1.
On Aug 06 there was another update
On Monday Aug 26 the update to 128.1.1esr occurred and the problem with the compact happened shortly there after.

This is a POP3 account setup (INBOX is intact on the server but SENT Mail only resides on the desktop)
I don't have a recent backup of the mailbox prior to the problem so I cannot say how big it should be.

We are running WIN 10 22H2 and use ESET Anti-Virus.

Comment 4

[Alessandro Castellani [:aleca]]

NI Ben so he's aware of this

Comment 5

[Magnus Melin [:mkmelin]]

Attachment: Bug 1915166 - more explicit 64bit integers in MboxMsgInputStream. r=BenC

Not sure if it would cause this bug or not.
The way mUnused in particular was used seems risky.

Might be worth making these more explicitly 64bit. (For 32bit platforms, it's a slight waste.)

Comment 6

[Ben Campbell]

(I'm pretty sure the patch in comment #5 won't help - comments in phab)

Is it possible that the truncation actually occurred before Aug 28 (when the move to 128.1.1.esr happened), and just wasn't picked up until afterwards?

Comment 7

[Wayne Mery (:wsmwk)]

(In reply to Ben Campbell from comment #6)

Is it possible that the truncation actually occurred before Aug 28 (when the move to 128.1.1.esr happened), and just wasn't picked up until afterwards?

David ^

Comment 8

[DavidW]

Hello,

It seems unlikely because this person uses their email a lot and would quickly notice that they are not able to load old messages.

They were on vacation the week of Aug 19 to Aug 23. I checked and this computer was never on during that week.

The previous version was updated on Aug 6th (as per the update history). The only thing that could have happened is if this previous version caused the problem on Friday August 16th right before they shutdown. It seems unlikely.

It seems too much of a coincidence that there was an update upon their return on Aug 26th and they noticed the problem shortly there after, later that same day.

Unfortunately they do not back up their email although some are maintained on the server. The last back up I have of their email (2023) shows an INBOX at 6.9 GB and SENT box at 3.9 GB.

Comment 9

[Ben Campbell]

The truncation wouldn't have affected the list of messages shown for the affected folder, and new messages would still come in (and be added at the end of the truncated mbox file). It just would have screwed up trying to display any messages that were in the truncated section.

Anyway, I've not yet been able to replicate it. I installed the 32bit windows build of 128.1.1esr (which immediately upgraded itself to 128.2, but that's OK - there's been no changes I'd suspect in to cause this kind of issue since the fix in 128.1.1esr).
I set up an account with pop3, then shut TB down.
I generated a large (>8GB mbox file) using my (fakemail)[https://github.com/bcampbell/fakemail) tool, and copied it over the Inbox (and deleted the Inbox.msf file to force a rescan).
I then deleted a message or two and then started a folder compaction, which worked fine. The mbox file stayed over 8GB.

I tried some other things involving moving incoming pop3 messages out of the Inbox with a filter (that's a bit of code I would suspect), but that seemed to work fine too. I just couldn't get it to truncate the mbox.

I think I need some other people to try some things out and see if we can catch some steps to cause truncation.
(I mentioned my fakemail tool above - if you give it a directory of big files it'll randomly use them as message attachments, so it's easy to build an mbox of whatever size you need)

I'm pretty confident of the folder compaction code itself. But I don't totally trust the filtering code for pop3, so that's where I'd be looking - maybe there's some odd combination of rules or conditions that cause that code to truncate the mbox? I don't know. But it'd be good to have some steps to reliably replicate this.

Comment 10

[Corey Bryant]

Switching to tracking-thunderbird_esr128? for now until we can confirm this issue still exists.

Comment 11

[Corey Bryant]

Hello David, are you still seeing this issue on new versions of 128?

Comment 12

[Wayne Mery (:wsmwk)]

(In reply to Corey Bryant from comment #11)

Hello David, are you still seeing this issue on new versions of 128?

Perhaps unlikely given comment 8