Closed Bug 1930847 Opened 4 months ago Closed 2 months ago

Using POP3 "fetch headers only" and then using "get selected messages" on multiple messages, messages get incorrectly deleted

Categories

(MailNews Core :: General, defect)

Thunderbird 128
defect

Tracking

(thunderbird_esr115 affected, thunderbird_esr128+ fixed, thunderbird134 affected)

RESOLVED FIXED
135 Branch
Tracking Status
thunderbird_esr115 --- affected
thunderbird_esr128 + fixed
thunderbird134 --- affected

People

(Reporter: goodwin1204, Assigned: welpy-cw)

References

Details

(Keywords: dataloss, Whiteboard: STR in bug 1936526 comment 5)

Attachments

(1 file)

Steps to reproduce:

When I manually compact my POP3 folder (right-click on my POP3 folder and choose compact) sometimes a recent e-mail gets deleted from my inbox. I have the list view showing my inbox and I see a recent e-mail disappear from the list after compacting is finished.

Actual results:

A recent e-mail gets deleted. It isn't there when I restart Thunderbird and it is not in my trash folder either. I use the undelete function with online MS Outlook and check my mail again to get it back in my Thunderbird inbox. When I compact my POP3 folder again it usually doesn't get deleted again...

Expected results:

The folder should compact without deleting the contents...
Have had this happen with all versions of Thunderbird 128, last time it happened was with 128.4.2 (haven't tried 128.4.3). Have downgraded today to 128.3.3 since there was a fix for messages that could become corrupted during folder compaction.

Summary: When compacting mail file sometimes e-mails are deleted in inbox → When compacting POP3 mail folder sometimes e-mails are deleted in inbox

Before or after your downgrade, have you seen this happens with 128.3.3?

Flags: needinfo?(goodwin1204)

(In reply to Robert Goodwin from comment #0)

(haven't tried 128.4.3).

128.4.3 has some fixes https://www.thunderbird.net/en-US/thunderbird/128.4.3esr/releasenotes/

(In reply to Wayne Mery (:wsmwk) from comment #1)

Before or after your downgrade, have you seen this happens with 128.3.3?

It hasn't happened yet since I downgraded to 128.3.3, but that was only yesterday... I can't really remember if it happened after I upgraded to 128.3.3 from 128.3.2. I remember reading the fixes and thinking that it was fixed in 128.3.3, but can't remember if it happened.
Will be staying on 128.3.3 for a while now to check if it happens or not...
It has happened a quite a few times in 128.4.x so I thought I must report it and go back to 128.3.3.

Flags: needinfo?(goodwin1204)

(In reply to Wayne Mery (:wsmwk) from comment #2)

(In reply to Robert Goodwin from comment #0)

(haven't tried 128.4.3).

128.4.3 has some fixes https://www.thunderbird.net/en-US/thunderbird/128.4.3esr/releasenotes/

Saw the fixes with 128.4.3, but thought that the fixes in 128.3.3 were closer to my case (fixed - Messages could become corrupted during folder compaction).
As I replied to your first comment, I will stay on 128.3.3 for a while and see if it happens...

That does not like a very good idea. 128.3.3 had issues, but 128.4.3 should have at least less.

(In reply to Magnus Melin [:mkmelin] from comment #5)

That does not like a very good idea. 128.3.3 had issues, but 128.4.3 should have at least less.

I will stay on 128.3.3 for a while to test if I experience the bug with it, then try 128.4.3 - next week sometime. If I experience the bug before that, I'll go straight to 128.4.3 and give it a try...

I forgot to mention that I'm on Windows 10 Pro 22H2 19045.5011 64 bit...

It finally happen again today with 128.3.3, so I have upgraded to 128.5.0. I saw in the 128.5.0 release notes that there was a "fix for message corruption" when compacting folders.
Since reporting the bug, when I check my mail and download the messages, I have always run compact to see if the messages would be affected...

It also happened with 128.5.0. I checked my e-mail, downloaded them, then ran compact folders. The newest e-mail that wasn't download in the last e-mail check was deleted from my inbox...

I aslo tried compacting folders after I downloaded the e-mail that was deleted and after compacting the inbox view changed from the newest e-mail highlighted to an much older e-mail highlighted...

Still happening on 128.5.1.
When I downloaded my e-mails then compacted my folders (right click on main folder heading and select compact) one of my new e-mails was deleted from the inbox...

Lost one of my latest e-mails when opening and closing Thunderbird (there were 3 e-mails from the same sender with the same title, just received at different times with different content) - lost the latest one, the next latest was listed twice when starting Thunderbird again.

(In reply to Robert Goodwin from comment #9)

It also happened with 128.5.0. I checked my e-mail, downloaded them, then ran compact folders. The newest e-mail that wasn't download in the last e-mail check was deleted from my inbox...

Robert, sorry to hear about your trouble. I'd like to understand your configuration, so we could try to reproduce here.

When you say the newest email wasn't downloaded, but then it was deleted - does that you that you have Thunderbird configured to only download headers for your new messages? Which means, Thunderbird downloaded the headers for your newest message, and showed it in the message list, but otherwise the message contents had not been downloaded yet?

Would you please be able to send a screenshot of your account settings / server settings tab ?

You may omit/redact the top sections server type/name/username, security settings, and also the bottom "local directory".

I'm interested in all the server settings starting with "check for new messages at startup" until "message store type".
Thanks in advance!

Can you please also say:

  • in junk settings for this account, do you have the top checkbox enabled?
  • do you have active thunderbird filters for this pop3 account?

And in general settings (not account settings), general, at the bottom, indexing, do you have "enable global search and indexer" enabled?

(In reply to Kai Engert [:KaiE:] from comment #13)

(In reply to Robert Goodwin from comment #9)

It also happened with 128.5.0. I checked my e-mail, downloaded them, then ran compact folders. The newest e-mail that wasn't download in the last e-mail check was deleted from my inbox...

Robert, sorry to hear about your trouble. I'd like to understand your configuration, so we could try to reproduce here.

When you say the newest email wasn't downloaded, but then it was deleted - does that you that you have Thunderbird configured to only download headers for your new messages? Which means, Thunderbird downloaded the headers for your newest message, and showed it in the message list, but otherwise the message contents had not been downloaded yet?

I think you misunderstood - I checked my e-mail twice and the newest e-mail from the first batch was deleted after I had downloaded the full e-mail and compacted the folders.
I do download headers only so that I can see what's in my online mail-box, but then I always download the full e-mail before I do anything else - I check my e-mail by downloading the headers first, mark them and "get" the full e-mail, then eventually compact my folders.

(In reply to Kai Engert [:KaiE:] from comment #14)

Would you please be able to send a screenshot of your account settings / server settings tab ?

You may omit/redact the top sections server type/name/username, security settings, and also the bottom "local directory".

I'm interested in all the server settings starting with "check for new messages at startup" until "message store type".
Thanks in advance!

Not too sure how to post a screen shot here.
Under Account settings, on the Server settings tab, under Server settings, the only item I have "ticked off" is "Fetch headers only"

(In reply to Kai Engert [:KaiE:] from comment #15)

Can you please also say:

  • in junk settings for this account, do you have the top checkbox enabled?
  • do you have active thunderbird filters for this pop3 account?

And in general settings (not account settings), general, at the bottom, indexing, do you have "enable global search and indexer" enabled?

In Junk settings I have no "tick boxes" ticked.
No, no filters at all (to be honest, I had to check what that was. Called rules in old MS Outlook...)

Version 128.5.1 is even worse. When I close the program and open it later one of the latest e-mails missing (usually the last one). Happened twice now. Last time (mentioned earlier) the second to last e-mail replicated itself, now the last one just disappeared...

(In reply to Robert Goodwin from comment #17)

Under Account settings, on the Server settings tab, under Server settings, the only item I have "ticked off" is "Fetch headers only"

I think this means you have also checked "leave messages on server", and "for most x days".
What value do you have in the days field?

(In reply to Kai Engert [:KaiE:] from comment #20)

(In reply to Robert Goodwin from comment #17)

Under Account settings, on the Server settings tab, under Server settings, the only item I have "ticked off" is "Fetch headers only"

I think I misunderstood, I think when you say "ticked off" - you actually mean "is marked as checked with a checkmark".

Because you also said, you do download headers only.

If I understand correctly, "fetch headers only" is the only setting that you have enabled/checked.

That means, you have the following disabled (not checked):

  • check for new messages at startup
  • check every n minutes
  • auto download new messages
  • leave messages on server

In other words, you ask Thunderbird to delete messages from the server after downloading.

(In reply to Robert Goodwin from comment #16)

I do download headers only so that I can see what's in my online mail-box, but then I always download the full e-mail before I do anything else - I check my e-mail by downloading the headers first, mark them and "get" the full e-mail, then eventually compact my folders.

When you say, you "mark them" and you "get" the full email, does that mean:

  • you click each of the messages one after the other
  • when clicking one of the messages, I am told that the message isn't downloaded yet, and I'm offered a link to click to download the rest of the message. Is that what you do to "get" the full e-email, as you say?
  • compacting usually should only have an impact after at least one message was deleted. So, in order to see the effect you were describing, do you usually delete at least one message?

So far, I haven't been able to reproduce your report.

I'm using a POP3 mailbox, I've filled that folder with a lot of messages to simulate some regular usage,
I'm downloading headers only, I manually click on download for each new message, afterwards I compact the folder.

No messages disappear on my side.

(In reply to Kai Engert [:KaiE:] from comment #21)

(In reply to Kai Engert [:KaiE:] from comment #20)

(In reply to Robert Goodwin from comment #17)

Under Account settings, on the Server settings tab, under Server settings, the only item I have "ticked off" is "Fetch headers only"

I think I misunderstood, I think when you say "ticked off" - you actually mean "is marked as checked with a checkmark".

Because you also said, you do download headers only.

That is correct. "Ticked off" is the box with a checkmark in it...

(In reply to Kai Engert [:KaiE:] from comment #22)

If I understand correctly, "fetch headers only" is the only setting that you have enabled/checked.

That means, you have the following disabled (not checked):

  • check for new messages at startup
  • check every n minutes
  • auto download new messages
  • leave messages on server

In other words, you ask Thunderbird to delete messages from the server after downloading.

I don't have access to my computer right now so I can't check 100%, but yes, Thunderbird deletes my messages on the server after fetching the e-mail, and the list looks about right.

(In reply to Kai Engert [:KaiE:] from comment #23)

(In reply to Robert Goodwin from comment #16)

I do download headers only so that I can see what's in my online mail-box, but then I always download the full e-mail before I do anything else - I check my e-mail by downloading the headers first, mark them and "get" the full e-mail, then eventually compact my folders.

When you say, you "mark them" and you "get" the full email, does that mean:

  • you click each of the messages one after the other
  • when clicking one of the messages, I am told that the message isn't downloaded yet, and I'm offered a link to click to download the rest of the message. Is that what you do to "get" the full e-email, as you say?
  • compacting usually should only have an impact after at least one message was deleted. So, in order to see the effect you were describing, do you usually delete at least one message?

If I receive more than one e-mail I highlight all of them, right click, and choose the "get" option in the right click menu. If I only have one new e-mail I open it and click on the get e-mail link where the e-mail would be displayed.

OK, I thought that compacting also meant applying a compression algorithm in addition to cleaning up deleted e-mails (like in the desktop version of MS Outlook). I seldom delete e-mails from my inbox since I check the headers first and sort through them first (if necessary).

(In reply to Kai Engert [:KaiE:] from comment #24)

So far, I haven't been able to reproduce your report.

I'm using a POP3 mailbox, I've filled that folder with a lot of messages to simulate some regular usage,
I'm downloading headers only, I manually click on download for each new message, afterwards I compact the folder.

No messages disappear on my side.

OK. One important thing I should add is that the e-mails that have disappeared have usually been from the same sender with the same subject as earlier e-mails I have (i.e. an e-mail discussion with the two participants just pressing reply each time). The e-mails are usually around the same size. Yesterday though, I lost an e-mail from a completely different sender that was between two e-mails from the same sender (with the same subject). If I remember correctly I have between 550 - 600 e-mails in my inbox...
When the e-mails disappear I just undelete them on the server and check my mail again to get them back...

I'm using the Mbox format for my mail file (not Maildir).

I noticed something strange today.
Got home and downloaded all my new messages (not just headers). I moved 5 messages from my inbox to the sent folder and noticed that the total messages number at the top of the inbox hadn't changed.
Thinking that it need time to register I selected my sent folder then selected my inbox. The total had changed to a higher number. I decided to check the number and selected all my e-mails in the inbox (ctrl-A) and numbers didn't match (out by 6 - 580 messages, 274 selected). When I did the same in my sent folders the numbers matched...
I compacted my inbox and the total e-mail number didn't change. Restarted Thunderbird, still the same number discrepancy...

I can confirm that Kai's list of the Server settings disabled (from yesterday) is correct.

To correct my above message:
... (out by 6 - 580 messages, 574 selected)...

I can reproduce the corruption.

Status: UNCONFIRMED → NEW
Ever confirmed: true
Keywords: dataloss

I can also reproduce with version 115.x, so not a recent regression.

To reproduce:

  • use a POP3 account
  • enable "download headers only"
  • disable "leave messages on server" and disable "check automatically"
    (not sure those are necessary, but that's what we use)
  • check mail
  • wait until you have at least 2 new messages
  • select all
  • remember the count of messages that is shown on screen above the list of messages, total and selected
  • right click, get selected messages
  • quit thunderbird
  • restart thunderbird
  • the number of messages shown is less than what we had previously.

So you are able to reproduce the message count error or the random loss of inbox e-mails (or both)?
With the 115 and 128 versions?

both in both

(adding some pop and support folk)

We probably don't have telemetry on this, but I've encountered many users that use "keep on server" as a proxy for imap and "sharing" messages with users on other computers. I would not surprise me if "fetch headers only" also has a good percentage of users - one example reason given is to be able to delete spam and possibly "dangerous" messages before they get downloaded.

Yes, I have setup field offices (using satellite coms) where messages are left on the server so that all users can access the common e-mails (unless online mail is used). I use the message header function to give myself the opportunity to manually filter my e-mails (after Microsoft has done their thing).

(In reply to Kai Engert [:KaiE:] from comment #34)

To reproduce:
[...]

  • enable "download headers only"
  • disable "leave messages on server" [...]
    (not sure those are necessary, but that's what we use)

Apparently exactly the combination of those two options shows the unwanted behavior:

Only in this case https://searchfox.org/comm-central/rev/2cf7d6d45ae722859ec1e5358713c3861ca06a96/mailnews/local/src/Pop3Client.sys.mjs#1592 is called after a body for a partial messages has been fetched, leading eventually to https://searchfox.org/comm-central/rev/2cf7d6d45ae722859ec1e5358713c3861ca06a96/mailnews/local/src/Pop3Client.sys.mjs#1427 where nsPop3Sink::EndMailDelivery() is called. This calls nsPop3Sink::CheckPartialMessages() right away. Unfortunately, this finds all just completed messages already absent from the server and therefore removes them from the database.

Anyway, commenting out https://searchfox.org/comm-central/rev/2cf7d6d45ae722859ec1e5358713c3861ca06a96/mailnews/local/src/nsPop3Sink.cpp#194 seems to mitigate the problem. Could you verify this, Kai? For a more robust solution, we should check whether and when both nsPop3Sink::FindPartialMessages() and nsPop3Sink::CheckPartialMessages() are to be called for pop3-js. Furthermore, with "leave messages on server" enabled, nsPop3Sink::EndMailDelivery() is never called. I don't know if that's entirely correct.

Flags: needinfo?(kaie)

Do not purge the message database of partial messages that no longer exist on the server when bodies for such messages are downloaded.

Assignee: nobody → h.w.forms
Status: NEW → ASSIGNED

(In reply to Hartmut Welpmann [:welpy-cw] from comment #40)

Furthermore, with "leave messages on server" enabled, nsPop3Sink::EndMailDelivery() is never called. I don't know if that's entirely correct.

Hmm. That sounds a bit dodgy. I assume BeginMailDelivery() is called? It seems pretty clear they should be paired?
(I'm still trying to understand all the pop3 code - in case it's useful to anyone, my personal notes are at https://github.com/bcampbell/tb-notes/blob/master/pop3.md )
(In reply to Hartmut Welpmann [:welpy-cw] from comment #41)

Created attachment 9443264 [details]
Bug 1930847 - Fix "Get Selected Messages" with "Leave Messages on Server" disabled. r=#thunderbird-back-end-reviewers

Do not purge the message database of partial messages that no longer exist on the server when bodies for such messages are downloaded.

I haven't gone through all the partial-message handling code yet, but just reading your comment there, doesn't that mean we'd end up with two entries in the database? The original partial (header-only) message, and the full message? (I'm assuming not, because that'd be really obvious in the GUI - you'd see the same message twice)

(In reply to Ben Campbell from comment #42)

(In reply to Hartmut Welpmann [:welpy-cw] from comment #40)

Furthermore, with "leave messages on server" enabled, nsPop3Sink::EndMailDelivery() is never called. I don't know if that's entirely correct.

Hmm. That sounds a bit dodgy. I assume BeginMailDelivery() is called?

Yes.

It seems pretty clear they should be paired?

The comment says so. Otherwise this doesn't seem to influence the issue here, so maybe this should be investigated in another bug.

(In reply to Hartmut Welpmann [:welpy-cw] from comment #41)

Created attachment 9443264 [details]
Bug 1930847 - Fix "Get Selected Messages" with "Leave Messages on Server" disabled. r=#thunderbird-back-end-reviewers

Do not purge the message database of partial messages that no longer exist on the server when bodies for such messages are downloaded.

I haven't gone through all the partial-message handling code yet, but just reading your comment there, doesn't that mean we'd end up with two entries in the database? The original partial (header-only) message, and the full message? (I'm assuming not, because that'd be really obvious in the GUI - you'd see the same message twice)

No, this is already taken care of in nsPop3Sink::IncorporateComplete().

nsPop3Sink::FindPartialMessages() and CheckPartialMessages() on the other hand are functions to get rid of partial messages that the user hasn't downloaded and that no longer reside on the server—an entirely separate mechanism for regular clean-up.

(In reply to Ben Campbell from comment #42)

(In reply to Hartmut Welpmann [:welpy-cw] from comment #40)

Furthermore, with "leave messages on server" enabled, nsPop3Sink::EndMailDelivery() is never called. I don't know if that's entirely correct.

Hmm. That sounds a bit dodgy. I assume BeginMailDelivery() is called? It seems pretty clear they should be paired?

Sorry, I just tested this again: EndMailDelivery() does get called after one or more messages have been completed, regardless of the "Leave Messages on Server" setting, it's just different code paths in Pop3Client.sys.mjs.

Your attached patch is an improvement (thanks!), but doesn't fix the issues completely yet.

It fixes the message counts during the "get messages" step,
before and after, the counts are the same, and the total equals the selected count.

It fixes disappearing messages during "get message", I don't "visibly" lose messages during this step.
Immediately after the "get message" step, the message list still shows the correct number of messages.

However, despite that display, the mailbox is already in a broken state.

Simply switching to another folder, then going back to this inbox, several messages are gone already.
Still gone after TB restart.

In my scenario, 7 total, 3 messages are gone, 4 messages shown.
After repair folder I have 14 - every original message duplicated (once downloaded, once not downloaded).

The message duplication can be prevented by executing "compact" prior to "repair".

Flags: needinfo?(kaie)

Note I'm testing on the esr128 branch

(In reply to Hartmut Welpmann [:welpy-cw] from comment #40)

Anyway, commenting out https://searchfox.org/comm-central/rev/2cf7d6d45ae722859ec1e5358713c3861ca06a96/mailnews/local/src/nsPop3Sink.cpp#194 seems to mitigate the problem. Could you verify this, Kai?

I confirm this.

Well, when running BOTH your attached patch, and also removing the above line to CheckPartialMessages(),
then the issue is completely fixed for me.

Wait. I'm confused.

Did I make a mistake when testing?

I can no longer reproduce what I said in comment 45.

Did I mess up what I have tested?

I'm repeating my tests.

Comment 45 was wrong.

The attached patch fully fixes the issue from me.

Great work Hartmut, thanks a lot !

Duplicate of this bug: 1936526
Whiteboard: STR in bug 1936526 comment 5
Summary: When compacting POP3 mail folder sometimes e-mails are deleted in inbox → Using POP3 "fetch headers only" and then using "get selected messages" on multiple messages, messages get incorrectly deleted

Does this need an official r+ from Ben, or can we start testing on comm-central as is?

Let's get it landed.
Would be nice with a test as this like it could easily break.

Target Milestone: --- → 135 Branch
See Also: → 1937069

With reference to the bug 1937069. In my case I have always displayed my messages unthreaded and still lost messages when selecting the downloaded headers and using the right click menu, get selected messages...

(In reply to Robert Goodwin from comment #53)

With reference to the bug 1937069. In my case I have always displayed my messages unthreaded and still lost messages when selecting the downloaded headers and using the right click menu, get selected messages...

Yes, confirming. Bug 1937069 is simply about TB not trying to download some messages at all. With or without the fix from this bug 1930847, after get messages, there are still several messages which show the "not downloaded" information when clicked.

Pushed by mkmelin@iki.fi:
https://hg.mozilla.org/comm-central/rev/d0fb11668688
Fix "Get Selected Messages" with "Leave Messages on Server" disabled. r=kaie

Status: ASSIGNED → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED

Comment on redacted Comment 45...

In my scenario, 7 total, 3 messages are gone, 4 messages shown.
After repair folder I have 14 - every original message duplicated (once downloaded, once not downloaded).
The message duplication can be prevented by executing "compact" prior to "repair".

This is very reminiscent of a major bug in TB 102 that dragged on for awhile. Early releases were always acting this way, duplicating downloads, etc -- but it was eventually fixed around the time v115 was released (which we're still using). Running REPAIR prior to COMPACT became necessary, so leaving COMPACT as an automatic function was not possible. Was THIS bug a regression? Or maybe the 102/115 case was never really fixed? Personally, I've been hesitant to switch to v128, given all the problems like this I'm seeing reported.

Blocks: 1937555

A follow up to my comment no. 30.
After reading the proceeding comment here (no. 56) I found where the repair command was located and ran in om my inbox. I then selected all my messages (Ctrl-A) and the totals matched. I then went through my inbox and found that the messages that had disappeared (and I had downloaded again) were duplicated.
I guess that in reality the messages were still in the inbox folder but weren't displayed because of the corrupted folder, and by re-downloading them they became duplicated (but were not visible because of the corruption)...

Has this been fixed in 128.6.0esr? (Have upgraded.) Couldn't see it in the release notes...

(In reply to Robert Goodwin from comment #58)

Has this been fixed in 128.6.0esr? (Have upgraded.) Couldn't see it in the release notes...

No, this will be in upcoming Beta 135 and should be uplifted to ESR 128 soon.

(In reply to Hartmut Welpmann [:welpy-cw] from comment #59)

(In reply to Robert Goodwin from comment #58)

Has this been fixed in 128.6.0esr? (Have upgraded.) Couldn't see it in the release notes...

No, this will be in upcoming Beta 135 and should be uplifted to ESR 128 soon.

OK, thanks for the information, and thank you for foxing it (and for Thunderbird ;-) ). I am assuming that it will be listed in the release notes when it is included...

Has this lived on beta long enough to be considered for our next uplift 128.6.1 on Tuesday?

Comment on attachment 9443264 [details]
Bug 1930847 - Fix "Get Selected Messages" with "Leave Messages on Server" disabled. r=#thunderbird-back-end-reviewers

Yes, I think this should be okay to uplift.

Attachment #9443264 - Flags: approval-comm-esr128?

Hartmut, would you be able to fill out the uplift template for this?

Flags: needinfo?(h.w.forms)

Comment on attachment 9443264 [details]
Bug 1930847 - Fix "Get Selected Messages" with "Leave Messages on Server" disabled. r=#thunderbird-back-end-reviewers

[Approval Request Comment]
User impact if declined: If "Leave messages on server" is disabled, partial POP3 messages that have just been completed are immediately removed from the database, resulting in data loss when the message folder is compacted.
Testing completed (on c-c, etc.): c-c and beta
Risk to taking this patch (and alternatives if risky): low

Flags: needinfo?(h.w.forms)

Or did you mean the relnote-thunderbird?

Comment on attachment 9443264 [details]
Bug 1930847 - Fix "Get Selected Messages" with "Leave Messages on Server" disabled. r=#thunderbird-back-end-reviewers

[Triage Comment]
Approved for esr128

Attachment #9443264 - Flags: approval-comm-esr128? → approval-comm-esr128+
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: