1781792 - When a filter copies multiple messages into local subfolders, the copied messages are empty!!

Status: NEW

(Thunderbird :: Filters, defect, P2)

Description

[aetfor]

Attachment: thunderbird.png

Steps to reproduce:

I have filters on the inbox folder to automatically copy some kinds of messages into local subfolders on the computer.

Actual results:

When I open Thunderbird and the emails are downloaded from the server, some filters automatically run to copy the messages into local folders and subfolders. The messages copied in the subfolders are correctly listed (see top-right panel in the image attached), but actually they are empty (see bottom-right panel in the image)!! The "copied" messages have all the same empty size of only 0.2 kB, no text, no headers (sender, addressees, date, attachments, etc.).
Please note that this bug happens only when downloading messages at the opening of thunderbird. On the contrary, if thunderbird is already open and a new message is received, then the filter correctly copy the whole message into the local subfolders.

Comment 1

[Magnus Melin [:mkmelin]]

What kind of account are they moved from? IMAP?
If you didn't already, enable the filter log. Does it mention any problem?

Comment 2

[aetfor]

Yes, emails are downloaded from an IMAP account. Filter log is enabled and it does not report any problem.
Please note that the problem happens only when copying emails into subfolders. On the contrary, when emails are copied into the local main inbox (i.e. from IMAP inbox into local main inbox folder), the messages are copied correctly.

Comment 3

[Wayne Mery (:wsmwk)]

aetfor,
What happens if you do "Repair" of the target local subfolder?

Comment 4

[aetfor]

Using "repair" on the target local subfolder does not correct the wrongly copied messages while, actually, it even worsens the situation. Indeed, the corrupted messages still remain empty and, additionally, their date/time in the message list panel (that at least was correct befere the repair action) get corrupted as well, displaying the date/time of the repair instead of the original date/time.

Further notice: I see that the malfunctioning only happens when the filters copy multiple message at the same time in the subfolder. If only one message is filtered, then it is copied correctly. If more than one message is copied in the subfolder, the first message (i.e. the oldest one) is copied correctly, while all the others are copied as corrupted.

I'm really puzzled with this issue...

Comment 5

[Wayne Mery (:wsmwk)]

Any better with 102.1.2 ?

Comment 6

[gene smith]

This works for me with self-built daily updated recently:
Appended (copied) 6 messages to be filtered to imap inbox with imap access "disabled" (no checks for new messages enabled).
Created a subfolder of an existing folder in Local Folders called sf as the filter destination.
Create filter to move messages from imap Inbox to sf.
Shutdown TB and restarted.
Selected imap account Inbox which triggered fetch of the 6 new messages that get moved to sf.
6 messages are now in sf, all are readable.

Comment 7

[Dan Pernokis]

Careful with the wording. Reporter consistently says "copy" (and variants), whereas the test filter does a MOVE. Copying should clone the message -- but supposedly doesn't -- whereas MOVE in effect does a copy from the inbox, then deletes the original. The two copy processes might not be exactly the same. MOVE typically destroys the original (gets deleted after copy) whereas COPY leaves the original intact when creating the copy.

Also, I've seen and/or documented elsewhere a couple of bugs which appear only at the start of TB, so this doesn't surprise me. Proper initialization of flags & variables can be tricky business when converting to a new language.

Comment 8

[aetfor]

Attachment: Actions in filter causing issues

Comment 9

[aetfor]

@Wayne Mery:
I'm using the latest stable version of TB, that is 102.0.3.

@gene smith and @Dan Pernokis:
Dan is right. My problem is with "copy". I didn't try with "move" because I don't want to delete messages from the IMAP account. Apart from the difference with copy/move, I'm not sure that Gene's simulation is the same as in my case, because he has firstly made a copy of 6 messages already existing, while I'm working with real new messages received on the IMAP account (I don't know if this can make a difference...).

In order to give you further details, I've attached a new screenshot in this bug report, exactly showing one of the concerned filters (I have two similar filters causing exactly the same issues) with actions in the same order as they should be applied by TB. As you can see, the filter firstly copies the messages into the local subfolder and then moves them to an IMAP subfolder. The messages copied in local subfolder are corrupted (except for the oldest one in the cue that is copied correctly) while those copied in the IMAP subfolder are ok.

Comment 10

[gene smith]

Dan is right. My problem is with "copy"

Yes, I missed that you said "copy" to local folders. I'll retry it with copy. Also, I'll add a "move" to an imap folder to my test. I don't know if it matters but are you moving to an imap folder on the same account as the source folder for the filter? It might since if the move is to another account a imap APPEND is used and the source message is marked \deleted, if to the same account, just an imap MOVE is probably done (if your imap server supports MOVE).

I'm working with real new messages received on the IMAP account (I don't know if this can make a difference...).

It shouldn't matter since the filter triggers the copy/move when new messages are fetched from the server and this won't happen until I click on Inbox even though I imap APPENDed the 6 messages to the server earlier.

I'll try again and report my results.

Comment 11

[gene smith]

Confirmed! The messages appear OK in the imap folder (same account as filter source folder) but are blank, just as you describe, in Local Folders.

Comment 12

[gene smith]

aetfor, By any chance are your Local Folders using maildir storeage format and not the default mbox? I have my Local Folders set to maildir for some reason I don't remember. Also, I can see one of the 6 messages copied and the other 5 are empty/blank. Looking at the maildir storage files, they are all mostly empty except for the one I can see.
I also see this each time I cause the filter to run using multiple messages. So it's not just a startup problem.

Comment 13

[aetfor]

Gene, thank you very much for your efforts.

I don't know if it matters but are you moving to an imap folder on the same account as the source folder for the filter?

Sorry for my late reply. I deduce that you have already found the answer, anyway yes, the main folder and its subfolders are on the same IMAP account.

are your Local Folders using maildir storeage format and not the default mbox?

I don't know how to check it. If this may help, looking to the local folders properties I see that their address is of the form "mailbox:///C:/Users/....".

I also see this each time I cause the filter to run using multiple messages. So it's not just a startup problem.

Sorry, my mistake. Since when TB is already connected to the IMAP account (after startup), I usually receive one message at a time and the filter does not show any issue, I supposed that it was only a startup issue. Thanks for checking that it's not only at startup. For this reason, I've modified the bug title.

Finally, I could image a possible hypotesis for the problem happening:
When a batch of multiple messages fitting the filter conditions is downloaded from the IMAP account, TB starts the first filter action (copy into local subfolders). As soon as the first (i.e. the oldest) message is copied in the local folder, TB probably immediately triggers the second action (move into IMAP subfolder) even before the completion of the copy of all the other messages into the local folder. As a consequence, the subsequent messages of the batch cannot be copied correctly because they have been already moved away from the main folder...

Could it be reasonable??

Comment 14

[gene smith]

Sorry for the late reply, busy with other stuff today... I wrote most of this yesterday but never posted it:

The same thing happens with 91.11 so it's not a recent regression. Also, I tested 91.11 with all accounts using mbox (including Local Folders) so maildir is not causing it.

It looks like all but the first source message gets deleted in storage (mbox/maildir) before the file copy occurs since I see the "From " delimiters and the "X-Mozilla-*" headers for the two empty messages following the 1st and only visible message in destination mbox file in Local Folders:

:
Delivery options: https://thunderbird.topicbox.com/groups/addons/subscripti=
on  <end of only non-empty and visible message>

From - Wed Aug 10 23:27:50 2022
X-Mozilla-Status: 0000
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:                                                                                 


From - Wed Aug 10 23:27:50 2022
X-Mozilla-Status: 0010
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:                                                                                 

<end of mbox file>

I did see one new thing with my daily: I selected 3 messages and copied them to Inbox from another folder in the same imap account. Only 1 message got copied to Inbox. I could see this in the IMAP log that only one UID was copied. This is a separate bug that needs to be reported. I don't see this with 91.11.

When a batch of multiple messages fitting the filter conditions is downloaded ...
Could it be reasonable??

Yes, it does seem to be something like you propose, not sure. Need to look at it closer.
I did find a possible workaround and with it all the messages are visible in both the imap and Local Folders destinations:
Modify the filter so that the actions are reversed:

1. Copy the matching message to imap folder
2. Move the matching message to the Local Folder.

Don't know why this works but the opposite order of actions doesn't.

Comment 15

[Dan Pernokis]

Gene, my gut says you're right -- that if the MOVE happens, then there is no source file remaining to be copied from -- in essence, what Aetfor proposed.

There are plenty of examples where order is important, and if you follow the steps logically, you discover that TB is actually doing what you tell it to do. I'd say then that this is not so much a bug as "undefined behaviour". In this case, we may think -- but nothing actually says -- that, after a MOVE, the original source is still available.

(And as a separate warning, one should use the STOP EXECUTION step if subsequent filters are not to be applied -- such as moving here, then moving there, when really only one MOVE is desired. But I digress... )

Comment 16

[gene smith]

This is present in 102 but definitely not something new. Turns out that this is a very old bug reported 9 years ago here, Bug 892424, and was never fixed. Even earlier there was a partial fix for this but it only applied when the filter is run manually, not on receipt of new messages, bug 448337 I think it is.
So this could be closed and "resolved" as a duplicate of Bug 892424, but I hate to do that.

FWIW, another workaround suggested is to keep the reporter's rule order and just run the filter manually (or maybe on the built-in 10 minute interval). I tested running manually (had to disable run on new mail) and the messages appear in both destinations correctly.

Comment 17

[aetfor]

Attachment: filter order

Comment 18

[aetfor]

Thanks Gene and Dan. Here are some replies to your comments:

I did find a possible workaround and with it all the messages are visible in both the imap and Local Folders destinations:
Modify the filter so that the actions are reversed:

1. Copy the matching message to imap folder
2. Move the matching message to the Local Folder.

Don't know why this works but the opposite order of actions doesn't.

There are plenty of examples where order is important, and if you follow the steps logically, you discover that TB is actually doing what you tell it to do. I'd say then that this is not so much a bug as "undefined behaviour". In this case, we may think -- but nothing actually says -- that, after a MOVE, the original source is still available.

I'm afraid that reversing the order does not work. Indeed, originally my filter was exactly in the opposite order (1. move + 2. copy) because indeed it is the most logical behavior, but unfortunately TB forces the actions to be run in a different order (I don't know why!!): see third screenshot added in the bug now.

(And as a separate warning, one should use the STOP EXECUTION step if subsequent filters are not to be applied -- such as moving here, then moving there, when really only one MOVE is desired. But I digress... )

Yes, but in this case I need both actions to be executed.

This is present in 102 but definitely not something new. Turns out that this is a very old bug reported 9 years ago here, Bug 892424, and was never fixed. Even earlier there was a partial fix for this but it only applied when the filter is run manually, not on receipt of new messages, bug 448337 I think it is.
So this could be closed and "resolved" as a duplicate of Bug 892424, but I hate to do that.

Why should the bug be set as "resolved" even if it is not? :-)

FWIW, another workaround suggested is to keep the reporter's rule order and just run the filter manually (or maybe on the built-in 10 minute interval). I tested running manually (had to disable run on new mail) and the messages appear in both destinations correctly.

It could be a workaround maybe, but this will nullify the usefulness of automatic actions. The idea is that, as soon as TB is connected to the IMAP account and downloads all new messages, I automatically get all the copies that I need in my local folders... I don't like (and for sure I will not remember) to manually run the filter every time that I receive a batch of new messages :-)

Comment 19

[Dan Pernokis]

@aetfor...

I'm afraid that reversing the order does not work. Indeed, originally my filter was exactly in the opposite order (1. move + 2. copy) because indeed it is the most logical behavior, but unfortunately TB forces the actions to be run in a different order (I don't know why!!): see third screenshot added in the bug now.

But that's exactly what I'm saying: LOGICALLY you want to do that (move it AND copy it). But TB does just what you say: Moves it, then tries to copy it. There's nothing that actually says the message is tracked when it is moved elsewhere, so copying it copies nothing. I agree it shouldn't work that way, but it does (and so do several other order-dependent examples). The rules need to be clarified & defined so we know one way or the other and can program accordingly.

Regarding STOP EXECUTION: Yes, but in this case I need both actions to be executed.

Sorry, this didn't pertain to your example where both actions occur in one filter. I meant it as a related warning to cases in general where the action of subsequent (additional) FILTERS is unexpected but nonetheless executes and causes problems. The STOP EX prevents run-on into the next filter if & when the current filter satisfies all.

Comment 20

[gene smith]

I did find a possible workaround and with it all the messages are visible in both the imap and Local Folders destinations:
Modify the filter so that the actions are reversed:

1. Copy the matching message to imap folder
2. Move the matching message to the Local Folder.

I didn't just mean reverse the rules. The rules that are failing are like this:

1. Copy the matching message to Local Folder
2. Move the matching message to imap Folder

The workaround, as quoted above, is to just to reverse the destination for each action. Sorry for the poor explanation. I should have said "...so the destinations are reversed". Edit: maybe better: "...so the destinations are swapped."
Yes, if you try to do the move first, you get a warning that copy occurs first.
When I change it to copy to imap and then move to local, as quoted above, all the messages appear in imap and in local.

Why should the bug be set as "resolved" even if it is not? :-)

That's often done when a bug is written up in multiple reports so it can be resolved in a single bug report. Anyhow, not sure what can be done about this bug since previous "experts" were stumped by it too.

Comment 21

[Thomas D. (:thomas8)]

We have Severity field and keyword: dataloss for severity.

Comment 22

[aetfor]

The workaround, as quoted above, is to just to reverse the destination for each action.

Gene, sorry for my misunderstanding and thanks for your further clarification. Yes, your suggestion is a possible workaround, even if, of course, the optimal solution should be to correct TB behaviour in order to execute the filter actions in the proper order. I'll try to use this workaround (or to set the filter to periodically run every 10 minutes, as you have also suggested), hoping that the issue will be definitely solved soon in the TB functioning :-)

Comment 23

[Thilo [:ThiloteE]]

Tests were done on a soft-fork of Thunderbird called Betterbird; Version 102.3.0-bb17 (64-bit) connecting via IMAP and by moving/copying 5 messages that fulfill filter conditions at the same time from local folder to Imap Inbox. Mailbox is stored in file per folder (mbox)

Summary of results:

Filter (A) fails with

1. Copy the matching message to Imap Sub-Folder
2. Move the matching message to Local Folder

--> All mails are moved to Local Folder and the body is fine, but only ONE (the first) mail was copied to imap Sub-Folder

Filter (B) fails with

1. Copy the matching message to Local Folder
2. Move the matching message to imap Sub-Folder

--> All mails are copied to Local Folder, but all mails lack body, except first mail. All mails are moved correctly to imap Sub-Folder.

Filter (C) fails with

1. Move the matching message to Local Folder
2. Copy the matching message to imap Sub-Folder

--> All mails are moved to Local Folder and the body is fine, but only ONE (the first) mail was copied to imap Sub-Folder

I will check, if same happens on Thunderbird daily.

Comment 24

[Thilo [:ThiloteE]]

Same failing behavior with Thunderbird Daily 107.0a1 (2022-09-28) (64-Bit) Linux Mint (Cinnamon) as described in comment 23

Comment 25

[Thilo [:ThiloteE]]

Solution?:

I found a fix using the "Move later" filter action provided by the Filtaquilla Addon

Following actions in the given order work without dataloss:

1. "Move later" the matching message to Imap Sub-Folder
2. Copy the matching message to Local Folder

To Do:

Test

• if this creates problems with additional other filter actions.
• what happens, if switching to offline mode, sending messages, running filters and then switching into online mode.

How to fix regular Thunderbird?

Maybe Thunderbirds code could be adapted to fall back to "move later" behaviour, if both the move and copy filter actions are set at the same time (such as described in comment 23)? Code by Filtaquilla could serve as inspiration?

Since we are dealing with data loss, I would like to ask for a fix in Thunderbird core.

Additionally, once this is fixed, I think adding this case to unit tests to prevent future regressions would be nice.

Comment 27

[Pedro [:pedrov]]

Attachment: dataloss.log

Moving here from bug 462156 as while the underlying issue should be the same, this appears to be the filter-specific data loss problem.

Describing the issue here too to some degree, more details can be found starting at https://bugzilla.mozilla.org/show_bug.cgi?id=462156#c180 ( https://bugzilla.mozilla.org/show_bug.cgi?id=462156#c194 should be the most technically helpful pointer).

Reproducing the issue I'm about to describe requires multiple mails to be processed at the same time which is currently achieved by using Gmail with bug 1834167 , resulting in batches of a couple of minutes.
Could be also Gmail-specific, filters on the Inbox can be triggered by copying from Trash to Inbox which may not rely on batching, and with a filter resulting in mails getting moved to Trash, there's additional ease of bug reproducing even multiple times.

Wanted to set up a filter to suppress new mail notification for mostly notification kind of emails, and figured at the same time they may as well get moved to a local folder as they would end up there later anyway. Bug 399475 made the Move operation alone unappealing, so I went with the usual Copy+Delete combination.

As the uploaded log shows, there's a problem with these 2 filter actions not getting processed sequentially, instead roughly the following happens:

• First mail gets copied
• All affected mails get deleted
• Following mails are attempted to get copied without success

While all headers get fetched properly, the problem is that while only the first email gets a (body) fetch query queued up during initial filter processing, the Delete step queues up a coalesced move for all emails, so as the (asynchronous?) copy filter keeps progressing, it keeps adding fetch queries after the coalesced move already made the mails disappear.

While not seen in the uploaded log file, quite oddly I've observed coalesced fetches too occasionally, always issued after the coalesced move.
In theory a coalesced fetch queued up for all mails early in the Copy step would deal with the issue of mails disappearing as the move operation could only happen after fetching is done.

Apparently coalescing is done for moving only, so the issue can be reproduced with Copy+Move and Copy+Delete (assuming Delete moves, might not be the case for all configurations), but not with a Copy+Copy filter which processes mails one by one. However the introduced delay of Copy+Copy results in failure to suppress new mail notification.

Unresolved observation: Rarely, but sometimes even the mail header is lost, resulting in a completely blank mail dated to 1970-01-01.

Considering this is a data loss issue, isn't S1 the appropriate severity here?

Comment 28

[Pedro [:pedrov]]

Didn't have time to test further, but feeling important to report that Copy+Copy isn't reliable either.

Settled on the following action list before stopping:

• Copy to local folder
• Mark as read
• Copy to Trash

This at least dealt with the earlier mentioned desire of suppressing notifications, but exposed yet another oddity: local folder copies were also marked as read, as if the copy happens after the mail is marked as read.

While keeping track of mails was quite silly this way, at least I haven't seen contents disappearing, until today.
Thunderbird was stopped more than 2 days ago, starting it with quite some work to do after this much time. This led to more than half of the copied mails becoming empty, therefore I can conclude that there's no reliable workaround after all even with caution, the data loss is simply unavoidable.