1767190 - In maildir POP account, body matching rule filter applied before junk mail classification when new messages are received will fail to move message

Status: UNCONFIRMED

(Thunderbird :: Filters, defect)

Description

[EarlgreyTea]

Attachment: file not found.jpg

User Agent:
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.8.1
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:101.0) Gecko/20100101 Thunderbird/101.0a1

Steps to reproduce:

1. Create a POP account using maildir storage.
2. Create the following message filter rule.
   • Apply filter when:
     • Getting New Mail: "Filter before Junk Classification"
   • Match any of the following
     • Body contains "hogehoge"
   • Perform these actions:
     • Move Message to "test" folder
3. Send an e-mail to this account with "hogehoge" in the body.

Actual results:

Added to the message list in "test" folder, but also added to Inbox.
The message in "test" folder hasn't been moved and says "File not found".
The filter log is as follows:

[2022/05/01 19:21:48] Applied filter "Body contains: hogehoge" to message from fuga <[e-mail]> - filter test at 2022/05/01 19:20:55 moved message id = [e-mail] to mailbox://[e-mail]/test

[2022/05/01 19:21:48] Filter action failed: "Move failed" with error code=0x8055000f while attempting: Applied filter "Body contains: hogehoge" to message from fuga <[e-mail]> - filter test at 2022/05/01 19:20:55 moved message id = [e-mail] to mailbox://[e-mail]/test

[2022/05/01 19:21:48] Filter action failed: "Failed applying the filter action" with error code=0x8055000f while attempting: Applied filter "Body contains: hogehoge" to message from fuga <[e-mail]> - filter test at 2022/05/01 19:20:55 moved message id = [e-mail] to mailbox://[e-mail]/test

[2022/05/01 19:21:48] Message from filter "Body contains: hogehoge": Applying filter actions failed

[2022/05/01 19:21:48] Filter run failed

Expected results:

New messages should be moved to the destination folder by the message filter.

Additional Information:

This issue does not occur in any of the following cases:

• Message Store Type: "File per folder (mbox)"
• Getting New Mail: "Filter after Junk Classification"
• Manually Run
• IMAP account

Comment 1

[EarlgreyTea]

Attachment: #1767190.moz_log

I got the log with "set MOZ_LOG=Filters: 5,timestamp".

Comment 2

[EarlgreyTea]

The Linux version doesn't seem to have this issue.
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.8.1
( I used the WSL2/Ubuntu 18.04 LTS environment instead of the actual Linux )

Comment 3

[EarlgreyTea]

It has been two years since I reported this bug.
I believe this bug is a serious problem for Windows users trying to use maildir with Thunderbird.
Please confirm this bug.

Comment 4

[Wayne Mery (:wsmwk)]

Unfortunately I am not in a position to confirm - I have only one pop account, and it is not maildir.
There are very few reporters in bugzilla citing maildir, so I don't have anyone to recommend to test - bug maybe Gene. has encountered this.

Comment 5

[gene smith]

I'm not a big filter or POP3 user and I don't have maildir on any of the POP3 accounts I have for test. Also, I'm on linux and seldom boot up my Win11 drive (but it does have WSL2 but haven't used it in a long time either).
Since this this maildir specific, maybe BenC could look at this?

Comment 6

[Ben Campbell]

I just had a go at replicating this (with a local dovedot pop3 server) and couldn't trigger a problem.

Filter log looks fine:

[27/08/24, 13:24:52 GMT+12] Applied filter "Body contains: dolorem" to message from [e-mail] - Proactive heuristic migration at 4/10/15, 00:39:00 GMT+13 moved message id = [e-mail] to mailbox://bob@localhost/test 

Not to say there isn't a problem - the pop3 filtering code is pretty hairy, and there's lots of scope for bugs in there...
But my stripped-down test case (one filter-triggering message and one not) didn't seem to trigger the bug, so there's likely other contributing factors.

Comment 7

[EarlgreyTea]

(In reply to Ben Campbell from comment #6)

Not to say there isn't a problem - the pop3 filtering code is pretty hairy, and there's lots of scope for bugs in there...
But my stripped-down test case (one filter-triggering message and one not) didn't seem to trigger the bug, so there's likely other contributing factors.

This bug should be easy to reproduce. I can do it in under 5 minutes starting with a new profile.
I feel like something isn't getting through to you correctly.
I'll try to provide more details on the steps to reproduce.

1. Start Thunderbird with the new profile on Windows PC.

"C:\Program Files\Mozilla Thunderbird\thunderbird.exe" -profile "%HOMEPATH%\Desktop\test"

2. Select Menu Bar > Tools > Settings > General.
3. Set Message Store Type for new accounts: to "File per message (maildir)".
4. Set up any POP account ([e-mail])
   I checked with the following POP servers:
   pop.mail.yahoo.co.jp, pop.mail.yahoo.com, pop.gmail.com, pop.zoho.com
5. Create a folder named "test".
6. Open Menu Bar > Tools > Message Filters.
7. Create the following filter.
   • Apply filter when:
     • Getting New Mail: "Filter before Junk Classification"
   • Match all of the following
     • body contains "test test test"
   • Perform these actions:
     • Move Message to "test on [e-mail]"
8. Send the following email to [e-mail].
   • Subject: test
   • body: test test test
9. Receive the email in Thunderbird.

Comment 8

[EarlgreyTea]

The above steps were done on Windows 10.
Not tested on Windows 11.

Comment 9

[Ben Campbell]

This is a frustrating one. I did manage to trigger this morning, using more or less the steps in Comment #7.
I couldn't get it to happen by delivering just a single matching message - I had to get a few messages onto the server for pop3 pickup, a mix of matching and non-matching ones. But I definitely saw screwed-up messages - looking at the raw message data on disk, they were blank except for the extra POP3 headers (X-Account-Key: and X-UIDL:) and X-Mozilla-* headers that get added.
This was on Linux, using a tame local pop3 server, and I'm pretty sure (like 90% sure) I managed to get it to happen when running on mbox too...

Then I went to dig into it deeper this afternoon, and for the life of me I couldn't get it to happen again! Doing those same steps over and over, but no dice. It just kept working and was utterly frustrating.
I'll try again with a remote POP3 server (timing could potentially be an issue) and failing that I'll try and dig up a windows setup to try out (although I don't see anything in the symptoms or the code which screams "platform-specific" to me).

I'd like to find out:

1. does this also happen with mbox? (@EarlGreyTea - have you tried the same steps on mbox?)
2. does it only happen on windows?

Wayne: are there some people you could ask to run through the steps in Comment #7 (on any platform, and on both mbox and maildir)? It'd be nice to get some more people to try and replicate this.

Comment 10

[Ben Campbell]

In the meantime, on the assumption that it is maildir-specific, I've done a little patch to remove the icky little shortcut the filter code takes to do message moves on maildir. There's a try build running here:
https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=12dbaa6b7cde2302ba0df13c17d0e456b6c63d35

@EarlGreyTea: would you be able to try out a custom build with this patch to see if it helps?
@Wayne: Do we have any end-user instructions on how to install custom builds from treeherder? If not, could you explain it here and I'll aim to add a page to developers.thunderbird.net! I never know how to find the correct installer...

Comment 11

[EarlgreyTea]

(In reply to Ben Campbell from comment #9)

1. does this also happen with mbox? (@EarlGreyTea - have you tried the same steps on mbox?)
2. does it only happen on windows?

Ans.1
In my environment, the occurrence rate of this issue is 10/10 for maildir POP account, but 0/10 for mbox POP account.

Ans.2
I only have a Windows 10 PC. So it is difficult to give a precise answer to that question.
This issue did not occur when I tested it in the WSL2/Ubuntu 18.04 LTS environment two years ago (I did it several times).

(In reply to Ben Campbell from comment #10)

@EarlGreyTea: would you be able to try out a custom build with this patch to see if it helps?

I don't have a development environment so I can't test the patch.
If you provide an executable I'd be happy to test it.

On the MozillaZine.jp forum, there are four persons, including me, who have encountered this issue.

Comment 12

[Ben Campbell]

(In reply to EarlgreyTea from comment #11)

(In reply to Ben Campbell from comment #9)

1. does this also happen with mbox? (@EarlGreyTea - have you tried the same steps on mbox?)
2. does it only happen on windows?

Ans.1
In my environment, the occurrence rate of this issue is 10/10 for maildir POP account, but 0/10 for mbox POP account.

Great, thanks!

Ans.2
I only have a Windows 10 PC. So it is difficult to give a precise answer to that question.
This issue did not occur when I tested it in the WSL2/Ubuntu 18.04 LTS environment two years ago (I did it several times).

(In reply to Ben Campbell from comment #10)

@EarlGreyTea: would you be able to try out a custom build with this patch to see if it helps?

I don't have a development environment so I can't test the patch.
If you provide an executable I'd be happy to test it.

You can get a build with the patch from that treeherder try build I linked in comment #10.
Click on the green "B" beside the platform you want ("Windows 2012 x64 opt", I'd think), then click on the "Artifacts and Debugging Tools" tab.
In that list there are a list of files, and one of them should be the installer (setup.exe would be my guess. There's also target.install.exe?).
Sorry, I'm a bit fuzzy on the exact details here!

If you try it out, NI me back! If it fixes things, then I'll tidy it up into a proper patch.

Comment 13

[Ben Campbell]

Wayne sent me some proper instructions for installing test builds from treeherder (which I'll try and get onto a proper page we can point to on developers.thunderbird.net, but for now I'll just paste it here):

When the build at https://treeherder.mozilla.org/jobs?repo=try-xxxxxxxxxx is complete (normally takes 1-2 hours)

1. Click the green "B" (for binary) next to one of the following: "Windows 2012 x64 opt", "Linux x64 opt", "OS X Cross Compiled shippable opt" (unless instructed to use a debug build)
2. In the black header below click "Artifacts and Debugging"
3. In the Artifacts section, to download the install file click on target.installer.exe (Windows), target.tar.bz2 (Linux), or target.dmg (Mac)
4. Install the downloaded file

Comment 14

[EarlgreyTea]

(In reply to Ben Campbell from comment #12)

You can get a build with the patch from that treeherder try build I linked in comment #10.
Click on the green "B" beside the platform you want ("Windows 2012 x64 opt", I'd think), then click on the "Artifacts and Debugging Tools" tab.
In that list there are a list of files, and one of them should be the installer (setup.exe would be my guess. There's also target.install.exe?).

I got the target.installer.exe and installed it, then did same testing with it.
The result was 0/10, no errors.
The patch seems to be correct.