Filters on Yahoo! and AOL accounts filters only first message of a match. Manual filter works
Categories
(Thunderbird :: Filters, defect)
Tracking
(thunderbird_esr115 affected, thunderbird122 affected)
People
(Reporter: drghughes, Assigned: gds)
References
Details
Attachments
(1 file)
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/119.0
Steps to reproduce:
-
Set up a free Yahoo! or AOL account in Thunderbird using IMAP.
-
Create a filter on the account that is set to run when Getting new mail.
-
Send at least 2 emails to the account that will match the filter.
-
Get mail for the account.
Actual results:
The last email to arrive that matches the filter will be processed, but all other emails that match the filter will be ignored. However, if you run the filter manually (assuming this option is selected), the remaining mail that matches the filter will be processed correctly.
Expected results:
All emails that match the filter should be processed when getting new mail, the same as happens with filters for other email accounts.
Note that this problem started at about the same time as Bug 1782719 when Yahoo! and AOL updated their systems, so there may be some overlap. Prior to that, the filters that I have on these accounts worked as expected.
|
||
Comment 1•7 months ago
|
||
But 1782719 was "fixed" a year ago and it has no regressions marked against it. On that basis I'd think it is unrelated.
Sounds partly like bug 1818775
Regarding my comment on Bug 1782719, this bug appeared/existed prior to Bug 1782719 being reported, let alone be fixed. But I just haven't had time to report it until now. And as I said, they both appeared when Yahoo! and AOL updated their systems. Hence my comment that there may be some overlap since the underlying cause appears to be the same.
I did hope that the fix for Bug 1782719 would also fix this problem, but it didn't. Maybe that was because I didn't see the problem in Bug 1782719.
And I don't think that the updated summary describes the problem correctly, since if you have multiple filters on an account, the "first message of a match" isn't processed for each filter. The only message that seems to be processed by the filters system is the last message to arrive.
|
Assignee | |
Comment 3•6 months ago
|
||
I think I see the reason for this. It appears that yahoo based imap servers (yahoo, aol, verizon etc) have stopped (or maybe never have) set the \RECENT flag on new messages. You can see on other server types when new messages arrive in an imap inbox that the brand-new messages have a little orange dot beside them to indicate that they are new and not yet read/seen messages. I don't know exactly how filtering works but I suspect that it uses this same indication (whatever causes the dot to appear) to know to only deal with the "new" messages when applying the filter. But without the \RECENT flag, it only sees the newest message (highest imap UID) as having the orange "dot" so it only moves/copies that message and not the other new messages with a lower UID that arrived in the most recent biff or imap IDLE triggered fetches.
I don't know why the highest UID is seen as new (shows the "dot" and/or gets filtered) even thought it also doesn't have the \RECENT flag.
And I'm not even sure how the \RECENT flag causes the "orange dot" to appears, but it seems to since the orange dot and filtering work as expected with other non-yahoo imap servers that do put the \RECENT flag on all newly arrived messages.
Note: the \RECENT flag is only set/reset by the server so it's a "read-only" flag to TB. It only appears one time when a message's flags are fetched for the first time. On subsequent fetches for a given message it is never set.
Thanks gene!
That makes a lot of sense, and matches what I see. I'm guessing that it means that there is no solution as far as Thunderbird is concerned??
For now, I'll try changing the filters to run periodically and see how that goes. Hopefully it does create too much of an overhead.
|
|
Assignee | |
Comment 5•6 months ago
|
||
It may be possible to fix this in TB's imap but it will takes some research to figure out how.
Looking at the the next proposed version of IMAP, https://datatracker.ietf.org/doc/rfc9051/, the \recent flag is deprecated. So eventually, TB needs to operate without any dependence on \recent flag or untagged "recent" response, which is also deprecated. This is assuming that the proposed RFC for IMAP4rev2 is eventually approved and adopted by the imap server providers. Currently, only IMAP4rev1 (rfc3501, which supports "recent") is in active use by imap server makers and supported by TB.
|
|
Assignee | |
Comment 6•6 months ago
|
||
At this time I have pretty much eliminated the lack of "\RECENT flag and untagged RECENT response to SELECT" as the cause. I commented out all the imap code that handles these and everything still works OK with non-Yahoo servers and has no effect on Yahoo servers. So my theory about "recent" stuff is probably a "red herring" w.r.t. this bug.
Another thing I see strange about the yahoo/aol imap response is the fetch of flags occurs in reverse order compared to other server types I observe. I.e., the newest messages is listed first when flags are fetched with command UID FETCH 1:*. With all others, the fetch occurs from oldest with newest last. Right off, I don't see any reason why this would break new message marking (the orange dot) or break filtering, but I need to look closer.
FWIW, in Bug 1481102 it is mentioned about filters not working for yahoo. I think the reporters were trying to make a filter to prevent messages getting marked as Junk incorrectly but the filter never worked. But there was another bug that fixed the main complaint so that filters not working was a side issue that maybe got ignored until reported in comment 0 above.
|
|
Assignee | |
Comment 7•6 months ago
•
|
||
This is definitely caused by yahoo based servers returning fetched UIDs in descending order. The current TB codes assumes ascending order. I have a fix that I need to test some more and see if it breaks any tests.
This bug is actually causing two problems:
- As reported, only the first "new" message gets filtered. The other new messages don't.
- Only the first "new" message is shown with the orange dot and bold unread status. The other new messages only show bold unread (no dot).
|
|
Assignee | |
Comment 8•6 months ago
|
||
This is caused by a fairly recent change by yahoo that returns UID
fetch items with highest sequence/UID first instead of last. Most (all?)
other server types return fetches with lowest seq/UID first.
The changes in nsMsgDatabase.c independently fix the new message detection.
The changes in nsImapMailFolder.cpp independently fix the filtering.
|
||
Updated•6 months ago
|
Thanks for your work on this gene!
I'll look out for it in the beta stream and report back.
|
|
Assignee | |
Comment 10•6 months ago
|
||
Made the final recommended changes and re-tested.
Requesting check-in.
(In reply to drghughes from comment #9)
I'll look out for it in the beta stream and report back.
Once it's checked in it will first appear in daily. If you could try the daily it would be great!
|
||
Updated•6 months ago
|
|
||
Comment 11•6 months ago
|
||
Pushed by mkmelin@iki.fi:
https://hg.mozilla.org/comm-central/rev/0b057d91cb17
Yahoo imap fix for filtering and new message detection. r=mkmelin
|
Reporter | |
Comment 12•5 months ago
|
||
I did a quick test of this in v123 beta 1 and the filter now works as expected. This was just on an AOL account and filter that I was using to monitor this issue.
I'll do some more testing over the next few days, i.e. more than one filter and adding a Yahoo! account to the profile.
Thanks again gene!
|
|
||
Comment 13•3 months ago
|
||
(In reply to drghughes from comment #12)
I'll do some more testing over the next few days, i.e. more than one filter and adding a Yahoo! account to the profile.
How did that go?
|
|
Assignee | |
Comment 15•3 months ago
|
||
(In reply to Wayne Mery (:wsmwk) from comment #14)
Good for 115?
Yes, it's good. It's been in beta a while and reporter has also tested the beta at comment 12.
|
|
Reporter | |
Comment 16•3 months ago
|
||
(In reply to Wayne Mery (:wsmwk) from comment #13)
(In reply to drghughes from comment #12)
I'll do some more testing over the next few days, i.e. more than one filter and adding a Yahoo! account to the profile.
How did that go?
I think that it is good. I've tested a couple of times with multiple filters and both AOL and Yahoo! accounts and I haven't been able to find any problems. Initial testing was a few weeks ago, and then more testing today with v125 beta 3.
Thanks
Description
•