1481102 - [v52.9.1] Junk Filter Resets Mail Status on Every Refresh, Filters Are Ignored

Status: RESOLVED DUPLICATE of bug 1260059

(Thunderbird :: Filters, defect)

Description

[formyer]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36

Steps to reproduce:

First off: I am reporting these two phenomena together because I suspect that the first is just a symptom of the second.
I occasionally get web notifications from a website (that I WANT to receive). They are automatically marked as junk and moved to the junk folder. I set up a filter that is supposed to kick in before the junk filter, moving the mail into the inbox and marking it as "Not Junk". I also tried running it after the junk filter, disabling the junk filter and disabling all Thunderbird add-ons (only Lightning is installed, no plug-ins).


Actual results:

The emails are still marked as junk and go into the junk mail folder no matter how the filter is configured, but even more than that:
If I manually mark them as "Not Junk", they are immediately re-categorized as junk the moment the inbox updates (pressing F5, waiting a bit, closing and restarting the application, ...).


Expected results:

The emails should have arrived in the inbox not marked as junk, or should at least not show up as junk anymore after I manually unmark them as such.

Comment 1

[Wayne Mery (:wsmwk)]

How long has this been happening?
Can you post a screen shot of the message list of your Junk folder with this message in it?

Comment 2

[formyer]

This has been happening for at least a year now (probably longer). Since this is an IMAP email account, I also checked my inbox through my provider's website, but they seem not to have any sort of spam filtering in place except manually moving an email to a folder named 'Spam' (which is different from the local folder 'Bulk Mail' where my messages end up in).

I'm sorry, I'm not really comfortable sharing the subject line or sender of the mail in question because they contain possibly sensitive info from other people (but it's not a website that people would block due to 'newsletters' or other unwanted notifications if that's what you were thinking). Feel free to ask for specific information, though!

And to clear something up: The emails do NOT get returned to the Bulk Mail folder upon refreshing, they simply regain their Junk status. I'm sorry that I was unclear on this:
 - The problem that I am having that needs fixing is that the emails are put in the Bulk Mail folder UPON ARRIVAL since I don't get a notification when that happens, defeating the whole purpose of the email.
 - The problem that most probably (IMHO) causes the first one is that the emails' Junk status seems to be PERMANENTLY set to "Yes, this is Junk" even though the emails stay in the inbox after moving them there.

What I have witnessed as well is that whenever I set all messages from the website in question to "Not Junk", the next notification usually ends up in the inbox, but after some time it always reverts to its faulty behaviour.

Comment 3

[Wayne Mery (:wsmwk)]

Do you see this when using version 60, downloadable from getthunderbird.com ?

Comment 4

[formyer]

I cannot easily check whether emails are automatically moved into the Bulk Mail folder upon arrival, but the manual mark/unmark as spam still DOES NOT work (and since marking an email as spam in the inbox does move it to the Bulk Mail folder automatically, and vice-versa, I would say yes, the unwanted behaviour is still there).

Comment 5

[chris]

I'm experiencing the same behaviour (Junk Filter Resets Mail Status on Every Refresh, Filters Are Ignored) using TB v60.0. using imap with a yahoo.com mail account. Even though I have disabled all settings in TB (that I could find) so it shouldn't do any spam filtering at all, certain messages that are not spam end up in Bulk mail folder.

When selecting one of these in Bulk mail folder and clicking [Not Spam] button the message is moved back to Inbox having the flames icon disabled (grey). But the flames icon turns back to orange as soon as looking for messages next time (eiter automatically or by clicking [Get Messages] and no matter if some New messages were received or not). Although the messages previously moved from Bulk mail folder to Inbox do not get moved back to the Bulk mail folder the spam indication on the messages reappearing in is annoying.

After doing some searching and reading various threads I started to think it may be related to the mail provider (yahoo.com) marking these messages somehow. And indeed in the message header is the following field:
"X-YahooFilteredBulk: <some IP>"

There are 2 issues connected to this:
1) I do not want TB to react on this header information at all, because all spam handling should be turned off already, so it should just not care. But it seems to take action anyway.

2) At least I should be able to undo the junk classification by applying a filter 'after spam classification' when e-mail is fetched from the server.

So, I created a filter looking for a custom header field "X-YahooFilteredBulk" containing a . (single dot) because any IP given would contain a dot and setting it to run 'after spam classification'.

Running the filter manually on Inbox works fine and toggles the notification flames icon to grey on affected messages. Yet clicking on [Get new mail] or when mail is retrived automatically next time, all the messages previously unmarked from being junk by the filter action, are again shown as junk. Of course manually applying the filter once more would remove the junk status, but this is pointless because on next retrive they automatically become junk marked again. The filter does not seem to run 'AFTER spam classification' as desired. 

I've also tried to switch the TB 'mail.imap.filter_on_new' setting to false, then restarting TB.
Resulting in no difference in behaviour.
And tried to run the filters 'BEFORE spam classification', then restarting TB.
Resulting in no difference in behaviour, neither with 'mail.imap.filter_on_new' set to false or true.
Clearly there is some bug hiding somewhere. Hope this information helps you on the quest of locating it :)

Comment 6

[krusea]

Here's another one experiencing this behaviour.
My mail provider ist AOL and the junk functionality (especially classifying mails as "no junk") worked flawlessly until I upgraded to Thunderbird 60.
I don't have records on when I made that update, but I also found that the protocol for selflearning filter (in my german version: "Junk-Protokoll für selbstlernenden Filter") has it's last entry on September 13th. That might very well be the day of updating.
I can also confirm that these mails do contain a header named "X-YahooFilteredBulk".

Comment 7

[Wayne Mery (:wsmwk)]

so this issue is limited to yahoo/aol ?

If so, it's likely a duplicate

Comment 10

[Wayne Mery (:wsmwk)]

Gene, does this issue match to Bug 1260059 - spam e-mail from yahoo imap server marked as junk doesn't keep junk flag ?

Comment 11

[gene smith]

(In reply to Wayne Mery (:wsmwk) from comment #10)

Gene, does this issue match to Bug 1260059 - spam e-mail from yahoo imap server marked as junk doesn't keep junk flag ?

I'm sure it will help but maybe not completely. All it addresses is allowing the junk classification to remain in place when the user marks a yahoo/aol (or possibly other providers) message as spam or not spam. It doesn't address anything directly with filtering. So it won't fix messages getting automatically classed as spam even when spam filtering turned off; that's potentially a different bug.

So it should fix this from comment 0:

(In reply to formyer from comment #0)

If I manually mark them as "Not Junk", they are immediately re-categorized
as junk the moment the inbox updates (pressing F5, waiting a bit, closing
and restarting the application, ...).

Comment 12

[gene smith]

First off, I don't see in the tb code that the custom header X-YahooFilteredBulk is used anywhere so I don't think that is a factor.

What I think is happening is that yahoo (and aol that uses the same server) is classifying emails as spam, as indicated by the X-YahooFilterBulk header, but it is also including the imap FLAG/keyword "$Junk" when the message is fetched. So tb sees the message as spam even though tb didn't determine that it really is spam. So this may be causing the message to be moved by tb to the configured spam folder. Most imap providers don't automatically set spam/not-spam keywords but let the client set the keywords. Yahoo/aol take a different approach.

So, on 2nd thought, maybe the fix in Bug 1260059 will mostly fix this bug too.

Comment 13

[Wayne Mery (:wsmwk)]

Please test the beta https://www.thunderbird.net/en-US/channel/ and report your results

Comment 14

[chris]

Have just tested with latest beta: 67.0b1 (32-bit) on a win 10 64-bit system.

Had fortunately a few old e-mails still marked with the "sticky" junk flag. Tested to unmark the junk flag on those. Junk status flag did not reappear any more after refreshing the messages list with "Get Messages". Also tried to close Thunderbird and reopen followed by "Get Messages" the junk flag did not reappear. So to me it seems the bug is fixed now.

Comment 15

[krusea]

My test environment also was the latest beta, but on Windows 8.1.

I can also confirm that the issue of the junk mail flag disappearing right away is gone - the flag now does stay once it is set. This is also true if used in the trash folder.
On the other hand, the "adaptive junk filter" functionality still seems to be broken - I had cleared the log entries for it in December and there wasn't a new entry for the new junk mail after I marked it as junk.

Comment 16

[Wayne Mery (:wsmwk)]

Thanks for the tests.