1367684 - junk mail always checks all messages marked as 'junk'

Status: RESOLVED INVALID

(Thunderbird :: Untriaged, defect)

Description

[Christian Riechers]

User Agent: Mozilla/5.0 (X11; Linux i686; rv:54.0) Gecko/20100101 Firefox/54.0
Build ID: 20170523002428

Steps to reproduce:

Problem started with Thunderbird 52.
IMAP account, subscribed to the 'Spam' folder. I.e. all junk messages received on that account are downloaded, including those already classified as 'junk' by the provider's filter on the server.
Selecting the accounts 'Spam' folder in the folder pane.
Menu Button - Tools - Run Junk mail Controls on Folder
This triggers the Thunderbird Bayesian filter to run locally and eventually classifies messages as 'junk'.


Actual results:

The problem is, after a Thunderbird restart always all messages in the Spam folder are checked, regardless whether they are already marked as 'junk' or not.
This is particularly annoying and takes time when there are a lot of messages.
Subsequent checks only run on new messages not (yet) marked as 'junk'.


Expected results:

Even after a reboot only new messages not (yet) marked as 'junk' should be checked.
That was the behavior with TB45.

Comment 1

[Jorg K (CEST = GMT+2)]

Could that be related to bug 308148 and bug 1320832?

Christian, could you set the preference mail.imap.filter_on_new to false, which was the original value in TB 45.

We've also just received bug 1367443.

Comment 2

[Christian Riechers]

(In reply to Jorg K (GMT+2) from comment #1)
> Christian, could you set the preference mail.imap.filter_on_new to false,
> which was the original value in TB 45.

Changing mail.imap.filter_on_new to false makes no difference. TB is still checking all messages in Spam after startup.

Comment 3

[Kent James (:rkent)]

Could you install the JunkQuilla extension, and enable the column "Junk Status +"? Then check if the server-classified messages are being detected as such (which will show up as a little flag icon meaning that the server flagged it).

Normally only the Inbox will be checked for Junk, so it is not clear to me that the behavior you are seeing is related to junk processing. You might confirm, with JunQuilla, that none of the messages in the Spam folder are being processed locally for junk (which would show by them being flagged as either junk or spam with the capital sigma icon).

Is the problem fixed by disabling offline use for this folder (Right click on folder, folder Properties, Synchronization, disable "select this folder for offline use").

Is this spam folder marked as he Junk folder, or is it a normal folder?

Comment 4

[Christian Riechers]

(In reply to Kent James (:rkent) from comment #3)
> Could you install the JunkQuilla extension, and enable the column "Junk
> Status +"? Then check if the server-classified messages are being detected
> as such (which will show up as a little flag icon meaning that the server
> flagged it).

I did, but I don't think I really understand what I'm supposed to check.

> Normally only the Inbox will be checked for Junk, so it is not clear to me
> that the behavior you are seeing is related to junk processing.

As stated above, I'm triggering junk processing of the spam folder manually:
Selecting the accounts 'Spam' folder in the folder pane.
Menu Button - Tools - Run Junk mail Controls on Folder

> You might confirm, with JunQuilla, that none of the messages in the Spam folder are
> being processed locally for junk (which would show by them being flagged as
> either junk or spam with the capital sigma icon).

I've attached a screenshot of the contents of the spam folder prior to manually triggering junk processing. The top 6 messages are new and haven't been processed locally yet. All others with the flame icon and the flame/capital sigma icon have been processed already during the previous session a day before.
The problem is, they are being downloaded and processed again, when manually triggering junk processing for the spam folder.

Also attached is a screenshot of the spam folder after manually triggering junk processing.
 
> Is the problem fixed by disabling offline use for this folder (Right click
> on folder, folder Properties, Synchronization, disable "select this folder
> for offline use").

"select this folder for offline use" is not checked for the spam folder.

> Is this spam folder marked as he Junk folder, or is it a normal folder?

It is marked as the Junk folder.

Comment 5

[Christian Riechers]

Attachment: before_junk_processing.png

Comment 6

[Christian Riechers]

Attachment: after_junk_processing.png

Comment 7

[Kent James (:rkent)]

"I've attached a screenshot of the contents of the spam folder prior to manually triggering junk processing. The top 6 messages are new and haven't been processed locally yet. All others with the flame icon and the flame/capital sigma icon have been processed already during the previous session a day before.
The problem is, they are being downloaded and processed again, when manually triggering junk processing for the spam folder."

Manually triggering junk processing on a folder is intended to process all messages on a folder, so it is expected behavior that all of the existing marked messages are also downloaded and checked. This behavior is useful if, for example, the inbox has a lot of spam messages that are not being caught, you then train a few messages, and you want to rerun the spam filter (including over previously processed messages) to catch spam messages that are now detected with the improved training.

I don't really understand why you want to run the spam filter on spam messages that get copied to that folder. But I believe that you could accomplish your goals by setting up a virtual folder that only shows messages not marked as junk in the junk folder, then running the spam filter on that virtual folder.

I'm tempted to close this bug as INVALID since it claims that expected behavior is a bug.

Comment 8

[Christian Riechers]

(In reply to Kent James (:rkent) from comment #7)
> Manually triggering junk processing on a folder is intended to process all
> messages on a folder, so it is expected behavior that all of the existing
> marked messages are also downloaded and checked.

Yes, but download and check doesn't have to happen for all messages again and again, even if they have been downloaded and checked previously.

> This behavior is useful if,
> for example, the inbox has a lot of spam messages that are not being caught,
> you then train a few messages, and you want to rerun the spam filter
> (including over previously processed messages) to catch spam messages that
> are now detected with the improved training.

That is exactly what I'm doing, except that the IMAP server has moved those messages to the spam folder already.

> I don't really understand why you want to run the spam filter on spam
> messages that get copied to that folder.

For the reason you outlined above.

> But I believe that you could
> accomplish your goals by setting up a virtual folder that only shows
> messages not marked as junk in the junk folder, then running the spam filter
> on that virtual folder.

I don't see what good a virtual folder would do and what problem it would address.

> I'm tempted to close this bug as INVALID since it claims that expected
> behavior is a bug.

I'm not the expert. All I can say is that the behavior I'd expect was the way it did work up to Thunderbird 45, and changed after that.

Comment 9

[Kent James (:rkent)]

(In reply to Christian Riechers from comment #8)
> (In reply to Kent James (:rkent) from comment #7)
> > Manually triggering junk processing on a folder is intended to process all
> > messages on a folder, so it is expected behavior that all of the existing
> > marked messages are also downloaded and checked.
> 
> Yes, but download and check doesn't have to happen for all messages again
> and again, even if they have been downloaded and checked previously.

You have to recheck the message if the spam training has changed.

If you don't want those messages to be downloaded again, you can enable "select this folder for offline use"for the spam folder.


> 
> I don't see what good a virtual folder would do and what problem it would
> address.

If you only want to run spam processing on messages in a folder that have not had previous spam processing, you could setup a virtual folder that only shows the folders that you want to check, then run spam processing on that virtual folder. That should accomplish what you are aiming for.

> I'm not the expert. All I can say is that the behavior I'd expect was the
> way it did work up to Thunderbird 45, and changed after that.

I would not expect this behavior to change from 45 to 52, other than the issues that Jörg already asked you to investigate. So I am not sure why this changed.

Comment 10

[Wayne Mery (:wsmwk)]

Christian, 
Does this still happen?
And if you return to version 45 does problem go away?

Comment 11

[Christian Riechers]

(In reply to Wayne Mery (:wsmwk) from comment #10)
> See Also: → bug 1435192

I think bug 1435192 is an entirely different issue.

> Does this still happen?
> And if you return to version 45 does problem go away?

I haven't tried this again with v45. I've turned synchronization back on for the junk folder, so spam messages are downloaded once, but not over and over again. I think I can live with that.
Closing this bug as INVALID.