242744 - Junk mail preference delete junk after x days doesn't work if "Junk Purge" is enabled for multiple accounts

Status: VERIFIED FIXED

(MailNews Core :: Filters, defect)

Description

[WADA:World Anti-bad-Duping Agency]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7b) Gecko/20040421
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7b) Gecko/20040421

Automatic Junk Mail Purge is not kicked if Junk Mail Purge condition occurs on
multiple accounts at same time.

This is apparantly DUPE of other bug reports on "Junk Mail Purge" failure when
Junk Mail Control is enabled for multiple accounts or when "Junk Mark" is lost
in Junk folder.
But I can not find which is the exact bug because too many bugs were found for
this type of problem.
This bug's purpose is to supply NSPR log and report problem recreation procedure.
Test was done with relase version of Mozilla 1.7RC1(installer version) on
Win-2K(SP4). 

(Problem-1)

When "[N] purging! searching for messages older than X days" conditions occured
on multiple account, Junk Mail purging was not kicked.
In my case, both pop3.ops.dti.ne.jp and mail.spymac.com had Junk mails in Junk
folder.
This is probably Mozilla's bug. 
See [First Junk Deletion] and [Second Junk Deletion]

(Problem-2) 

If "Junk Mark" is lost in Junk folder, Junk Purge does not delete mails in Junk
folder because junkScore becomes insufficient.
In my case, "Junk mark" was lost due to manual "Junk.msf" deletion, and this
"Junk mark" loss is reasonable.
See [Second Junk Deletion] and [Third Junk Deletion]
This is design and works well, I think.

However, some other bugs report loss of "Junk Mark" by other than user himself.
If "Junk mark" is only saved in ".msf", and if "Junk Mark" is reset on automatic
".msf" recreation due to some kind of error or special occasion such as
timestamp inconsistency,
manual recovery procedure of marking Junk again should be documented at somewhere.

Reproducible: Always
Steps to Reproduce:




<Following is summary of NSPR log in attachment>

[Before start of Mozilla]
(1) Junk Control is active on all POP accounts including Local Folders
(2) Junk.msf of pop3.ops.dti.ne.jp is deleted to force .msf recreation
     ==> All mails in Junk folder looses "Junk" mark.
(3) Junk folder of mail.spymac.com also contains Junk mails.

[Restart of Mozilla]

  [293f20]: mail.purge.timer_interval=5 minutes
  [293f20]: setting to check again in 5 minutes

(3) Click Junk folder and view some mails in Junk folder of pop3.ops.dti.ne.jp
     ==> All mails in Junk folder are marked "Not Junk" due to .msf deletion.

[First Junk deletion (5 minutes after restart of Mozilla)]

  [293f20]: performing purge
  [293f20]: 10 servers

  [293f20]: [0] news.mozilla.org (nntp)

  [293f20]: [1] pop.ops.dti.ne.jp (pop3)
  [293f20]: [1] purging! searching for messages older than 1 days

  [293f20]: [2] Dummy.POP3.Server.DTI (pop3)
  [293f20]: [3] bird.zero.ad.jp (pop3)
  [293f20]: [4] news3.dti.ne.jp (nntp)
  [293f20]: [5] Dummy.POP3.Server.DDI-Pocket (pop3)
  [293f20]: [6] Local Folders (none)
  [293f20]: [7] newsall.dti.ne.jp (nntp)
  [293f20]: [8] news.opera.com (nntp)

  [293f20]: [9] mail.spymac.com (pop3)
  [293f20]: [9] purging! searching for messages older than 14 days

  [293f20]: lastPurgeTime is now Thu May 06 07:49:25 2004
  [293f20]: on new search
  [293f20]: setting to check again in 5 minutes
  [293f20]: 0 messages to delete

(4) Disabled Junk Mail Control for all POP3 accounts except pop3.ops.dti.ne.jp
    Note: All mails in Junk folder of pop3.ops.dti.ne.jp are still "Not Junk".

[Second Junk deletion (10 minutes after restart of Mozilla)]

  [293f20]: performing purge
  [293f20]: 10 servers
  [293f20]: [0] news.mozilla.org (nntp)
  [293f20]: [1] pop.ops.dti.ne.jp (pop3)
  [293f20]: [1] purging! searching for messages older than 1 days

  [293f20]: [2] Dummy.POP3.Server.DTI (pop3)
  [293f20]: [3] bird.zero.ad.jp (pop3)
  [293f20]: [4] news3.dti.ne.jp (nntp)
  [293f20]: [5] Dummy.POP3.Server.DDI-Pocket (pop3)
  [293f20]: [6] Local Folders (none)
  [293f20]: [7] newsall.dti.ne.jp (nntp)
  [293f20]: [8] news.opera.com (nntp)
  [293f20]: [9] mail.spymac.com (pop3)

  [293f20]: lastPurgeTime is now Thu May 06 07:54:25 2004

  [293f20]: on new search

  (All mails are checked but not deleted because they are not marked as Junk)

  [293f20]: setting to check again in 5 minutes
  [293f20]: 0 messages to delete

(5) Marked all mails in Junk folder of pop3.ops.dti.ne.jp

[Third Junk deletion (15 minutes after restart of Mozilla)]

  [293f20]: [1] pop.ops.dti.ne.jp (pop3)
  [293f20]: [1] purging! searching for messages older than 1 days

  [293f20]: on new search
  (Many Junk mails were nominated as mails to delete)
     [293f20]: junkScore=100 (if empty or <= 50, don't add to list delete)
     [293f20]: added message to delete

  [293f20]: setting to check again in 5 minutes
  [293f20]: 352 messages to delete
  [293f20]: delete messages

Congrturations!
Junk Mails were successufully deleted!

Comment 1

[WADA:World Anti-bad-Duping Agency]

Attachment: NSPR log for MsgPurge:5

Comment 2

[WADA:World Anti-bad-Duping Agency]

"Junk Mark" loss on ".msf" recreation will be resolved when Bug 195737 will be
fixed.   
> Bug 195737 Junk mail flag should be stored in mbox, instead of msf.

Comment 3

[WADA:World Anti-bad-Duping Agency]

Change summary according to new discovery.

(New)
> Automatic Junk Mail Purge's action of "purging! searching for messages older
than NN days" is executed only last account among Junk Purge enabled accounts.

(Old)
> Automatic Junk Mail Purge is not kicked if Junk Mail Purge condition occurs
> on multiple accounts at same time.

Sequence number(order) of account used in Junk Purge was not order of ID number
of account definition in prefs.js.
So "last account" is not predictable.

Note:
Similar problem was reported in the past for Junk filterig.
 - Bug 187504 Bayesian junk mail filters work with only 1 account
But this problem did not occur and does not occur in my environment.
Junk filtering worked well and is working well for all accounts on latest trunk
nightly.

Comment 4

[WADA:World Anti-bad-Duping Agency]

Attachment: MsgPurge NSPR log(Junk mail search on last account only)

'purging! searching for messages older than NN days' were issued for 4
accounts( [4], [7], [9], [18] ), but Junk mail search was done only on last
accout([18] in this test).

Summary of NSPR log(MsgPurge) is as follows.

(Junk mail count in Junk mail folder of each account)
  Account [4] : 141, Account [7] : 0, Account [9] : 757, Account [18] : 3

(1. First Junk Purge is kicked)

0[283f80]: [4] purging! searching for messages older than 14 days
0[283f80]: [7] purging! searching for messages older than 7 days
0[283f80]: [9] purging! searching for messages older than 30 days
0[283f80]: [18] purging! searching for messages older than 7 days

0[283f80]: lastPurgeTime is now Fri Nov 19 20:43:38 2004
0[283f80]: on new search
0[283f80]: messageId=[e-mail]
0[283f80]: messageId=[e-mail]
0[283f80]: messageId=[e-mail]

0[283f80]: setting to check again in 5 minutes
0[283f80]: 0 messages to delete
	   ==> Above three mails are mails in Junk folder of Account [18]
	   ==> Not-deleted is normal because Junk mark is lost due to other bug


(2. Delete Junk mails of Account [18])
    2-1. Create a folder nmaed "Temp"
    2-2. Move all mails in "Junk" folder to "Temp" folder
    2-3. "Compact folder" on "Junk"

(3. Second Junk Purge is kicked)

0[283f80]: [4] purging! searching for messages older than 14 days
0[283f80]: [7] purging! searching for messages older than 7 days
0[283f80]: [9] purging! searching for messages older than 30 days
0[283f80]: [18] purging! searching for messages older than 7 days

0[283f80]: lastPurgeTime is now Fri Nov 19 20:48:38 2004
0[283f80]: on new search
	   ==> No search result is avilable!

0[283f80]: setting to check again in 5 minutes
0[283f80]: 0 messages to delete

(4. Third Junk Purge is kicked)

0[283f80]: [4] purging! searching for messages older than 14 days
0[283f80]: [7] purging! searching for messages older than 7 days
0[283f80]: [9] purging! searching for messages older than 30 days
0[283f80]: [18] purging! searching for messages older than 7 days
0[283f80]: lastPurgeTime is now Fri Nov 19 20:53:38 2004
0[283f80]: on new search
	   ==> No search result is avilable!

0[283f80]: setting to check again in 5 minutes
0[283f80]: 0 messages to delete
0[283f80]: performing purge

(5. Copy back Junk mails of Account [18])
    5-1. Copy all mails in "Temp" to "Junk" folder
    5-2. "Compact folder" on "Junk"

(6. Third Junk Purge is kicked)

0[283f80]: [4] purging! searching for messages older than 14 days
0[283f80]: [7] purging! searching for messages older than 7 days
0[283f80]: [9] purging! searching for messages older than 30 days
0[283f80]: [18] purging! searching for messages older than 7 days

0[283f80]: lastPurgeTime is now Fri Nov 19 20:58:38 2004
0[283f80]: on new search
0[283f80]: messageId=[e-mail]
0[283f80]: messageId=[e-mail]
0[283f80]: messageId=[e-mail]

0[283f80]: setting to check again in 5 minutes
0[283f80]: 0 messages to delete
	   ==> Above three mails are mails in Junk folder of Account [18]
	   ==> Not-deleted is normal because Junk mark is lost due to other bug

Comment 5

[WADA:World Anti-bad-Duping Agency]

Test in Comment #4 is done with Mozilla Suite latest-trunk 2004111805(Win-2K).

Comment 6

[WADA:World Anti-bad-Duping Agency]

After test of Comment #4, I disabled "Junk Mail Control" of accounts.
 (1) Disabled all dummy IMAP accounts
     => Not purged
 (2) Disabled all dummy POP accounts(no logon, including 'Local Folders')
     => Not purged
 (3) Disabled all accounts which are multiple accounts on same POP3 server
     => Not purged
 (4) Disabled all accounts(logon is done, exept account [4] [9])
     => Not purged
 (5) Disabled account [9] (only accout [4] is enabled)
     => Expired Junk mails of account [4] was purged successufully.
 (6) Disabled account [4], enabled accout [9] (only accout [9] is enabled)
     => Expired Junk mails of account [9] was purged successufully.
This result also indicates that Junk Purge works only for an account(same result
as test in Comment #0).

Comment 7

[WADA:World Anti-bad-Duping Agency]

Change summary because "on new search" produced no mail listing when "Junk mail
control" was enabled for only two accounts(accout [4] and [9] in comment #4).

Comment 8

[WADA:World Anti-bad-Duping Agency]

Change summary again, because it is too long.

Problem can be said as ;
  Automatic Junk Mail Purge is not done properly,
  because "on new search" action based on condition of
  "purging! searching for messages older than NN days" is executed only one time
  (probably only for the last condition)
  even when the condition occurs on multiple accounts
  and searching Junk mails is done on incorrect Junk folder.

Comment 9

[Jacek Piskozub]

Confirming the bug. I've hit the same problem a few days ago after switching on
Junk Mial Controls on another POP3 account.

After finding this bug, checking off Junk Mail Control on all accounts but one
and restarting Mozilla (1.7.5 nightly build 20041126), I had 378 messages moved
to Trash after 5 minutes (meaning it worked as it is supposed to work). 

Also reassigning to bienenu as he is probably the guilty party (as admitted in
bug 228512 comment #21).

Comment 10

[David :Bienvenu]

Attachment: proposed fix

this part is the fix (the rest is renaming some vars to make the code a little
easier to read, and moving a log statement)

+	       // if we've never purged this folder, do it...
+	       if (curJunkFolderLastPurgeTime == nsInt64(0))
+		 break;

the problem was that if a folder haad never been purged, we'd never purge it if
there was another folder eligible for purging.

Comment 11

[Jacek Piskozub]

I see "fixed-aviary1.0". Then according to Asa mozilla 1.7.5 should also be
fixed (they are supposed to have the same gecko version down to the smallest patch).

Comment 12

[David :Bienvenu]

this has nothing to do with gecko...but I'm not opposed to putting this into 1.7.5

Comment 13

[Jacek Piskozub]

I know it is not gecko. However, per divers' decision on 1.7.5 and Aviary 1.0
equivalence see also bug 268231 comment #26 by David Baron.

Comment 14

[David :Bienvenu]

that comment refers to FireFox 1.0, not thunderbird 1.0. SM 1.75 will not match
thunderbird 1.0, afaik.

Comment 15

[WADA:World Anti-bad-Duping Agency]

I transfered to Mozilla suite latest-trunk 2004112906(Win-2K), then my old Junk
Mails of 2 POP3 accounts have been moved to Trash successufully.
It's very long time since bug open, but only a moment since David has come.
I should have invited David to this bug when I opened this bug...

Anyway, VERIFIED.  

Comment 16

[Asa Dotzler [:asa]]

1.7.5 has shipped. Moving request to 1.7.6.

Comment 17

[(not reading, please use [e-mail] instead)]

Comment on attachment 167263 [details] [diff] [review]
proposed fix

a=sspitzer

Comment 18

[Christopher Aillon (sabbatical, not receiving bugmail)]

plussing.  dveditz says bienvenu will land this.

Comment 19

[David :Bienvenu]

Attachment: fix that builds on 1.7 branch

this is what I'll check into the 1.7 branch.