194090 - [junk] purge of designated junk folder will purge not junk messages in that folder

Status: VERIFIED FIXED

(MailNews Core :: Backend, defect)

Description

[esther]

Using trunk builds 20030218 on winxp, maco osx and linux if I have the purge
option enabled, all messages that meet the date requirment for purging will be
purged including message that are NOT labeled as Junk.  

1. Launch app that has a mail account set up with Junk Controls on, a designated
Junk folder and the option for deleting junk mail from junk folder after [1]
days enabled.
2. View the Junk folder to see that you have a combination of  Junk Mail and Not
Junk mail in that folder with dates greater than 24 hrs.  Wait 5 minutes for the
purge process to happen. 

Result: All messages Junk and Not Junk dated greater than 24 hrs. are purged
from that folder when purge fires.
Expected:  Only messages marked as Junk dated greater than 24 hrs should be
purged, Not Junk messages should remain in folder

Comment 1

[esther]

Nominating, this is data loss especially bad if user has selected Trash as
designated Junk folder because not all mail in Trash is Junk and a delete from
Trash is not recoverable unless it's done right away with Undo.

Comment 2

[Samir Gehani]

Mail triage team: nsbeta1+/adt2

Comment 3

[(not reading, please use [e-mail] instead)]

the fix is to the code that searches the junk folder, I need to append a search
term.

Comment 4

[(not reading, please use [e-mail] instead)]

> the fix is to the code that searches the junk folder, I need to append a search
> term.

actually, that won't work for imap.  I can't search on junk status, as not all
servers support keywords, and that's how we store the junk status on the server,
and we are doing an online (server) search.

instead, as I get back search hits, I'll check the msg header and verify it is
actually a junk message.

fix in hand.

Comment 5

[(not reading, please use [e-mail] instead)]

Attachment: fix for this problem, and clean up of how the purge code works

I still need to test it, and I'll make to write up something on how this code
works and how QA can use the prefs

Comment 6

[(not reading, please use [e-mail] instead)]

Attachment: new patch

Comment 7

[(not reading, please use [e-mail] instead)]

Attachment: add prlogging for QA purposes

Comment 8

[(not reading, please use [e-mail] instead)]

Attachment: updated patch, fix bad prlogging.

Comment 9

[(not reading, please use [e-mail] instead)]

Attachment: correct patch

Comment 10

[(not reading, please use [e-mail] instead)]

fixed.  this has r/sr=bienvenu

QA (esther), for testing, use these:

NSPR_LOG_FILE=c:\purgelog.txt
NSPR_LOG_MODULES=MsgPurge:5

user_pref("mail.purge.min_delay",5);  (default is 480 minutes, or 8 hours)
user_pref("mail.purge.timer_interval",1); (default is 5 minutes)

here's how this works:

every mail.purge.timer_interval minutes (starting when mail accounts are 
loaded), looping through all accounts and

find the first server that:
  can get incoming messages (if you can't, we don't bother, since this type of 
account won't show up in the JMC)
  and is enabled to do purging
  and has a valid junk folder that exists
  and hasn't been purged in mail.purge.min_delay minutes.
  and isn't busy (like if this was a pop account, and we were getting new mail)
  and if we require authentication to open the junk folder (like on imap) and 
we're authenticated

if we find one, we stop looking and search that folder for any messages older 
than the per-JMC purge interval.
on the search results, we delete all messages that are marked as spam 
(according to the msg db, not according the imap server)
now update when that account was last purged.  (this is stored on the junk mail 
folder, so if the junk mail folder is shared, we do the right thing)

we only purge one account per timer firing.

if I have 5 accounts that are set up for purging, 
it will take 5 timer firing to get to them all.  
(the purge service can only do one at a time.)


I'll add this to a doc on mozilla.org

Comment 11

[(not reading, please use [e-mail] instead)]

fixed. the dataloss was also addressed.

Comment 12

[(not reading, please use [e-mail] instead)]

the doc is now at http://www.mozilla.org/mailnews/arch/spam/purge.html (will be
live in a few hours)

esther sent me some questions, and I'll put the answers in that doc.

Comment 13

[esther]

Using trunk build 20030305 on winxp only at this time was tested with a single
mail account profile and a multiple mail account profile. Purging happened as
expected in both mail accounts at the intervals and delays set as mentioned
above. Passed, and verified (with exception of mail servers that don't support
search command as noted in bugscape bug 22828)
Data loss of non-junk mail is fixed and verified.
Need to test linux and macosx now.   

Comment 14

[esther]

Using trunk build 20030311 on linux and macosx and windows I found that POP
accounts purge correctly in profile with single mail account and multiple mail
accounts.  However, IMAP purge seems to purge once in a session then stops
purging. I'm testing this now with just IMAP accounts to see if the purge on POP
interfers with the IMAP purge. 

Test scenario delay set to 10, interval set to 2
Profile has 4 accounts in this order  IMAP, POP, web and other type of IMAP.
Accounts 1 & 2 have purge feature checked with a value of older than 1 day.
Accounts 2 & 3 do not have purge feature checked. 
Biff is set to 30 minutes for all accounts. 

Purge happens on account 1  2 minutes after logging in =pass
Purge happens on account 2  4 minutes after loggin in =pass

I move expired trash messages into Junk folders for both accounts 1 & 2
No purge on account 1 after 15 minutes in same app session. failed
Purge happens again on account 2   15 minutes later in same app session. passed

I move expired trash messages into Junk folders for both accounts 1 & 2 
Still no purge on account 1 after 26 minutes in sam app session  failed 
Purge happens again on account 2   26 minutes later in same app session. passed

Comment 15

[esther]

Working on Mac and linux multiple accounts in profile purging now.  Basically,
this feature is working, however as pointed out in previous comment, purge
doesn't always happen when expected and some messages don't get purged the first
time but are caught the next time around.  If this is the case for Mac and linux
too, I will verfify this and write up a new bug for the irratic behavior.   

Comment 16

[esther]

Mac OSX, Linux and Winxp, basically does purge junk mail at delay and interval
times for specified folder for single account profiles and multiple account
profiles.   Only junk mail is purged, so that portion of this bug is fixed too. 
As noted above, sometimes when the purge fires, junk mail is not purged during
that firing, however is purged at the next firing on IMAP accounts.  That needs
more investigation as to why and I will log a new bug when I find out.  This bug
as stated is fixed and verified.