191042 - [junk] IMAP mail flagged as Junk in Inbox, lose the flag when server is pinged

Status: VERIFIED FIXED

(SeaMonkey :: MailNews: Message Display, defect)

Description

[esther]

Using trunk build 20030128 on winxp, mac osx and linux and an IMAP mail account,
if the user flags a message as Junk in the Inbox, then clicks Get Msg the junk
flags are removed, however the message is still flagged as Junk on server.  

1. Launch app, launch Mail, log into a IMAP mail account.
2. Select a message that has never been flagged as Junk before and flag it as
Junk.  
Result: Notice the message has a flag- expected.
3. Click Get Msg  
Result: That same message loses it's Junk flag, however the flag is still on the
message on the server because if you exit the app and relaunch the flag shows up
in the thread pane.
Expected: Not to lose the Junk flag

Comment 1

[esther]

nominating, this needs to work 

Comment 2

[Michael Baffoni]

I'm still seeing this on 2003013108.

Comment 3

[Daniel Veditz [:dveditz]]

Don't see any evidence of active work on a patch, doesn't otherwise match the
1.3b stop-ship criteria, minusing on behalf of drivers.

Comment 4

[Samir Gehani]

Mail triage team: nsbeta1+/adt2

Comment 5

[Ken Roser]

I'm seeing something similar to the original report.  I can set multiple
messages as junk on my imap Inbox and within a few minutes, some of the junk
status flags (but not always all) might be turned off.  I haven't yet been able
to determine when/if this will occur.  Sometimes it works fine.  As I'm writing
this I wonder if the flags get cleared when my client queries the imap server to
see if there is new email.  I'll report back with further information as I
obtain it.  I've seen this behavior on 1.3b and also Build 2003021804

Comment 6

[Ken Roser]

Another note.  I can set the junk status on many messages on my imap inbox, but
if  I then hit the Get Msgs button, all the junk status flags I had just set get
cleared.

Comment 7

[(not reading, please use [e-mail] instead)]

sarah sees this too.

Comment 8

[(not reading, please use [e-mail] instead)]

*** Bug 196599 has been marked as a duplicate of this bug. ***

Comment 9

[Ere Maijala (slow)]

How about blocking 1.3? This bug makes it almost impossible to train the spam
filter and will definitely confuse any new users who try it (I have some real
world experience of it). I believe the bug will have very high visibility if
shipped in 1.3. The spam filter is supposed to be one the shining new features,
isn't it?

Comment 10

[Ere Maijala (slow)]

Attachment: Patch to clear old custom flags correctly

Sorry for jumping in like this, I'd just like to see this fixed :)

Comment 11

[(not reading, please use [e-mail] instead)]

ere, thanks for jumping in.

this is badness, but 1.3 might already be out the door.

I hope to review and test your fix today.

Comment 12

[Ere Maijala (slow)]

Yep, I know. Too bad, but I'll be using nightlies anyway :)

Comment 13

[(not reading, please use [e-mail] instead)]

thanks again for the patch, ere.

here's the problem:

a message can get into a state where we have both Junk and NonJunk flags, since
we aren't clearing them.

to reproduce the problem:

take an imap message, mark as junk, mark as not junk, mark as junk.
then, hit get msg.  the junk state will clear.

I've tested the fix, and will go land it.

re-assign to [e-mail], for credit

Comment 14

[(not reading, please use [e-mail] instead)]

Comment on attachment 117116 [details] [diff] [review]
Patch to clear old custom flags correctly

r/sr=sspitzer

I've checked in the change with an additional comment, and changed
ClearCustomFlags() to clearCustomFlags() [to follow interCaps]

Comment 15

[(not reading, please use [e-mail] instead)]

fixed.  huge thanks to ere for squashing this bug.

Comment 16

[Ere Maijala (slow)]

Thanks :) Good point about the interCaps. I got distracted by other methods in
that interface. 

Comment 17

[esther]

Reopening, when Seth was working on this with me we saw a case where manually
analyzing messages would mark the messages as junk, then Get Msg unmarked them.
  We believe this case needs to be fixed, still testing other criteria around
this case. 

Comment 18

[(not reading, please use [e-mail] instead)]

re-assigning, to look into the problem esther mentions.

Comment 19

[(not reading, please use [e-mail] instead)]

esther's problem is with manually analyzing messages.

looking at the code, I see the problem.

but the good news is I can fix this, and another problem (about manual analyzing
not doing the same thing as the JMC controls, whitelisting, moving, etc) at the
same go.

accepting.

Comment 20

[(not reading, please use [e-mail] instead)]

> and another problem (about manual analyzing
> not doing the same thing as the JMC controls, whitelisting, moving, etc) at the
> same go.

not sure about that yet.  let me start with just this bug.

Comment 21

[(not reading, please use [e-mail] instead)]

here's what I wasn't sure about.

currently, manually analysis (Run Junk Mail Controls on Selected Messages...)
will determine if a message is junk or not.

it will not move messages and it will not whitelist.

Esther, Ninoschka and I were discussing this recently, and we thought that it
should follow your JMC settings.

But here's the problem with that:

1) what do you do with messages in Local Folders?  (No JMC controls apply for those)
2) what do you do if the user wants to analyze message a message from someone
they know (remember the klez virus)

After giving it some more thought, I think we want to keep it as is, except to
change it from:

"Run Junk Mail Controls on Selected Messages..."
to
"Analze Folder for Junk Messages..." (or however we word it.)
and add a message context menu item for
"Analyze For Junk"

Comment 22

[(not reading, please use [e-mail] instead)]

the tools menu item is now "Run Junk Mail Controls on Folder" (see bug #198749)

but we still have this problem on manually anaylzed messages (but not on incoming).

Comment 23

[Ken Roser]

I'm having trouble understanding the reasoning for changing the Tools menu item
from "Run Junk Mail Controls on Selected Messages" to "Run Junk Mail Controls on
Folder".

With this change how am I to run the junk mail controls on a subset of messages
in my folder.  I'm an IMAP user and often have a lot of messages in a folder. 
At times to test how effective my training is, I'd like to run the junk mail
controls on a subset of this folder.  Now there's no way to do it unless I do
the whole folder;  that's a lot of message downloading from my IMAP server.

I wish you'd reconsider adding back the ability to run junk mail controls on the
selected messages in the folder.

Comment 24

[(not reading, please use [e-mail] instead)]

over to david

Comment 25

[David :Bienvenu]

How do I recreate this bug? I've tried manually analyzing messages in my inbox,
and hitting get new mail frequently, and I don't ever see the junk status get lost.

Comment 26

[Michael Baffoni]

I'm not seeing this on current builds, currently I'm on Mozilla/5.0 (Windows; U;
Windows NT 5.0; en-US; rv:1.4b) Gecko/20030428 .  I'd suggest marking WFM.

Comment 27

[(not reading, please use [e-mail] instead)]

I'll see if I can still reproduce it.

I know esther and I were able to reproduce this pretty easily.

Comment 28

[Ken Roser]

Here's my input.
Since 1.3 things have improved in the 1.4a and 1.4b builds.  I can no longer
reproduce the problem on my Inbox, but I still can on a sub-folder in 1.4b. 
Here are the steps that show the symptom:

Set up Junk Mail Controls to move junk mail to the Junk folder.
Accumulate some messages in the Junk folder over time as junk mail arrives.
Select the junk folder.  At this point you might see some of the messages that
were classified as junk no longer has the junk flag visible.
Select all the messages in the folder that aren't flagged as junk and mark them
as junk.  Now all messages should show a junk flag.
Click Get Msgs.  Some of the junk flags will turn off.

In case it's significant my user.js file has the line:
user_pref("mail.check_all_imap_folders_for_new", true); 

Comment 29

[David :Bienvenu]

Ken, can you attach or e-mail me a protocol log showing these symptoms by
following these instructions? Thx.

http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap

yes, user_pref("mail.check_all_imap_folders_for_new", true); is very
significant. How many folders do you have? What's your check for new mail
interval set to in minutes? Does this happen every time you follow those steps?
Is it the case that every time you open your junk folder, not all the messages
have the junk flag set? Do you know what kind of imap server you're using? the
log will tell me if you just want to send me that. I'm wondering if your imap
server supports user-defined keywords. If not, we might have to reclassify your
junk messages (not sure if we classify in the junk folder).

Comment 30

[Ken Roser]

Attachment: IMAP protocol trace (.zip format) while junk flags are being cleared erroneously

Comment 31

[Ken Roser]

I've just attached a protocol trace (imaplog.zip) of the Junk flags being
cleared on many of the messages in my Junk (IMAP) folder.  After a bunch of
experimentation it seems the problem of the flags being cleared is easiest to
reproduce as follows:

Select the Junk folder
Hit Get Msgs
While my numerous folders are being checked select the Inbox then the Junk folder.
Upon returning to the Junk folder you can see the flags get cleared.

Perhaps the background periodic checking of folders for messages is what usually
causes this problem.

One other thing.  If I exit Mozilla then return to the Junk folder, the junk
flags are now set again.

Comment 32

[David :Bienvenu]

I've looked at that log. Nothing looks odd. I've tried those steps with my
machine, and not had a problem. Are the messages in your junk folder that lose
their junk status only "new" messages? Or are they messages that have been in
the folder for a while? I'd ask for a test account on the machine with the
problem, but I don't see anything odd in the log that would make me think it's
server-specific.

Comment 33

[esther]

David, this bug as originally reported and commented in #19 is not as easy to
reproduce. I have given you one of our test accounts that is showing a junk icon
changing from on to off just by moving down the folder list displaying each
folder then moving back up the list displaying the folders thread pand again. 
Let me know if you need more info.

Comment 34

[David :Bienvenu]

Attachment: proposed fix

reset the custom flags hash when switching folders

Comment 35

[(not reading, please use [e-mail] instead)]

Comment on attachment 123233 [details] [diff] [review]
proposed fix

r/sr/a=sspitzer

Comment 36

[David :Bienvenu]

fix checked in.

Comment 37

[esther]

Using trunk build 20030519 on winxp, macosx and linux this is fixed.  I verified
this with the latest test case I gave to david as mentioned in comment 33.  

Comment 38

[David :Bienvenu]

thx very much to Esther for finding a reproducible test case!