Closed Bug 195787 Opened 22 years ago Closed 16 years ago

Read messages revert to unread status

Categories

(MailNews Core :: Networking: IMAP, defect)

Product:
MailNews Core
Component:
Networking: IMAP
Platform:
x86
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: david.e.herr, Assigned: Bienvenu)

Details

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.0) Gecko/20020529
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.0) Gecko/20020529

Some of my messages revert to the unread status after I have read them
(from INBOX connected to a imap server). Annoying to say the least. Is
this a "feature" that can be turned off, or a bug? Mozilla build id is
2002052918.

I haven't been able to descern a pattern as to which messages are
effected. Immediatley after leaving the message the unread counter is
decremented and the message line in the inbox indicates that the
message has been read, but then a few seconds later the message line
is bolded again and the unread message counter increments.

The Imap server is Netmail 3.01D. This product was previously known as NIMS
and is a product of Novell. The problem occurred with the previous version
of the product as well.

Reproducible: Sometimes

Steps to Reproduce:
1.Open a unread message
2.Close the message
3.The message is marked as read
4.After a short interval of time (from a few seconds to a few minutes) the
  message reverts to Unread Status


Expected Results:  
Once a message is opened it should stay marked as read unless the user
specifically asked that it be re-marked as Unread
reporter try a more recent build please then let us know if
your bug is still there.

http://www.mozilla.org/releases/

currently at 1.3b
NIMS is probably one of those imap servers that don't correctly say whether or
not there are any permanent flags (or one that doesn't return any permanent
flags) - please try updating to a newer version of mozilla where we handle those
cases correctly. I'm marking this invalid for now, since we don't accept 1.0
bugs anymore, unless they occur in post 1.2 builds or later. If this occurs in
1.3b, please re-open. Thx!
Status: UNCONFIRMED → RESOLVED
Closed: 22 years ago
Resolution: --- → INVALID
I've installed mozilla 1.3b and the problem is still occurring. When I first
invoked the mailer all of the message that I have read, came up correctly as
having been read. However as new messages come in, messages that I have read
since invocation are reverting to Unread status.

David Herr
Status: RESOLVED → UNCONFIRMED
Resolution: INVALID → ---
*** Bug 199433 has been marked as a duplicate of this bug. ***
I am experience similar problems on mail moved to local folders accounts.
It seems that mail from approximately before november 2002 is marked as unread
again.
Some additional info:
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4a) Gecko/20030331
I'am not using IMAP but pop accounts.
When I mark the messages as read, the stay marked as read.
The date seems to be between 13 and 20 november 2002.
If you believe this is another bug I will file it as a new one.
Ok, since bug 199433 has been marked as a duplicate of this one (incorrectly
IMHO), I will keep the discussion here.
It seems that it has something to do with the cyclical check for new messages,
if I set the time to 1 minute it is much worse (I refer to the symptoms in bug
199433, not this one).
BTW, I downloaded the latest nightly and the problem is still there.
I also tried user_pref("mail.server.server1.max_cached_connections", 1); but
apparently made no difference.
Still happening in build 2003041508
Minotaur (first trunk build, Apr 9) seems to work fine.

I tried build 20030314 and see the same behavior. In addition, some messages
don't load at all, I need to force them by double-clicking and opening the
message in a new window. Just says "done".

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.3) Gecko/20030314

I did not see that problem with version 1.2 (still use that one on my laptop).


anyone seeing this with IMAP, if they could attach a client-side imap protocol
log  by following these instructions, that would be very helpful

http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap
I reported this with IMAP in bug 199433.
Today I will be out of the office, tomorrow I'll try to get a client log.

WOW, before leaving the office I tried to get a log file. In 5 minutes it
generated a 20 megabytes file!.
It's no wonder that mozilla mail is barely usable over a slow link.
It's bug 18266 rearing its ugly head. See bug 18266 comment 20. And bug 101219.
Anyway at a quick glance it seems this bug isn't apparent from the log (besides
the verbisity it seems normal). Tomorrow I'll check again.

I would not turn on the "check other folders for new mail feature" over a slow
link. It was not designed to work over a slow link.

One guess for why you're losing the read state on messages is that your imap
server  is dropping the connection and losing mailbox state. If your server is
unhappy about having a bunch of mail folders selected (which is what the "check
other folders for new mail" feature does), it makes sense that it could be the
server dropping state. It's also possible that we're making multiple
simultaneous connections to the INBOX, and this causes your server to drop the
first connection, and the state changes made in that connection, like the UW
server does. We go to a lot of trouble to avoid this, but it would be something
to look for in the log.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Ok, finally I didn't go out of the office, so I had some time to study the
client side imap log.
Since I use two servers with a lot of mailboxes, in order to reproduce and
analize the problem I proceeded as follow:

1) injected a couple hundred fake (and with an empty body) messages in a folder
2) skimmed through the messages and at the same time triggered a "get new
messages" various times, until I saw the behaviour described in 199433 (read
messages going back as unread and then, after a while, went back to read).
3) once things stabilized, I reopened one of the messages that went through the
read->unread->read dance, in order to see the message-id and with it to know its
uid (in my server the uid is the same as the filename of the message, so I just
had to grep for the message-id).
4) filtered the client log to see only communications relating to that folder
and looked for all interchanges related to the incriminated uid.

I repeates points 2 to 4 variuos times.

I can only say that the log seems normal, e.g.

a) at first the message (let's say it's uid 414 in case I forget) has just the
\Recent flag
b) then it has both the \Recent and NonJunk flags for a couple of readings
c) then it has just the NonJunk flags for various readings
d) then when the message has been read mozilla will store the \Seen flag once[*]
e) from now on the message has the \Seen and NonJunk flags in each reading


[*]just once (with uid 359) in my variuos attempts mozilla wrote the \Seen flag
twice and in the same thread (the first number in the log is the thread id, right?).

Anyway, it seems that the read->unread->read dance is internal to mozilla and
has nothing to do with the exchange with the server.

I'm not attaching the log because it's huge (15megabytes the whole log,  just
2megs the part with the affected mailbox), but I can upload the filtered one if
someone thinks my analisys is flawed and want to check the data.

BTW, as well as getting the flags instead of using status, preloading the
messages (something I wasn't aware of) will slow things down considerably over a 
slow link. Is this new starting from 1.3 or was it there all the time?


> I would not turn on the "check other folders for new mail feature" over a slow
> link. It was not designed to work over a slow link.

It would be even worse to do client side filtering. Since I do server side
filtering I need that option. Using "status" would make that almost immediate
("status" result is one line, regardless the number of messages). I don't care
that status returns messages deleted but not expunged: that's the correct thing
to do (unless you're using the trash but you shouldn't anyway with imap).

> One guess for why you're losing the read state on messages is that your imap
> server  is dropping the connection and losing mailbox state.

Nope, see comment 14.

If you have client-side junk mail filters enabled, we will "pre-load" imap
messages when you open a folder, or check for new mail runs. That's new in 1.3 -
you probably want to turn it off. I'm a little surprised it's on by default, but
if you didn't turn it on yourself, I think it must be on by default - Tools |
Junk Mail Controls will allow you to turn it off on a per-account basis. Yes,
you're right, you don't want to do client-side junk mail filters over a slow
link. Doing Status would result in a lot less network traffic, no question. I
just don't have the time to work on that.

Re your log - the first number in the log is analagous to a thread id, yes. It's
actually the connection id. Re whether this is just an internal mozilla dance,
if you shut down and restart, are the messages read or unread? If they're read,
then yes, it's probably internal to mozilla.  Now, is it just some but not all
of the messages that you read in a given session that lose their read state?

The other test you could try once you see this happen, is to toggle the offline
state, and then reselect your inbox, and see if the messages are read or unread
(that test is similar to shutting down and restarting in that it causes us to
close our connections and re-open them)

Re the log, is it the case that we're only making one connection to the INBOX?
I.e., there's only one Select "INBOX"?
> Re whether this is just an internal mozilla dance,
> if you shut down and restart, are the messages read or unread? If they're read,
> then yes, it's probably internal to mozilla.  Now, is it just some but not all
> of the messages that you read in a given session that lose their read state?

This is interesting: I closed mozilla as soon as saw the dance (normally I just
waited until the message was read again) and effectively upon reopening mozilla
the message was unread. Guess what? Mozilla *never* sent a "uid store xxx +Flags
(\Seen)" for that message

> The other test you could try once you see this happen, is to toggle the offline
> state, and then reselect your inbox, and see if the messages are read or unread
> (that test is similar to shutting down and restarting in that it causes us to
> close our connections and re-open them)

Didn't try this but I don't think it's necessary.

> Re the log, is it the case that we're only making one connection to the INBOX?
> I.e., there's only one Select "INBOX"?

Yes there's only one of them but I'm seeing the problem on other folders and
cyrus-imapd deals correctly with various concurrent connections to the same
folder (even to INBOX).

Anyway, I'll try to deactivate junk mail filtering (I swear it was on by
default) and see if it changes anything.
BTW, each time I press ok in the junk mail settings, no matter if it's on or
off, mozilla will try to create a toplevel "Junk" folder and that's a no-no with
cyrus in its default configuration (all personal folders should be under INBOX,
top level is for shared folders). Mozilla should respect the namespaces provided
by the server (yes, I've the corresponding option set correctly).

Probably it's to soon to jump to conclusions, but with junk mail filtering
disabled I cannot reproduce the problem. Only, since the message hasn't been
preloaded, when I trigger the "Get new messages", reading the message will take
forever (probably all threads are busy reading the flags for all mailboxes), but
no flickering of the read state.
Do you think it's possible that, when the messages are preloaded, mozilla shows
them as read since it retrieves them from the cache, but defers storing of the
\Seen flag until later when there is an available thread and, from time to time,
it forgets to store the flag?
Bear in mind that we don't have to store the /SEEN flag explictly if we fetch
the message normally (i.e., w/o using PEEK). However, we could be getting
confused about whether we need to store the /SEEN flag explicitly, or we could
be trying to store it, but it could be hung up in the queueing mechanism we have
to prevent multiple connections to the folder. In other words, we could a
pending url that will mark the message /SEEN, but we haven't got to run it yet
because sometimes the queueing mechanism stalls (though I think it got a lot
better in 1.3 final and 1.4). If you shut down, then we wouldn't get to run it.
But I think it would explain the problem, if I'm understanding the scenario
correctly.

E.g., 
1. Read a message that's already cached
2. We try to store the seen flag, but for some reason, the url gets queued.
3. Then, check for new mail runs, which means we sync our front end flags with
the imap connection flags, which don't think the message is read.
4. Eventually, the url from step 2 unstalls and marks the message read, and it
shows up as read.

Now, the way the url queue works is that when we finish a url, we run the next
url in the queue. When that url finishes, we run the next url, etc.  Because of
our multi-threaded implementation, there might be a race condition where you can
add a url to the queue because we think the folder is busy but after we check if
there are any more urls to run in the queue. But I was not able to reproduce this.

However, if this were such a general problem, I'd expect to see it on my system.
I have the junk mail filtering enabled, which means most messages get read from
the cache, and we have to explicitly store the /SEEN flag. I also have check for
new mail turned on for some other folders (but not all). Your usage pattern
might be contributing to the problem.
> Bear in mind that we don't have to store the /SEEN flag explictly if we fetch
> the message normally (i.e., w/o using PEEK). 

Yes, but when you preload the message you use PEEK, so when the message is
actually read you have to explicitly store the \Seen flag, so the sequence
you're describing is possible. In fact, I have enough patience, all messages
eventually show up as read, it's just the flickering that's annoying (and
misleading). That's why I said that this bug and bug 199433 aren't really
duplicates :-)
Yes, in general, if we're pre-fetching messages, then we'll find them in the
memory cache and won't have to fetch them from the server. But, sometimes
messages get evicted from the memory cache and we do have to fetch them from the
server. Looking at bug 199433 again, it looks exactly like this bug, unless I'm
misreading one of them.
Well, this bug has been originally reported for 1.0 and 1.0 didn't do message
preload (at least you told me that it was new in 1.3). In fact I had no such
problem until 1.3.
So they couldn't possibly be the same bug.
Even the symptoms are different:

this bug:    read->unread
bug 199433:  read->unread->read


I'm confused. I thought you said that eventually, the messages show up as read.
Didn't you say this?

"In fact, I have enough patience, all messages eventually show up as read"

meaning this bug is also read->unread->read or more accurately, both bugs are

unread->read->unread->read. That would be consistent with the theory that we're
trying to mark the messages read but the url queue is getting stalled. Things
like "check for new mail" will unstall the queue, so if you have the "check for
new mail" interval set to a small number, like one minute, the stall can be
relatively short.

> I'm confused. I thought you said that eventually, the messages show up as read.
> Didn't you say this?
> 
> "In fact, I have enough patience, all messages eventually show up as read"

Yes I did. This is the bug I reported as 199433

> meaning this bug is also read->unread->read or more accurately, both bugs are

No, this bug, bug 195787, is about read->unread and staying this way

> unread->read->unread->read. That would be consistent with the theory that we're

well, I omitted the initial "unread" since that's the status of the message as
it enters the mailbox.

> trying to mark the messages read but the url queue is getting stalled. Things
> like "check for new mail" will unstall the queue, so if you have the "check for
> new mail" interval set to a small number, like one minute, the stall can be
> relatively short.

Actually a "check for new mail" exhacerbates the problem (this is the way I
reproduced it) since (I think) it would tie threads for a longer time, so there
would be less opportunities to mark the message (preloaded) as seen.
This is what I think is happening:

1) the message is preloaded in the cache
2) the user selects the message for display, the message won't be fetched again,
and mozilla displays it as read but...
3) at the same time there's a check for new mail running, all threads are busy
so 2) cannot mark the message as read in the server immediately, the order to
mark it as read is queued
4) the check for new mail fetches the flags from the server. Since the update
from 2) is still pending, it will see the message as unread and display it that way
5) eventually 3) terminates so 2) can update the server. The message displays
again as new.

The time between 2) and 5) can be very short (the flicker will be barely
noticeable) or can be various seconds.
Of course I know nothing about mozilla internals so I could be totally wrong.
At home I never experienced this problem, it turns out I had junk mail disabled.


When you say check for new mail, you mean "check all folders for new mail",
which, I agree, causes all sorts of problems. I meant the standard sense of just
checking the inbox for new messages, which tends to unstall the queue.
can you try a trunk build from the past couple days, just to make sure this is
still happening? There's a slim chance that this is fixed by a fix to the
multiple connection protection code I made recently.
David, do you have Mozilla configured to check all or some folders for new mail
(other than the inbox)?
> can you try a trunk build from the past couple days, just to make sure this is
> still happening?

Seeing that after this message you reopened bug 199433 comment 3, do you think I
should really try?
Anyway, since I switched off junk mail controls (see comment 18) everything is ok.

Luca, no, I think bug 199433 is well-understood and not fixed. I meant anyone
experiencing this bug where messages go back to unread and stay that way might
try a new build.
Product: MailNews → Core
(In reply to comment #29)
> Luca, no, I think bug 199433 is well-understood and not fixed. I meant anyone
> experiencing this bug where messages go back to unread and stay that way might
> try a new build.

Everyone here seems to be gone, so => incomplete. But If you still see a problem and it's not bug 19433, please comment.
Status: NEW → RESOLVED
Closed: 22 years ago → 16 years ago
Resolution: --- → INCOMPLETE
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.