874815 - IMAP messages shown as read but they're not

Status: RESOLVED INCOMPLETE

(Thunderbird :: Untriaged, defect)

Description

[Anton Gavrilov]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:16.0) Gecko/20100101 Firefox/16.0 (Beta/Release)
Build ID: 20121010223843

Steps to reproduce:

I use Thuderbird to read corporate mail on an IMAP server (dovecot, over SSL).  I always set up Thunderbird not to mark my messages as read as soon as I set it up.
However, if I highlight a message, switch away, do some work and come back, every now and then I discover that the message is marked as read.  But this is an entirely client-side thing: if I restart Thunderbird or check mail from another machine, I find the message is still "unread", as it should be.

This bug that has been around for years and I've witnessed it on all sorts of OSes and hardware.  It just happened an hour ago, in Thunderbird 17.0.2 on Ubuntu 12.10 x64.

What I can't tell you is the exact steps to reproduce it.  It never seems to happen when you're staring at the message.  You leave the mail client running, switch to another task, come back and find the message is "read".  You restart the program and see the message is in fact still "unread".

Comment 1

[WADA:World Anti-bad-Duping Agency]

(In reply to Anton Gavrilov from comment #0)
> However, if I highlight a message, switch away, do some work and come back,
> every now and then I discover that the message is marked as read.
> But this is an entirely client-side thing:
> if I restart Thunderbird or check mail from another machine,
> I find the message is still "unread", as it should be.

(Q1) You say "you never changed the mail from Unread to Read by Tb".
     Even so, why the mail is shown at Tb as Read?
(Q2) Do you share the IMAP Mbox with other IMAP clients?

(Q3) Following, isn't it?
0. A mail is Unread status(no \Seen flag)
   => Status in Tb == Unread
1. By other mail client, you changed the mail from Unread to Read
   => \Seen flag is stored 
   Tb knows about the \Seen flag
   => Status in Tb == Read
2. By other mail client, you changed the mail from Read to Unread
   => \Seen flag is removed 
3. Tb doesn't know about removal of \Seen flag yet
   => Status in Tb == still Read
4. By third mail client, check mail status
   => Because of no \Seen flag, status in third client == Unread
5. Restart Tb.
   Mbox is re-opened.
   => Because of no \Seen flag, status in Tb == Unread

If so, phenomenon of step 3 is perhaps same as bug 512745 and bug 693204.

(Q3) Problem on IMAP Ibox?
(Q4) If yes, do you see your problem with (a) max cached connections = 1, and (b) click(open) other IMAP Mbox then click(re-open) Inbox again.

Comment 2

[Anton Gavrilov]

No, this is different.  It definitely happens to me when I only have the mailbox open in one mail client.  No one else has access to this mailbox, and I never use anything but Thunderbird, except for experiment  (I end sticking with Thunderbird as it works the best for me, despite a few shortcomings).

I don't remember with 100% certainty but I don't think the mistakenly-marked-as-read messages get corrected after clicking another mailbox and re-opening the original one, or clicking "Get Mail".

But restarting Thunderbird has always helped.

The reason I notice these things (but no one else seems to) is the way I organize my work.  I receive several work related mails ever day, and each one contains a small task for me (usually involving updating the website).  I read the message, switch to the browser to do the job, return to the mailbox and mark the message as read.  So my inbox is my TODO list that manages itself without any extra effort from my part, except for pressing M when I'm done with a task.

As you can see it is very efficient -- except the bug hijacks my workflow.

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to Anton Gavrilov from comment #2)
> It definitely happens to me when I only have the mailbox open in one mail client.
> No one else has access to this mailbox, and I never use anything but Thunderbird, except for experiment

If so, who can change mail status from Unread(no \Seen flag) to Read(\Seen flag is stored) is;
 (a) Thunderbird 
 (b) IMAP server(and applications run at server).

What operation do you call by "if I highlight a message," in comment #0?
What option do you set at Tools/Options/Advanced/Reading&Display?
What prefs.js setting for it?
(prefs.js entries written in bug 449905 comment #2. Check via Tools/Options\Advanced/Geraral, Config Editor) 

> I don't remember with 100% certainty but I don't think the
> mistakenly-marked-as-read messages get corrected
> after clicking another mailbox and re-opening the original one, or clicking "Get Mail".
> But restarting Thunderbird has always helped.

If Tb or someone changed mail status of the mail from Unread to Read(store \Seen flag) after your operation of "I highlight a message",
and if other new mails arrived at the Mbox(perhaps Inbox in your case) after the mail,
and if "IDLE command use" is enabled(Account Settings/Server Settings/Advanced),
observed phenomenon is probably Bug 693204 and/or Bug 512745.
(1) The new mail : UID=(N), Highest UID=(N), Next-UID=(N+1)
(2) Tb store \Seen flag to mail of UID=(N),
    or \Seen flag is stored to mail of UID=(N) at server.
(3) Because Tb requests "uid (N+1):* fetch Flags" by new mail check,
    and because many IMAP servers usually return flag status of
    "currently highest UID" to the command, 
    Tb can know \Seen flag of UID=(N), even it's stored by IMAP server.
(4) New mail of UID=(N+1) arrives, and Tb knows it.
    The newest mail : UID=(N+1), Highest UID=(N+1), Next-UID=(N+2)
(5) At server side, \Seen flag of UID=(N) is removed.
(6) Because Tb requests "uid (N+2):* fetch Flags" at connection
    where Inbox is selected(select "Inbox" is done by Tb and IDLE is
    issued), Tb can't know UID=(N)'s status change of "\Seen is 
    removed", unless server notifies the status change via IDLE.

And, Bug 513309　is actual example of "status change, header change, by IMAP server".

Do you see phenomenon of "Unread status change at IMAP server on UID=(N) in above is not reflected" by following?
- Max cached connections=1 (Server Settings/Advanced), and restart Tb
- When you see problem, click(open) other Mbox, then click Inbox again.

Comment 4

[Anton Gavrilov]

You're correct in pointing out that the problem might lie not in Thunderbird, but rather in the IMAP server sending spurious "status changed to read" messages for an unknown reason.  But I have to point out that the problem has never happened to me with any messages other than the message currently selected.  This would suggest the IMAP server knows what the current message is.  Does it know that?


In Preferences/Advanced/Reading&Display I have "Automatically mark messages as read" unchecked and the two lines below it ("Immediately on display", "After displaying for 999 seconds") greyed out.
In Config Editor I see
mailnews.mark_message_read.auto = false
mailnews.mark_message_read.delay = true 
mailnews.mark_message_read.delay.interval = 999


Let me make another attempt to describe the steps to reproduce the bug, because you seem to keep disbelieving me and pointing me to unrelated bugs. So,
1) Start Thunderbird
2) In the left pane, click on the Inbox corresponding to the IMAP account
3) In the message list (top-right pane), click on any unread message.
4) The message text loads in the bottom-right pane.  The message remains unread (header shown in bold) in the message list, because my Thunderbird is configured not to mark messages as read automatically.
5) Minimize Thunderbird, open a browser and surf the web for an hour.
6) Switch back to Thunderbird and discover that the message appears read (!!!)
7) Close Thunderbird and open it again.  Click on Inbox.  The message is now unread again, as it should be.

Note that the above steps do NOT guarantee that the bug will reappear.  Sometimes it does, sometimes it does not.


I have set "max cached connections" to 1 in the account's settings as you suggest, and I will try to remember to "click(open) other Mbox, then click Inbox again" next time I see the problem.

Comment 5

[WADA:World Anti-bad-Duping Agency]

(In reply to Anton Gavrilov from comment #4)
> But I have to point out that the problem has never happened to me
> with any messages other than the message currently selected.

If the "currently selected message" in your usage is "newest message at the time", and if messages are changed to Read and changed back to Unread again by server, it's a phenomenon of Bug 693204 / Bug 512745.
  Newest message     : UID=(N)   Next-UID=(N+1)
  Non-newest message : UID=(N-1)
  Tb issues "uid (N+1):* fetch Flags",
  and server returns flags of UID=(N) if newer message doesn't exist,
  then Tb can know "Unread to Read of UID=(N)",
  but Tb can't know "Unread to Read of UID=(N-1)",
  because it's not notified via IDLE.

UID of mail is shown in "Order Received" column when IMAP folder.
What happpens when the "selected message" is mail of non highest UID?

Comment 6

[Anton Gavrilov]

Attachment: The problem

The highlighted message appears "read", but it's not, as can be verified by connecting to the same account with a different client.

Comment 7

[Anton Gavrilov]

Attachment: What it should have looked like

Screenshot taken shortly after I clicked on a different folder and switched back to Inbox.   The selected message appears unread now -- as it should be.

Comment 8

[Anton Gavrilov]

It happened again and I was better prepared this time.

As you can see in the screenshots, the message is not newly received: it's almost a week old.

Again what happened is I selected the message (the message was "unread"), then alt-tabbed to the browser and worked for about an hour.  I switch back to Thunderbird and see this: https://bugzilla.mozilla.org/attachment.cgi?id=755833

I had "Max cached connections=1"!

Then I log in to my account from my phone (Using Android 2.2.3's standard "Email" app) and I see that the message is still "unread" on the server.

I click on a subfolder on my Inbox, then click back on Inbox.  The message in question is still "read" (Disclaimer: I'm not 100% sure about this. Maybe 95% sure).

What I do next is click on a sub-subfolder in the Inbox.  I.e. I click on a + icon to the left of a subfolder, then click on one of its subfolders.  I return to Inbox, and this time the message is back to "unread"!   https://bugzilla.mozilla.org/attachment.cgi?id=755834

Comment 9

[Anton Gavrilov]

For what it's worth, I have a few other IMAP accounts on the same server, and I have them attached in my Thunderbird.  I did not change their "Max cached connections" setting.  I don't use them often;  I did not use any of them today.

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to Anton Gavrilov from comment #8)
> As you can see in the screenshots, the message is not newly received:
> it's almost a week old.

Because we are talking about phenomenon like Bug 693204 / Bug 512745, "newest" in that context == "highest UID".　It's never "newest timestamp value in Date: header".

Comment 11

[WADA:World Anti-bad-Duping Agency]

According to your check result with max cached connections=1, next phenomenon is perhaps actually relevant to bug which I pointed, which you call "I keep pointing you to unrelated bugs".
- Status of a mail checked by Android == Unread
  (no \Seen flag at this time)
- But status of the mail shown in Tb == Read(Tb thinks \Seen is stored)

Trick in max cached connections=1 is;
  Because only one cached connection is usable,
  "select other-than-Inbox" is issued at the only cached connection.
  So, when Inbox is clicked again, Tb has to issue "select Inbox" again,
  because "selected status of Inbox" is cleared by
  previous "select other-than-Inbox"(IMAP spec).
  So, Tb issues "uid 1:* fetch Flags" for inbox to re-synchronize,
  then flag status of all mails in Inbox is fetched again.
For performance reason, if max cached connections>1, one cached connections is reserved for Inbox, and Inbox is always selected at a cached connection, and Tb relies on "message status change notification from server" according to "SHOULD" in IMAP spec.
However, IMAP server doesn't always do it, according to "it's not MUST" in IMAP spec...

Another workaround is;
  Never use Inbox as repository of many mails to keep long time,
  use Inbox as letter box, mail drop, mail tray, ...,
  even if it's IMAP and it's not POP,
  with not so many cached connections.
  (If max cached connections>2, Tb reserves second connection for Trash)
  I think many ordinal peoples won't hold all mails in mail drop... :-)
 
There are several cases.
(i)   Tb stored \Seen flag, then other than Tb removed \Seen flag.
      Tb can't know "removal of \Seen flag".
(ii)  Tb changed mail's status in Tb to Read, 
      but Tb failed to store \Seen flag in mail held in server's Mbox.
      There is no chance to re-synchronize \Seen flag status of the mail
      with actual status in server's Mbox.
(iii) Other than Tb stored \Seen flag,
      then other than Tb removed the \Seen flag.
      Tb could know "\Seen flag is stored".
      But Tb couldn't know "removal of \Seen flag at server".
(iv)  Other

Comment 12

[WADA:World Anti-bad-Duping Agency]

FYI.
If Inbox is still opened internally when Inbox is clicked again, re-synchronization may not be invoked.
To force re-synchronize of Inbox with max cached connections=1, "wait for a while, until Inbox is closed internally", "force folder rediscovery" and so on may be required.

Comment 13

[Anton Gavrilov]

Okay.  Thank you for explaining the technicalities, it's all very interesting to know.

However this does nothing to help me with the original problem: a message spontaneously getting its Seen flag set.

The argument that I shouldn't be using my Inbox the way I use it doesn't convince me, to be honest.

Comment 14

[WADA:World Anti-bad-Duping Agency]

(In reply to Anton Gavrilov from comment #13)
> The argument that I shouldn't be using my Inbox the way I use it
> doesn't convince me, to be honest.

To be honest, (a) move all mails put in Inbox to In_box by server side filter, (b) See In_box instead of Inbox at all your PC and SmartPhone, (c) if message filter by Tb is needed, enable message filter for other than Inbox(hidden pref), is not so inconvenient, if problem of this bug is so critical for you.
Why "folder name of Inbox" is so important in using E-mail? It's merely a kind of defact-standard name of "mail drop" in E-mail system, isn't it? I believe any name of "letter box", "mail drop", "mail tray", ...,  can be used for it.

> However this does nothing to help me with the original problem:
> a message spontaneously getting its Seen flag set.

Here is bugzilla.mozilla.org. Here is not support forum. Because you claim "culprit is Tb" at bugzilla.mozilla.org, "who is responsible to show evidence of it" is you.
I suspect (iii) since initial even though I never neglect (iv), (i), (ii) and I never say "it's never (i) nor (ii)".
Please note that similar phenomenon to (iii) actually occured and occurs in some servers, and is already reported in other bug(s).
  Known phenomena.
    After new mail arrival and presenting it to IMAP client,
    server side app stores \Deleted flag, and perhaps adds or alters
    message headers(perhaps by virus scan or spam detection).
    After it, some servers simply removes \Deleted flag without
    UID change, and other servers add the mail as mail of newer UID.

Comment 15

[WADA:World Anti-bad-Duping Agency]

If you can see your problem frequently, get IMAP log, and check \Seen flag exchange between Tb and server.
(1) max cached connections = 1
Before select an Unread mail and keep Tb minimized for long time,
(2) Terminate Tb, enable IMAP loging with timetamp, restart Tb.
(3) Show "Order Received" column.
    Select Unread mail of non-Highest UID at Thread Pane.(call UID-X)
    Write down UID of mail of highest-UID(call UID-Y).
After it, 
(4) Minimize Tb for long time, as you did usually.
(5) If your problem occurs,
    click other than Inbox then click Ibox again,
    and confirm Unread status is set again.
(6) Terminate Tb, keep log file, check log file content by Text Editor,
    and search for \Seen flag exchange log for UID-X and UID-Y.
    If problem doesn't occur, discard log file.
See bug 402793 comment #28 for getting IMAP log.
> Win example : SET NSPR_LOG_MODULES=timestamp,imap:5

Comment 16

[WADA:World Anti-bad-Duping Agency]

Needless to say, because you are getting log of communication between Tb and IMAP server for Mboxes and mails, log file is huge if many mails are held in Mbox, and because IMAP logging writes log data for mail data, if download/upload of mail is logged, pretty huge log file is pretty easily produced.
To see overview of Tb<->IMAP sever communication by IMAP log, get minimum log first and check the minimum log.
(1) Disable any automatic IMAP server access
    (disable new mail check of all IMAP accounts).
    Create a folder for getting log(call FolderX),
    and copy a few pretty small text mails to FolderX.
    To force "open FolderX after restart of Tb",
    keep "FoldeX selected at Folder Pane" and terminate Tb,
    or to force "no folder is opened after resart of Tb",
    select the IMAP accont at Folder Pane,
    and terminate Tb.
(2) With IMAP logging enabled, start Tb.
(3) Click FolderX at Folder Pane, view a mail in FolderX,
    then termiate Tb.
(4) Keep backup of log file, and view log file content by Text Editor.

Comment 17

[Wayne Mery (:wsmwk)]

Do you still see this when using version 31?
If yes, protocol log will be greatly appreciated (comment 16).

Comment 18

[Phoenix]

Resolved per whiteboard