532059 - Random old IMAP messages are being marked as unread (IMAP server supports CONDSTORE)

Status: RESOLVED DUPLICATE of bug 517461

(MailNews Core :: Networking: IMAP, defect)

Description

[Sander Marechal]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.4pre) Gecko/20090915 Thunderbird/3.0b4

Thunderbird 3b4 is marking (what appears to be) random old messages as unread. It doesn't happen very often but it is persistent when it does happen.

I have a Dovecot 1.2.6 IMAP server running on Debian Squeeze. My account has some 40 folders in it. Occasionally when I browse to a certain folder in Thunderbird there will suddenly be one or more old messages that are unread. I can mark them as read but as soon as I click "Get Mail" or as soon as Thunderbird checks for new messages it will become unread again.

The only way to fix this is to restart Thunderbird.

When I mark the message as read, the Thunderbird Activity Manager shows nothing. An entry briefly pops up but disappears again in a flash. Too fast to read.

I have tried capturing the TCP traffic between Thunderbird and my IMAP server but Wireshark doesn't like my SSL key, so I cannot decode it. I have now switched Thunderbird and Dovecot to use plain IMAP without security. When the issue pops up again I will capture the TCP traffic and post the results here.

Reproducible: Sometimes

Steps to Reproduce:
1. Click on a folder. Unread messages appear. These are old messages.
2. Mark the message as read.
3. Get Mail.
Actual Results:  
The message becomes unread again.

Expected Results:  
Old messages should stay read.

Comment 1

[Sven van Heel]

I have exactly the same problem as described here.

The same problem, with identical behavior occurs on two different clients:
Thunderbird 3.0b4 on Fedora 12 (latest package: thunderbird-3.0-3.9.b4.fc12.i686)
Thunderbird 3.0rc1 on Windows 7

The server is a Fedora 12 server, most recent Dovecot Fedora package:
dovecot-1.2.6-4.fc12.i686

I'd be happy to post some log-files or wireshark dumps if needed.
The problem occurs quite regularly.

Comment 2

[Sander Marechal]

> The problem occurs quite regularly.

Yes, but not all the time. I think I get it about once every 1-2 days. I have the impression that it starts to happen after Thunderbird has been running for some time. I have never seen it happen when I start up Thunderbird and check my mail. But it often happens when I have left TB running all day or all night.

Comment 3

[Sander Marechal]

I have some more information. I think that Thunderbird is confusing UIDs.

I have a folder for the debian-devel-announce list. According to my server there are three unread messages in there. They are UID 183, 184 and 185 according to Dovecot. Here's the TCP dump from when I select the folder:

3562 select "INBOX/zz_lists/debian-devel-announce" (CONDSTORE)

* OK [CLOSED] Previous mailbox closed.

* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)

* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.

* 186 EXISTS

* 0 RECENT

* OK [UNSEEN 183] First unseen.

* OK [UIDVALIDITY 1256125433] UIDs valid

* OK [UIDNEXT 187] Predicted next UID

* OK [HIGHESTMODSEQ 8] Highest

3562 OK [READ-WRITE] Select completed.

3563 UID fetch 187:* (FLAGS)

* 186 FETCH (UID 186 FLAGS (\Seen))

3563 OK Fetch completed.

As you can see, Dovecot says that the first unseen message is UID 183. Thunderbird shows 3 unread messages in that folder, but it has entirely the wrong UIDs for them. I marked all three messages as read. Here's the TCP dump:

4339 uid store 169 +Flags (\Seen)

4339 OK Store completed.

4340 IDLE

+ idling

DONE

4340 OK Idle completed.

4341 uid store 120 +Flags (\Seen)

4341 OK Store completed.

4342 IDLE

+ idling

DONE

4342 OK Idle completed.

4343 uid store 32 +Flags (\Seen)

4343 OK Store completed.

4344 IDLE

+ idling

So, the three messages that Thunderbird thought were unread were UID 169, 120 and 32. The wrong UID's.

I have checked the Dovecot UID map. Thunderbird does pick the right UID for each message. That is, when I select the message in the folder pane and click "Mark as read", the UID that Thunderbird sends (in this case 169, 120 or 32) is the correct UID. When I manually map that back on the server I see the right file with the right e-mail message.

Where it goes wrong is when Thunderbird decides to display the folder pane with the list of all messages and tries to figure out which messages are read and unread.

Comment 4

[Ludovic Hirlimann [:Usul]]

(In reply to comment #3)

> So, the three messages that Thunderbird thought were unread were UID 169, 120
> and 32. The wrong UID's.
> 
> I have checked the Dovecot UID map. Thunderbird does pick the right UID for
> each message. That is, when I select the message in the folder pane and click
> "Mark as read", the UID that Thunderbird sends (in this case 169, 120 or 32) is
> the correct UID. When I manually map that back on the server I see the right
> file with the right e-mail message.
> 
> Where it goes wrong is when Thunderbird decides to display the folder pane with
> the list of all messages and tries to figure out which messages are read and
> unread.

Is that with 3.0b4 ? if so we fixed a few Imap issues in RC2 - can you check that the issue is still here. If still present you can use our intenal tool to generate an imap log - documentation is at https://wiki.mozilla.org/MailNews:Logging.

Comment 5

[Sander Marechal]

Yes, this is with 3.0b4. I'm downloading 3.0rc2 now and I will try to reproduce it with that.

Comment 6

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #3)
> 3562 select "INBOX/zz_lists/debian-devel-announce" (CONDSTORE)

Same problem as bug 530551?

> * 186 EXISTS
> * OK [UNSEEN 183] First unseen.
> * OK [UIDNEXT 187] Predicted next UID
> * OK [HIGHESTMODSEQ 8] Highest

186 mails and UIDNEXT=187 => UID 1 to 186, no expunged mails, and HIGHESTMODSEQ=8. Very clean and small folder.
Can you check minimum IMAP log for next simple test?
0. Prepare 2 Tb profiles with single IMAP account only.
   Disable Gloda. Disable auto-sync, offline use. Disable IDLE command use.
   Unsubscribe non-needed folders.
   Show "Order Received" column(UID if IMAP).
1. Start Tb-1 with -no-remote using prof-1, with NSPR logging(add timestamp)
   NSPR_LOG_MODULES=timestamp,imap:5  (see bug 402793 comment #6) 
2. Start Tb-2 with -no-remote using prof-2(or Sm), with NSPR logging
   NSPR_LOG_MODULES=timestamp,imap:5
3. At Tb-1, repeat next several or many times.
   - Change read/unread status of some mails.
   - Add a few mails.
   - folder click and other empty folder click (folder open/close)
4. At Tb-2, repeat folder click and other empty folder click, and watch status.

Comment 7

[WADA:World Anti-bad-Duping Agency]

Adding CONDSTORE to bug summary for ease of search. Remove it if irrelevant to problem, please.

Comment 8

[Sander Marechal]

I can confirm that the bug still exists with TB 3.0rc2. Unfortunatly I do not have a minimal test case yet. The problem is that it only happens sometimes on some folders. It does not happen all the time on all folders. So, a minimum test case with only one folder subscription seems to only have a very slight chance of the bug appearing while my normal profile (with 50 or so folders) suffers pretty much daily from the bug.

Comment 9

[WADA:World Anti-bad-Duping Agency]

Re-open please, if not DUP.