358635 - [Regression] last IMAP message not shown

Status: VERIFIED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Udo Richter]

I've found an issue with IMAP folders, where sometimes at least one mail is missing from the IMAP folder view while the mail is present on the server. The regression window is between 2006102704 and 2006102803 on 1.8 branch. The only mailnews related checkins in this time frame are bug 353841 and bug 350179, where only bug 350179 is directly related to IMAP.

Symptoms:
One mail, often the one that was added last, is missing from the mail folder, and the total/unread count is off by the same amount. The missing mail appears/disappears when switching between 2704 and 2803 builds. In case that the missing mail is a new mail that arrived on server, the total/unread counter decreases by one on folder opening. Re-building the index does not fix this.

I've verified this with a clean profile. The server is Courier IMAP 3.0.8-4sarge5 of Debian. I have full access to the server and can do some sniffing if needed.

Confirmed by forum report:
http://forums.mozillazine.org/viewtopic.php?t=477182&start=32

Comment 1

[David :Bienvenu]

an imap protocol log would be very helpful here.

Comment 2

[Udo Richter]

Attachment: problematic imap folder and tcpdump before/after regression

I've isolated a hidden mail (in fact, its the bugzilla mail notification for your answer) into an otherwise empty folder, and the bug is still triggered. The attachment contains this folder in Maildir format, plus two tcpdump files of the first access to that folder after starting thunderbird. (password removed.) 

One tcpdump is from build 2006102303, where everything was fine. The mail count increased from 0 to 1 here. The other is from 2006102902. Here the mail was very briefly shown in the mail list, before the folder update decreased the mail count from 1 to 0 again.

The important part of the 2303 dump:
> 7 UID fetch 1:* (FLAGS)
> 
> * 1 FETCH (UID 1 FLAGS (\Seen NonJunk))
> 7 OK FETCH completed.
> 
> 8 UID fetch 1 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type)])
> 
> * 1 FETCH (UID 1 RFC822.SIZE 2136 FLAGS (\Seen NonJunk) BODY[HEADER.FIELDS ("From" "To" "Cc" "Subject" "Date" "Message-ID" "Priority" "X-Priority" "References" "Newsgroups" "In-Reply-To" "Content-Type")] {288}
> Date: Sun, 29 Oct 2006 17:28:43 -0800
> Message-Id: <[e-mail]>
> Subject: [Bug 358635] [Regression] Not all mails shown on IMAP
> In-Reply-To: <[e-mail]/>
> To: [e-mail]
> From: [e-mail]
> 
> )
> 8 OK FETCH completed.

The 2909 is the same, just without the 8 UID fetch 1 command.

Comment 3

[Udo Richter]

Attachment: tcpdump of folder access

This one is even more interesting. I've filled the folder with three copies of the mail, and made sure tb knows nothing about this folder by re-subscribing and going offline. The dump captures the next access to the folder. 

> 5 UID fetch 1:* (FLAGS)
> 
> * 1 FETCH (UID 1 FLAGS (\Seen NonJunk))
> * 2 FETCH (UID 19 FLAGS (\Seen NonJunk))
> * 3 FETCH (UID 20 FLAGS (\Seen NonJunk))
> 5 OK FETCH completed.
> 
> 6 UID fetch 1:19 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type)])
> 

Although 1, 19 and 20 were announced, only 1-19 get fetched.

Comment 4

[Ian Neal]

This seems to be very visible of Sent Folders on IMAP.

Comment 5

[David :Bienvenu]

I'll fix this asap.

Comment 6

[David :Bienvenu]

Attachment: proposed fix

d'oh, off by one in figuring out how many flag+uid pairs we've seen

Comment 7

[David :Bienvenu]

fixed on trunk and branch, summary changed

Comment 8

[Stephen Donner [:stephend] Not actively reading bugmail]

This is Verified FIXED using build 2006-10-30-18 of SeaMonkey trunk under Windows XP with:

hemlock.uits.indiana.edu Cyrus IMAP4 Murder v2.2.10 server
CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ MAILBOX-REFERRALS NAMESPACE UIDPLUS ID NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE IDLE STARTTLS AUTH=GSSAPI SASL-IR

Comment 9

[David :Bienvenu]

*** Bug 358748 has been marked as a duplicate of this bug. ***

Comment 10

[David :Bienvenu]

*** Bug 358751 has been marked as a duplicate of this bug. ***

Comment 11

[David :Bienvenu]

*** Bug 359041 has been marked as a duplicate of this bug. ***