1123617 - Because Biff requests "uid fetch NextUID:* Flags" for new mail check even when CONDSTORE is used, flag state of non-HighestUID can not be obtained by "uid fetch 1:* flags (CHANGEDSINCE highestMODSEQ)" which is used by Mbox open

Status: NEW

(MailNews Core :: Networking: IMAP, defect)

Description

[WADA:World Anti-bad-Duping Agency]

+++ This bug was initially created as a clone of Bug #1123094 +++

Because Biff requests "uid fetch NextUID:* Flags"(without CHANGEDSINCE, even when CONDSTORE is used by Tb) for new mail check, if both "flag status change on uid of Non-HighestUID" and "flag status change on uid of HighestUID2 occurs, Thunderbird can not get "flag status change on uid of Non-HighestUID" when "uid fetch 1:* flags (CHANGEDSINCE LargestMODSEQ_Tb_Knoiws)" is issued upon next Mbox open.

[Steps to reproduce]
(0) Define two Gmail imap accounts of same userID in Tb
      with host=imap.googlemail.com and host=ima.gmail.com.
         account1 = host=imap.googlemail.com  : no IDLE comand use, no new mail check
         account2 = host=imap.gmail.com           : IDLE comand is used. new mail check intervaal = Every 1 minute.
      max cached connections=1 for ease of testing.
(1) account2/Inbox  : Repair Folder => 4 mails are fetched
                                    Call UID=AA, UID=B, UID=C, UID=D (A<B<C<D)
(2) account1/Inbox  : Change flag of mails (add tag, remove tag, change Read/Unread state etc.
                                    Call MODSEQ-1-1 to MODSEQ-1-X
(3) new mail check at account2 on Inbox : uid fetch (D+1):* flags
      => flag stats of UID=D only is returned from server
(4) Repeat (2)/(3) several times.
(5) Repeat next :
         account1/Inbox : Change flag status of UID=D only
         account2/Inbox : new mail check => uid fetch (D+1):* flags
                                      => MODSEQ which Tb knows increases each time.
                                      Call highest one "MODSEQnn"
(6) account2 : Click other folder, hen click Inbox again at folder pane (open Inbox)
                        => Select Inbox (CONDSTORE)
                              uid fetch 1:* Flags (CHANGEDSiNCE  MODSEQnn)
     Because MODSEQ of any flag state change on UID=A/B/C < MODSEQnn for UID=D,
     flag state of UID=A/B/C is not retured.
     If "Read/Unread status change" is involved, mismatch of Read/Unread count can occur.
     Bug 517461 was perhaps for this phenomenon.

What is bad = "uid fetch NextUID:* flags" for new mail check by Biff.
If CONDSTORE is used, it should be "uid fetch 1:* flags (CHANGEDSINCE known_highest_MODSEQ)".
And, if "uid fetch 1:* flags (CHANGEDSINCE known_highest_MODSEQ)" is used by new mail check by Biff, this cn be used as solution of problem of Bug 693204.

Comment 1

[WADA:World Anti-bad-Duping Agency]

Attachment: imap log

Comment 2

[WADA:World Anti-bad-Duping Agency]

Attachment: Summay of imap log

4 mails in Inbox
 * 1 FETCH (UID 788 MODSEQ (185185) FLAGS ($label1 NonJunk \Seen)) 	
 * 2 FETCH (UID 789 MODSEQ (185183) FLAGS ($label1 NonJunk)) 	
 * 3 FETCH (UID 790 MODSEQ (185179) FLAGS ($label1 NonJunk)) 	
 * 4 FETCH (UID 791 MODSEQ (185181) FLAGS ($label1 NonJunk \Seen)) 

No new mail is added, so new mail check at imap.gmail.com is always :
   UID fetch 792:* (FLAGS) 	
         * 4 FETCH (UID 791 MODSEQ (185???) FLAGS ( ... ... ... )) 
Because flag of UID=791 is always changed at imp.googlemail.com,
modseq for UID=791 is incresed upon each new mail check at imap.gmail.com.

Last new mail check at imap.gmail.com.
  580 UID fetch 792:* (FLAGS) 	
  * 4 FETCH (UID 791 MODSEQ (185270) FLAGS ($label1 NonJunk)) 	
  580 OK Success
So, after Select Inbox for re-open of Inbox at imap.gmail.com, following is issued, 
then flag status change of UID=788,789,790 is not notified to Tb by server.
  585 UID fetch 1:* (FLAGS) (CHANGEDSINCE 185270) 	
  585 OK Success

Comment 3

[WADA:World Anti-bad-Duping Agency]

Possible solution.
(a) Save [HIGHESTMODSEQ MODSEQ_Upon_Previous_Select] upon previous "SELECT Mbox".
      After next Select Mbox,
      issue "uid fetch 1;* Flags (CHANGEDSINCE  MODSEQ_Upon_Previous_Select)"
      instead of  "uid fetch 1;* Flags (CHANGEDSINCE  MODSEQ_Upon_Last_NewMailCheck)"
(b) When new mail check at cached connection where Mbox is already selected ,
      issue "uid fetch 1;* Flags (CHANGEDSINCE  LargestMODSEQ_Which_Tb_Knows)".

From perspective of "Resynch", (a) is perhaps better.
But, as a solution of proble like Bug 693204, (b) is better, and it's reasonable CONDSTORE use.
Further, because followig is needed for Bug 1123094 unless QRESYNC is available,
      "uid fetch 1:* Flags without  CHANGEDSINCE"
(b) is needed to resolve problem like Bug 693204 by utilzing CONDSTORE.

Comment 4

[WADA:World Anti-bad-Duping Agency]

Code to determine "1:*" or not, parameter such as Flags, CHANGEDSINCE or not, looks  following.
http://mxr.mozilla.org/comm-central/source/mailnews/imap/src/nsImapProtocol.cpp#4039

Comment 5

[gene smith]

CONDSTORE needs to be revisited since it seems like a useful feature, especially for users having huge mailboxes, but has been disabled by default since it caused other problems (e.g., detection of new mail via IDLE).