462880 - IMAP folder displays mesages already deleted from web Gmail (If all mail in a folder is deleted. Response of "*0 EXISTS")

Status: VERIFIED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[DP]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.3) Gecko/2008092414 Firefox/3.0.3
Build Identifier: 3.0b1 pre

Using the 11/3 nightly build.

Using Web Gmail, delete mail from the inbox.  Goto Shredder, the mail is still in the box.  If you try to delete again, you get the error: "The current command did not succeed.  The mail server responded: No messages match. (Failure).".  Same error if you hit send/receive to update the folder.  Once you delete the mail, no error messages.

Reproducible: Always

Steps to Reproduce:
1.
2.
3.

Comment 1

[Nikolay Shopik]

Not seeing problem using on Windows with this build.
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b2pre) Gecko/20081031 Shredder/3.0b1pre
Message deleted 1-2 seconds thereafter I delete from web.

Comment 4

[MaTias BarleTTa]

I can reproduce the same thing in the Linux installation.  Thunderbird 3b2 and nightly build from 23/4/2009.

Exactly the same way.

1. Receive mails with TB (inbox)
2. Go to webmail and move messages away from inbox.
3. Go to the inbox and message is still there. TB tries to get the message but never finish.

thanks :)

Comment 5

[Ludovic Hirlimann [:Usul]]

MaTias BarleTTa : could you reproduce while you are logging imap (as described at : https://wiki.mozilla.org/MailNews:Logging ) and attach the imap log to this bug ?

Comment 6

[MaTias BarleTTa]

Attachment: Thunderbird Imap.log

I did the process stated before. Same result... attaching the log .. hope it helps..

Comment 7

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #4)
> 2. Go to webmail and move messages away from inbox.

What is your "move messages away from inbox" operation?
"Delete" button of Gmail's Web Interface? Or "Move to" button? 

(In reply to comment #6)
> Thunderbird Imap.log

I assume "Delete" button at Gmail Web Interface.
What is UID of mail which you expected to disappear from Inbox folder of Tb?
(Show "Order Received" column. Value in it is UID of mail if IMAP folder)
Where is flow which corresponds to next in the log log file?
> 3. Go to the inbox and message is still there. 
>    TB tries to get the message but never finish.

Comment 8

[WADA:World Anti-bad-Duping Agency]

Attachment: IMAP log (Add Gmail Label then delete the Gmail label, via Gmail Web Interface)

(A) Gmail IMAP notifies about delete(remove of Gmail Label) via IDLE,
    but Tb doesn't care for it. (if notify of new mail, Tb issues fetch)
(B) Although Gmail IMAP notifies delete(remove of Gmail Label) of mail via IDLE,
    (B-1) Gmail IMAP returns OK to uid xxx fetch with no mail data.
    (B-2) Gmail IMAP returns OK to uid xxx copy Trash, uid xxx store \Deleted.
          And, nothing is copied to Trash.

Comment 9

[WADA:World Anti-bad-Duping Agency]

Attachment: IMAP log. Remove of Gmail Lable=>never deleted mails on TB => Add Gmail Label(==new mail arrives)

After (A) in Comment #8,
(C) Add a Gmail Label(Test-003/A-00 in this case) to a mail
    at Gmail Web Interface.
    => New mails is notified via ILDE.
       * 1 EXISTS
       DONE
   => Tb execute re-sync.
       31 UID fetch 1:* (FLAGS) 	
       * 1 FETCH (UID 8 FLAGS (\Seen)) 	
       31 OK Success 	
   => Tb knows "UID=5,6,7, doesn't exist".
   => 3 mails disappaer.

Comment 10

[WADA:World Anti-bad-Duping Agency]

I think problem of (A) in Comment #8 and (C) in Comment #9 can be said.
 (1) If delete notification via IDLE is "0 EXISTS",
     or if select(by initial folder open) response is "0 EXISTS",
     Tb doesn't issue "fetch N:* (flags)".
     Then "all mails are alredy deleted by other" situation is not reflected to
     MailDB and thread pane display.
It's not Gmail IMAP particular issue. It can occur on any IMAP server if mail is deleted&expunged by other IMAP client.

And I guess one of next.
  If "fetch N:* (flags)" issued but no response lines(already "no mail in
  folder" status at Gmail IMAP/Gmail),
  (2) Tb possibly waits forever even after OK response.
  (3) Gmail IMAP doesn't return OK, then Tb waits forever.
I guess (3), because I saw "never ending folder open" several times and I saw error message of "System Failure" upon folder open after "remove of Gmail Lable at Gmail Web Interface".
I think (B) in comment #8 and above is still remainig Gmail IMAP issue of next, which I pointed in Bug 402793 Comment #12.
> http://mail.google.com/support/bin/answer.py?answer=96926&topic=12922
> Updates not reflected in multiple clients

Confirming my test result of (A) in comment #8 and (C) in comment #9.

Comment 11

[WADA:World Anti-bad-Duping Agency]

Attachment: Imap log(Gmail IMAP). If non-deleted mail exists in the folder("N EXISTS" where N>0 in IDLE/SELECT response), "deleted-by-other" mail disappers.

Comment 12

[WADA:World Anti-bad-Duping Agency]

(In addition to comment #11)

Following is log for "Removal of Gmail Label from mail-1(UID=1)" when UID=1/2/3 exist in folder.
In this case, mail of UID=1(mail-1) was removed from thread pane.
How Tb determined that UID=1 was deleted from Gmail IMAP response of "* 2 FETCH (UID 3 FLAGS (\Seen))" only?
Gmail's response to "33 UID fetch 4:* (FLAGS)" is proper?

> [5] Remove Gmail Lable of Test-003/A-00 from mail-1(UID=1), notifiled via ILDE
>     => mail-1(UID=1) disappeared
> 00000294  * 1 EXPUNGE 	
> 00000296  * 2 EXISTS 	
> 00000297  DONE
> 00000299  30 OK IDLE terminated (Success) 	
> 00000312  33 UID fetch 4:* (FLAGS) 	
> 00000314  * 2 FETCH (UID 3 FLAGS (\Seen)) 	
> 00000316  33 OK Success

Comment 13

[WADA:World Anti-bad-Duping Agency]

(In addition to comment #11)

Number of n in "* n FETCH (UID 3 FLAGS (\Seen))" seems "Message Sequence Number". 
> http://tools.ietf.org/html/rfc3501#section-2.3.1.2
> 2.3.1.2. Message Sequence Number Message Attribute

Possibly following algorithm.
 (1) "* 2 FETCH (UID 3 FLAGS (\Seen))" to "33 UID fetch 4:* (FLAGS)" 
     => UID=3 == second mail(Message Sequence Number==2)
 (2) * 1 EXPUNGE / * 2 EXISTS
     => 1 mail is deleted/expunged (Tb knew 3 mails of UID=1/2/3 existed) 
 (3) By (1) && (2), Tb can know one of UID=1 & UID=2 is deleted/expunged.
But how can Tb know deleted/expunged mail is UID=1 instead of UID=2?

Comment 14

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #4)
> 3. Go to the inbox (snip) ... . TB tries to get the message but never finish.

It's independent/different problem from main issue of this bug(==never removed mail even though deleted/expunged at server). Phenomenon that looks "never ending folder open" is probably problem of bug 490556(see bug 490556 comment #5).
MaTias BarleTTa(poster of comment #4), watch that bug, please.

Comment 15

[pacshady]

Attachment: IMAP Log - Marked for deletion, followed by move to separate folder

I've also go this problem, using Shredder 3.0b3. Either deleting from Gmail web interface or separate client, or marking as deleted and expunging, emails still stay in the folder they were in. Deletion only updates when an email is moved into that folder (either by new incoming email or being moved in there). Emails delete properly in TB when set to move to Trash rather than simply marking, but if emails are deleted from another client or Gmail web interface this doesn't update. Only seems to happen with Gmail, not with my other IMAP account.

Comment 16

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #15)
> IMAP Log - Marked for deletion, followed by move to separate folder

As seen in log, Gmail IMAP automatically expunges mail flag as "\Deleted", and returns "* 0 EXISTS" if there is no active mail any more in the folder.
> S-INBOX:SendData: 20 uid store 7078 +Flags (\Seen \Deleted)
> S-INBOX:CreateNewLineFromSocket: * 1 FETCH (FLAGS (\Seen \Deleted) UID 7078)
> S-INBOX:CreateNewLineFromSocket: * 1 EXPUNGE
> S-INBOX:CreateNewLineFromSocket: * 0 EXISTS
> S-INBOX:CreateNewLineFromSocket: 20 OK Success

(Q1) Can you reproduce problem even when active mails exists in folder after "delete/expunge by other IMAP client or Gmail Web"?
(Q2) Can you reproduce problem even when "automatic expunge" is disabled by 
"Advanced IMAP Controls" in Gmail's "Labs"?

> Only seems to happen with Gmail, not with my other IMAP account.

(Q3) I think you can see phenomenon with your other ordinal IMAP accounts by;
     (a) Disable IDLE, in order to avoid notification of "* n EXPUNGE" etc.
     (b) Delete all mails & Expunge by other IMAP client or Web Interface.
     Is it wrong?
  1. Tb knows mails in a folder. Shutdown Tb.
  2. Delete all mails in folder & Expunge by other IMAP client or Web Interface.
  3. Restart Tb, and click the folder.
     => * 0 EXISTS, because no mails anymore in the folder.

Comment 17

[MaTias BarleTTa]

Shredder beta3pre is working correctly. I no longer have a problem with deleted messages in GMAIL not being removed from thunderbird.

thanks!!

Comment 18

[Ludovic Hirlimann [:Usul]]

(In reply to comment #17)
> Shredder beta3pre is working correctly. I no longer have a problem with deleted
> messages in GMAIL not being removed from thunderbird.
> 
Dypark @ gmail . com can you confirm that too ?

Comment 19

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #17)
> Shredder beta3pre is working correctly. I no longer have a problem with deleted
> messages in GMAIL not being removed from thunderbird.

MaTias BarleTTa, with which build?
Really no problem even when "* 0 EXISTS" response to Tb"(no mail exists in folder any more) after "delete of mails which Tb already knew & expunge by other client(or Web Interface of Web mail)"?

Comment 20

[WADA:World Anti-bad-Duping Agency]

Bug 486477 is opened for ordinal IMAP server. Setting dependency to it, because "* 0 EXISTS" itself is not Gmail particularity.

Comment 21

[WADA:World Anti-bad-Duping Agency]

I still could see problem with Tb trunk 2009/6/21 build.
  - Delete model: Just mark it as deleted
  - Shift+Delete of all mail in a folder => All are marked with red cross
  - These deleted & already expunged mails won't disappear
    until new mail is copied to the folder(*N EXISTS response where N>0).

Comment 23

[Matěj Cepl]

Attachment: log according to https://wiki.mozilla.org/MailNews:Logging

I can reproduce this very well with mail.messagingengine.com (the IMAP server of fastmail.fm service). The other email server in this log (mail.corp.redhat.com, which is a Zimbra server) works just fine.

Comment 24

[Matěj Cepl]

Forgot to add ... this log has been generated with the binary from ftp://ftp.mozilla.org/pub/thunderbird/nightly/3.0b4-candidates/build4/linux-i686/en-US/thunderbird-3.0b4.tar.bz2

Comment 25

[Matěj Cepl]

Bug 509902 has been closed as a duplicate of this bug, but I am really not sure whether this isn't (or shouldn't be) GMail IMAP specific. I guess their implementation of IMAP is unfortunately quite special to deserve it.

Comment 26

[Dan Mosedale (:dmosedale, :dmose)]

Does this problem also happen with Thunderbird 2?

Comment 27

[David :Bienvenu]

not blocking, wanted.

Comment 30

[kk.ss]

This problem does not happen in Thunderbird 2 as far as to the version before the official 3.0 one.

Comment 31

[David :Bienvenu]

Attachment: proposed fix

this will fix the problem. now to attempt a unit test.

Comment 32

[David :Bienvenu]

Attachment: fix with unit test

Comment 33

[David :Bienvenu]

forgot to mention, this cleans up a couple things in the test_flagChanges test as well, which I cloned this test from...Neil, if you don't want to review the test, you can leave that for standard8's sr.

Comment 34

[[e-mail]]

Comment on attachment 421702 [details] [diff] [review]
fix with unit test

I take it this only affects CONDSTORE servers, as otherwise the flag state is never partial?

Comment 35

[David :Bienvenu]

(In reply to comment #34)
> (From update of attachment 421702 [details] [diff] [review])
> I take it this only affects CONDSTORE servers, as otherwise the flag state is
> never partial?

No, sadly, both kinds of servers are affected. The issue is that ResetFlagInfo sets the flag state to partial, because that's a safer assumption in general, at least for condstore servers. But in this case, we're resetting the flag info because we know there are zero messages, but because we know there are zero messages, we need to clear the partial flag.

Comment 37

[Ján Rusnák]

also TB 3.0.1 suffers from this bug.
I can reproduce it on gmail and also dovecot imap server.
steps:
1. sent mail to an account.
2. mail appears in Inbox folder.
3. Close TB.
4. delete mail from other client (TB, web ui gmail).
5. start TB, and already deleted mail still seen in Inbox.
important settings TB settings:
account setting: synchronization of messages -> leave messages from account on this computer !OFF!
global setting: global search and indexing: !DISABLED!

this is only one serious bug which I find on new 3.0 TB release.

Comment 38

[David :Bienvenu]

(In reply to comment #37)
> also TB 3.0.1 suffers from this bug.
Yes, this bug has not been fixed yet. The patch is waiting for review. It will be fixed in nightly trunk builds soon, and in 3.02.

Comment 40

[David :Bienvenu]

ping on the sr - we've had several dupes of this...

Comment 41

[David :Bienvenu]

fixed on trunk and 3.0x branch

Comment 45

[Tom Guyette]

I'm seeing this symptom as of a couple weeks ago -- haven't seen *0 EXISTS (don't know where to look), but deleting email from my Android phone does not result in the e-mail disappearing from my Thunderbird inbox.  TB 24.1.0 Win7, IMAP to Gmail.  I haven't noticed whether it's always or sometimes.  Using Get Mail / Get All New Messages button does nothing.

Comment 46

[Wayne Mery (:wsmwk)]

(In reply to Tom Guyette from comment #45)
> I'm seeing this symptom as of a couple weeks ago -- haven't seen *0 EXISTS
> (don't know where to look), but deleting email from my Android phone does
> not result in the e-mail disappearing from my Thunderbird inbox.  TB 24.1.0
> Win7, IMAP to Gmail.  I haven't noticed whether it's always or sometimes. 
> Using Get Mail / Get All New Messages button does nothing.

Tom, If you can still reproduce in version 31 beta[1], please file a new bug if there is not a bug already covering this issue.

beta at http://www.mozilla.org/en-US/thunderbird/channel/