Last Comment Bug 462880 - IMAP folder displays mesages already deleted from web Gmail (If all mail in a folder is deleted. Response of "*0 EXISTS")
: IMAP folder displays mesages already deleted from web Gmail (If all mail in a...
Status: VERIFIED FIXED
[no l10n impact][has repro steps in c...
: fixed-seamonkey2.0.3, regression
Product: MailNews Core
Classification: Components
Component: Networking: IMAP (show other bugs)
: Trunk
: PowerPC All
: -- normal with 1 vote (vote)
: ---
Assigned To: David :Bienvenu
:
Mentors:
: 463090 464478 486477 528233 537676 537990 540891 541782 545874 (view as bug list)
Depends on: 500144
Blocks: tb-gmailWIP 465800 486477
  Show dependency treegraph
 
Reported: 2008-11-03 10:37 PST by DP
Modified: 2014-08-10 17:20 PDT (History)
23 users (show)
mozilla: blocking‑thunderbird3-
mozilla: wanted‑thunderbird3+
mozilla: in‑testsuite+
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
.2+
.2-fixed


Attachments
Thunderbird Imap.log (27.81 KB, text/plain)
2009-04-24 23:09 PDT, MaTias BarleTTa
no flags Details
IMAP log (Add Gmail Label then delete the Gmail label, via Gmail Web Interface) (22.24 KB, text/plain)
2009-04-25 03:39 PDT, WADA
no flags Details
IMAP log. Remove of Gmail Lable=>never deleted mails on TB => Add Gmail Label(==new mail arrives) (9.34 KB, text/plain)
2009-04-26 19:35 PDT, WADA
no flags Details
Imap log(Gmail IMAP). If non-deleted mail exists in the folder("N EXISTS" where N>0 in IDLE/SELECT response), "deleted-by-other" mail disappers. (64.55 KB, text/plain)
2009-04-27 01:35 PDT, WADA
no flags Details
IMAP Log - Marked for deletion, followed by move to separate folder (178.83 KB, text/plain)
2009-05-15 21:54 PDT, pacshady
no flags Details
log according to https://wiki.mozilla.org/MailNews:Logging (117.35 KB, text/plain)
2009-09-19 14:35 PDT, Matěj Cepl
no flags Details
proposed fix (876 bytes, patch)
2010-01-14 13:26 PST, David :Bienvenu
no flags Details | Diff | Splinter Review
fix with unit test (7.39 KB, patch)
2010-01-14 14:08 PST, David :Bienvenu
neil: review+
standard8: superreview+
Details | Diff | Splinter Review

Description DP 2008-11-03 10:37:01 PST
User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.3) Gecko/2008092414 Firefox/3.0.3
Build Identifier: 3.0b1 pre

Using the 11/3 nightly build.

Using Web Gmail, delete mail from the inbox.  Goto Shredder, the mail is still in the box.  If you try to delete again, you get the error: "The current command did not succeed.  The mail server responded: No messages match. (Failure).".  Same error if you hit send/receive to update the folder.  Once you delete the mail, no error messages.

Reproducible: Always

Steps to Reproduce:
1.
2.
3.
Comment 1 Nikolay Shopik 2008-11-03 11:35:16 PST
Not seeing problem using on Windows with this build.
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b2pre) Gecko/20081031 Shredder/3.0b1pre
Message deleted 1-2 seconds thereafter I delete from web.
Comment 2 Matthias Versen [:Matti] 2008-11-04 13:45:07 PST
*** Bug 463090 has been marked as a duplicate of this bug. ***
Comment 3 Nikolay Shopik 2008-11-13 00:29:48 PST
*** Bug 464478 has been marked as a duplicate of this bug. ***
Comment 4 MaTias BarleTTa 2009-04-24 01:59:01 PDT
I can reproduce the same thing in the Linux installation.  Thunderbird 3b2 and nightly build from 23/4/2009.

Exactly the same way.

1. Receive mails with TB (inbox)
2. Go to webmail and move messages away from inbox.
3. Go to the inbox and message is still there. TB tries to get the message but never finish.

thanks :)
Comment 5 Ludovic Hirlimann [:Usul] 2009-04-24 02:19:52 PDT
MaTias BarleTTa : could you reproduce while you are logging imap (as described at : https://wiki.mozilla.org/MailNews:Logging ) and attach the imap log to this bug ?
Comment 6 MaTias BarleTTa 2009-04-24 23:09:58 PDT
Created attachment 374571 [details]
Thunderbird Imap.log

I did the process stated before. Same result... attaching the log .. hope it helps..
Comment 7 WADA 2009-04-24 23:57:06 PDT
(In reply to comment #4)
> 2. Go to webmail and move messages away from inbox.

What is your "move messages away from inbox" operation?
"Delete" button of Gmail's Web Interface? Or "Move to" button? 

(In reply to comment #6)
> Thunderbird Imap.log

I assume "Delete" button at Gmail Web Interface.
What is UID of mail which you expected to disappear from Inbox folder of Tb?
(Show "Order Received" column. Value in it is UID of mail if IMAP folder)
Where is flow which corresponds to next in the log log file?
> 3. Go to the inbox and message is still there. 
>    TB tries to get the message but never finish.
Comment 8 WADA 2009-04-25 03:39:53 PDT
Created attachment 374586 [details]
IMAP log (Add Gmail Label then delete the Gmail label, via Gmail Web Interface)

(A) Gmail IMAP notifies about delete(remove of Gmail Label) via IDLE,
    but Tb doesn't care for it. (if notify of new mail, Tb issues fetch)
(B) Although Gmail IMAP notifies delete(remove of Gmail Label) of mail via IDLE,
    (B-1) Gmail IMAP returns OK to uid xxx fetch with no mail data.
    (B-2) Gmail IMAP returns OK to uid xxx copy Trash, uid xxx store \Deleted.
          And, nothing is copied to Trash.
Comment 9 WADA 2009-04-26 19:35:21 PDT
Created attachment 374717 [details]
IMAP log. Remove of Gmail Lable=>never deleted mails on TB => Add Gmail Label(==new mail arrives)

After (A) in Comment #8,
(C) Add a Gmail Label(Test-003/A-00 in this case) to a mail
    at Gmail Web Interface.
    => New mails is notified via ILDE.
       * 1 EXISTS
       DONE
   => Tb execute re-sync.
       31 UID fetch 1:* (FLAGS) 	
       * 1 FETCH (UID 8 FLAGS (\Seen)) 	
       31 OK Success 	
   => Tb knows "UID=5,6,7, doesn't exist".
   => 3 mails disappaer.
Comment 10 WADA 2009-04-26 21:04:31 PDT
I think problem of (A) in Comment #8 and (C) in Comment #9 can be said.
 (1) If delete notification via IDLE is "0 EXISTS",
     or if select(by initial folder open) response is "0 EXISTS",
     Tb doesn't issue "fetch N:* (flags)".
     Then "all mails are alredy deleted by other" situation is not reflected to
     MailDB and thread pane display.
It's not Gmail IMAP particular issue. It can occur on any IMAP server if mail is deleted&expunged by other IMAP client.

And I guess one of next.
  If "fetch N:* (flags)" issued but no response lines(already "no mail in
  folder" status at Gmail IMAP/Gmail),
  (2) Tb possibly waits forever even after OK response.
  (3) Gmail IMAP doesn't return OK, then Tb waits forever.
I guess (3), because I saw "never ending folder open" several times and I saw error message of "System Failure" upon folder open after "remove of Gmail Lable at Gmail Web Interface".
I think (B) in comment #8 and above is still remainig Gmail IMAP issue of next, which I pointed in Bug 402793 Comment #12.
> http://mail.google.com/support/bin/answer.py?answer=96926&topic=12922
> Updates not reflected in multiple clients

Confirming my test result of (A) in comment #8 and (C) in comment #9.
Comment 11 WADA 2009-04-27 01:35:21 PDT
Created attachment 374742 [details]
Imap log(Gmail IMAP). If non-deleted mail exists in the folder("N EXISTS" where N>0 in IDLE/SELECT response), "deleted-by-other" mail disappers.
Comment 12 WADA 2009-04-27 01:53:21 PDT
(In addition to comment #11)

Following is log for "Removal of Gmail Label from mail-1(UID=1)" when UID=1/2/3 exist in folder.
In this case, mail of UID=1(mail-1) was removed from thread pane.
How Tb determined that UID=1 was deleted from Gmail IMAP response of "* 2 FETCH (UID 3 FLAGS (\Seen))" only?
Gmail's response to "33 UID fetch 4:* (FLAGS)" is proper?

> [5] Remove Gmail Lable of Test-003/A-00 from mail-1(UID=1), notifiled via ILDE
>     => mail-1(UID=1) disappeared
> 00000294  * 1 EXPUNGE 	
> 00000296  * 2 EXISTS 	
> 00000297  DONE
> 00000299  30 OK IDLE terminated (Success) 	
> 00000312  33 UID fetch 4:* (FLAGS) 	
> 00000314  * 2 FETCH (UID 3 FLAGS (\Seen)) 	
> 00000316  33 OK Success
Comment 13 WADA 2009-04-28 02:39:49 PDT
(In addition to comment #11)

Number of n in "* n FETCH (UID 3 FLAGS (\Seen))" seems "Message Sequence Number". 
> http://tools.ietf.org/html/rfc3501#section-2.3.1.2
> 2.3.1.2. Message Sequence Number Message Attribute

Possibly following algorithm.
 (1) "* 2 FETCH (UID 3 FLAGS (\Seen))" to "33 UID fetch 4:* (FLAGS)" 
     => UID=3 == second mail(Message Sequence Number==2)
 (2) * 1 EXPUNGE / * 2 EXISTS
     => 1 mail is deleted/expunged (Tb knew 3 mails of UID=1/2/3 existed) 
 (3) By (1) && (2), Tb can know one of UID=1 & UID=2 is deleted/expunged.
But how can Tb know deleted/expunged mail is UID=1 instead of UID=2?
Comment 14 WADA 2009-04-30 22:56:07 PDT
(In reply to comment #4)
> 3. Go to the inbox (snip) ... . TB tries to get the message but never finish.

It's independent/different problem from main issue of this bug(==never removed mail even though deleted/expunged at server). Phenomenon that looks "never ending folder open" is probably problem of bug 490556(see bug 490556 comment #5).
MaTias BarleTTa(poster of comment #4), watch that bug, please.
Comment 15 pacshady 2009-05-15 21:54:37 PDT
Created attachment 377831 [details]
IMAP Log - Marked for deletion, followed by move to separate folder

I've also go this problem, using Shredder 3.0b3. Either deleting from Gmail web interface or separate client, or marking as deleted and expunging, emails still stay in the folder they were in. Deletion only updates when an email is moved into that folder (either by new incoming email or being moved in there). Emails delete properly in TB when set to move to Trash rather than simply marking, but if emails are deleted from another client or Gmail web interface this doesn't update. Only seems to happen with Gmail, not with my other IMAP account.
Comment 16 WADA 2009-05-15 22:59:07 PDT
(In reply to comment #15)
> IMAP Log - Marked for deletion, followed by move to separate folder

As seen in log, Gmail IMAP automatically expunges mail flag as "\Deleted", and returns "* 0 EXISTS" if there is no active mail any more in the folder.
> S-INBOX:SendData: 20 uid store 7078 +Flags (\Seen \Deleted)
> S-INBOX:CreateNewLineFromSocket: * 1 FETCH (FLAGS (\Seen \Deleted) UID 7078)
> S-INBOX:CreateNewLineFromSocket: * 1 EXPUNGE
> S-INBOX:CreateNewLineFromSocket: * 0 EXISTS
> S-INBOX:CreateNewLineFromSocket: 20 OK Success

(Q1) Can you reproduce problem even when active mails exists in folder after "delete/expunge by other IMAP client or Gmail Web"?
(Q2) Can you reproduce problem even when "automatic expunge" is disabled by 
"Advanced IMAP Controls" in Gmail's "Labs"?

> Only seems to happen with Gmail, not with my other IMAP account.

(Q3) I think you can see phenomenon with your other ordinal IMAP accounts by;
     (a) Disable IDLE, in order to avoid notification of "* n EXPUNGE" etc.
     (b) Delete all mails & Expunge by other IMAP client or Web Interface.
     Is it wrong?
  1. Tb knows mails in a folder. Shutdown Tb.
  2. Delete all mails in folder & Expunge by other IMAP client or Web Interface.
  3. Restart Tb, and click the folder.
     => * 0 EXISTS, because no mails anymore in the folder.
Comment 17 MaTias BarleTTa 2009-05-17 12:40:45 PDT
Shredder beta3pre is working correctly. I no longer have a problem with deleted messages in GMAIL not being removed from thunderbird.

thanks!!
Comment 18 Ludovic Hirlimann [:Usul] 2009-05-18 00:40:40 PDT
(In reply to comment #17)
> Shredder beta3pre is working correctly. I no longer have a problem with deleted
> messages in GMAIL not being removed from thunderbird.
> 
Dypark @ gmail . com can you confirm that too ?
Comment 19 WADA 2009-05-18 01:32:29 PDT
(In reply to comment #17)
> Shredder beta3pre is working correctly. I no longer have a problem with deleted
> messages in GMAIL not being removed from thunderbird.

MaTias BarleTTa, with which build?
Really no problem even when "* 0 EXISTS" response to Tb"(no mail exists in folder any more) after "delete of mails which Tb already knew & expunge by other client(or Web Interface of Web mail)"?
Comment 20 WADA 2009-06-06 23:20:16 PDT
Bug 486477 is opened for ordinal IMAP server. Setting dependency to it, because "* 0 EXISTS" itself is not Gmail particularity.
Comment 21 WADA 2009-06-23 20:54:15 PDT
I still could see problem with Tb trunk 2009/6/21 build.
  - Delete model: Just mark it as deleted
  - Shift+Delete of all mail in a folder => All are marked with red cross
  - These deleted & already expunged mails won't disappear
    until new mail is copied to the folder(*N EXISTS response where N>0).
Comment 22 WADA 2009-08-12 02:03:13 PDT
*** Bug 509902 has been marked as a duplicate of this bug. ***
Comment 23 Matěj Cepl 2009-09-19 14:35:43 PDT
Created attachment 401666 [details]
log according to https://wiki.mozilla.org/MailNews:Logging

I can reproduce this very well with mail.messagingengine.com (the IMAP server of fastmail.fm service). The other email server in this log (mail.corp.redhat.com, which is a Zimbra server) works just fine.
Comment 24 Matěj Cepl 2009-09-19 14:37:00 PDT
Forgot to add ... this log has been generated with the binary from ftp://ftp.mozilla.org/pub/thunderbird/nightly/3.0b4-candidates/build4/linux-i686/en-US/thunderbird-3.0b4.tar.bz2
Comment 25 Matěj Cepl 2009-10-02 06:52:53 PDT
Bug 509902 has been closed as a duplicate of this bug, but I am really not sure whether this isn't (or shouldn't be) GMail IMAP specific. I guess their implementation of IMAP is unfortunately quite special to deserve it.
Comment 26 Dan Mosedale (:dmose) 2009-10-06 10:41:38 PDT
Does this problem also happen with Thunderbird 2?
Comment 27 David :Bienvenu 2009-10-06 13:11:12 PDT
not blocking, wanted.
Comment 28 Ludovic Hirlimann [:Usul] 2010-01-07 07:31:33 PST
*** Bug 537990 has been marked as a duplicate of this bug. ***
Comment 29 Ludovic Hirlimann [:Usul] 2010-01-07 07:39:03 PST
*** Bug 537676 has been marked as a duplicate of this bug. ***
Comment 30 kk.ss 2010-01-14 06:58:27 PST
This problem does not happen in Thunderbird 2 as far as to the version before the official 3.0 one.
Comment 31 David :Bienvenu 2010-01-14 13:26:04 PST
Created attachment 421692 [details] [diff] [review]
proposed fix

this will fix the problem. now to attempt a unit test.
Comment 32 David :Bienvenu 2010-01-14 14:08:28 PST
Created attachment 421702 [details] [diff] [review]
fix with unit test
Comment 33 David :Bienvenu 2010-01-14 14:11:05 PST
forgot to mention, this cleans up a couple things in the test_flagChanges test as well, which I cloned this test from...Neil, if you don't want to review the test, you can leave that for standard8's sr.
Comment 34 neil@parkwaycc.co.uk 2010-01-14 15:55:01 PST
Comment on attachment 421702 [details] [diff] [review]
fix with unit test

I take it this only affects CONDSTORE servers, as otherwise the flag state is never partial?
Comment 35 David :Bienvenu 2010-01-14 15:58:47 PST
(In reply to comment #34)
> (From update of attachment 421702 [details] [diff] [review])
> I take it this only affects CONDSTORE servers, as otherwise the flag state is
> never partial?

No, sadly, both kinds of servers are affected. The issue is that ResetFlagInfo sets the flag state to partial, because that's a safer assumption in general, at least for condstore servers. But in this case, we're resetting the flag info because we know there are zero messages, but because we know there are zero messages, we need to clear the partial flag.
Comment 36 David :Bienvenu 2010-01-20 12:02:20 PST
*** Bug 540891 has been marked as a duplicate of this bug. ***
Comment 37 Ján Rusnák 2010-01-20 14:59:02 PST
also TB 3.0.1 suffers from this bug.
I can reproduce it on gmail and also dovecot imap server.
steps:
1. sent mail to an account.
2. mail appears in Inbox folder.
3. Close TB.
4. delete mail from other client (TB, web ui gmail).
5. start TB, and already deleted mail still seen in Inbox.
important settings TB settings:
account setting: synchronization of messages -> leave messages from account on this computer !OFF!
global setting: global search and indexing: !DISABLED!

this is only one serious bug which I find on new 3.0 TB release.
Comment 38 David :Bienvenu 2010-01-20 15:01:31 PST
(In reply to comment #37)
> also TB 3.0.1 suffers from this bug.
Yes, this bug has not been fixed yet. The patch is waiting for review. It will be fixed in nightly trunk builds soon, and in 3.02.
Comment 39 David :Bienvenu 2010-01-24 08:23:09 PST
*** Bug 541782 has been marked as a duplicate of this bug. ***
Comment 40 David :Bienvenu 2010-01-26 07:23:22 PST
ping on the sr - we've had several dupes of this...
Comment 41 David :Bienvenu 2010-02-01 10:45:56 PST
fixed on trunk and 3.0x branch
Comment 42 WADA 2010-02-09 23:13:19 PST
*** Bug 528233 has been marked as a duplicate of this bug. ***
Comment 43 WADA 2010-02-09 23:19:08 PST
*** Bug 486477 has been marked as a duplicate of this bug. ***
Comment 44 Saïvann Carignan 2010-02-12 22:40:14 PST
*** Bug 545874 has been marked as a duplicate of this bug. ***
Comment 45 Tom Guyette 2013-11-05 22:55:49 PST
I'm seeing this symptom as of a couple weeks ago -- haven't seen *0 EXISTS (don't know where to look), but deleting email from my Android phone does not result in the e-mail disappearing from my Thunderbird inbox.  TB 24.1.0 Win7, IMAP to Gmail.  I haven't noticed whether it's always or sometimes.  Using Get Mail / Get All New Messages button does nothing.
Comment 46 Wayne Mery (:wsmwk, NI for questions) 2014-06-19 05:07:59 PDT
(In reply to Tom Guyette from comment #45)
> I'm seeing this symptom as of a couple weeks ago -- haven't seen *0 EXISTS
> (don't know where to look), but deleting email from my Android phone does
> not result in the e-mail disappearing from my Thunderbird inbox.  TB 24.1.0
> Win7, IMAP to Gmail.  I haven't noticed whether it's always or sometimes. 
> Using Get Mail / Get All New Messages button does nothing.

Tom, If you can still reproduce in version 31 beta[1], please file a new bug if there is not a bug already covering this issue.

beta at http://www.mozilla.org/en-US/thunderbird/channel/

Note You need to log in before you can comment on or make changes to this bug.