Closed Bug 512745 Opened 15 years ago Closed 11 years ago

same mail in gmail imap folders(Gmail Label) are listed as separate messages, but they sometimes have different read/unread status (IDLE in Gmail IMAP does not send unsolicited responses for flag changes)

Categories

(MailNews Core :: Networking: IMAP, defect)

Product:
MailNews Core
Component:
Networking: IMAP
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 693204

People

(Reporter: bekirserifoglu, Unassigned)

References

(Blocks 1 open bug)

Details

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.13) Gecko/2009080315 Ubuntu/9.04 (jaunty) Firefox/3.0.13
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.4pre) Gecko/20090825 Shredder/3.0b4pre

If you get a mail that with your gmail imap account and it is listed under different labels, Thunderbird sees that one mail as multiple mails. And the real problem is even if you read the one in the inbox, other copies of the mail listed under different label do not get marked as read automatically.

Reproducible: Always
Do you use gmail over imap or pop3 ?
Blocks: tb-gmailWIP
Version: unspecified → 3.0
as I said in the bug report, I use imap with gmail.
(In reply to comment #0)
> "listed under different labels" (snip)

What do you mean by the "labels"?
  (a) At Tb. IMAP folder accessed via Gmail IMAP.
  (b) At Gmail's Web Interface. Gmail Label.
"Still unread" is phenomenon at which?
As you may know gmail uses labels instead of folders. And they show up as folders in thunderbird. I use several labels with my gmail account, I created them using the web interface and they show up in thunderbird with no problem. As you may guess since gmail has labels instead of folders, it stores only one copy of a mail in its archive and applies labels to that mail. In fact, what you see as "inbox" in gmail is just a label. if you archive a mail, it will just lose the label "inbox" and it will be kept in "all mail". Thunderbird 3 sees these labels as folders, it probably stores the mails in that order. and when you get one mail, it shows up in different folders. in fact, even if you don't create labels, sometimes one mail shows up as two, because it's included in "inbox" and "all mail". the more annoying thing is even if you read a copy of a mail, the other copies listed under different labels do not get marked as read. Of course, gmail web interface does not have this problem, the problem is with thunderbird 3.
Thunderbird 2 does not have this problem either.
Actually, I reproduced the bug with thunderbird 2 too. Sorry.
The problem may be about Gmail. I have been using thunderbird 2 for a long time with gmail and didn't have any problem. I switched to Thunderbird 3 a few weeks ago and this problem began to occur. It's probably a coincidence because now I can reproduce the bug with thunderbird 2 too. And one more important thing is I just now enabled some lab feature about imap and this time one mail does not show as multiple but you don't see it's label until you click on a folder.
Well, i think this is certainly expected, though undesired. Would need to be fixed server side though.
Summary: Mail with Multiple Gmail Labels → mail in gmail imap folders ("labels") are listed as separate messages - not one with several labels
What do you mean "expected"?
Magnus, your bug summary is not accurate for "Unread/Read status of mail" part.

Problem on "Unread/Read status" is next;

(0) Gmail Label of FOLDER-A and FOLDER-B is added to a mail(via Gmail IMAP
    using IMAP client), or to a conversation(via Gmail Web Interface).
    Assume intital status == "Unread" (no \Seen flag).
(1) The mail(or a mail in the conversation) is presented to IMAP client as;
    (a) mail of UID-A-m in FOLDER-A, without \Seen flag 
    (b) mail of UID-B-n in FOLDER-B, without \Seen flag
(2) When Tb changes (a) from Unread to Read,
    Tb issues "uid UID-AAA-m store flag \Seen" at FOLDER-A.

After that, design/implementation of Gmail/Gmail IMAP is special.
(3) Because single copy of mail is held in "All Mail", status of "Read" is saved in data for the single copy in "All Mail".
(4) Because single copy of mail is held in "All Mail", status of "Read" is automatically propagated to mail in "[Gmail]/All Mail" via Gmail IMAP.
(5) Because single copy of mail is held in "All Mail", status of "Read" is automatically propagated to (b) too.
(6) Because of (5), event of "\Seen is added to (b)" is notified to IMAP client via Gmail IMAP, by ILDE, by next fetch flag by Tb, and so on.

Above step (4)/(5) usually works normally. But Gmail/Gmail IMAP still has next issue, and step (4)/(5) doesn't work well sometimes.
> http://mail.google.com/support/bin/answer.py?answer=96926&topic=12922
> Updates not reflected in multiple clients
  Note: Above step (3) is action by Gmail, not by Gmail IMAP himself.
        So "multiple clients" is applicable to "single IMAP client access" case.
Sometimes, collapse/re-expand of account or parent folder resolves problem, if reason is simply a delay. But if issue in Gmail/Gmail IMAP, restart of Tb is required as written in above document.

Bug opener's case is:
  FOLDER-A == Inbox
  FOLDER-B == a user defined Gmail Label, and/or "[Gmail]/ALL Mail"

I don't know which phenomenon in comment #4 is this bug's problem.
WADA: feel free to fix the summary as appropriate
This problem is critic with a important number of tags, but it's probably not necessary to add a new functionnality (add tags) to thunderbird. With Evolution, I can download and see my messages in folders, and evolution folder size is about 1,3 Go. But with thunderbird 3.0.3, now the size is 12 Go ! It's a real bug !
bug 518581 reported with detailed description same problem as comment #0. Setting dependency to bug 518581 insted of duping for ease of searrch/tracking.
Depends on: 518581
Summary: mail in gmail imap folders ("labels") are listed as separate messages - not one with several labels → same mail in gmail imap folders ("labels") are listed as separate messages, but they sometimes have different read/unread status
(In reply to comment #11)
> This problem is critic with a important number of tags, (snip)

For "tag" case, see  Bug 450246 comment #5. Read/unread(\Seen flag) and Tag(keyword=user deined flag) looks to have same issue, but they are better to be analized in different bug for ease of diagosis. 

> But with thunderbird 3.0.3, now the size is 12 Go !

One problem per bug is rule of B.M.O.
Thibaud, please read bug 532323 and bug 537498 before add additional complaint to this bug for absolutely different problem.
 Bekir, is it still and issue in v3.1.4?

wada in comment #9:
>...
> Bug opener's case is:
>   FOLDER-A == Inbox
>   FOLDER-B == a user defined Gmail Label, and/or "[Gmail]/ALL Mail"
> 
> I don't know which phenomenon in comment #4 is this bug's problem.

so is this an issue with our backend?  or our front end?
Isn't the unread count and message status updated correctly when you select the other folder? That's what I see happening. It sounds like you want us to automatically update every gmail folder that might potentially be affected whenever you change a flag on a message, and I don't think we're going to do that.
Aha!

If next condition, Read/Unread staus change at folder-B is not automatically detected at folder-A by Tb, because Tb uses uid X:* fetch flags(X=largest UID+1, or next_UID) for new mail check of already opened IMAP folder.
(1) folder-A is opened by Tb. folder-A is selected at folder pane, thread pane
    is shown. SELECT Inbox is already issued a cached connection, and is kept.
(2) folder-B. Read/Unread status of mails which are same mails in folder-A
    is changed at folder-B.
(2-a) folder-B(IMAP folder) is not opened by Tb
(2-a-1) folder-B(Gmail Label) at Gmail Web.
(2-a-2) folder-B(IMAP folder) is opened by other mail client(s9 including
        Thunderbird on same PC or on different PC.
(2-b) folder-B(IMAP folder) is opened by Tb
(2-b-1) folder-B(IMAP folder) of Tb is opened by Tb at differenet Tab or
        Tb Window

To know mail's status(IMAP flag) change at folder-A, next is required.
  Click other folder at all Tab/Tb Window at which folder-A is opened.
  => folder-A is closed.
     at a cached connection for folder-A, SELECT other-folder is issued,
     then folder-A is changed to unselected status.
  After it,
  (i)  If automatic new mail check is enabled for folder-A, STATUS folder-A is
       issued by new mail check, then unread mail count of folder-A is updated.
  (ii) If automatic new mail check is not enabled for folder-A,
       upon next folder open of folder-A, uid 1:* fetch flags is issued,
       then Read/Unread status of all mails is obtained.
       So, unread mail count of folder-A at folder pane is refreshed.

It's not applicable to folder-A==Inbox case, because Inbox is always kept SELECTED status at a cached connection(If max cached connections>=3, Trash too. I knew it by David's comment in other bug). So, uid X:* fetch flags is used for new mail check. Thus, unread mail count of Inbox is not updated by new mail check. In my test, "unread mail count update of not-opened Inbox by new mail check" was observed only with max cached conections=1.
I don't know above is applicable when other than Inbox is clicked(Inbox is closed?) and Inbox is cliked again.
For folder-A==Inbox case, Work Offline->Work Online is probably simplest workaround.

See bug 594106 comment #15 and bug 518581, please.
It's same phenomenon as bug 518581.

In bug opener's case, folder-B==Inbox and folder-A==other than Inbox, Trash.
I think one of "manual folder re-open of folder-A" and "enable new mail check of folder-A" updates unread mail count of folder-A at folder pane.

For "no problem in Tb 2".
David, did Tb 2 issue "SELECT folder + uid 1:* fetch flags (\Unseen only?)" upon new mail check?
The imap server should tell us about changes made from other connections as an unsolicited response.

I don't know what SELECT folder + uid 1:* fetch flags (\unseen only) means. We fetch the flags for all messages. There's no way to fetch just unseen message flags...other than doing a search, which we don't do.
(In reply to comment #17)
> I don't know what SELECT folder + uid 1:* fetch flags (\unseen only) means.

Oh, sorry for invald comment. \Unseen? was "query for mail of '\Seen is not set' only?, if new mail check".

> The imap server should tell us about changes made from other connections as an
> unsolicited response.

Gmail IMAP's RFC violation and/or Gmail/Gmail IMAP's bug?

Assuming that both folder-A and folder-B are opened by Tb and both are SELECT'ed at two cached connections, and IDLE is enabled, in this bug's case.
(In my test IDLE, is killed to make it simple.)

As Gmail IMAP, UID-a in folder-A of Gmail IMAP and UID-b in folder-B of Gmail IMAP is same mail of Gmail-Mail-ID-mmm-and-Conversion-ID-ccc in Gmail's central Mail DB. So, "uid UID-a store flag \Seen" at folder-A via cached-connection-A is saved in Gmail-Mail-ID-mmm-and-Conversion-ID-ccc, and is propagated to UID-b in folder-B of Gmail IMAP sooner or later by Gmail/Gmail IMAP.

From perspective of IMAP, UID-a in folder-A of Gmail IMAP and UID-b in folder-B of Gmail IMAP are absolutely independet and different mail. So, flag propagation to UID-b in folder-B of Gmail IMAP is identical to flag change of UID-b in folder-B of Gmail IMAP by other mail client.

IMAP server has to notify the flag change by an IMAP client on UID-b in folder-B of Gmail IMAP to other IMAP client who selects folder-B of Gmail IMAP via an cached connection using IDLE by unsol response of IDLE?
(In reply to comment #18)
> (In reply to comment #17)
> 
> > The imap server should tell us about changes made from other connections as an
> > unsolicited response.
> 
> Gmail IMAP's RFC violation and/or Gmail/Gmail IMAP's bug?

IDLE in Gimap does not send unsolicited responses for flag changes.  It is a missing feature, but not a bug or RFC violation.  (The RFC only requires unsolicited messages for new or deleted messages, and those are sent.)
Thunderbird cannot learn about flag changes without asking.
(In reply to comment #19)

> 
> IDLE in Gimap does not send unsolicited responses for flag changes.  It is a
> missing feature, but not a bug or RFC violation.  (The RFC only requires
> unsolicited messages for new or deleted messages, and those are sent.)
> Thunderbird cannot learn about flag changes without asking.

right, should send, not must send :-) Perhaps even "should" is too strong, but that's the way IMAP is designed to work, and it's definitely left up to the server's discretion.
Gotcha!
I've understood why similar phenomenon were reported to some bugs for GMail IMAP even though IDLE is enabled, and why I could always observe same phenomenon by intensional test with IDLE disabled using Gmail IMAP.
I'll try to check using non Gmail IMAP with IDLE enabled.
Thnaks to David and Brian for your technical supports.
Status: UNCONFIRMED → NEW
Component: General → Networking: IMAP
Ever confirmed: true
OS: Linux → All
Product: Thunderbird → MailNews Core
QA Contact: general → networking.imap
Hardware: x86 → All
Summary: same mail in gmail imap folders ("labels") are listed as separate messages, but they sometimes have different read/unread status → same mail in gmail imap folders ("labels") are listed as separate messages, but they sometimes have different read/unread status (IDLE in Gmail IMAP does not send unsolicited responses for flag changes)
Version: 3.0 → Trunk
Checked with my ISP's IMAP - similar to Gmail, for single Mail Box of my account, access via POP3 server, via IMAP(non-SSL) server, via IMAP(SSL) server, are provided.

0. Max cached conections=3, IDLE is enabled, New mail check is disabled.
   Auto-sync of Inbox=off(offline-use=off).
1. Copy a mail to Inbox of IMAP(SSL) => Automatically detected by IMAP(non-SSL)
2. Say Largest UID=P, Next UID=Q
2-1. IMAP(SSL) : Add tag-01 to UID=P => IMAP(non-SSL) : Nothing happens.
2-2. Copy a mail to IMAP(SSL), UID=Q
     IMAP(non-SSL) : New mail is detected, tag-01 of UID=P is detected
3. Say UID=A, B, C is smaller than P
3-1. IMAP(SSL) : Add tag-01, tag-02, tag-03 to UID=A, B, C, respectedly
     => IMAP(non-SSL) : Nothing happens.
3-2. Copy a mail to IMAP(SSL)
     IMAP(non-SSL) : New mail is detected, Tag of UID=A, B, C is not detected
4. IMAP(non-SSL) : Click Trash, then click Inbox again
   => Tag of all mails is detected.

(A) Unsol response via IDLE is for new(and deleted) mail only, not for Tag.
(B) By manual re-open of Inbox, re-sync is normally executed,
    even if a cached connection is reseved for Inbox.

I think it's same as Gmail IMAP with IDLE enabled. I'll check using Gmail IMAP with IDLE enabled(test with IDLE disabled/max chached connections=1 etc. is to find a way for automatic Tag status synchronization of Inbox by automatic new mail check only)
also. each message is downloaded ATLEAST twice (both INBOX and ALL MAIL), if it has more labels it will be downloaded additionally..
No longer depends on: 518581
Summary: same mail in gmail imap folders ("labels") are listed as separate messages, but they sometimes have different read/unread status (IDLE in Gmail IMAP does not send unsolicited responses for flag changes) → same mail in gmail imap folders(Gmail Label) are listed as separate messages, but they sometimes have different read/unread status (IDLE in Gmail IMAP does not send unsolicited responses for flag changes)
Bug 693204 is for same phenomenon by same cause in different and generic situation.
xref that bug.
Depends on: 693204
If I know the folder that is wrong (in my case its [Gmail]/Important or INBOX) is their someway to refresh the read/unread for just that folder without doing a complete compact or switching between offline/online? Thanks.
"IDLE does not send unsolicited responses for flag changes" is not Gmail IMAP only issue.
Issue of "Tb is not torelant with 'IDLE does not send unsolicited responses for flag changes'" is not Gmail IMAP only issue.
Because bug 693204 was processed after analysis of many bugs, that bug is crispy than other bugs.
So, dupin this bug to bug 693204.
If duping is wrong, re-open this bug, please.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → DUPLICATE
No longer depends on: 693204
No longer blocks: 594106
No longer blocks: 544439
No longer blocks: 518581
You need to log in before you can comment on or make changes to this bug.