Open Bug 693204 Opened 13 years ago Updated 13 days ago

Long delay of emails being flagged as read \Deleted, \Seen, tagged or untagged when mails is read from another device/location (IDLE does not send unsolicited responses for flag changes)

Categories

(MailNews Core :: Networking: IMAP, defect)

Product:
MailNews Core
Component:
Networking: IMAP
Type:
defect

Tracking

(Not tracked)

People

(Reporter: johndouglaspark, Unassigned)

References

(Depends on 1 open bug, Blocks 2 open bugs)

Details

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202 Safari/535.1

Steps to reproduce:

When reading an email or emails on another device/location, those emails are marked as read on that device and the IMAP server. I am using gmail & an IMAP connection on both Thunderbird and the other device.


Actual results:

When Thunderbird is left open, it sometimes takes 2-3 hours or even longer for Thunderbird to mark those emails as read. I have tried clicking on the "Get Mail" button to refresh the IMAP folder but TB still doesn't mark those emails as read. I have checked to see whether those emails are flagged as read or unread but using the gmail web client and they are indeed being marked as read but Thunderbird will not refresh their read/unread status.


Expected results:

Each time Thunderbird attempts to get mail it should check the status of read and unread emails from the IMAP server and mark those emails as read if applicable. The only way to force TB to refresh the read/unread status is to completely close TB and re-open which is obviously very annoying.
Do you enable IDLE command use?
If yes, same problem as bug 512745.
  bug 512745 :
    Gmail Label of A and B is added to a mail, and has flag \Seen(or no \Seen).
    The mail is shown in both Gmail IMAP folder named A and B, 
    and Tb knows flag \Seen(or \no \Seen) of both mails.
    Mail in Gmail IMAP folder named A is changed to no \Seen(or has flag \Seen).
    Mail in Gmail IMAP folder named B is also changed to no \Seen(or has flag
    \Seen) at Gmail IMAP server, because both are copy of same mail each other.
    This flag status change at Gmail IMAP folder named B is not notified to
    IMAP client like Tb via IDLE.
  Your report :
    A mail in a Gmail IMAP folder has flag \Seen(or no \Seen).
    Tb1(or Smart Phone etc.) and Tb2 knows flag \Seen(or \no \Seen) of the mail.
    The mail is changed to no \Seen(or has flag \Seen) by Tb1(or Smart Phone).
    The flag status change at Gmail IMAP folder is not notified to Tb2 via IDLE.
These are same phenomenon.

> I have checked to see whether those emails are flagged as read or unread
> but using the gmail web client and they are indeed being marked as read
> but Thunderbird will not refresh their read/unread status.

If the mail is "a mail in a conversation of Gmail which contains multiple mails", phenomenon of bug 478996 occurs. Don't use "read/unread status of a conversation at Gmail Web interface" as evidence of "read status or unread status of a mail", unless it's sure that the conversation of Gmail has single mail only or that all mails in a conversation of Gmail has same status of read or unread.

See also bug 518581, if Gmail IMAP.

> Expected results:
> Each time Thunderbird attempts to get mail it should check the status of read
> and unread emails from the IMAP server and mark those emails as read if applicable. 
> The only way to force TB to refresh the read/unread status is
> to completely close TB and re-open which is obviously very annoying.

To avoid excess network traffic due to "fetch of flags of all mails in IMAP folder upon each folder access",
(imagine 1,000,000 mails in a folder * 10,000 folders, new mail check of all mails folders each one minute)
Tb currently relies on "flag status change notification from server via IDLE", but many IMAP servers including Gmail IMAP unfortunately don't look to do so.
Following is a possible workaround at Tb in your case, if your any IMAP folder is never huge.
  - Disable IDLE command use
    Max chached connections = 1
    (I think Smart Phone's setting is similar to: no IDLE, max=0)
    (Tb doesn't permit max=0. Always uses default of 5 if max=0 is specified)
  - When you want to know flag status change by other IMAP cilent,
    or when you want to know flag status change propagation to
    same mail in other Gmail IMAP folder,
    click different folder, then click needed folder at folder pane of Tb.
Can you see flag status change by other IMAP client like Smart Phone, using Tb without restart?
Status: UNCONFIRMED → NEW
Ever confirmed: true
Component: General → Networking: IMAP
OS: Windows 7 → All
Product: Thunderbird → MailNews Core
QA Contact: general → networking.imap
I tried disabling IDLE and setting max cached connections to 1 but there appeared to be no change to the issue.

I also changed max cached connections to 0 and still no change.

I tried selecting multiple folders (Sent Items, Drafts etc) to try and force TB to refresh the read flags within the Inbox folder but no joy.

Interestingly, if I read an email in TB, that email is almost instantly flagged as read on my iPhone.

There is still a delay of many hours if I read an email on my iPhone or web client to be marked as read in TB.
Just noticed your notes regarding max cached connections = 0 not being a valid option. Changed back to 1 and still no change in result.

The only way I can get TB to check for read/unread status is to exit TB and relaunch.
(In reply to John Park from comment #2)
> I tried disabling IDLE and setting max cached connections to 1 but there
> appeared to be no change to the issue.

Did you restart Tb after the setting change?
Effect by the setting change is not immediate, so restart of Tb is needed to see effective or not.

(In reply to John Park from comment #3)
> Just noticed your notes regarding max cached connections = 0 not being a
> valid option. Changed back to 1 and still no change in result.

Read next in my comment, please.
>    (Tb doesn't permit max=0. Always uses default of 5 if max=0 is specified)
I've restarted TB and it appears the emails are now being marked as read.

Thank you for your help!

Is this an issue with TB or google?
It's an issue with how google handles synchronizing state between multiple connections. It's up to the server how frequently it does that synchronization...
(In reply to John Park from comment #2)
> Interestingly, if I read an email in TB, that email is almost instantly
> flagged as read on my iPhone.

Is this true for a mail in other than Inbox?
  At Tb, move(not copy) some(not one) mails from Inbox to other than Inbox
    (call Test folder, not subfoler of Inbox)
  At iPhone, see the mails in folder named Test. 
  At iPhone, see other folder than the Test folder, such as Inbox.
  At Tb, show "Order Received" column of Test folder.
  At Tb, change Read/Unread status of a mail in Test folder
         which has smallest "Order Received" column value.
Summary: Long delay of emails being flagged as read within Thunderbird when the emails are read from another device/location → Long delay of emails being flagged as read within Thunderbird when the emails are read from another device/location (IDLE does not send unsolicited responses for flag changes)
Per folder setting like next may be a solution.
  [ ? ] When automatic new mail check is invoked for this folder,
        always checks flag/keyword status of all mails in this folder.
Tagging the old report as dup of the new one is the best way to make devs think the bug is recent (and irrelevant).

Yes, an other TB3 profile on the same host.

But bug can also be reproduced with any other Imap client, like my mobile phone: just after reading an email on Gmail, I can sync my phone, and it will show mail as read; but clicking GetMail on TB3 will now ... untill TB restart. This is making 568775#c6 irrelevant.
An other point against TB, prooving that it's not Google's fault: bug appeared during the week after migration from TB2 to TB3 ... it would be a very strange hasard if Google changed their server conf just that week ...
FYI.
Another workaround, if "max cached connections=1" is not practical in your environment, and if you are eager to know flag/keyword change of existent mails by other mail client even though your IMAP server doesn't notify flag/keyword change to Tb via IDLE :
  Go "Work Offline mode", then go back to "Work Online mode",
  and, if folder you want to know flag/keyword change by other is FolderX,
  click an account or other folder, then click FolderX.
FYI.
Gmail IMAP started to support CONDSTORE from this year, so problem in Tb's CONDSTORE support is perhaps exposed to all Gmail users(see bug 912216, bug 885220).
IIUC, workaround of "Going offline then online" is affected by the problem and the workaround perhaps doesn't work as expected(see bug 885220 cmment #76).
Disable Tb's CONDSTORE support, please.
  mail.server.default.use_condstore = false (setting for all IMAP accounts)
  mail.server.server#.use_condstore = false (per account setting)
FYI.
If server supports CONDSTORE, and if CHANGEDSINCE of CONDSTORE is used by Tb upon new mail check, problem of this bug can be resolved. 
1. Upon first open(select at cached connection) of Mbox after restart.
   select "Mbox"  (CONDSTORE)
   * OK [HIGHESTMODSEQ 120418]
   UID fetch 1:* (FLAGS)
2. Upon second or later open(select at cached connection) of Mbox after restart of Tb.
   select "Mbox" (CONDSTORE)
   * OK [HIGHESTMODSEQ 120418] 
   UID fetch 1:* (FLAGS) (CHANGEDSINCE 120418)
3. Upon each new mail check at cached connection where Mbox is already selected.
   UID fetch 1:* (FLAGS) (CHANGEDSINCE 120418)
    * 836 FETCH (UID 3292 MODSEQ (120420) FLAGS (\Seen))
No longer blocks: 512745
Summary: Long delay of emails being flagged as read within Thunderbird when the emails are read from another device/location (IDLE does not send unsolicited responses for flag changes) → Long delay of emails being flagged as read(or flagged as \Deleted, \Seen is removed, tagged or untagged) within Thunderbird when the emails are read from another device/location (IDLE does not send unsolicited responses for flag changes)
Hardware: x86_64 → All
Thanks for the information, WADA.  The easy way to disable the CONDSTORE as suggested on Comment #13 is by doing this:

1. [On Linux] Edit -> Preferences // [On Windows] Tools -> Options...
2. Advanced -> Config Editor...
3. Search for "CONDSTORE" (with no quotes, and it shouldn't matter if it's lowercase).
4. Double-click on "mail.server.default.use_condstore" to change the 'Value' to "false".
5. Close the Config Editor (the search window).
6. Restart Thunderbird.

I hope this works, as I am just trying this fix out myself.

(See my old bug comment here: https://bugzilla.mozilla.org/show_bug.cgi?id=594106#c43)
Again, disabling CONDSTORE does not always help. There should probably be two separate bugs, one for CONDSTORE related, one for not related to CONDSTORE.. a bunch of different bugs all got rolled into one bug, so if somebody fixes CONDSTORE, hopefully they will not close this :/
(In reply to Nye Liu from comment #24)
Nye Liu, please don't confuse following.
 (a) problem of this bug(notification of flag status change)
 (b) problems due to CONDSTORE support in Tb
 (c) Tb's behavior in CONDSTORE support which may affect on workaround of this bug;
       - Going Work Offline, going back Work Online, then re-open folder by folder click
       - max cached connection=1, re-open folder by folder click
 (d) A possible solution of this bug by utilizing CONDSTORE (by correctly using)

"Disabling CONDSTORE support of Tb" in this bug is only for (c), even though the "disabling CONDSTORE support of Tb" is automatically applied to (b) always.
Purpose of workaround of this bug is to force Tb to issue "uid fetch 1:* flags" upon "folder click" step in workaround.
However, as I already pointed in comment #13, Tb may issue "UID fetch 1:* (FLAGS) (CHANGEDSINCE mod-seq-value)" or "uid fetch HighestUID:* (FLAGS)" upon the folder click in the workaround when CONDSTORE support of Tb is enabled. And, it may be different between Inbox and other Mbox. It may depend on timing of events during workaroud execution. Inconcistency of phenomenon during workaround execution surely produces user's confusions.
Purpose of "Disabling CONDSTORE support of Tb" in this bug is:
  Surely force "uid fetch 1:* flags" upon "folder click" step of workaround.
  Avoid user's confusions during workaround execution.

Because problem of this bug itslef is absolutely irrelevant to CONDSTORE support by Tb, even if CONDSTORE extension can be utilized to resolve problem of this bug, I'm sure that developers never confuse this bug with CONDSTORE support related problem.
(In reply to Nye Liu from comment #24)
> There should probably be two separate bugs, one for CONDSTORE related, one for not related to CONDSTORE..
> a bunch of different bugs all got rolled into one bug, (snip)

Nye Liu, have you actually read and understood all bugs which is duped to this bug by me?
Which phenomenon in which dup'ed bug is not problem due to "flag status change is not notified from IMAP server via IDLE"?

Do you actually understand problems by CONDSTORE support of Tb or problems in Tb around CONDSTORE?
Have you read and understand Bug 885220　which I already pointed in comment #13, which is for problem occurs only when CONDSTORE support of Tb is enabled?
Do you understand a change in the past which is one of causes of Bug 885220?
Have you read and understand bug 885220 comment #88?
Do you understand why the "change in the past" was backed out?
Do you understand what problem may occur again by the backuot?
Do you understand what problem occurred by the backout?
Have you read and understand bug 912216 which I already pointed in comment #13?

These problem relevant to "CONDSTORE extention and Tb's CONDSTORE support" and problem relevant to "flag status change is not notified from IMAP server via IDLE" are clearly isolated already.
Quick summary of this bug, to avoid confusions by some peoples, or to avoid irrelevant comments due to confusion with problems in Tb's CONDSTORE support.

-Problem of this bug;
  - Because server doesn't notify flag status change via IDLE,
    Tb can't know flag status change of mails at server Mbox via IDLE.
  - Tb's expectaion of "server notifies flag status change via IDLE" is never
    RFC violation, because "flag status change notification via IDLE" is SHOULD.
  - Server's behavior is never RFC violation, because "flag status change notification 
    via IDLE" is SHOULD, is never MUST.
  - Unfortunately, some major IMAP servers don't notify flag status change via IDLE,
    and Gmail IMAP is an example of such server.
- Problems in current "Tb's CONDSTORE suport" is irrelevant to above problem of this bug
  which is produced by IDLE command use.
 
Please never confuse following.

- If CODNSTORE extention is well utilized, problem of this bug can be resolved by simple
  method.

- Current Tb's CONDSTORE support has some problems, so "disabling Tb's CONDSTORE"
  is better. Rather, Tb user MUST disable it until problems will be resolved,
  if Tb user uses IMAP server who supports CONDSTORE extension.

- Gmail started to support CONDSTORE extesion this year,
  and Tb's default is mail.server.default.use_condstore=true.
  So problem in "Tb's CONDSTORE support" is exposed to Tb/Gmail IMAP user from this year.
  Because number of Tb+Gmail IMAP user is not so small,
  default of mail.server.default.use_condstore should be changed to false.

  "dup'ed bugs to this bug which is relevant to Gmail IMAP" was opened before Gmail
  started to support CONDSTORE extension. How can problem reported to these dup'ed bugs
  be relivant to Tb's CONDSTORE support and/or CONDSTORE extesion?

- Because wrokaround of this bug is affected by CONDSTORE extension and Tb's current
  CONDSTORE support, Tb user who experienced problem of this bug should disable
  Tb's CONDSTORE support, if Tb user uses IMAP server which supports CONDSTORE,
  in ordre to make the workaround usable,
  in order to avoid confusion produced by problem in Tb's CONDSTORE support.

- Because "disabling Tb's CONDSTORE support" does do nothing
  if IMAP server doesn't support CONDSTORE or server is not IMAP server,
  Tb user can disable Tb's CONDSTORE support freely, safely, any time,
  regardless of "used IMAP server supports CONDSTORE or not",
  regardless of "IMAP server is used or not".
(In reply to WADA from comment #27)
>   - Tb's expectaion of "server notifies flag status change via IDLE" is never
>     RFC violation, because "flag status change notification via IDLE" is
> SHOULD.

I'd say that is only true if "flag status change notification via IDLE" is MUST.

Seems to me that unless the server MUST provide flag status changes via IDLE, TB cannot rely on the server to provide flag status changes.
Years ago it worked... I guess it was from the Thunderbird 2-3 times.

Are we sure the problem is related to IDLE flag updates? Why do other IMAP clients detect deletions almost instantaneously?

I have Thunderbird 24.1.0 and K9Mail on android both connected to the same IMAP account on Zimbra 8.
If I delete a message on Thunderbird I see the action on the phone in 2-3 seconds.
If I delete a message on the phone, i never see the message disappear on Thunderbird.

Going to offline mode and back doesn't seem to do it for me.
Guess what, this isn't even limited to Gmail. It happens on my own IMAP server as well. It's running Dovecot on Linux. See bug 712595. Offline and online mode does not help. Changing folders does not work. Only restarting Thunderbird helps for sure. Until then, I regularly see unread or read messages in my inbox at home that I've already read and/or deleted through webmail or smartphone. I also have the impression that it's not a long delay, but instead it's a now or never. Disabling IDLE is definitely no option because that's the future (well, present already) and what everybody else uses successfully. I have the impression that there's some race condition inside Thunderbird that nobody will ever be able to find without rewriting the whole thing.
(In reply to Yves Goergen from comment #31)
> Guess what, this isn't even limited to Gmail. (snip)

What is reason to add new "merely 'me too' comment without required data for problem analysis" to this bug, even after reading my bug 712595 comment #24?
There is no need of such "guess" in this bug. In this "Bug report at B.M.O for developers to resolve Tb's actual bug(flaw in Tb's code)", "Why problem occurs", "What hapens" etc. is pretty clear already.
Please note that here is never support forum nor Help Center to saying complaint to Tb developers. Here is B.M.O for helping developers.
All "comuication between IMAP cliet(Thunderbird) and IMAP server" is done by "IMAP command and responce". How can software developers know "what actually happens in your environment" without required data?
4 x windows xp machines running thunderbird 24.1.1
2 x iphones (ios 6 and 7)

All sharing 4 different mailboxes.

Up until recently we used ms but changed due to roaming issues without exchange.

Our email is hosted by gmail.

All email was replicated instantly using outlook so gmail is not the problem.

Results are hit and miss, some machines update immediately, others will not update unless tb is restarted, offline/online / different folder / manually checking for mail does not solve.

Settings on all machines / users / mailboxes are mirrored.  IDLE is enabled accross but i have disabled and makes no difference.  I have disabled firewalls incase it was stopping the server updating but still nothing.
(In reply to delid4ve from comment #33)
> (snip) offline/online / different folder / manually checking for mail does not solve. (snip)
> IDLE is enabled accross, but i have disabled and makes no difference.

It's pretty simple. Merely one of next,
  (i)  Your problem is not this bug. Your problem is absolutely different problem from this bug.
  (ii) Your problem is this bug, but your operation is wrong.
because this bug is for following problem(and only for following problem).
  Server doesn't notify flag change via IDLE,
  so Tb can't know flag change status of already fetched mail via IDLE.

Example of (i) :
- If CONDSTORE is used, UID fetch 1:* (FLAGS) (CHANGEDSINCE ...), UID fetch Highest:*
  (CHANGEDSINCE ...) is used by Tb. This may produce problems in getting flag status of already
  fetched mail.
  Read bug 885220 for IMAP flow for this, please.
  This kind of issue is one of reasons why bug 912216 was opened.
  Note: Both bugs are alreeady pointed in this bug.
Example of (ii) :
  - IDLE command is issued after end of IMAP command execution at cached connection.
    "Use IDLE command" setting is for "issue IDLE or not, at end of IMAP command execution".
    "Use IDLE command" setting is never for "terminate current IDLE status at a cached connection".
  - Upon periodical new mail check, if an Mbox is already selected at a cached connection.
    Tb uses "uid fetch KnownHighestUID:* Flags".
    Inbox is selected at a cached connection always, except when max cached connection=1.
  Even though these, even though an Mbox is still IDLE status at a cached connection after
  disabling "Use IDLE command" setting, or even though an Mbox is still selected at a cached
  connection after some actions, write useless comment to a bug.
  Note:
  "go Work Offline" step of operations which is written in this bug is for;
     Force close of all cached connections, in order to force Tb to do "open connection, login,
     select Inbox(and other Mbox), uid 1:* fetch (flags)" again after "go Work Online".
  "disabling CONDSTORE support" is for;
     Force Tb to use "uid 1:* fetch (flags)" instead of "uid fetch 1:* (flags) (CHANGEDSINCE ...)"
     even when server supports CONDSTORE extension.
Would it be possible to at least add some sort of menu option next to "work offline" ... like "force sync" or something?
*(In reply to plop from comment #34)

Yeah, plop, that's why I switched to the Evolution mail program instead.  It seems to work better with Gmail than Thunderbird.  I don't wish Thunderbird to die out or anything, but I probably won't go back any time soon.

Although, I am slowly getting away from using Gmail as my main provider, since I do not really care for people having easy access to see all of my e-mail, not that I have anything to hide, but on principle.  I have just been using my website's e-mail service instead.
Blocks: tb-tagsmeta
(In reply to WADA from comment #27)
> -Problem of this bug;
>   - Because server doesn't notify flag status change via IDLE,
>     Tb can't know flag status change of mails at server Mbox via IDLE.
>   - Tb's expectaion of "server notifies flag status change via IDLE" is never
>     RFC violation, because "flag status change notification via IDLE" is
> SHOULD.
>   - Server's behavior is never RFC violation, because "flag status change
> notification 
>     via IDLE" is SHOULD, is never MUST.
>   - Unfortunately, some major IMAP servers don't notify flag status change
> via IDLE,
>     and Gmail IMAP is an example of such server.

Dear WADA,

How do other email clients deal with this situation on the commonly huge IMAP folders of Gmail ?

I think this is a rather critical bug of Thunderbird. As you say, "some major IMAP servers don't notify flag status change via IDLE", and Tb should be able to deal with this situation OOB. If I understand correctly, the workaround only works well with small IMAP folders.

Are there plans to fix this 2.5 years old bug ?

Regards,
Yes, I too have become exceedingly frustrated with this problem to the point of considering a move to another email client.   Given that a very painful manual workaround is available and has been for some time, it seems criminal that these workaround steps haven't been automated within Thunderbird.   Would it really be too resource intensive to simply perform these additional steps each time Thunderbird checks for mail updates?  Maybe that is what every other email client does and that is why they do not have this problem.  Thank you.
(In reply to Nye Liu from comment #36)
> Would it be possible to at least add some sort of menu option next to "work offline" ... like "force sync" or something?

A possible way.
- Install "Custom Buttons" addon, add your own Toolbar Button labeled "force sync".
- Put small JavaScript code like "click Work Offline, reply if required, wait for a while, click Work Online, reply if required".
  /*CODE*/
  // var MenuId="goOfflineMenuitem" ;  // Firefox
      var MenuId="goOfflineMenuItem" ;  // Thunderbird 
      var Menu=document.getElementById(MenuId) ;
      var Status=( "true" == Menu.getAttribute("checked")  ) ; // atribute value is String
      var OnCommand=Menu.getAttribute("oncommand") ;
      // oncommand="MailOfflineMgr.toggleOfflineStatus();"
      if( false == Status )
      {
          // go Work Offline
          eval(OnCommand);
          // do reply to dialog for download now,
          // go back Work Online, and do reply to dialog for send unsent messages
          // setTimeout(GoBackOnline,3*1000) ;
       }
      else if( true == Status )
      {
         // go back Work Online
         eval(OnCommand);
         // and do reply to dialog for send unsent messages
      }

This version doesn't do automatic reply to confirmtion dialog, doesn't do Toogle Back to Work Online mode.
So, you have to do: 1. Click the button, 2. No to dialog(download now), 3. click button, 4. Cancel to dialog(send unsent mail).
This is merely an example of your own pretty small Addon by your own ToolBar button.
id of menuitem, toolbarbutton , attributes of them etc. can be pretty easily known using DOM Inspector.
(In reply to Miguel from comment #38)
(In reply to Rich from comment #39)

If this bug is so critical for you and so important for you, why you never do one of next yet?
  (a) Use other than Thunderbird as your mailer which fully fulfills your pretty important requirement(s).
  (b) Use other sophisticated and clever IMAP server who supports feature of SHOULD even if the feature is not MUST in RFC. 
Why you never ask your IMAP server to support feature of SHOULD even if the feature is not MUST in RFC?
Even if SHOULD != MUST in RFC, why expecting feature of SHOULD can be a critical bug of a software?
SHOULD is sufficiently strong even if it's not MUST and SHOULD is weaker than MUST, isn't it?
(In reply to WADA from comment #40)
> - Put small JavaScript code like "click Work Offline, reply if required,
> wait for a while, click Work Online, reply if required".

In case you're not aware of it anymore, going offline and back online does not resolve the issue. At least for me. The only thing that does help is restarting Thunderbird, which I have to doo every few days when I deleted something either in the web mailer or my smartphone (K9 mail), with both work fine the other way around.

(In reply to WADA from comment #41)
>   (a) Use other than Thunderbird as your mailer which fully fulfills your
> pretty important requirement(s).

Thunderbird is the closest, unfortunately.

>   (b) Use other sophisticated and clever IMAP server who supports feature of
> SHOULD even if the feature is not MUST in RFC. 

This seems to be the Mozilla way of dealing with bugs: Blame it on the others, no matter how. While my comment may not be constructive, neither is yours. We're living in a real world that doesn't always work as designed.
(In reply to Yves Goergen from comment #42)
> In case you're not aware of it anymore, going offline and back online does not resolve the issue. At least for me. 
> The only thing that does help is restarting Thunderbird, which I have to doo every few days (snip)

Funny. It sounds that your Thunderbird is not standard one...
IIUC, IIRC, after Go Work Offline, Tb closes all connections, and after Go back Work Online, Tb executes login sooner or later, and after login, Tb opens Inbox(and others if requested) for new mail check by Biff, or open Inbox (or an Mbox) when selected, and upon first select Inbox/Mbox at a cached connection, Tb always issues "uid 1:* fetch Flags()" unless CONDSTORE is used. This is confirmed many times by getting IMAP log.
If  "uid 1:* fetch Flags()" is used, Tb should know all flags of all mails in an Mbox.
If not, it usually means that your Thunderbird is not properly configured.

Have you checked IMAP protocol level flow by yourself?
Do you correctly disable CONDSTORE support as stated in this bug more than one time?
Or if Inbox, did you do required "click Inbox while other than Inbox, for example account, is selected at Folder Pane"?
Because "new mail check by Biff" is not immediately invoked after going back to Work Online"(invoked at next new mail check interval), "time lag" exists between "going back to Work Online" and "first new mail check after going back to Work Online".

> Thunderbird is the closest, unfortunately.
It's sad. We will have to see complants by you, which surely won't help problem resolving,  many times in this bug....
rsx11m, Nikolay, do you have anything to add?
Flags: needinfo?(shopik)
Flags: needinfo?(rsx11m.pub)
I don't use gmail+imap, but I've seen some issues with marking many years ago (these was never been consistent)
Flags: needinfo?(shopik)
If I recall correctly, that's a case which the condstore feature is supposed to cover. However, since the MailNews implementation is broken (see bug 885220 and bug 912216) I wouldn't be surprised if it's an issue prompted by that. I don't see this for non-Gmail servers which are not using condstore, though changes in message status don't propagate immediately either.
Flags: needinfo?(rsx11m.pub)
Depends on: condstore-default
I've found that some extensions also cause problems with this. Two that I've found so far are:

- ImportExportTools (seems to do lots of concurrent IMAP operations with Tbird, often resulting in "Operation in progress" messages)
- Send Without Save(Copy Sent to Current has equivalent functionality and seems to be fine)

Once I disabled the above extensions, using a cyrus imapd 2.4 IMAP account that supports IDLE and CONDSTORE, Tbird changes from often requiring the Offline/Online trick, to usually updating immediately, sometimes requiring a folder change to see the updates, and occasionally requiring a Compact to see the updates. But as far as I have seen so far, Tbird now never requires Offline/Online to sync the folder, which is a huge improvement.

Disabling CONDSTORE further improves matters and Tbird more often (always?) updates the folder immediately. Obviously that is related to the separate CONDSTORE issues though.
(In reply to WADA from comment #43)
> (In reply to Yves Goergen from comment #42)
> > In case you're not aware of it anymore, going offline and back online does not resolve the issue. At least for me. 
> > The only thing that does help is restarting Thunderbird, which I have to doo every few days (snip)
> 
> Funny. It sounds that your Thunderbird is not standard one...

I don't know about Thunderbird, but I see the same thing (going offline does not work, I have to restart) in "standard" Seamonkey, and have done through many versions of Seamonkey.  In Seamonkey it is actually more annoying, because you need to shutdown all Seamonkey windows (i.e. browser windows and everything, not just mail and news windows).
(In reply to alister.hood from comment #50)> 
> but I see the same thing (going offline does not work, I have to restart) in "standard" Seamonkey, (snip)

IIUC, there is no difference between SeMonkey and Thunderbird, as for phenomeon of this bug.

(a) As repeatedly written, after go Work Offline the go back Work Online, explicit folder open by new folder click is mandtory to force server access again.  When did  go Work Offline the go back Work Online, did youa ctually do it?
      -  If Inbox is already selected at folder pane when "go back Work Online" is executed,
          click account then click Inbox again at folder pane.
(b) As repearedly written, Gmail IMAP started to enable CONDSTORE and Tb's CONDSTORE support is enaabled by default.
      When CONDSTORe is used, Tb may not use "uid fetch 1:* flags" or "uid fetch 1:* flags  CHANGEDSINCE nnnn".
      Tb may use "uid fetch <HighestUID>:* flags" or  "uid fetch <HighestUID>:* flags  CHANGEDSINCE nnnn".
      As repearedly written, to force  "uid fetch 1:* flags" always  in recovery operation for this bug,
      disbaling Tb's CONDSTORE support is needed for problem determination.
      Did you properly disabled Tb's CONDSTORE support?
      Read thru Bug 912216 and bug  885220, al repeatedly written.

Further, get IMAP log and check imap level flow by yourself, before adding comment like "annoying ..." which will never help activity for resolving problem.
Problem of this bug is pretty simple.
1. To know flag state change of already fetched mail of UID=NN<CurrentlyHighestUID, one of next is needed.
    1-1. flag staae change notification for the UID=NN from server via unsol response to lIDLE for selected Mbox.
            => Tb expects it, but maany IMAP servers including Gmail IMAP won't do this.
    1-2. uid fetch NN flags at connection where the Mbox is selected.
            => Tb uses "uid fetch CurrentlyKnownHighestUID:* flags" for new maail check of already selected Mbox.
                  So, flag state change of UID=NN can not be known untill "uid fetch 1:* flags" is issued again for the Mbox.
2. Recovery operation stated in this bug is simply "a procedure to try to force uid fetch 1:* flags".
    2-a. Unless you explicitly click  Mbox,  connection with server for the Mox won't be opened immediately.
            Without connection with imap server, any "uid fetch ..." won't be issued for the mbox by Tb.
    2-b.  Even when connection for the Mbox is established and the Mbox is selected at connection,
             Tb may not issue uid fetch 1:* flags". Tb may use "uid fetch HighestUID:* flags ...".
             Even if "uid fetch 1:* ..." is used by Tb, it may be "uid fetch 1:* CHANGEDSINCE nnnn"
             where nnnn is too large(may be newer than MODSEQ for flag change of UID=NN).
(In reply to WADA from comment #51)
...
> (a) As repeatedly written, after go Work Offline the go back Work Online,
> explicit folder open by new folder click is mandtory to force server access
> again.  When did  go Work Offline the go back Work Online, did youa ctually
> do it?

Yes, did that.

>       -  If Inbox is already selected at folder pane when "go back Work
> Online" is executed,
>           click account then click Inbox again at folder pane.

Did that too.

> (b) As repearedly written, Gmail IMAP started to enable CONDSTORE and Tb's
> CONDSTORE support is enaabled by default.
...
>       Did you properly disabled Tb's CONDSTORE support?
>       Read thru Bug 912216 and bug  885220, al repeatedly written.

Well, I disabled it in mail.server.default.use_condstore - there is no ...use_condstore setting for the account, so I guess Seamonkey doesn't implement it per account - or should I just create the setting?

> Further, get IMAP log and check imap level flow by yourself

Unfortunately that'll have to wait for another day.
But testing a few more times I discovered that it actually does work, just only every second time - i.e. I have to do "offline>online>other folder>back to folder>offline>online>other folder>back to folder" before it updates.
(In reply to alister.hood from comment #52)
> Well, I disabled it in mail.server.default.use_condstore - there is no
> ...use_condstore setting for the account, so I guess Seamonkey doesn't
> implement it per account - or should I just create the setting?

The implementation is identical between SeaMonkey and Thunderbird. If there is no server#-specific setting mail.server.id#.use_condstore then mail.server.default.use_condstore is used. Thus, setting mail.server.default.use_condstore to false should disable it for all configured IMAP servers unless specific preference settings for that server override it.
(In reply to alister.hood from comment #52)
> But testing a few more times I discovered that it actually does work, just only every second time
> - i.e. I have to do "offline>online>other folder>back to folder>offline>online>other folder>back to folder" before it updates.

It's perhaps "connection loss during IDLE".
  IDLE -> Sm/Tb(and server) enters "receive state" -> Work Offline -> Tb closes server connection(Socket, TCP/IP level)
  Tb perhaps closes IMAP connection. But for SeaMonkey, this is perhaps  "connection loss during IDLE".
  By click of Inbox again, Sm tries to send DONE(enters Send mode)
  -> because TCP session is already closed by Sm, nothing is sent to server, and Sm waits for timeout.
  -> go Work Offline/Online again -> send of DONE fails and imap connection is normally closed in Sm
  -> by next click of Inbox, TCP session is established and imap connection is established
It seems that error detection code is different between SeaMonkey and Thunderbird.

> so I guess Seamonkey doesn't implement it per account - or should I just create the setting?

There is no need of guessing. "CONDSTORE is aactually used or not" is always clearly known pretty easily by simply looking imap log.
Blocks: condstore-default
No longer depends on: condstore-default
FYI #1.
If CONDSTORE support of Tb is enabled(mail.server.default.use_condstore=true, or mail.server.server#.use_condstore=true), and if imap server supports CONDSTORE, bug 1123617 occurs even after go Work Offline/go Work Online again.
i.e. "Workaroudof this bug/recovery operaation of this bug",  go Work Offline/go Work Online again/explicitly open Mbox again, won't work due to bug 1123617 .
FYI #2.
If bug 1123617 will be fixed, this bug will be resolved by utilizing CONDSTORE, because change required for that bug is "uid fetch 1:* Flags (CHANGEDSINCE LargestKnowMODSEQ) upon each new mail check".
FYI.
I've opened Bug 1124924.
Bug 1124924 - If CONDSTORE is usable, use "uid fetch 1:* flags (CHANGDEDSINCE modseq)" for new mail check by Biff instead of "uid fetch NextUID:* flags", in order to resolve problem like Bug 693204, Bug 517461
All this discussion is very well and good, but how do we work around this problem?

I'm deleting e-mails from Gmail on my iPhone and still seeing them hours later in Thunderbird.

Work Offline > Work Online > Click another folder > Click Inbox seems to force resynch.

I'd really prefer not to do this by hand every hour.

What is the prescribed fix / workaround?
(In reply to Tom Guyette from comment #57)

Do you read about a workaround of "max cached connection=1"?

If shared with iPhone, I believe mail volume, number of mboxes, frequency of new mails, new mail volue, are not so large, 
If so, "max cached connection=1", "disable IDLE", "adequate new mail check intervlal", can be used.
Needless to say, "disabling CONSTORE" is mandatory.
1. Click other than Inbox(call FolderX) -> SELECT FolderX -> Inbox is unselected at only one connection
2. Soner or later, new mail check occurs -> SELECT Inbox -> Inbox is selected at  only one connection -> uid fetch 1:* Flags
3. When you need to snch flsag state with Gmail IMAP and iPhone, click FolderX, then click Inbox
     SELECT FolderX -> SELECT Inbox -> uid fetch 1:* Flags for Inbox is issued.
     As I repeadly staated, workaround explained in this bug is to force this "uid fetch 1:* Flags" for Inbox.

Another way: Use Inbox as mail drop, mail tray. Don't use Inbox as mail repository for log time.
                       Becuase of IMAP many Mboxes are lready supported. There is no need of folder named "Inbox" for mail viewing.

Because Inbox is mbox where new mails arrives, Tb always keeps Inbox "selected state" at a cached connection for Inbox only, and kepp Inbox open.
This is a reason why "uid fetch 1:* Flags" is not issued for Inbox while Tb is running.
If other folder than Inbox and Trash, connection for a FolderX is stolen by other FolderY when other FolderY is opened.
So, "max cached connections = 3 with folder switch of some folders someties" usually forces "uid fetch 1:* Flags" of FolderX, FolderY.

Why no problem in iPhone :
   Because of small/mobile device, connection is not kept for long time, and number of connections is usually one..
   Upon mail reading, login, SELECT Inbox, "uid fetch 1:* Flags" is executed.
Not sure what smartphones you have there but I have a permanent IDLE connection from mine, as on the desktop with Thunderbird. If disabling IDLE in Thunderbird is part of the workaround, then I think IDLE implementation is the problem. I'm not using IDLE for fun, I want to see new messages as they arrive. That's how modern event-driven systems work. Polling is from the 90s.

But since nobody was able to even locate the problem as of today, and given the open secret that Thunderbird is merely in maintenance mode for quite some time, I don't believe anybody is able to fix this issue (and others) without rewriting it all. If only I had more time...
(In reply to Yves Goergen from comment #59)
> Not sure what smartphones you have there but I have a permanent IDLE connection from mine,
> as on the desktop with Thunderbird. If disabling IDLE in Thunderbird is part of the workaround, then I think IDLE implementation is the problem. 
> I'm not using IDLE for fun, I want to see new messages as  they arrive. That's how modern event-driven systems work. Polling is from the 90s

Do you understand workaround of "max cached connection=1" and understaand IDLE?
If "max cached connection"=1, and if "other than Inbox, call FolderX" is selected at the only one connection, IDLE is effective only for FolderX. If Inbox is de-selectied at yhe only one cached connection, IDLE can do nothing on Inbox. If workround of "max cached connection"=1 is used, "IDLE works on Inbox sometime but IDLE won't work on Injbox other times" is pretty confusing, and it may produce useless bug open at B.M.O.

> I don't believe anybody is able to fix this issue (and others) without rewriting it all. If only I had more time.

You are absolutely free to believe anything.

Have you read and understood all bugs pointed in Bug 912216?
Because Gmail IMAP started to support CONDSTOREl last year, "solution of this bug utilizing CONDSTORE" is possible. As known by bugs pointed in Bug 912216, code change needed for the solution is not so huge. Rather, relatively small change than we thought.
Neeedless to say, as known by Bug 912216, there is problem in current CONDSTORE support in Tb, so it should be resolved.
And, for better solution, QRESYNC support by both Gmal IMAP and Tb is needed.
(In reply to WADA from comment #58)
> (In reply to Tom Guyette from comment #57)
> Do you read about a workaround of "max cached connection=1"?

Thanks WADA.  I read the whole thread, after reading another long thread that was closed as a duplicate of this one, so by the time I got here I was very tired and just wanted deleted messages to go away.

Max cached connections was the only part in all that reading that I understood, so I have max connections set to 1.  It slows down my client a bit, but I can accept that.

> If so, "max cached connection=1", "disable IDLE", "adequate new mail check intervlal", can be used.

I didn't understand what "disable IDLE" was until I started writing the proposed steps below.  I still don't understand what "adequate new mail check interval" means.  Bear in mind, I don't understand the client as a developer, I understand it as a casual user, so I need click-by-click, press-by-press, key-by-key instructions.  I suspect most other users who would come across this thread would need that too, and ideally it would be pinned to the top of the bug.  I've made an attempt at documenting the prescribed workaround below.

> Needless to say, "disabling CONSTORE" is mandatory.

I don't understand what that means, so it's not included in the steps below.

> 1. Click other than Inbox(call FolderX) -> SELECT FolderX -> Inbox is
> unselected at only one connection
> 2. Soner or later, new mail check occurs [...]

I did an experiment and I believe "sooner or later" can be replaced with "Restart Thunderbird."  This is my attempt at documenting the workaround step by step:  

FIRST, RECONFIGURE YOUR ACCOUNT:
1 - From the main Thunderbird screen, select "Tools" menu > "Account Settings" to launch the Account Settings window.
2 - On the left side of the Account Settings window, select "Server Settings" under your Gmail account name to view the Server Settings page.
3 - On the Server Settings page, press the "Advanced..." button to launch the Advanced Server Settings popup window.
4 - Set "Maximum number of server connections to cache" to 1.
5 - Deselect the checkbox for "Use IDLE command if the server supports it."
6 - Press the OK button to close the Advanced Server Settings window.
7 - Press the OK button to close the Account Settings window.

EVERY TIME YOU WANT TO REFRESH THE INBOX:
1 - Select the "Sent Mail" folder (or any other subfolder under the affected account).
2 - Close Thunderbird, including all mail viewing windows.
3 - Start Thunderbird.
4 - Select the "Inbox" folder of the affected account.  Thunderbird should completely refresh the Inbox.

> [...] -> SELECT Inbox -> Inbox is
> selected at  only one connection -> uid fetch 1:* Flags
> 3. When you need to snch flsag state with Gmail IMAP and iPhone, click
> FolderX, then click Inbox
>      SELECT FolderX -> SELECT Inbox -> uid fetch 1:* Flags for Inbox is
> issued.
>      As I repeadly staated, workaround explained in this bug is to force
> this "uid fetch 1:* Flags" for Inbox.
> 
> Another way: Use Inbox as mail drop, mail tray. Don't use Inbox as mail
> repository for log time.
>                        Becuase of IMAP many Mboxes are lready supported.
> There is no need of folder named "Inbox" for mail viewing.

I disagree that there is no need, or that telling users not to use it is an acceptable workaround.  Every standard mail client I know of has a view called Inbox.  Thunderbird itself creates one below the Gmail account name in the folder view.  Clearly it must have an important intended function.

Would it be possible to simply add a "Refresh folder" function to the "Get Messages" dropdown button that would perform a complete refresh of the current folder and Preferred Folder?  Better yet, add a Refresh button in the same row as Unread, Starred, Contact, Tags, and Attachment?
(In reply to Tom Guyette from comment #61)
> Would it be possible to simply add a "Refresh folder" function to the "Get Messages" dropdown button
> that would perform a complete refresh of the current folder and Preferred Folder?

Tb already has button for "Refresh folder from scratch". It's called "Repair Folder" button :-)

> Better yet, add a Refresh button in the same row as Unread, Starred, Contact, Tags, and Attachment?

You are right. 
There is no way to intentionally close mail folder in Tb.There is no way to force de-select of Mbox at a connection.
There is no way to force "uid fetch 1:* Flags" to recover from this bug by simple/easy operation.
Purpose of any workaround stated in this bug is  : force "uid fetch 1:* Flags" for Inbox.
    Go Offline/Go Online : Force logoff -=> Tb has to do login, SELECT Inbox, uid fetch 1:* Flags
    max cached connections=1 : FolderX steels connection from Inbox => When Inbox is accessed, Tb has to do SELECT Inbox, uid fetch 1:* Flags
If "close/reopen folder" like button/menu will be implemented, it's helpful for this bug.
But there is no need of such button in usual use. Button only for working around problem(in this case, in both Tb and Gmail) is not usually implemented.

About IDLE setting  : Server Settings/Advanced
About CONDSTORE : read Bug 912216, please.
About "adequate new mail check interval" :
   If "a few new mails per an hour", I believe "Check new messages Every 1 minute" is usually not mandatory.
   If "10000 new mails per an hour", I believe "Check new messages Every 1 minute" is not needed(rather annoying), 
   because you can know about many many new mails upon each mail viewing. :-)
(In reply to Tom Guyette from comment #61)
> I did an experiment and I believe "sooner or later" can be replaced with
> "Restart Thunderbird."  This is my attempt at documenting the workaround
> step by step:  

> EVERY TIME YOU WANT TO REFRESH THE INBOX:

> 2 - Close Thunderbird, including all mail viewing windows.
> 3 - Start Thunderbird.

If I understood your headings, I can safely reduce all of your steps to the ones I left in my quotation. I see deleted mails almost every day in my primary mailbox which is not Gmail. I just need to restart Thunderbird and everything works nicely again. No other configuration is required for this to work. Still it's cumbersome to restart applications to repair them, it might as well crash every day. This really should not be necessary at all.
Still seems to be an issue for me with Gmail. At least I assume this is the same issue. In my case, I filter all emails into folders (or "labels" as Gmail calls them) on the server. So all emails sent directly to *my* email address go to a folder/label called "Me".

== Steps to reproduce ==
1. Open Thunderbird.
2. Open arbitrary email app on phone/other computer. Send email to Gmail account.

== Expected results ==
Within seconds, Thunderbird displays the new mail in the "Me" folder, complete with a notification.

== Actual results ==
<nothing happens>

== Completely manual "workaround" ==
Click the "Me" folder. Thunderbird instantly displays the new message.

I note that even clicking "Get Messages" doesn't pull down the new message. What the heck? Clicking the folder gets a new message from the server but clicking "Get Messages" doesn't?

I have tried what I think is everything from this thread: disabled IDLE and changed max_cached_connections to 1. My CONDSTORE was actually "false" by default (Thunderbird 38.2.0 on OS X) so I left that as-is. I did a complete Exit of Thunderbird and restarted. No difference whatsoever.
Blocks: 987626
I have had 2 Thunderbird 45 instances for the same Gmail account open in the last weeks, viewing the All Mail folder while I was deleting mails from that folder. On several occasions, I noticed after deleting and/or tagging mails from one instance that the other instance wasn't updated. The deleted mails would still show, and would keep showing even after "Get Messages". There was no activity shown, and even Activity Manager showed nothing going on. If I remember correctly, this fixed between 1 and 5 minutes after going back to the outdated instance (Thunderbird would take the initiative to update, even though it had been asked before).

It seems like after a while Thunderbird goes in some kind of sleep mode, and has trouble waking up from that, but it hasn't happened often enough for me to explain the precise conditions needed for this to happen.

I do not fully understand Summary and am unable to parse several comments, and as it would take an eternity to go through all other comments, I will assume that the symptoms I experienced are results of the bug reported here and add a vote.
Blocks: 590791
Severity: normal → S3
Summary: Long delay of emails being flagged as read(or flagged as \Deleted, \Seen is removed, tagged or untagged) within Thunderbird when the emails are read from another device/location (IDLE does not send unsolicited responses for flag changes) → Long delay of emails being flagged as read \Deleted, \Seen, tagged or untagged when mails is read from another device/location (IDLE does not send unsolicited responses for flag changes)
See Also: → 591683
See Also: → 468490
You need to log in before you can comment on or make changes to this bug.