590791 - multiple TB not consistently updating read/unread status (and other header flags) viewing the same Inbox, despite all receiving the required real-time IMAP IDLE "push" of flag changes

Status: RESOLVED INCOMPLETE

(Thunderbird :: Mail Window Front End, defect)

Description

[bugzilla]

User-Agent:       Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; InfoPath.1; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C)
Build Identifier: 3.1.2 (and earlier)

If I open TB on 2 different PCs to view the same IMAP account at the same time, TB does not seem to reliably update message read/unread status in the other client - at least not consistently - I will expand...

Environment:
Latest TB 3.1.2 (and previous versions) - same issue seen with all versions
Hosting own IMAP email server - Dovecot (recent version)
Client OS - Windows XP Professional SP3

By and large our Dovecot IMAP installation works just fine, and I am confident that IMAP IDLE is properly configured on the server.

If 2 TB clients are open on separate PCs viewing the same email account then if new mail arrives both clients will see this email arrive at exactly the same time, real-time, i.e. TB and IMAP IDLE are working perfectly, 100% of the time in this respect.

Where TB does not appear to be handling IMAP IDLE updates properly is when email header flags are changed - e.g. most importantly - the read/unread status of an email.

Now, if a *recently* received email is opened on PC 1, PC 2 immediately reflects the read status of the mail by changing it from bold to unbold.  i.e. This occurs real-time as one would expect.

However, if I go to an *old* email in the Inbox on say PC 1 and change its status from read to unread (e.g. by opening it) then the TB Inbox displayed on PC 2 is *not* updated  :? 

The wierd thing is that if I then go to that *same* old email on PC 2 and start toggling it's read/unread status, this time PC 1 *does* recognise the status change real-time i.e. you see the message flit from bold to unbold in sync, real-time, with the change being made on PC 2 (i.e. the expected and desired behaviour).  If I then go back to PC 1 and toggle the same message's status there, again I see PC 2 updating correctly in real-time.  i.e. we have managed to temporarily "fix" the correct behaviour for this 1 old email by giving it a "kick" from both sides :)

So, in summary:

Picking on an old message in the Inbox and toggling its status on one PC doesn't get recognised by the other PC unless you follow the above scenario after which it works correctly just like newly arrived mail.  Of course, if you close the clients on both PCs and re-open them then updating the status of old messages is broken again.

Now, I found a good TB add-on called TBTracer that shows all the network traffic TB is sending.  So I installed this on PC 1 and PC 2.  It revealed that IMAP IDLE is indeed correctly pushing the status changes for all emails and more importantly that the remote TB client is also receiving these IMAP updates for 100% of the messages.  You can see in the log that TB/IMAP is pushing flags and including one it calls "Seen" whenever an email is marked as unread.  You can also see the remote PC 2 receiving these flag updates in the real-time TBTracer log, even though it is not acting on the updates it is being sent in the case of "old" emails as described above.  So, it is not a case of broken IMAP idle functionality, but something about TB that means it is not acting correctly on all IMAP messages it is receiving.

By "old" emails I essentially mean any email that has not been received recently or opened recently on the remote PC.  By recently, I probably mean since TB was opened on that PC (although it's possible that is too generous an assumption).

BTW, both clients are set up to dowload the headers+emails as they arrive and locally cache all emails in all folders.  Also, both clients are set to cache the default 5 connections to the IMAP server.  I don't believe the issue has anything to do with the number of cached connections, but include this information for completeness.

I feel quite confident that both our server and clients are correctly set up, and also feel quite confident that this has to be a TB bug.  And yet, I can't understand why I can't find 1000s of other people with the same issue on Google :-).  Surely this bug is a big deal and I can't be the only one seeing this behaviour!?  So I will be extremely grateful even to know that it has been possible to replicate the above and so confirm existence of bug.

It really makes TB unusable in a commercial environment where several people are sharing the same email box and replying to the same pool of emails (which is our situation).  Currently it's impossible to be sure whether someone else has already replied to an email!  Emails end up getting replied to twice etc etc.

Reproducible: Always

Steps to Reproduce:
1. Install Dovecot (maildir) on Linux box
2. Install TB on 2 separate PCs, configured to read exactly the same email account.  Enable IMAP IDLE and configure TB to download headers+message as mail arrives, and cache locally.
3. Move over some emails to the Inbox (preferrably 1000 or so just in case the number makes any difference).
4. Allow time for both TB client installations to download all messages locally and finish indexing them etc.
5. Close both TB clients and re-open them
6. Pick on an old message and toggle between read/unread and observe other TB client to see if real-time change to read/unread status is being updated in the other client.
7. If 6 seems to work keep picking on other emails until you find one where it doesn't
8. Unstall TBTracer to confirm that both clients are receiving the real-time flag updates even though the remote client is not reflecting these updates it is receiving.

These steps (and the scenario) are also explained in more detail in the "Details" section
Actual Results:  
For "old" emails (i.e. to be sure - those not received during the session since the TB client was opened), when read/unread status is toggled this status change is not seen in another open TB client installed on another PC and viewing the same inbox.

For "newer" mails i.e. In particular those that have just come through since the TB clients were opened, will probably behave as expected i.e. no problems - the read/unread status change will be seen real-time in multiple TB clients that may be open on various PCs and pointing at the same Inbox.

Expected Results:  
Toggling the read/unread status of *any* email should be seen in all open TB clients (on separate PCs) pointing at the same Inbox.  i.e. you should see the mail turn bold/unbold in the Inbox as its read/unread status is toggled on a remote TB client.

Comment 1

[bugzilla]

Server OS: CentosOS 5.5 so Kernel inotify used by Dovecot.  In any case, as I say above, the IMAP IDLE is doing its stuff correctly as TBTracer real-time log window shows *all* TB clients receiving "seen" flag update via IMAP (when a message is marked as "read"), and yet the remote TB clients are not always acting on this information they are receiving.

Also, note that if you kick the email from both ends e.g. toggle the *same* email on PC 1 and PC 2, then it will work properly for that email (until you close and re-open the TB clients).

Comment 2

[Ludovic Hirlimann [:Usul]]

What's the value of mail.server.default.autosync_offline_stores on your machines ?

Comment 3

[bugzilla]

mail.server.default.autosync_offline_stores = true (on all TB client PCs).

Comment 4

[bugzilla]

I mentioned that the behaviour is "Reproducible: Always".  Strictly what I mean is that it is not difficult for me to reproduce the problem - i.e. Open a couple clients and find an "old" email that doesn't seem to update (unless "kicked" from both ends).  However, it is not consistent in the sense that there are occasions where every email in the inbox appears to work fine between PC1 and PC2, but perhaps not also on PC3 at the same time.

In other words, you may need to test for a little while before you see the problem but I am confident if you do such tests you will see the problem without having to test for too long :-)

BTW, I have quite a few folders - perhaps 100, and many 1000s of emails (perhaps 2000 in the Inbox), with all folders ticked for off-line use (i.e. all emails downloaded as they arrive and cached locally).  I don't think the number of folders or number of emails is relevant to the issue - I only mention it for completeness.

As already mentioned - TBTracer is showing the IMAP flag update info arriving at the TB clients 100% of the time.  It's just that despite receiving the realtime flag updates via IMAP TB doesn't always act on this information.  As discussed there is more chance that a recently received email will behave correctly than an older one, but that's about much as can deduce.

Comment 5

[David :Bienvenu]

does your server support condstore? If so, does turning off condstore use in Thunderbird fix the issue? Look for condstore in the config editor and try toggling it off.

Comment 6

[bugzilla]

Yes, server does support condstore:
a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH] Logged in

As regards Thunderbird confg:
mail.server.default.use_condstore = True

So, I have set use_condstore = False on all TB clients.

So far, it looks like this workaround **may** have solved the problem, in that I haven't been able to reproduce the problem when testing briefly today.  However, I really can't be very sure until we have tried this setting for a full working week.  So at this point I will reserve my sincere thanks :-) and keep you posted.

I see there was a historic condstore issue with TB (seemingly random changes to read/unread state of emails) which was apparently fixed in TB 3.0.2.  So I guess it looks like this is a new TB condstore issue?  From my now limited understanding of condstore it would seem to stack up that this behaviour could also be a condstore issue.  Anyone any ideas what's causing TB to fail to act on all IMAP flag updates it receives when use_condstore = true?

Please let me know if there is anything specific I can do or provide to help uncover the source of this bug.

Comment 7

[it]

ive tried setting this variable to false but we still dont see the flag changes for over 10 seconds
if the message has its \seen flag toggled it updates imediatly

Comment 8

[WADA:World Anti-bad-Duping Agency]

> In bug summary : despite all TB clients receiving the required real-time IMAP IDLE "push" of flag changes

How did you confirm it?
Even big IMAP server such as Gmail IMAP, Yahoo! IMAP, server never notifies flag change via IDLE yet.
See Bug 693204.

Comment 9

[WADA:World Anti-bad-Duping Agency]

(In reply to bugzilla from comment #6)
> As regards Thunderbird confg:
> mail.server.default.use_condstore = True
> So, I have set use_condstore = False on all TB clients.
> So far, it looks like this workaround **may** have solved the problem, 8snip)

Current CONDSTORE support in Thunberbird can produce many unwanted problems, so default was changed to mail.server.default.use_condstore=false by Bug 912216. Read bugs pointed in that bug, please.

Comment 10

[Wayne Mery (:wsmwk)]

(In reply to WADA:World Anti-bad-Duping Agency from comment #8)
> > In bug summary : despite all TB clients receiving the required real-time IMAP IDLE "push" of flag changes
> 
> How did you confirm it?
> Even big IMAP server such as Gmail IMAP, Yahoo! IMAP, server never notifies
> flag change via IDLE yet.
> See Bug 693204.

afaict, none of the users commenting here are still around

Comment 11

[Erik Ohrnberger]

Hi.  I was very keen on this bug during the previous iteration of the shop's email system.

The email server was a Gentoo uw-imap, and this was a problem with an inbox shared with approx. 4 client PCs.

An email would come in and be noticed on the first PC client promptly.  The others lagged behind an unknown period of time.  Once noticed by the other PCs, the first PC would read the message, and a similar delay in updating the read status of the message would occur.

Watching Thunderbird's activity manager, you could see where it was tying to update the various accounts and folders.  ON the server side, I could see where inbox locks were being set and lost, as the next client tried to do their update.

Since then the email infrastructure has evolved to a Go Daddy email hosted solution, and the problem seems to have been reduced / eliminated with this back end migration.  This makes me think that the problem was with that particular imap server implementation, possibly.

Comment 12

[Wayne Mery (:wsmwk)]

Erik, thanks for your update.

=> incomplete for the original reporter's issue