Closed Bug 524902 Opened 11 years ago Closed 11 years ago

Thunderbird sometimes fetches read/unread flags from the wrong IMAP folder

Categories

(Thunderbird :: Folder and Message Lists, defect)

x86
Linux
defect
Not set
major

Tracking

(thunderbird3.0 .1-fixed)

RESOLVED FIXED
Thunderbird 3.1a1
Tracking Status
thunderbird3.0 --- .1-fixed

People

(Reporter: brian, Assigned: Bienvenu)

References

Details

Attachments

(2 files, 2 obsolete files)

Since transitioning to Thunderbird 3 nightly builds a few days ago, I've been having a problem with my main IMAP account where random folders will suddenly have a ton of unread messages in them.

This happens not in the initial mail check, but after Thunderbird has been running for a while. I can view the folder before the problem occurs and I won't see unread messages there. But later, without warning the folder display will suddenly indicate a hundred or so unread messages (depending on the folder). When I look in these folders, random messages new and old are in the unread state, and there's no discernible pattern to which messages wind up unread. Many message remain correctly marked as read.

If I right-click the folder, choose properties, and click 'rebuild index', then Thunderbird restores the proper unread count and no longer incorrectly sees all those messages as unread. So the server-side status is untouched.

My IMAP host is running Debian Lenny with dovecot-imapd 1:1.2.4-2~bpo50+1 from backports. It has IDLE support and that feature is enabled in Thunderbird, and I have some server-side filters sending certain messages directly to folders other than my inbox. I'll try to observe whether this is triggered by receiving mail in these folders or not, though I believe it has happened before when I didn't receive a message to the folder. The problem hasn't happened to my inbox so far.

I have a couple other IMAP accounts which are not exhibiting this problem. However, they also don't get much use. They aren't on servers I run, so I don't know which IMAP daemon they are using.
Yeah, this is definitely happening to folders even when they haven't received any mail.
Looks like signals are getting crossed here. The problem just happened again, so I changed to an unthreaded view, sorted by 'order received' and compared with my other folders. The folder that this happened to got the read/unread flags from another folder.

That is, I had the message pane open for folder A, with the first 27 messages visible, and 10 were incorrectly in the unread state. Folder B looked exactly the same: 10 unread messages all in the same positions according to 'order received'. Then I found folder C, which was actually the folder that info came from (where the same messages, by position, were marked as read even after 'rebuild index'.
Summary: Thunderbird randomly marks hundreds of messages as unread → Thunderbird sometimes fetches read/unread flags from the wrong IMAP folder
this is the first report I've heard of this ever which makes me highly suspicious of your imap server. But, you could try generating an IMAP protocol log

https://wiki.mozilla.org/MailNews:Logging

and annotate it with which folder was displaying read state incorrectly.
This sounds like the exact same problem I am facing.
My setup is also almost exactly the same. However I use dovecot-1.2.6-1.fc11.x86_64 and also IMAP IDLE as well as Sieve Message filtering.
I'm also facing this problem using the RC1 under Windows 7. Sieve and Dovecot are also working on my mailserver. Is there any solution or workaround yet?
This hasn't been solved in the final release :( Are there any more hints to the problem? In the meantime, I've also tried running Thunderbird in safe mode, but even here the status numbers gets scrambled.
I suppose at least one of us should make a log as suggested in comment 3. I might do that with a fresh account on my server so I don't get a ton of unnecessary (and personal) data flooding the log file. I'll see if I can reproduce this with only two folders existing. But I don't know when I'll be able to get to that.

Also, I've upgraded to dovecot-imapd 1:1.2.8-1~bpo50+1 now, and am still experiencing the issue.
You could try turning off condstore support in TB - see https://bugzilla.mozilla.org/show_bug.cgi?id=517461#c9 (and I think this bug is a dup of that bug, actually)
Turning off condstore solves my problems.
OK, thx for the diagnosis - I see the problem in the code; we're not clearing out the flag state completely when switching selected folders in cached connections. This doesn't matter if you're not using condstore, but it does matter with condstore. The fix should be easy, but I have to figure out how to add a unit test without our fakeserver supporting condstore.
Well, setting up an actual dovecot-server to test condstore shouldn't be that hard. It's supported since version 1.2 which was released this summer. And this could be the standard debian version in squeeze, released next spring...
Thx, yes, we have a dovecot server installed. But we don't run our tests against live servers, unfortunately...anyway, I'll try to attach a patch today.
Attached patch proposed fix (obsolete) — Splinter Review
this makes us clear the whole flag and uid state when we switch folders with a connection instead of partially clearing it, and switches to using an nsTArray for the flags.
Assignee: nobody → bienvenu
Attachment #417424 - Flags: superreview?(neil)
Attachment #417424 - Flags: review?(neil)
there's a patch waiting for review, and the bug won't be fixed until the review is done, and the code checked in.
Comment on attachment 417424 [details] [diff] [review]
proposed fix

>+  fFlags.InsertElementsAt(0, fNumberOfMessageSlotsAllocated, 0);
>+
>   fUids.InsertElementsAt(0, fNumberOfMessageSlotsAllocated, 0);
Nit: strange place to add a blank line

>+  fUids.SetCapacity(0);
>+  fFlags.SetCapacity(0);
SetCapacity(0) does nothing, as it's a lower bound. Did you mean Clear()?

>   for (counter = msgIndex; counter < (PRUint32) fNumberOfMessagesAdded; counter++)
>   {
>     fUids[counter] = fUids[counter + 1];
>-    fFlags[counter] = fFlags[counter + 1];                                  
>+    fFlags[counter] = fFlags[counter + 1];
Would it not work to call RemoveElementAt(msgIndex)?

>     nsTArray<nsMsgKey>      fUids;
>+    nsTArray<imapMessageFlagsType> fFlags;
Would it be worth making these an nsAutoTArray<T, kImapFlagAndUidStateSize>?
thx for the comments - I've addressed the comments except for the autoTArray one - I think imap folder sizes are so variable that it's not a win to use the autoTArray...
Attachment #417424 - Attachment is obsolete: true
Attachment #419597 - Flags: superreview?(neil)
Attachment #419597 - Flags: review?(neil)
Attachment #417424 - Flags: superreview?(neil)
Attachment #417424 - Flags: review?(neil)
Attachment #419597 - Flags: superreview?(neil)
Attachment #419597 - Flags: superreview+
Attachment #419597 - Flags: review?(neil)
Attachment #419597 - Flags: review+
fix checked in. I need to come up with a similar fix for 3.01, since technically we can't change the flag and uid interface, even though it's strictly internal.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Attached patch 3.01 fix (obsolete) — Splinter Review
The version for 3.01 just ignores the count that's passed to reset() (It's always zero anyway).
Attachment #419971 - Flags: approval-thunderbird3.0.1?
Depends on: 537551
Comment on attachment 419971 [details] [diff] [review]
3.01 fix

I'm thinking 3.01 needs a much simpler fix, instead of trying to cleanup the surrounding code.
Attachment #419971 - Flags: approval-thunderbird3.0.1?
Attached patch simpler 3.01 fixSplinter Review
Sorry to do this to you, Neil, but this fix seems a lot safer for 3.01 (code freeze for which is Friday night). If you are unlikely to be able to review this by then, I can switch it to Standard8.

I couldn't find an easier way to clear out values in an nsTArray w/o deallocating and reallocating the memory, or looping through assigning elements...
Attachment #419971 - Attachment is obsolete: true
Attachment #420394 - Flags: superreview?(neil)
Attachment #420394 - Flags: review?(neil)
Comment on attachment 420394 [details] [diff] [review]
simpler 3.01 fix

>   if (m_customFlagsHash.IsInitialized())
>     m_customFlagsHash.EnumerateRead(FreeCustomFlags, nsnull);
>+  memset(fFlags, 0, sizeof(imapMessageFlagsType) * fNumberOfMessageSlotsAllocated);
>   m_customFlagsHash.Clear();
Nit: strange positioning of this memset
Attachment #420394 - Flags: superreview?(neil)
Attachment #420394 - Flags: superreview+
Attachment #420394 - Flags: review?(neil)
Attachment #420394 - Flags: review+
Attachment #420394 - Flags: approval-thunderbird3.0.1?
This is an equally important condstore fix, for 3.01, I believe.
Attachment #420394 - Flags: approval-thunderbird3.0.1? → approval-thunderbird3.0.1+
fixed for 3.01
Target Milestone: --- → Thunderbird 3.1a1
I am still seeing this behavior using

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.7) Gecko/20100111 Thunderbird/3.0.1

my IMAP server is dovecot, and condstore is set true (default). I've tried Properties -> Rebuild Index: it'll fix the unread count immediately, but the bad count returns.
(In reply to comment #24)

Phil Pishioneri, please attach at least data for diagnosis by developers such as IMAP log, instead of simply adding "me too" comment or adding "problem still occurs in your envorinment" comment without detailed data for further problem analysis by developers.
(In reply to comment #25)
> Phil Pishioneri, please attach at least data for diagnosis by developers such
> as IMAP log, instead of simply adding "me too" comment or adding "problem still
> occurs in your envorinment" comment without detailed data for further problem
> analysis by developers.

1) Please give a little detail when requesting a log; don't assume that someone will look through every comment trying to glean the information. In the amount of text you took to chastise me, you could have asked for that.

2) If the log level being requested is too detailed, I'm not comfortable with attaching to the bug. Yes, I do know that passwords and such are obfuscated in the logs, but those aren't the issue.
(In reply to comment #26)

> 2) If the log level being requested is too detailed, I'm not comfortable with
> attaching to the bug. Yes, I do know that passwords and such are obfuscated in
> the logs, but those aren't the issue.

So this can be easily solved. Could you First create a new bug (and add a  ice description in it). Then you can send the logs by email to bienvenu@mozillamessaging.com who can analyse them. Make sure to include the new bug number in that email so he knows which bug to tackle.
(In reply to comment #27)

> Could you First create a new bug (and add a  ice description in it).

Bug 540846 opened.
Ugh, why is Ludo encouraging people to open duplicate bugs? :-)

Phil, thx for the log - I'll look at it, but I'm fairly sure this is the same condstore issue that I'm looking at for 3.1. and is covered by several bugs.

I'll re-open this, since it's obviously not fixed...
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to comment #29)
> Ugh, why is Ludo encouraging people to open duplicate bugs? :-)
> I'll re-open this, since it's obviously not fixed...

Cause I though you had fixed it ;-)
Duplicate of this bug: 540846
Note that Dovecot v1.2.7 fixed some bugs its CONDSTORE code. Before spending too much time on some log dump, make sure it's generated by >= v1.2.7.
(In reply to comment #32)
> Note that Dovecot v1.2.7 fixed some bugs its CONDSTORE code. Before spending
> too much time on some log dump, make sure it's generated by >= v1.2.7.

I'm running Dovecot 1.2.9, and the bug still occurs.
(In reply to comment #32)
> Note that Dovecot v1.2.7 fixed some bugs its CONDSTORE code. Before spending
> too much time on some log dump, make sure it's generated by >= v1.2.7.

Mine is 1.2.8.
Running dovecot 1.2.9, definitely still a problem in TB 3.0.1. I've been watching my folders with 8K messages suddenly become all unread when I go into folder. In TB 3.0 it would sometimes be a few messages, with 3.0.1 it's now entire folders for me.
David, hence the re-opening. I've got some 3.01 + bug fix builds here that should fix the problem - http://tinderbox.mozilla.org/showbuilds.cgi?tree=ThunderbirdTry&maxdate=1264042655&hours=24&legend=0&norules=1
Duplicate of this bug: 541180
(In reply to comment #36)
> David, hence the re-opening. I've got some 3.01 + bug fix builds here that
> should fix the problem -
> http://tinderbox....

I downloaded (green: bienvenu@nventure.com-1264029848) and am running it:

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.8pre) Gecko/20100120 Shredder/3.0.2pre

And some old messages just became unread (rebuild index fixed it). Do you want the log (I had it enabled)?
Phil - yeah, a log would be good, along with what folder the messages were in. 

Instead of rebuilding index, you could just try toggling the offline button.
Phil might still have some issues, but so far David your nightly shredder build seems to be working great for me. Before I was experiencing the two unread/read bugs almost constantly while moving between folders. Since the install about two hours ago I haven't seen it once. I have a log going just in case and will report any more problems.
I'm going to remark this as fixed. The issue appears to be specific to 3.0.x only builds as shown by the patch in bug 540554. Bug 540554 will therefore deal with fixing the unread/read issue.
Status: REOPENED → RESOLVED
Closed: 11 years ago11 years ago
Resolution: --- → FIXED
Depends on: 540554
This is probably a server side error with Cyrus IMAP.

http://www.mail-archive.com/info-cyrus@lists.andrew.cmu.edu/msg28046.html

Solution in server side:

    * Stop SMTP Postfix, then stop IMAP Cyrus
          _ /etc/init.d/postfix stop; /etc/init.d/cyrus-x.x stop 
    * Delete the .seen* user afected files under /var/lib/cyrus/user/X/
          _ Example: cd /var/lib/cyrus/user/j/; rm jperez.seen* 
    * Start IMAP Cyrus and then start SMTP Postfix
          _ /etc/init.d/cyrus-x.x start; /etc/init.d/postfix start
You need to log in before you can comment on or make changes to this bug.