524902 - Thunderbird sometimes fetches read/unread flags from the wrong IMAP folder

Status: RESOLVED FIXED

(Thunderbird :: Folder and Message Lists, defect)

Description

[Brian Rogers]

Since transitioning to Thunderbird 3 nightly builds a few days ago, I've been having a problem with my main IMAP account where random folders will suddenly have a ton of unread messages in them.

This happens not in the initial mail check, but after Thunderbird has been running for a while. I can view the folder before the problem occurs and I won't see unread messages there. But later, without warning the folder display will suddenly indicate a hundred or so unread messages (depending on the folder). When I look in these folders, random messages new and old are in the unread state, and there's no discernible pattern to which messages wind up unread. Many message remain correctly marked as read.

If I right-click the folder, choose properties, and click 'rebuild index', then Thunderbird restores the proper unread count and no longer incorrectly sees all those messages as unread. So the server-side status is untouched.

My IMAP host is running Debian Lenny with dovecot-imapd 1:1.2.4-2~bpo50+1 from backports. It has IDLE support and that feature is enabled in Thunderbird, and I have some server-side filters sending certain messages directly to folders other than my inbox. I'll try to observe whether this is triggered by receiving mail in these folders or not, though I believe it has happened before when I didn't receive a message to the folder. The problem hasn't happened to my inbox so far.

I have a couple other IMAP accounts which are not exhibiting this problem. However, they also don't get much use. They aren't on servers I run, so I don't know which IMAP daemon they are using.

Comment 1

[Brian Rogers]

Yeah, this is definitely happening to folders even when they haven't received any mail.

Comment 2

[Brian Rogers]

Looks like signals are getting crossed here. The problem just happened again, so I changed to an unthreaded view, sorted by 'order received' and compared with my other folders. The folder that this happened to got the read/unread flags from another folder.

That is, I had the message pane open for folder A, with the first 27 messages visible, and 10 were incorrectly in the unread state. Folder B looked exactly the same: 10 unread messages all in the same positions according to 'order received'. Then I found folder C, which was actually the folder that info came from (where the same messages, by position, were marked as read even after 'rebuild index'.

Comment 3

[David :Bienvenu]

this is the first report I've heard of this ever which makes me highly suspicious of your imap server. But, you could try generating an IMAP protocol log

https://wiki.mozilla.org/MailNews:Logging

and annotate it with which folder was displaying read state incorrectly.

Comment 4

[Felix Kaechele]

This sounds like the exact same problem I am facing.
My setup is also almost exactly the same. However I use dovecot-1.2.6-1.fc11.x86_64 and also IMAP IDLE as well as Sieve Message filtering.

Comment 5

[Nico Haase]

I'm also facing this problem using the RC1 under Windows 7. Sieve and Dovecot are also working on my mailserver. Is there any solution or workaround yet?

Comment 6

[Nico Haase]

This hasn't been solved in the final release :( Are there any more hints to the problem? In the meantime, I've also tried running Thunderbird in safe mode, but even here the status numbers gets scrambled.

Comment 7

[Brian Rogers]

I suppose at least one of us should make a log as suggested in comment 3. I might do that with a fresh account on my server so I don't get a ton of unnecessary (and personal) data flooding the log file. I'll see if I can reproduce this with only two folders existing. But I don't know when I'll be able to get to that.

Also, I've upgraded to dovecot-imapd 1:1.2.8-1~bpo50+1 now, and am still experiencing the issue.

Comment 8

[David :Bienvenu]

You could try turning off condstore support in TB - see https://bugzilla.mozilla.org/show_bug.cgi?id=517461#c9 (and I think this bug is a dup of that bug, actually)

Comment 9

[Nico Haase]

Turning off condstore solves my problems.

Comment 10

[David :Bienvenu]

OK, thx for the diagnosis - I see the problem in the code; we're not clearing out the flag state completely when switching selected folders in cached connections. This doesn't matter if you're not using condstore, but it does matter with condstore. The fix should be easy, but I have to figure out how to add a unit test without our fakeserver supporting condstore.

Comment 11

[Nico Haase]

Well, setting up an actual dovecot-server to test condstore shouldn't be that hard. It's supported since version 1.2 which was released this summer. And this could be the standard debian version in squeeze, released next spring...

Comment 12

[David :Bienvenu]

Thx, yes, we have a dovecot server installed. But we don't run our tests against live servers, unfortunately...anyway, I'll try to attach a patch today.

Comment 13

[David :Bienvenu]

Attachment: proposed fix

this makes us clear the whole flag and uid state when we switch folders with a connection instead of partially clearing it, and switches to using an nsTArray for the flags.

Comment 14

[David :Bienvenu]

there's a patch waiting for review, and the bug won't be fixed until the review is done, and the code checked in.

Comment 15

[[e-mail]]

Comment on attachment 417424 [details] [diff] [review]
proposed fix

>+  fFlags.InsertElementsAt(0, fNumberOfMessageSlotsAllocated, 0);
>+
>   fUids.InsertElementsAt(0, fNumberOfMessageSlotsAllocated, 0);
Nit: strange place to add a blank line

>+  fUids.SetCapacity(0);
>+  fFlags.SetCapacity(0);
SetCapacity(0) does nothing, as it's a lower bound. Did you mean Clear()?

>   for (counter = msgIndex; counter < (PRUint32) fNumberOfMessagesAdded; counter++)
>   {
>     fUids[counter] = fUids[counter + 1];
>-    fFlags[counter] = fFlags[counter + 1];                                  
>+    fFlags[counter] = fFlags[counter + 1];
Would it not work to call RemoveElementAt(msgIndex)?

>     nsTArray<nsMsgKey>      fUids;
>+    nsTArray<imapMessageFlagsType> fFlags;
Would it be worth making these an nsAutoTArray<T, kImapFlagAndUidStateSize>?

Comment 16

[David :Bienvenu]

Attachment: patch addressing most of the comments

thx for the comments - I've addressed the comments except for the autoTArray one - I think imap folder sizes are so variable that it's not a win to use the autoTArray...

Comment 17

[David :Bienvenu]

fix checked in. I need to come up with a similar fix for 3.01, since technically we can't change the flag and uid interface, even though it's strictly internal.

Comment 18

[David :Bienvenu]

Attachment: 3.01 fix

The version for 3.01 just ignores the count that's passed to reset() (It's always zero anyway).

Comment 19

[David :Bienvenu]

Comment on attachment 419971 [details] [diff] [review]
3.01 fix

I'm thinking 3.01 needs a much simpler fix, instead of trying to cleanup the surrounding code.

Comment 20

[David :Bienvenu]

Attachment: simpler 3.01 fix

Sorry to do this to you, Neil, but this fix seems a lot safer for 3.01 (code freeze for which is Friday night). If you are unlikely to be able to review this by then, I can switch it to Standard8.

I couldn't find an easier way to clear out values in an nsTArray w/o deallocating and reallocating the memory, or looping through assigning elements...

Comment 21

[[e-mail]]

Comment on attachment 420394 [details] [diff] [review]
simpler 3.01 fix

>   if (m_customFlagsHash.IsInitialized())
>     m_customFlagsHash.EnumerateRead(FreeCustomFlags, nsnull);
>+  memset(fFlags, 0, sizeof(imapMessageFlagsType) * fNumberOfMessageSlotsAllocated);
>   m_customFlagsHash.Clear();
Nit: strange positioning of this memset

Comment 22

[David :Bienvenu]

This is an equally important condstore fix, for 3.01, I believe.

Comment 23

[David :Bienvenu]

fixed for 3.01

Comment 24

[Phil Pishioneri]

I am still seeing this behavior using

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.7) Gecko/20100111 Thunderbird/3.0.1

my IMAP server is dovecot, and condstore is set true (default). I've tried Properties -> Rebuild Index: it'll fix the unread count immediately, but the bad count returns.

Comment 25

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #24)

Phil Pishioneri, please attach at least data for diagnosis by developers such as IMAP log, instead of simply adding "me too" comment or adding "problem still occurs in your envorinment" comment without detailed data for further problem analysis by developers.

Comment 26

[Phil Pishioneri]

(In reply to comment #25)
> Phil Pishioneri, please attach at least data for diagnosis by developers such
> as IMAP log, instead of simply adding "me too" comment or adding "problem still
> occurs in your envorinment" comment without detailed data for further problem
> analysis by developers.

1) Please give a little detail when requesting a log; don't assume that someone will look through every comment trying to glean the information. In the amount of text you took to chastise me, you could have asked for that.

2) If the log level being requested is too detailed, I'm not comfortable with attaching to the bug. Yes, I do know that passwords and such are obfuscated in the logs, but those aren't the issue.

Comment 27

[Ludovic Hirlimann [:Usul]]

(In reply to comment #26)

> 2) If the log level being requested is too detailed, I'm not comfortable with
> attaching to the bug. Yes, I do know that passwords and such are obfuscated in
> the logs, but those aren't the issue.

So this can be easily solved. Could you First create a new bug (and add a  ice description in it). Then you can send the logs by email to [e-mail] who can analyse them. Make sure to include the new bug number in that email so he knows which bug to tackle.

Comment 28

[Phil Pishioneri]

(In reply to comment #27)

> Could you First create a new bug (and add a  ice description in it).

Bug 540846 opened.

Comment 29

[David :Bienvenu]

Ugh, why is Ludo encouraging people to open duplicate bugs? :-)

Phil, thx for the log - I'll look at it, but I'm fairly sure this is the same condstore issue that I'm looking at for 3.1. and is covered by several bugs.

I'll re-open this, since it's obviously not fixed...

Comment 30

[Ludovic Hirlimann [:Usul]]

(In reply to comment #29)
> Ugh, why is Ludo encouraging people to open duplicate bugs? :-)
> I'll re-open this, since it's obviously not fixed...

Cause I though you had fixed it ;-)

Comment 32

[Timo Sirainen]

Note that Dovecot v1.2.7 fixed some bugs its CONDSTORE code. Before spending too much time on some log dump, make sure it's generated by >= v1.2.7.

Comment 33

[Nico Haase]

(In reply to comment #32)
> Note that Dovecot v1.2.7 fixed some bugs its CONDSTORE code. Before spending
> too much time on some log dump, make sure it's generated by >= v1.2.7.

I'm running Dovecot 1.2.9, and the bug still occurs.

Comment 34

[Phil Pishioneri]

(In reply to comment #32)
> Note that Dovecot v1.2.7 fixed some bugs its CONDSTORE code. Before spending
> too much time on some log dump, make sure it's generated by >= v1.2.7.

Mine is 1.2.8.

Comment 35

[David Halik]

Running dovecot 1.2.9, definitely still a problem in TB 3.0.1. I've been watching my folders with 8K messages suddenly become all unread when I go into folder. In TB 3.0 it would sometimes be a few messages, with 3.0.1 it's now entire folders for me.

Comment 36

[David :Bienvenu]

David, hence the re-opening. I've got some 3.01 + bug fix builds here that should fix the problem - http://tinderbox.mozilla.org/showbuilds.cgi?tree=ThunderbirdTry&maxdate=1264042655&hours=24&legend=0&norules=1

Comment 38

[Phil Pishioneri]

(In reply to comment #36)
> David, hence the re-opening. I've got some 3.01 + bug fix builds here that
> should fix the problem -
> http://tinderbox....

I downloaded (green: [e-mail]-1264029848) and am running it:

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.8pre) Gecko/20100120 Shredder/3.0.2pre

And some old messages just became unread (rebuild index fixed it). Do you want the log (I had it enabled)?

Comment 39

[David :Bienvenu]

Phil - yeah, a log would be good, along with what folder the messages were in. 

Instead of rebuilding index, you could just try toggling the offline button.

Comment 40

[David Halik]

Phil might still have some issues, but so far David your nightly shredder build seems to be working great for me. Before I was experiencing the two unread/read bugs almost constantly while moving between folders. Since the install about two hours ago I haven't seen it once. I have a log going just in case and will report any more problems.

Comment 41

[Mark Banner (:standard8)]

I'm going to remark this as fixed. The issue appears to be specific to 3.0.x only builds as shown by the patch in bug 540554. Bug 540554 will therefore deal with fixing the unread/read issue.

Comment 42

[emonk]

This is probably a server side error with Cyrus IMAP.

http://www.mail-archive.com/[e-mail]/msg28046.html

Solution in server side:

    * Stop SMTP Postfix, then stop IMAP Cyrus
          _ /etc/init.d/postfix stop; /etc/init.d/cyrus-x.x stop 
    * Delete the .seen* user afected files under /var/lib/cyrus/user/X/
          _ Example: cd /var/lib/cyrus/user/j/; rm jperez.seen* 
    * Start IMAP Cyrus and then start SMTP Postfix
          _ /etc/init.d/cyrus-x.x start; /etc/init.d/postfix start