482754 - Folders indicate new messages when no new messages present; message count incorrect as well

Status: VERIFIED FIXED

(Thunderbird :: Folder and Message Lists, defect)

Description

[Joshua Nixon]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b2) Gecko/20081201 Firefox/3.1b2
Build Identifier: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b4pre) Gecko/20090311 Shredder/3.0b3pre ID:20090311025302

Recent nightly builds are showing "new" messages and triggering the new message alert sound for folders that do not contain any new messages. Folders are highlighted in blue, as if new messages are present. Checking folder shows nothing new. When Extra Folder Columns add-on is installed, some of the affected folders are reporting negative message counts. I have three folders right now showing negative counts - two with -13, one with -271. Clicking the folders restores the actual new and total message counts.

Disabling Extra Folder Columns doesn't fix the erroneous new message indicator.

Reproducible: Always

Comment 1

[Matthias Versen [:Matti]]

Did you already try to rebuild the index of that folders ?

Comment 2

[Aaron Daubman]

I also just started experiencing this after 2-3 nightly builds ago. I have many (30+) folder being accessed via IMAP (from Exchange 2007 server) and most of them will highlight as containing new messages every time a scheduled or manual check for new messages occurs. The new message count badge on the Dock Icon also blows up sometimes to 750+ (when there are zero or few real new messages).

I have not tried to re-build the index since it would be for each folder, some of which are quite large, and would take a long time... is this a necessary step or a shot in the dark? Any way to tell before I have to re-index all folders (since it seems to occur to a different and random majority each time).

The only way I can see where a new (if any) message has arrived is to click on each individual folder, which then un-bolds if no real new messages, or gets updated with the real (correct) new message count a few seconds after I highlight it (doing this for all 30+ folders isn't fun).

I'm running: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b4pre) Gecko/20090312 Shredder/3.0b3pre

Thanks,
     Aaron

Comment 3

[Thomas Stache]

I'm seeing this on Windows Vista, using IMAP against the German GMX mail provider.

Comment 4

[Mark Banner (:standard8)]

We should at least investigate this apparent regression possibly caused by bug 428266 before the next beta. Setting blocking and assigning to David as he's probably the best one to look at this.

Comment 5

[Dave Miller [:justdave]]

I'm seeing this on Mac, using IMAP with Mozilla's Zimbra.

Comment 6

[Nikolay Shopik]

Also folder can become bold but don't show message numbers, while there new messages.

Comment 7

[Thomas Stache]

Attachment: Part of IMAP log

I attached a part from my logging session.

It begins after a new messages finished downloading. Shredder than sat idling, until the folder "Mailinglists/Frisbee" turned bold (without an unread count in parenthesis). Maybe someone can get some insight from the log, but I don't see anything from the server suggesting that.

Comment 8

[Thomas Stache]

I disabled IDLE support to eliminate possible conflicts between IDLE and auto-sync. Still I just triggered the bug like this:

1. fearing that bug 483859 could be there I selected almost every folder
2. installed nightly update, forcing a restart
3. upon startup TB checked for new messages, bolding all the folders
   previously selected

Interesting tidbit: to sort things out I don't have to select folders again, hovering with the mouse pointer helps as well, I suspect triggering the calculation of the extended tooltip, which notices that actually no messages are there.

Comment 9

[Thomas Stache]

Attachment: IMAP log where many folders appeared bold after app startup

This log was recorded while I experienced this bug, many folders were bolded, but had no new-message-count. This startup followed immediately after the recording of Attachment #369040 [details] in bug 483859.

Comment 10

[Aaron Daubman]

Just as an update (since it was previously suggested), re-indexing all folders did not help.

I recently followed these steps:
http://kb.mozillazine.org/Phantom_folders
(delete *.msf, delete panacea.dat) and shortly after re-launching Thunderbird, all but nine folders became bold/blue to indicate new messages.
(No 1000s of new messages count indicator this time, but that usually happened less frequently).

Running: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b4pre) Gecko/20090325 Shredder/3.0b3pre
Against a MS Exchange 2007 with IMAP server

Comment 11

[Joshua Nixon]

Latest nightly seems to be better - I discovered duplicated .msf files in profile, so deleted everything except message filter files. On rebuild things are working again, as expected, no more erroneous "new mail" notices or bad message counts. Don't know if the duplicated files were due to new nightly or not - there were folders like (for example) Drafts, Drafts-1 and associated .msf files; I deleted all of it and restarted.

If a folder rebuild is necessary to stop this from happening, it will likely need to be done automatically on upgrade to avoid confusion.

Comment 12

[David :Bienvenu]

Attachment: possible fix

I'd like to see if this helps - I still need to figure out the underlying issues, but we shouldn't be marking folders as having new if the unread count isn't > 0.

Comment 13

[David :Bienvenu]

I've been poking around this code a bit more - I haven't reproduced the bug, but I can see where things would go wrong. I think the relevant change in bug
428266 was to persist the server counts, which I believe exacerbated some issues in nsImapMailFolder::UpdateImapMailboxStatus. That code treated differences in the server unread counts and db unread counts as pending unread count changes, and in some situations the pending unread count is getting confused. It's getting negative when it shouldn't (but just being negative isn't an error per se - e.g., we could have five unread messages in the db, but you could have marked them read on an other machine - in that case, after doing a status command, we'd have a pending unread count of -5 so that we would show 0 unread in the folder pane - when the user selected the folder, we'd reconcile all the headers+counts).

Comment 14

[David :Bienvenu]

Attachment: proposed fix

This makes us maintain the invariant that the pending unread + the db unread == server unread. It also makes sure that we reflect changes to the db and panacea.dat.

I'd like to see if this helps with the problems folks have been seeing.

Comment 15

[Thomas Stache]

Does Mozilla Messaging have Tryserver infrastructure yet? Or could a dev please link to a test build with the patch?

Comment 16

[Mark Banner (:standard8)]

(In reply to comment #15)
> Does Mozilla Messaging have Tryserver infrastructure yet? Or could a dev please
> link to a test build with the patch?

We're actively setting it up. At this time we only have OS X builds available.

Comment 17

[[e-mail]]

Comment on attachment 370118 [details] [diff] [review]
proposed fix

>   PRInt32 previousUnreadMessages = (m_numServerUnseenMessages)
>     ? m_numServerUnseenMessages : GetNumPendingUnread() + mNumUnreadMessages;
>   if (numUnread != previousUnreadMessages || m_nextUID != prevNextUID)
>   {
>+    PRInt32 unreadDelta = numUnread - (GetNumPendingUnread() + mNumUnreadMessages);
>+    if (numUnread - previousUnreadMessages != unreadDelta)
>+       NS_WARNING("unread count should match server count");
      if (numUnread - previousUnreadMessages != unreadDelta)
=     if (numUnread - unreadDelta != previousUnreadMessages)
=     if (GetNumPendingUnread() + mNumUnreadMessages != previousUnreadMessages)
=     if (m_numServerUnseenMessages && (m_numServerUnseenMessages !=
          GetNumPendingUnread() + mNumUnreadMessages))
But even this belongs in an NS_WARN_IF_FALSE/NS_ASSERTION i.e.
NS_WARN_IF_FALSE(m_numServerUnseenMessages &&
                 (m_numServerUnseenMessages !=
                  GetNumPendingUnread() + mNumUnreadMessages),
                 "unread count should match server count");
You can now also get rid of previousUnreadMessages.

Comment 18

[Aaron Daubman]

I would be happy to test and provide feedback on any fixed builds as I am dealing with this problem constantly. I see mentioned that "At this time we only have OS X builds available" but am not sure where to get them... I've been running the latest nightlies for a while now.

Comment 20

[David :Bienvenu]

fix checked in - please let me know if this helps with the problem.

Comment 21

[Chris Malton]

I'm guessing this fix should be in the 20090407 build when it gets rebuilt tonight.  Still in the last 20090406 update I downloaded.

Comment 22

[Aaron Daubman]

I've only been running the nightly with the fix for about an hour but already my workflow is much smoother =)

Thanks for committing the fix - as far as I can tell it is working well. I also noticed that aside from NOT getting the invalid unread/new message notifications now, I am also getting more reliable correct unread message counts (e.g. a folder that has an older unread message in it would not always show up bold or with an unread message count before. Now it always has the correct markings...)

Thanks!!!

Comment 23

[Chris Malton]

Just restarted Shredder after installing latest nightly.  The first thing to happen was a folder came up in bold.  Click on it and no new messages.  Not fixed here.  Will keep running the latest build and report if there are any more issues, or the issue is resolved.

Comment 24

[Wayne Mery (:wsmwk)]

Counts and new mail folder decoration much improved for me as well. 

(In reply to comment #23)
> Just restarted Shredder after installing latest nightly.  The first thing to
> happen was a folder came up in bold.  

A folder will of course be bold if there are *unread* messages. Do you mean there are no unread messages in the bold folder?

Comment 25

[Chris Malton]

There were no unread messages in that folder.  Interestingly, the issue seems to be resolved now. It was just that one folder at the first startup after the update.  Since then, works brilliantly.  Well done on fixing it.  Finally I only get a "new mail notify" sound when I actually get mail.

Comment 26

[Nikolay Shopik]

Also looks good for me as well.
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b4pre) Gecko/20090407 Shredder/3.0b3pre