Closed Bug 274688 Opened 20 years ago Closed 15 years ago

The dock icon often shows an incorrect number of unread messages

Categories

(Thunderbird :: General, defect)

All
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
Thunderbird 3.0b4

People

(Reporter: jalemon, Assigned: humph)

References

(Blocks 1 open bug)

Details

(Whiteboard: [penelope_wants])

Attachments

(2 files, 8 obsolete files)

The dock icon displays the number of unread messages in the green circle in the
upper right of the icon. The number is often incorrect, both too large and too
small, and indicating unread messages when there aren't any or not indicating
them when they exist.
I can confirm. When I press Get Mail, the incorrect number corrects itself.

For me, the number is never too small. If there are unread messages in my inbox,
it always indicates a number that is >= the correct number.

On a possibly related note, I have it set so that my junk mail is automatically
moved to the Junk folder.
may depend on bug 261201.
I think there are two issues here:
1. Thunderbird is counting messages marked as junk, when it probably should not
be (and definatly should not be if junk messages are automatically deleted).
2. If there is already an icon showing and new mail comes in then Thunderbird
will not update that icon (this is possibly a side effect of the windows biff,
which you only want to be triggered the first time mail comes in).
(In reply to comment #3)

I have been experiencing this behaviour with various versions of Thunderbird on
both Mac OS X Panther (10.3) and Tiger (10.4).


Currently, I am witnessing it with Thunderbird 1.0.6 and 1.5 Beta 1 on Tiger,
when I have setup 2 seperate IMAP accounts and multiple mailbox subscriptions
for both accounts.

> I think there are two issues here:
> 1. Thunderbird is counting messages marked as junk, when it probably should not
> be (and definatly should not be if junk messages are automatically deleted).

I think I can discount this scenario. I use SpamAssassin on both my mail servers
to filter for spam, and it gets filtered off appropriately at delivery time. I
never use the Thunderbird junk mail filters.
(In reply to comment #3)

I use Thunderbird to access the same account on various different machines.  When my Mac has been 'sleeping' and I wake it up, and it fetches mail that has already been viewed on another machine, the inbox correctly shows the messages as read, but the dock icon counts them as new.  The problem is that the dock icon is not getting its information from the right source, like the inbox is.
Yes: I have the same issues with the Thunderbird icon for Mac OS X:

I am using a recent Thunderbird 1.5 (just recently downloaded on Mac OS 10.3.9.

Typically, when the first message comes in, the icon will display that I have 1 message. But subsequently, if there are other messages that come in and I haven't opened the Thunderbird e-mail, then the Thunderbird mail icon will just continue to show 1 message. It won't indicate that there  are now more than one e-mail in the inbox.

The other issue is that when I wake up the computer after being away from it... the total  number of messages do show up in the icon display ... but these also include the ones in my junk filter or trash ... so I might think that I have five e-mails, but in fact, they might all be junk mail ones in the trash. There should be an option to allow the icon to only show new mail that is in the Inbox and not junk or trash.

Small irritations, but it would be nice to have them fixed!
Similar behaviour on 10.4 with Thunderbird 1.5.0.8 (20061025), except count never exceeds 1. This is with multiple IMAP accounts and a POP account. As a side note the badge disappears after qutting and reopening even when messages remain unread. This is not the expected bahviour.
I see this daily using version 2.0pre (20070321). If 2 or more pieces of mail come into my m.c account, it shows '1' in the badge in the dock. Every time.

Is there a reason this bug isn't confirmed? Or is that only because it's potentially a dupe of bug 261201?
QA Contact: general
Same here. I am running MacOSX 10.4.10 with version 2.0.0.4 (20070604). Simple one account setup, and I constant have a one green circle "Unread" message count on my dock icon, while all message already marked READ.

This bug is too obvious that starts to get annoying. Can some one in dev team take a look?
I also see this on both of my macs running 10.4.10 and TB 2.0.  New mail notifications in the dock are usually the wrong count (1 or 2 when I have 10 unread messages) and after reading mail, the green circle won't go away until I manually click on Get Mail.
Ditto that this is really annoying -- it's almost enough for me to move to another client.

I haven't seen an unread count that's actually wrong -- what I'm seeing (over and over again) is unread counts that are out of date. The It seems that the unread count in the dock is only updated on:

* manual mail check
* exit and/or re-entry into IMAP mail box (selecting a different mailbox sometimes works; sometimes you need to click back to the mailbox that had or has unread mail)

The unread mail counts on the mailbox list are fine -- which makes me think it's just a question of updating the dock icon on more triggers.

(IMAP accounts on 10.4.10 and TB 2.0.0.6)
Confirming due to comments #9, #10, #11, #12.

Reporters, could you please report if the issue still occurs in the latest supported version 2.0.0.12 or trunk nightlies?
Status: UNCONFIRMED → NEW
Ever confirmed: true
MacOS X.4.9 - Thunderbird 2.0.0.12 AND Thunderbird 3.0a1pre (2008032603)

The issu still occurs.
For exemple :
- I have 3 unread messages. The green circle shows 3. I read one message -> The green circle disapears instead of showing 2.
- I have 2 unread messages, the green circle shows 2. I go in another box (trash, drafts or anything else) -> the circle disapears.
- After the disapearing of the circle (with 2 unread messages), if I receive one message, the green circle indicates 1 instead of 3.
(In reply to comment #14)
> The issue still occurs.

Yeah, I'd have to mention that I do notice the behaviour as well on Mac OS X 10.5, Thunderbird 2.0.0.12.

(I'm staring at 2 unread messages in Thunderbird but the dock icon doesn't show any numbers at all.)

Nominating -> wanted-thunderbird3 to keep it on the radar.

Emre, do you notice this? (I'm thinking it's pretty common)
Flags: wanted-thunderbird3?
I digged around in the code a bit an found what I think is the reason for the wrong number shown in the Dock icon:

To trigger it set up the following scenario:

- A mail filter rule that moves messages with "bugzilla" in the subject line to a folder "Bugzilla"
- Send yourself three mails:
  - Mail A matches the filter and is moved to the folder Bugzilla
  - Mail B and C do not match the filter and stays in the inbox
  - Mail A and B should be marked as junk by the junk filter. Both mails should be moved to the Junk-folder automatically.

This setup should result in 1 new message in the inbox but the icon showing "2".

As far as I could track the problem down, this is what happens:

The mails are downloaded, normal filters and junk-filtering runs. nsMsgLocalMailFolder::OnMessageClassified correctly detects the junk and moves them to the junk folder. However, mail A was moved to the folder "Bugzilla" and GetNumNewMessages (called at the end of OnMessageClassified) returns 0 instead of 1. OnMessageClassified then sets oldCount (0) - spamCount (1) with SetNumNewMessages for folder "Bugzilla".

When the Dock icon is regenerated, it uses GetNumNewMessages to find the following:

- Inbox: 2
- Bugzilla: -1

(It started as Inbox: 3 and Bugzilla: 0 and was correctly decremented by 1 for each Junk mail. Unfortunatelly moving mail A from Inbox to Bugzilla did not update the counts to Inbox: 2, Bugzilla: 1. This in turn would have resulted in the desired Inbox: 1, Bugzilla: 0)

In nsMsgDBFolder::GetNumNewMessages you will find a line "if (num > 0) // it's legal for counts to be negative if we don't know" which makes GetNumNewMessages ignore the -1 from "Bugzilla". Voila, there we have the wrong badge.

As I am writing this I realize that the root of this problem is not in GetNumNewMessages but in the filter code that does not update the number of new messages in "Bugzilla". It also seems to me that this bug is not Mac-only.
Blocks: 438257
This makes it hard to use GTD-style methods, where you're likely to want to keep the dock icon at zero; increasing severity because this seems like a reasonably common use case.  Marking as wanted+.
Assignee: mscott → nobody
Severity: minor → normal
Flags: wanted-thunderbird3? → wanted-thunderbird3+
This behavior is confirmed on Mac OS X 10.5.4 and Shredder 3.0a2pre (2008071903).

The badge number in my dock is almost always either a 1 or a 2 or completely absent.  I have 2 IMAP accounts and this is the only machine I use to access them.  The unread message count seems to be correct in the TBird interface, its just the badge that is habitually too small.
Attached patch Non-perfect patch (obsolete) — Splinter Review
The attached patch solves the problem for me. It does this in a dangerous way: It simply no longer ignores negative message counts but adds them to the number of new messages. This way the messages moved by a filter are no longer lost for the message count. Unfortunatelly, -1 is documented to be the "no message count available" flag. So -1 is used for real while it is not valid.

In this (rare? never happening?) case the number of messages is unknown so Thunderbird could not show the correct number anyways.
b2 would be a good time to get this in.
Target Milestone: --- → Thunderbird 3.0b2
I think the problem is more complicated than this. This patch also only deals with pop3 messages, not IMAP.

Also, are we trying to show an unread count, or a new count (messages arrived in the current session)? 
Also, rkent is working on overhauling our "new" message handling and might have some input here.
This might be a kludge you don't want, but what about adding a variable to nsMessengerOSXIntegration to remember the last badge count, and add to it the new count instead of doing a pure 'New' total each time, which will zero out the previous count?  This would give you the total number of messages since you last were in the app.  Fixing bug 459483, which I hope to do as well, should make this number more meaningful.

In reply to c14 above, I don't think it makes sense to decrease that count when you start clicking on messages in the app.  Clicking in the app gives you that info (i.e., the unread mail is there, bolded, etc.).  I think removing the icon badge when the app gets focus is the right behaviour.

I'm happy to do this if there isn't some other plan in the works deeper in the message handling code.  rkent?
Concerning "some other plan in the works deeper in the message handling code" please look at my proposal at https://wiki.mozilla.org/User:Rkentjames:NewCounts  (Briefly, I don't think that the problem is going to be fully solvable, including a number of related bugs, until you separate the concept  of "New" from "Needs Biff notification"). To do that requires more message flags. As a related issue, the current patch in bug 461479 introduces the new concept of volatile process flags, which would be the logical place to put some of these needed flags.

This is more of a proposal than a plan, however. So let me update the status of this from my perspective. I started working in this area of New Flag management as a community service project, which means it was not that important to me personally, but represented an area where I could contribute to bugs that were marked Wanted or Blocking. But the changes that I have proposed have not generated sufficient discussion or interest - not to mention something resembling a consensus - to justify my moving forward with the large, disruptive patch that would be required to implement this. So the proposal sits sort of as a half-way plan, not enough of a plan to actually move forward, but enough to discourage others from working in the area - not exactly an ideal situation. I guess what I am hoping is that the issues will come up again as the drivers reconsider the TB3 bugs, and generate enough interest then to have the discussion that IMHO is needed.

Unfortunately with the existing definitional confusion, it is difficult to predict the effectiveness of various little fixes designed to alleviate specific symptoms. In particular, I can't anticipate all of the consequences of your proposal, nor can I even test it since it is mac-specific.
I would value some comment on rkent's proposal above, and the likelihood that this, or something like this (e.g., such sweeping backend changes) will happen.  Clearly this would change what I'm going to say below.

The current status of the badge count on Mac is horrible, and something has to be done to make this better.  I can see some low-hanging fruit that should be picked in the short term at least so that we can get a more credible story with regard to current message count in the app.

Before I attempt it, my question is this: what should that number be?  I realize this is both a naive and at he same time obvious question.  To me, this number represents "Total number of New Mail since you lasted visited the app."  Which means that if I have multiple mail accounts, the cumulative total change over time since I was last working in the app.

Am I wrong?  What should it be, and I'd like to try and make it come closer to that mark.  Not getting this perfect for all edge cases can't stop us from making it work for the general case.  I'm motivated to try for this, at least.
It sounds like you want it to display new mail.  I want it to display unread mail.  I can unmark bug 374543 as a duplicate if you wish to treat that as a separate bug.
:humph asked in IRC about Mail.app badge behavior for comparison.  Here is what I found:

Two IMAP accounts (A & B), looking at inboxes only.  A had five new unread messages, B had no unread messages.  The badge showed 5.

Sent a mail to B account.  Counts in the side bar now show 5,1.  Badge shows 6.
Right clicked an old message in B and marked it as unread, sidebar now shows 5,2.  Badge now reads 7.

Went to a non-Inbox folder in B and marked a message in it as unread.  Does not show up in the Inbox section.  Shows up below in the sidebar with a 1.  The badge still reads 7.

When Mail.app is started, the sidebar shows the cached values for unread mail.  The badge does not appear in the dock icon.  After a few moments, Mail.app checks for mail.  After that check, the new mail notification sound plays and the badge now shows the updated unread count, even if it is the same as what was previously counted.
Assignee: nobody → david.humphrey
Depends on: 465381
Target Milestone: Thunderbird 3.0b1 → Thunderbird 3.0rc1
Attached patch Fix dock counts patch (obsolete) — Splinter Review
This uses the stuff I recently did to add the NewMail atom, such that all new mail triggers notification, not just on setting biff.  I'm using the new count for this session, so it will be cleared when the user returns to the app, and then zeroed.
Attachment #331402 - Attachment is obsolete: true
Attachment #362040 - Flags: review?(bienvenu)
(In reply to comment #29)
> so it will be cleared when the user returns to the app, and then zeroed.

I'd like someone to clarify whether this bug is limited to fixing the new mail counter, or whether it includes having an unread mail counter.  I duped bug 374543 to this one on the assumption that it included the latter.  I will un-dupe if not.
(In reply to comment #30)
> (In reply to comment #29)
> > so it will be cleared when the user returns to the app, and then zeroed.
> 
> I'd like someone to clarify whether this bug is limited to fixing the new mail

Well, un-duping the other one just delays this conversation.  I'm not sure there is a right answer here, but I know that I see it differently from you.  Do we want a pref to let users decide between the two cases?  I'm not even sure it's that simple

Tell me what should happen, in your case, pretending a user with 2 accounts:

Account A: 5 Unread, no new mail
Account B: 10 Unread, no new mail

Now, the following occurs:

* Account A gets 2 new mails (Unread count = 7, new mail count = 2) 

What should be on the dock?

1) 17
2) 2
3) 7

Also, when does the dock get badged with this count?  If it isn't related to new mail number, should it happen at startup?

It's possible for me to choose between either of nsMsgDBFolder::GetNumUnread and nsMsgDBFolder::GetNumNewMessages when seeding my local count.

Obviously my vote is what's in the patch.  What do others think?
#1: 17. That's how almost all other mail clients I've ever used have worked. I often leave unread messages in my inbox to look at later. I want the notification.

This bug is about the number of "unread messages" (as filed anyway). Not sure when it changed to be about only "new" messages.

Really though, most users with unread messages will assume they're new; those things mean the same to end users and changing that metaphor will be confusing (and you'll get a bunch of bugs filed on it).
Yes, from my point of view, 17 is the right answer.  And I agree with Sam that I've never used a mail client that gives you any other answer.

In fact, I was so confused when I filed bug 374543 that it was only after spending time trying to document the behavior (so I could file a bug) that I realized what it was trying to do.  In fact, I only realized that after I filed the bug, until then it just felt completely random to me.

Part of the problem is that it's not really feasible to display "new mail", because "new" is not something you can calculate on its own--there is an implicit "since X" that's inherent in the concept.  So, you have chosen to display "new since the last time I happened to command-tab over to Thunderbird except in the case where Thunderbird was the frontmost app already", but why is that the correct event to count from?  You're really trying to approximate "new since the last time you were paying attention to the number", but of course you can't calculate that.

I'd wager, though, that even if you were to require gaze-tracking and have me wear a headset so you can monitor my alertness, I'd still just want the number of unread messages.  Although you'd probably be able to approximate the "new" number more closely, it doesn't matter in my workflow how many mails "just arrived".  When I decide to read mail I will process the entire set of unread messages (GTD-style).
(In reply to comment #33)
> Yes, from my point of view, 17 is the right answer.  And I agree with Sam that
> I've never used a mail client that gives you any other answer.

Cool.  It's helpful to see that others feel the same way as you.  My goal isn't to be right, but to fix this.  So if this is "the way" then let's figure out how to get it done properly.  I'm motivated to finish this before tb3. 
 
> Part of the problem is that it's not really feasible to display "new mail",
> because "new" is not something you can calculate on its own--there is an
> implicit "since X" that's inherent in the concept.  So, you have chosen to
> display "new since the last time I happened to command-tab over to Thunderbird
> except in the case where Thunderbird was the frontmost app already", but why is
> that the correct event to count from?  You're really trying to approximate "new
> since the last time you were paying attention to the number", but of course you
> can't calculate that.

"Since x" is clearly defined, at least in the code, and that was what I was working from: x is "since biff."  So, yes, we can say that you have received N new mail messages, and when we do, we mean "since biff."  That concept is bolted deep into how the mail is stored in the backing mail db.  And this gets cleared not in an arbitrary way, but when biff is cleared.  That's what happens now, and I was trying to make it work properly.

I also notice the unrest over at postbox over the same thing: http://getsatisfaction.com/postbox/topics/mac_osx_dock_icon_unread_count_incorrect

So if Mail.app is our baseline for what Mac users expect, help me understand what we should do here (I've never used it).  Do we count all folders?  Just the Inbox?  If we exclude some folders, how do we do it with sane defaults such that you don't force users to go and hand-pick folders to include/exclude (fine for an extension, but crappy for the general case, imho).

Do we badge based on the count rather than an event?  That is, do we badge at startup vs. biff?  (For bienvenu, or others who can help me think about the code for this...which events do we then care about, in order to refresh this count?  Can we just ask the root folders for counts that go deep, or do we need to keep our own count, updating based on events from the folders we care about?).

Bryan/DavidA, I'd love some input from you guys here.  I'd really like to solve this mess, as it's embarrassing for users who try the app and are instantly greeted with an erroneous count.

> arrived".  When I decide to read mail I will process the entire set of unread
> messages (GTD-style).

I'll be honest that I don't use GTD-style, nor did I know it by this name.  However, it was neat to read about it, and it made me think that you could do some cool workflow stuff in TB based on it.
(In reply to comment #34)
> So if Mail.app is our baseline for what Mac users expect, help me understand
> what we should do here (I've never used it).  Do we count all folders?  Just
> the Inbox?  If we exclude some folders, how do we do it with sane defaults such
> that you don't force users to go and hand-pick folders to include/exclude (fine
> for an extension, but crappy for the general case, imho).

We count every Inbox (if there are multiple accounts, we count all of them).

> Do we badge based on the count rather than an event?  That is, do we badge at
> startup vs. biff? 

Mail.app will show me new messages on startup and keep the number up-to-date throughout my session. When that gets to 0 (and the badge disappears), it will start based on any new email. I'm not sure if that's what "we're" supposed to do, but that's what I expect (and, fwiw, it's what other mail apps do).
(In reply to comment #34)
> "Since x" is clearly defined ... x is "since biff."

humph, I have not been following your patches closely, but I thought I would chime in here since there are levels of complication that I do not see being discussed.

IMHO, "Since x" is not so clearly defined. There are at least three possible variations of this.

First, what exactly do you mean by BIFF? It is sometimes defined as the process of periodically checking for new mail, and sometimes as the notification that new mail has arrived. That's two variations. I think in the current code, if you request a manual check for mail, then the NEW flags get cleared on a folder, and the NEW counts are reset, even though no BIFF notification has occurred (I could be wrong on this, or it could have changed with recent patches).

The third variation is that the NEW flag is cleared when a user opens and then leaves a folder. This is useful when you want to use unread as a proxy for "I still need to process this in some way" but still be able to see when something new arrives in a folder.

So you really need to distinguish three concepts within the existing code: the unread flag, the new flag, and the new count. The new count is NOT the same as the count of messages that have their NEW flag set in a folder. The unreadcount IS always the same as the count of messages with the UNREAD flag.

For the current bug, you are moving toward using the unread count as the number in the dock icon. That should be fairly straightforward to do, but you are not taking advantage of the NEW statuses, which in spite of their confusion, are a decent concept, particularly for people who do not want to clear UNREAD for emails.

So here's the typical scenario that I think the code was designed to support, even though it doesn't really work. Youe code is checking regularly for new mail. Whenever a new mail occurs, a small popup occurs that shows the subject of new mails. For each new mail, it should only show up once (this does not work at the moment). Meanwhile, you've been ignoring your client (or perhaps you were away when the notification occurred), so a number of emails have been received since you've looked at things (defined as opening then closing a folder with new mail in it.) The count in the dock icon should show you the accumulated number of new mail notifications that you have received but not processed in any way. When you finally get around to looking over your emails, then you start opening folders to see what is there. Some you read, others you don't read. When you exit the folder, then the NEW flags are reset on all emails in the folder. IMHO, the dock icon should then change its count by that amount. The dock icon count should go to zero when you have no mails left with the NEW flag set.

In other words, IMHO the dock icon should represent the sum of NEW flag counts for all folders. NEW flag should also not be reset when you manually check for new mail, but only when you exit a folder.
> First, what exactly do you mean by BIFF? It is sometimes defined as the process
> of periodically checking for new mail, and sometimes as the notification that

I'm not having a theoretical discussion about how people use the term, whether correctly or incorrectly.  I'm talking about the mechanics of how nsMsgDBFolder::kBiffStateAtom gets used, and rightly or wrongly, it is pretty clear when it gets set.

> So you really need to distinguish three concepts within the existing code: the
> unread flag, the new flag, and the new count. The new count is NOT the same as
> the count of messages that have their NEW flag set in a folder. The unreadcount
> IS always the same as the count of messages with the UNREAD flag.

If we agree that Unread is what we care about, this complexity is not an issue.

> For the current bug, you are moving toward using the unread count as the number
> in the dock icon. That should be fairly straightforward to do, but you are not
> taking advantage of the NEW statuses, which in spite of their confusion, are a
> decent concept, particularly for people who do not want to clear UNREAD for
> emails.

I think this will only server to further confuse users.  If we do Unread, let's do Unread as is most commonly understood.

> So here's the typical scenario that I think the code was designed to support,
> even though it doesn't really work. Youe code is checking regularly for new
> mail. Whenever a new mail occurs, a small popup occurs that shows the subject
> of new mails. For each new mail, it should only show up once (this does not
> work at the moment). Meanwhile, you've been ignoring your client (or perhaps

See bug 460287 which changes all this, and has landed.

> you were away when the notification occurred), so a number of emails have been
> received since you've looked at things (defined as opening then closing a
> folder with new mail in it.) The count in the dock icon should show you the
> accumulated number of new mail notifications that you have received but not
> processed in any way. When you finally get around to looking over your emails,

While I agree with you, and my patch does just this, Dan/Sam don't think this is the expected behaviour for most Mac users.

> In other words, IMHO the dock icon should represent the sum of NEW flag counts
> for all folders. NEW flag should also not be reset when you manually check for
> new mail, but only when you exit a folder.

Yet another interpretation of what this single number should represent!  I'm tempted to get in line behind Dan, and forget tracking New based on Biff or some other event.

So last night I did an experiment along the lines of what Dan is suggesting, and I can get the info I need.  However, I have two technical issues I need advice on:

1) On startup when I seed the unread count, are there types of accounts I should ignore?  Should this include NNTP or other non-mail accounts?

2) Given the set of folders for 1) I need to know which folders to exclude, if any.  Right now TotalUnreadMessages atoms come in for every folder, and I expect this is going to wrong.
> We count every Inbox (if there are multiple accounts, we count all of them).

Alright, I'll do a patch for this and we can see if it does what people want.
This patch does what Dan/Sam suggest, and maintains a persistent badge count for unread mail in inboxes.  This is no longer tied to biff or new mail.
Attachment #362040 - Attachment is obsolete: true
Attachment #362217 - Flags: ui-review?(clarkbw)
Attachment #362217 - Flags: superreview?(bugzilla)
Attachment #362217 - Flags: review?(bienvenu)
Attachment #362040 - Flags: review?(bienvenu)
(In reply to comment #37)
> > In other words, IMHO the dock icon should represent the sum of NEW flag counts
> > for all folders. NEW flag should also not be reset when you manually check for
> > new mail, but only when you exit a folder.
> 
> Yet another interpretation of what this single number should represent!  I'm
> tempted to get in line behind Dan, and forget tracking New based on Biff or
> some other event.

Well, it's the correct interpretation ;)
I think it's very logical to show the NEW messages, it's only on startup that's not as relevant. Maybe we should be showing both counts, like "3 NEW (now 26 UNREAD total)", with some better phrasing of course...
(In reply to comment #39)
> Created an attachment (id=362217) [details]
> Badge unread count for all inboxes
> 
> This patch does what Dan/Sam suggest, and maintains a persistent badge count
> for unread mail in inboxes.  This is no longer tied to biff or new mail.

Thanks, David.  I don't know Tb internals to tell if your patch is correct, but from your description it is exactly what I want. Though, I'm sad I apparently wasn't able to convince you that it's actually the right thing to do, just that many people want it.

I'm not sure why you're framing the unread count badge as a Mac-specific behavior, TBH.  Every mail, news, rss client, and biff program I can remember using on *nix, windows, and on the web also use some variation of an unread count whenever a persistent count is displayed.  Though I haven't used windows for quite some time, is this "new" behavior now common in mail clients there?

Saying "new" is "new since biff" is kind of missing the point.  Obviously you have some algorithm (buggy or not) to calculate what "new" means to Tb, but I was talking about the behavior as it appears to the user.  No one looks at the dock and thinks "oh, I have 3 new mails since nsMsgDBFolder::kBiffStateAtom got cleared."  So from "since biff" is just an implementation detail.  I still think that you're more or less just approximating "new since the last time you looked."

By the way, I think some "new" count would actually work pretty well as a toast/growl notification.  It could be something similar to what was suggested in comment #40, "3 new messages just arrived".
> Thanks, David.  I don't know Tb internals to tell if your patch is correct, but
> from your description it is exactly what I want. Though, I'm sad I apparently
> wasn't able to convince you that it's actually the right thing to do, just that
> many people want it.

Don't be sad.  I'm a firm believer in "When in Rome," and perhaps my many years of using TB, with a broken count indicator, has just warped me ;)  I want users to use TB and get what they expect.  So this was a valuable discussion to have, and if I didn't agree with you, I wouldn't have changed my patch.

> I'm not sure why you're framing the unread count badge as a Mac-specific
> behavior, TBH.  Every mail, news, rss client, and biff program I can remember
> using on *nix, windows, and on the web also use some variation of an unread
> count whenever a persistent count is displayed.  Though I haven't used windows
> for quite some time, is this "new" behavior now common in mail clients there?
> 
> Saying "new" is "new since biff" is kind of missing the point.  Obviously you
> have some algorithm (buggy or not) to calculate what "new" means to Tb, but I
> was talking about the behavior as it appears to the user.  No one looks at the
> dock and thinks "oh, I have 3 new mails since nsMsgDBFolder::kBiffStateAtom got
> cleared."  So from "since biff" is just an implementation detail.  I still
> think that you're more or less just approximating "new since the last time you
> looked."

Yeah, this is moot, I've put up the new patch.  My comments were more a discussion of the code as is vs. what should be.

> By the way, I think some "new" count would actually work pretty well as a
> toast/growl notification.  It could be something similar to what was suggested
> in comment #40, "3 new messages just arrived".

Agreed.  I've done this for growl alerts and you can get it in nightlies.

Thanks for a good discussion about this issue.  I'm pleased with this outcome.
I'm not likely to have much time for this bug in the near future, especially while driving the b2 release.  With a light scan of this bug I like the current direction, but I haven't spent time to investigate the consequences of the change.  I'm glad everyone is remaining civil in such a large bugzilla discussion. :)
Comment on attachment 362217 [details] [diff] [review]
Badge unread count for all inboxes

In general this looks good, and I agree we should give the new numbers a try and see what folks think.

>+  // the CGImage with the numbers. Only badge if unread count > 0.
>+  if (mUnreadTotal < 1)
>+    return NS_OK;

What abount un-badging if unread count = 0?

>+  nsCOMPtr<nsISupportsArray> mFoldersWithUnreadMail;

Please use nsCOMArray here - there's an ongoing set of bugs to remove nsISupportsArray, so I don't want to add another instance of it unnecessarily.
Attachment #362217 - Flags: superreview?(bugzilla) → superreview-
If I get 10 new messages and they all get moved to other folders via incoming filters, will that make my badge count 0? That doesn't seem terribly useful for filters users...
(In reply to comment #45)
> If I get 10 new messages and they all get moved to other folders via incoming
> filters, will that make my badge count 0? That doesn't seem terribly useful for
> filters users...

In my case, that would actually be the desired behavior (I don't want to be notified of stuff I am auto-archiving), but I can see how some users might want the badge to display all unread mail, regardless of folder.

I could probably get used to either, TBH, though given a pref I would set it to inbox-only.
I'd want (or at least am used to) the complete opposite to comment #46 :) Maybe this should be a hidden pref? Or at least something that someone can write an extension to change.
The question of filtering makes me wonder what happens to messages auto-marked as junk, would they add to the count?

My interpretation of the meaning of the count on the dock icon is to indicate the number of 'things' in need of attention.  I tend to believe this number correlates to the number of unread messages in the Inbox.  And if a person is filtering messages away from the Inbox it is so that those messages are not attention grabbers.  David makes a great point about how some filter users would also expect otherwise.  I'm wondering if we should be expecting growl to handle the case of filters or some kind of preference is needed?

In general I think I can ui-r+ this assuming we understand what happens to the alternate use case.
junk messages are marked as read, so they don't count - we try very hard not to count junk messages in any case.

I mostly filter important e-mail away from my Inbox - bug mail, mail directly to me from people I know, various mailing list mail, etc. What's left in the inbox is usually unsolicited and of little value.

If we had a pref that would let the badge count be just the sum of counts of new messages, I'd be happy with that.
Ditto (well, assuming that "new" was a typo for "unread").

I know of three separate ways of working: filter good stuff into what are essentially separate inboxes the way bienvenu and I do, filter out uninteresting crap the way Thunder does, filter out uninteresting crap and mark it read (plus the combinations, because some people have to be difficult).

Sum of counts would cover 1 and 3, and haven't I seen a separate bug for a "don't notify" filter action to cover Thunder's 2 without having to mark read? A binary pref might be all we have the time to write, but I don't think "never notify for filtered mail" would be anywhere near as useful as a filter action, since with my bugmail filtered I would then be unable to not be notified about a boring mailing list, and Thunder would be unable to be notified if he started filtering one class of interesting mail. (Of course, then having a legacy binary pref would complicate things when someone did get the time to write a filter action...)
(In reply to comment #50)
> haven't I seen a separate bug for a
> "don't notify" filter action to cover Thunder's 2 without having to mark read?

FiltaQuilla includes a "don't notify" action that simply decrements the new message count. I have a mix of these cases, that is there are some important messages that I filter, and some unimportant messages that I filter. Sometimes I tag instead of copy to folder. So count me as a difficult person. But I also use a play sound filter action (from ToneQuilla) for notifications of important messages, so I control it pretty precisely.

You are not going to make everyone happy unless you support all cases. You could do that with a preference whether the badge count reflects new or unread, plus filter actions to adjust those on a per message basis. The filter actions are easily implemented as custom filter actions if the relevant counts are available in js-reachable attributes.
"notify" and "action" are not words that I associate with the unread message count.  Those words only make sense when talking about 'new' messages.

A global option such as 'show unread from all folders' would be sufficient to solve the use-case described in comment #45.  If you wish to make it more flexible (at the cost of being more complex/confusing), you can make it a per-folder preference: 'include this folder when counting global unread messages'.  Neither of these require or are affected by actions or notifications, you just have to decide how complicated you want the UI to be.

My suggestion is to pick a default (count only inbox, or count all folders), and make sure the other option is possible via about:config/an extension (bonus points for making the per-folder one possible via extensions too).
(In reply to comment #52)
> My suggestion is to pick a default (count only inbox, or count all folders),
> and make sure the other option is possible via about:config/an extension (bonus
> points for making the per-folder one possible via extensions too).

The intention of inheritedFolderProperties in bug 478360 is precisely to make it easy for extensions to modify options that might be global, per-server, or per-folder.
Comment on attachment 362217 [details] [diff] [review]
Badge unread count for all inboxes

I like using this behaviour for the default.

I also think we'll need a preference for 'show unread from all folders' as well.  Not sure if that should be this bug or not.

Beyond that I think we want to rely on extensions like the *Quilla family to provide really fine grain tools.

I'm making this breakdown of use cases because I think it maps well to the amount of work involved in each setup.  

For the simplest setup with no filters, a count of the Inbox is the clear choice.  For the more complex setup of a person who has created filters that move messages to other folders I believe we can rely on them also finding the preference to count all unread messages.

And for a mail setup beyond that our power mail users should be comfortable with using extensions.

- Also this reminds me we should get some of the *Quilla family into a.m.o. and on the recommended lists.
Attachment #362217 - Flags: ui-review?(clarkbw) → ui-review+
Nominating for blocking, as this is such an in-your-face part of our UI for mac users.  I also think I can work the above into a new patch, and will do so as soon as I can clear some other work from my plate.
Flags: blocking-thunderbird3?
Just a comment; the PostBox team have implemented a fix for this that works great on the Mac platform.  It works exactly how I expected it to.  Just thought you might like to know.

Later...
  Richard
David's analysis seems right to me; marking as blocking+.
Flags: blocking-thunderbird3? → blocking-thunderbird3+
Comment on attachment 362217 [details] [diff] [review]
Badge unread count for all inboxes

clearing review request based on standard8's minus, and the other comments.
Attachment #362217 - Flags: review?(bienvenu)
Attached patch Patch v3 (obsolete) — Splinter Review
Based on discussions above, and following Bryan's outline, here is a new patch.  This patch works like this:

1) By default, report unread counts for all inboxes
2) Allow users to switch counts to be for all folders with a pref
3) Allow extensions to explicitly ignore folders from being counted, and also to modify the text on the badge icon (including suppressing it completely).

There seems to be an issue that I hit every now and again, where the values cached in panacea.dat are not right, and I don't get a proper unread count right away--I think this is another issue.  Also, I sometimes find that it takes tb a moment to "warm-up" and start getting unread count updates.  This may be related to the earlier issue.  In any event, those problems, if real and not related to just me and my testing, aren't part of the changes I've made.

Other details

* added pref - "mail.notification.count.inbox_only" (not sure if this name is good), so that users can get count for all folders or inboxes only.  Default is inboxes only.

* removed mBiffIconVisible, mFoldersWithNewMail, CountNewMessages(), uses of and includes for nsIWeakReference and nsISupportsArray

* Added hooks for extensions (observer service notifications):
** "before-count-unread-for-folder" - chance to stop a folder (uri) from being counted in the total?
** "before-unread-count-display" - chance to modify the string going on the badge (e.g., change "1000" to "100+" or suppress altogether with "")

* various whitespace fixes
Attachment #362217 - Attachment is obsolete: true
Attachment #370052 - Flags: superreview?(bugzilla)
Attachment #370052 - Flags: review?(bienvenu)
(In reply to comment #59)
> There seems to be an issue that I hit every now and again, where the values
> cached in panacea.dat are not right, and I don't get a proper unread count
> right away--I think this is another issue.  Also, I sometimes find that it
> takes tb a moment to "warm-up" and start getting unread count updates.  This
> may be related to the earlier issue.  In any event, those problems, if real and
> not related to just me and my testing, aren't part of the changes I've made.

I've been looking into this after our discussion on IRC. I tried doing a mark all read on some of my gmail folders that accumulate unread messages, and I ran into the problem pretty quickly. mNumPendingUnreadMessages goes negative, which it should not, and that causes various issues. It also seems that marking all read isn't quite working with gmail imap - many if not all the messages show up as unread when we startup again. I'm not sure if this is because we're not actually storing the /seen flag on all messages (though I think we are), or if gmail is losing the flag changes (perhaps we're not cleaning up the connection on shutdown, and gmail is losing the state changes).
(In reply to comment #60)
> into the problem pretty quickly. mNumPendingUnreadMessages goes negative, 

likely this is what bug 485789 was for me.
Status: NEW → ASSIGNED
Version: unspecified → Trunk
Whiteboard: [penelope_wants]
Comment on attachment 370052 [details] [diff] [review]
Patch v3

I'll try this out on my Mac (I don't run my Mac that often, so I'm not the best tester for this, however).

One nit - +    PRBool countInboxes = PR_TRUE;

this should be called something like onlyCountInboxes or countInboxesOnly.
> I'll try this out on my Mac (I don't run my Mac that often, so I'm not the best
> tester for this, however).

I'll keep writing Mac patches until you run your Mac by default ;)

re: nit -- I'll wait until you hit me with other things to fix and do it all at once.
(In reply to comment #64)
> 
> I'll keep writing Mac patches until you run your Mac by default ;)
Keep 'em coming! :-)
Comment on attachment 370052 [details] [diff] [review]
Patch v3

Thx for the patch, sorry for the delay in looking at it. I played with this a bit - here's what I think it does:

The badge count is the sum of unread messages across all inboxes/folders that had their unread count change in the current run of Thunderbird.  So at startup, it's the sum of unread in inboxes/folders that got new mail at startup, and as the user opens other folders and changes the unread counts in those folders, their unread counts start to contribute to the total.

So you could very well have 100 unread messages in an inbox when you shutdown, restart, and not get new mail in that inbox, and have the badge count go up when you open that inbox.

I suspect you want to make an initial pass at all the unread counts for all the inboxes/folders in order to make the count consistent. Thoughts?
Attached patch Patch v4 (obsolete) — Splinter Review
After discussions with David on irc, here's a new patch:

1) On startup (using "mail-startup-done"), count unread for inboxes or all folders, depending on pref.  NOTE: I special case Junk and Trash to be ignored by default (see below).
2) Monitor changes to folders thereafter and update (not tied to biff)
3) Allow users to switch counts to be for all folders with a pref
4) Allow extensions to explicitly ignore folders from being counted, and also
to modify the text on the badge icon (including suppressing it completely).

Other details

* added pref - "mail.notification.count.inbox_only" (not sure if this name is
good), so that users can get count for all folders or inboxes only.  Default is
inboxes only.

* removed mBiffIconVisible, mFoldersWithNewMail, CountNewMessages(), uses of
and includes for nsIWeakReference and nsISupportsArray

* Added hooks for extensions (observer service notifications):
** "before-count-unread-for-folder" - chance to stop a folder (uri) from being
counted in the total?
** "before-unread-count-display" - chance to modify the string going on the
badge (e.g., change "1000" to "100+" or suppress altogether with "")

* Added special cases for Junk/Trash folders to be ignored, but give extensions the chance to include.  Should I add any others?

* Filed bug 487820 to get "mail-startup-done" added to SM.  Not sure if this needs to block on that?

* various whitespace fixes
Attachment #370052 - Attachment is obsolete: true
Attachment #372085 - Flags: superreview?(bugzilla)
Attachment #372085 - Flags: review?(bienvenu)
Attachment #370052 - Flags: superreview?(bugzilla)
Attachment #370052 - Flags: review?(bienvenu)
I haven't read your patch, so maybe it is obvious, but what is the method that a custom filter action could take to suppress this counting on a per-message basis? The custom filter action with have the nsIMsgDBHdr object available, and will be running from js.
sorry for the long delay - I'm back on my mac today and running with this patch.
I ran with this about a week ago, and I would still see some negative unread counts (e.g., -80 unread). I didn't have time to look into it, however.
Apologies for not looking at this sooner. In an effort to try this patch out I've produced a try server build:

http://s3.mozillamessaging.com/build/try-server/2009-06-22_04:32-bugzilla@standard8.plus.com-st8-docicon/bugzilla@standard8.plus.com-st8-docicon-mail-try-mac.dmg

So far I have noticed the following problems with the patch:

- Have Thunderbird open, get one new email into inbox, don't read it. Restart Thunderbird, dock icon displays 1, go to inbox and select the mail, dock icon displays -1.

- When marking unread on IMAP folders, count is doubled (though goes back down correctly after a short delay if you mark it as read again).

- Setting mail.notification.count.inbox_only
doesn't count rss feeds on initial start up, but then when you read on it decrements the count (hence negative possible).

I'll keep testing for a day or so and see if I spot anything else.
Hardware: PowerPC → All
Does this interact with the "hit space bar for next unread message" function? I note that it too has ..issues... I can know there is an unread message, but I don't get asked about moving to that folder.
Attachment #372085 - Flags: superreview?(bugzilla) → superreview-
Comment on attachment 372085 [details] [diff] [review]
Patch v4

So far, the only other thing I've picked up on are that "unread" mails in drafts and outbox get picked up as unread on the doc icon (if the pref is flipped).

Given that space, or Next, doesn't move onto those emails, I'm not sure they should be counted by the doc icon.
Attached patch Patch v5 (obsolete) — Splinter Review
This patch addresses your issues, and adds some assertions for the case of negatives (which I think I've gotten rid of).  Please have a look and let me know if this is ok.
Attachment #372085 - Attachment is obsolete: true
Attachment #390043 - Flags: superreview?(bienvenu)
Attachment #390043 - Flags: review?(bugzilla)
Attachment #372085 - Flags: review?(bienvenu)
From initial testing generally looking a lot better, though for some reason unread emails in my IMAP inbox are counted twice. No other folders appear to suffer from this problem (it is the same for 4 accounts across 3 servers).
(In reply to comment #76)
> From initial testing generally looking a lot better, though for some reason
> unread emails in my IMAP inboxes are counted twice. No other folders appear to
> suffer from this problem (it is the same for 4 accounts across 3 servers).

Note: I had mail.notification.count.inbox_only set to false when testing this, however the outcome is the same with it set to true.
Comment on attachment 390043 [details] [diff] [review]
Patch v5

I've just taken a quick look through the patch and it is looking good.

The main issue is the inbox problem mentioned above. I've not spotted where that problem is yet, but could it be the special root folder is counting the same as the inbox?

I've just noticed this code also appears to be wrong:

+    if (mOnlyCountInboxes)
+    {
+      nsCOMPtr<nsIMsgFolder> inboxFolder;
+      rootFolder->GetFolderWithFlags(nsMsgFolderFlags::Inbox, getter_AddRefs(inboxFolder));
+      if (inboxFolder)
+        GetTotalUnread(inboxFolder, &numUnread);
+    }
+    else
+      GetTotalUnread(rootFolder, &numUnread);

GetTotalUnread will count the number of unread in the folder passed and its sub-folders. So on startup with the inbox_only pref set to true, you'll count unread in the sub-folders of inbox (as can happen with some imap installations, e.g. courier imap), but then won't update the count when we read in those sub-folders.

>+  nsCOMPtr<nsIObserverService> os =
>+    do_GetService("@mozilla.org/observer-service;1", &rv);
>+  NS_ENSURE_SUCCESS(rv,rv);

nit: you've got a few instances of these without a space after the comma.
Attachment #390043 - Flags: review?(bugzilla) → review-
OK, thanks for the review.  I don't know what this double-counting issue is, as I'm not seeing it on the 2 imap servers I use.  Are you using a server I can test against?  I plan to try and finish this tomorrow.
(In reply to comment #79)
> OK, thanks for the review.  I don't know what this double-counting issue is, as
> I'm not seeing it on the 2 imap servers I use.  Are you using a server I can
> test against?  I plan to try and finish this tomorrow.

I see it against Gmail, Courier IMAP and I think the other one is Zimbra.
Attached patch Patch v6 (obsolete) — Splinter Review
I think this is done now:

* Inbox count no longer includes child folders
* Virtual folders are not counted in updates (that was your double count)
* nits addressed
* other changes as listed above
Attachment #390043 - Attachment is obsolete: true
Attachment #391620 - Flags: review?(bugzilla)
Attachment #390043 - Flags: superreview?(bienvenu)
Comment on attachment 391620 [details] [diff] [review]
Patch v6

>+void
>+nsMessengerOSXIntegration::InitUnreadCount()
>+{
...
>+  nsCOMPtr<nsISupportsArray> servers;
>+  accountManager->GetAllServers(getter_AddRefs(servers));

I just spotted this, we should be checking this for success.

This all looks good now. Thanks for sticking with it.

r=Standard8
Attachment #391620 - Flags: review?(bugzilla) → review+
Target Milestone: Thunderbird 3.0rc1 → Thunderbird 3.0b4
Attached patch Patch v7 (obsolete) — Splinter Review
Thanks for catching that, I've fixed it.
Attachment #391620 - Attachment is obsolete: true
Attachment #391688 - Flags: superreview?
Attachment #391688 - Flags: superreview? → superreview?(bienvenu)
Depends on: 487820
Comment on attachment 391688 [details] [diff] [review]
Patch v7

thx for sticking with this. No need for an other review, but some little nits:

should just return shouldCount directly; no need for assignment to rv first:

+  rv = shouldCount->GetData(aCountFolder);
+  return rv;


inconsistent brace:
+  if (deep) {

probably cleaner just to add the &rv param to do_CreateInstance along with NS_ENSURE_SUCCESS here:

+  nsCOMPtr<nsISupportsString> str
+    (do_CreateInstance(NS_SUPPORTS_STRING_CONTRACTID));
+  if (!str)
+    return NS_ERROR_OUT_OF_MEMORY;
Attachment #391688 - Flags: superreview?(bienvenu) → superreview+
Attached patch Patch v8 (Final)Splinter Review
Thanks, fixed your nits.  This one can land.
Attachment #391688 - Attachment is obsolete: true
Keywords: checkin-needed
Checked in: http://hg.mozilla.org/comm-central/rev/b626c86b3f40
Status: ASSIGNED → RESOLVED
Closed: 15 years ago
Keywords: checkin-needed
Resolution: --- → FIXED
Depends on: 508001
Attached image screenshot of miscount
I'm seeing problems with this - attached is a screenshot that shows the dock icon with a '2' while I only have 1 unread (new mail). This is after I have read everything, and there is no # on the dock.

Will open a followup bug if needed - let me know.
(In reply to comment #87)

How many Inboxes do you have?  By default it will count any and all Inboxes.  I can't tell from this screenshot if there's another with 1 in it.
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.4pre) Gecko/20090903 Lightning/1.0pre Shredder/3.0b4pre

I've been noticing the same thing once in a while since this patch landed, but because I have three inboxes, I wasn't sure whether it was right or not.  The past few days, I was paying more attention when it happened and I saw that only one inbox had new messages, but the badge still reported 2.  I'll see if I can get a screencast next time it happens.

Should this be a new bug or should we wait and see if it might be a problem with this patch?
> Should this be a new bug or should we wait and see if it might be a problem
> with this patch?

New bug, please, and you can mention the number here.
Chris or Daniel,

Have either of you posted a new followup bug yet?  I've been seeing this problem on SeaMonkey 2.0 beta versions (both 1 and 2) and thought it might be caused by common MailNews code.  For example, I have SeaMonkey set to check for new messages on an IMAP account when I launch the program, and when I received three new messages in addition to the two marked unread in my inbox, the dock icon displays a count of eight unread messages (as if the new messages are double-counted).  At least now when I read a message the count goes down rather than up.
Using 3.0 release and STILL experiencing this issue. Was on nightly builds of Shredder prior and this problem was always present.
(In reply to comment #93)
> Using 3.0 release and STILL experiencing this issue. Was on nightly builds of
> Shredder prior and this problem was always present.

Possible you're seeing bug 516477.  You're not seeing this bug.
You need to log in before you can comment on or make changes to this bug.