Closed Bug 1396495 Opened 7 years ago Closed 3 years ago

Unread messages count not updated in folders when receiving new e-mail involving server-side filters

Categories

(MailNews Core :: Networking: IMAP, defect)

Product:
MailNews Core
Component:
Networking: IMAP
Platform:
Unspecified
All
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 1708981

People

(Reporter: erik.dercole, Unassigned)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

michael_log_1
184.76 KB, application/octet-stream
Details 
User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36

Steps to reproduce:

I have several IMAP accounts with server-side filters.
When Thunderbird receives new messages, unread messages count in each (involved) folder doesn't get updated.
To see the correct count, I have to manually click on each folder.
I've been experiencing this bug since years, on any version and platform (both Windows and Mac)



Expected results:

Folders which get new messages, should report updated unread e-mails count.
This is the standard behaviour in other e-mail clients.
Gene, could you please give some advice here.
Flags: needinfo?(gds)
With new messages received into several account INBOXes I see the unread count go up almost instantly for servers that support IDLE (a.k.a., push). For non-idle servers the count goes up when request for new messages occurs on 10 minute default timer. No clicking on account INBOX is needed.

However, I think you are not just concerned with INBOX unread count but for unread messages copied/moved to other folders by the filter function of tb (not sure if this is called "server side" filtering). I set up a filter to move a newly received message with subject "test" on account A to a folder on account B. When the message was sent I saw the unread count on account B's (destination) folder show 1 without changing focus from account A. Account B's folder showed 2 when I sent another message with subject "test". So it seems to work without clicking on it.

Maybe I don't understand what you mean by "server side filter" and how they are being used. Please be a bit more specific with which folders are not updating unread count and how this relates to filtering. Also, information on whether you are using the "get new messages" timer (or get new messages button at the top) and/or IDLE (*) function, that affects how quick new messages are received into tb, might be helpful. 

(*) IDLE, if supported by server, is enabled in tb with the server setting check box "Allow immediate server notification when new messages arrive"; it does not explicitly say "enable idle". This is enabled by default since you probably want to user IDLE if your server supports it.
Flags: needinfo?(gds)
By "server side" I mean, for example, GMail filters/rules, or any kind of server-side filters created in a webmail interface, e.g. filters in Horde.
Let's make a simple example. On the account xyz@gmail.com I have a folder named "Invoices" and a GMail filter which moves e-mails, sent by certain senders, inside the folder "Invoices".
At 09:00, Invoices folder shows no unread messages (0).
At 09:05, new mails arrive. One of these is an invoice (the sender verifies the filter I talked about before).
Invoice folder doesn't get automatically updated: I still see no unread messages (0).
As soon as I click on Invoice folder, unread count gets updated: I see (1) and the new email gets updated.
This is what happens with Thunderbird.
With other clients, such as Outlook, unread count gets updated as soon as new email arrive.

Yes, I'm using the "get new messages" timer and "Get new messages button": the problem happens in both cases.
Yes, I'm using IDLE function.

Thank you
Thanks for the example. In my gmail account I have an existing folder/label called [gmail]/lxr . I set up a filter at google webmail to "apply the label" to [Gmail]/lxr when a message received from email address gds@chartertn.net (my usual address). Then I send a message from my usual address to my gmail address from another tb instance on device A. Looking at my gmail account with tb instance on device B, while parked at my usual email account, I see gmail account Inbox and [Gmail]/lxr folder unread number increment when the mail comes in (almost immediately). So I don't see a problem here either. No clicking on gmail's Inbox or [Gmail]/lxr folders in tb are needed to see the increased unread count.

Am I still missing something here? It seems to work with my tb version which is 52.2.1 (64-bit). The only thing I see that would affect this is that when the filter is set up there is a non-default option to mark the message as read after the filter is applied, so you might check that this is not set (probably OK since clicking shows the right unread count).
Ok, I see the problem on yahoo when I set up a filter. Only clicking on filter destination folder causes unread count to increase (get new mail doesn't affect it).
To fix this there is a "pref" that can be set to cause *all* folders to be checked for new email and not just inbox. It is at Preferences(or Options)/Advanced/Config Editor...  In "about:config" widget, search for "all_folders" and set "mail.server.default.check_all_folders_for_new" to true by double-clicking on it and restart tb. This fixes yahoo for me so now I see the unread count go up when a new filtered message arrives in the destination folder.

There doesn't seem to be a UI for setting this for individual accounts/servers so it's either all or none (*). This feature was added at Bug 289208 but the UI for making it server specific was never done it seems. Since it will check every folder this might slow down tb some but probably not noticeable on a fast connection.

This is probably not a problem on servers that support IDLE since the server will report on its own any folder that has a new unread message. This is why gmail works (at least for me) since it supports IDLE but yahoo failed because it doesn't support IDLE. 

(*) Maybe you could create new "prefs" for individual servers/accounts that you want all folders checked on, e.g., "mail.server.serverX.check_all_folders_for_new" and set them true and then reset back to false "mail.server.default.check_all_folders_for_new". Not user-friendly (you have to determine X) and I haven't tried this so not sure it works. 
Jorg, do you think this needs more work?
Flags: needinfo?(jorgk)
You know the discussions we had about new hard to understand options being added to the "Advanced Account Settings".

I think the answer is the same: Detect whether the server can do it and set the option automatically, so:
mail.server.serverX.check_all_folders_for_new [auto, no, yes, no-auto, yes-auto].
Flags: needinfo?(jorgk)
I don't think it is a matter of "can the server do it" (I think all can) but does the user need it. Apparently the reporter needs it. Just not sure if he needs it for all or just certain servers. Bug 289208 provides a UI patch that was never landed that is not an "advanced" setting that just asks: "Do you want only INBOX checked for new messages for this account?" which should be on by default. If they uncheck it then all folders are checked for new messages when timer expires or check new messages button clicked.
Or maybe you mean always check all folders if the server doesn't support IDLE or if the option to use IDLE is un-checked? This assumes IDLE support always fixes the reporters problem which I am not yet 100% certain of since his example was gmail (although it seems to work for me).
Looking at bug 289208 there are two patches. One to do a per-server pref, and the second, to add some UI. That patch received ui-r- with the words: I think we can get a better UI for this feature than just adding a checkbox.

So I think the "backend" patch should be refreshed, reviewed and landed. But maybe it needs a modification.

mail.server.default.use_idle relates to "Allow immediate server notification ...".

So what I'd do is this:
If "Allow immediate server notification ..." is checked, the user expects immediate update on all folders. So if we know that doesn't work since the server doesn't do IDLE (to be confirmed), we go and do the necessary steps to make it work.

And that might well mean transitioning mail.server.serverX.check_all_folders_for_new from "auto" to "yes-auto".

Please correct me if that doesn't make sense. The point I'm trying to make is that we can live with yet another checkbox if we can assure that immediate notifications work on all folders when chosen.

But before we do anything further, we should find out what's going on with the reporter's Gmail account.
(In reply to gene smith from comment #5)
> Ok, I see the problem on yahoo when I set up a filter. Only clicking on
> filter destination folder causes unread count to increase (get new mail
> doesn't affect it).

That's *exactly* the problem I'm experiencing, but with different GMail accounts and also with my company's mail server (it's based on Cyrus IMAP).

As I said, it happens on several workstations, both Windows (7) and Mac OS. I regularly update Thunderbird to the latest available version; I even tried nightly builds.

I'm sure you understand that this behaviour is really frustrating. Sometimes I miss important emails, or I read them lately, because I've got to remember to click on every single subfolder to see it update.

The only way I found to avoid this huge problem is using local filters, but for me it's extremely uncomfortable, because I read my emails from at least 3 or for different computers, so I should configure filters on every machine and keep them syncronized (and on my smartphone I don't use Thunderbird).
Thank you Gene Smith and Jorg K for your time, I really appreciate.

"mail.server.default.check_all_folders_for_new" is set to true on all my Thunderbirds, it's one of the first options I tweaked and I read about searching on the Internet for this problem.

I also checked "When downloading new messages, always check this folder" (sorry, I'm running Thunderbird in Italian, so I don't know what is the right translation - but I'm sure you both understood which checkbox I'm talking about) for *EVERY* folder/subfolder in *EVERY* account (GMail or not).


Right now, I've tried to add "mail.server[X].default.check_all_folders_for_new" (set to true) for every [X] server I've configured on my Thunderbird. I don't expect this will fix the problem, but this is a quick try I can do.
(In reply to Erik from comment #12)

> Right now, I've tried to add
> "mail.server[X].default.check_all_folders_for_new" (set to true)

mail.server.server[X].default.check_all_folders_for_new

(sorry, I forgot a part of the string).



Jorg K, you wrote:

"I think the answer is the same: Detect whether the server can do it and set the option automatically, so:
mail.server.serverX.check_all_folders_for_new [auto, no, yes, no-auto, yes-auto]."

Isn't this parameter a boolean? In my setup "mail.server.default.check_all_folders_for_new" is a boolean, so I can't use values such as auto, no, yes, etc.
Am I missing anything?

Thank you
It's: When getting new messages for this account, always check this folder.
> Isn't this parameter a boolean? ... Am I missing anything?
Right, we'd have to turn this into a string or use a different preference.

I think before we talk about code changes, we should get to the bottom of why things don't work for your Gmail account when they work for Gene's. Do I have to try it myself? I'm not a great Gmail fan, but I can try a different IMAP server, where I'd have to set up server-side filters.
(In reply to Jorg K (GMT+2) from comment #14)
> It's: When getting new messages for this account, always check this folder.
> > Isn't this parameter a boolean? ... Am I missing anything?
> Right, we'd have to turn this into a string or use a different preference.
> 
> I think before we talk about code changes, we should get to the bottom of
> why things don't work for your Gmail account when they work for Gene's. Do I
> have to try it myself? I'm not a great Gmail fan, but I can try a different
> IMAP server, where I'd have to set up server-side filters.

What information can I provide about my configuration?

Anyway, I'm pretty sure this issue is not related to GMail accounts only. As I said, I'm experiencing the same problem with my company's mail server.
Does inbox always report new messages or is the problem *only* on filter destination folders?
(In reply to gene smith from comment #16)
> Does inbox always report new messages or is the problem *only* on filter
> destination folders?

Inbox folder hasn't any problem. The problem is *only* on filter desetination folders.
Are the destination folders subfolders or top level? With subfolders of inbox I had to have at least two unread before it showed (2).
Also do you have any other "hidden" prefs set? You might try setting up a new test profile with just 1 account like just gmail and see if that works. (sorry for terse I'm on a phone)
(In reply to gene smith from comment #18)
> Are the destination folders subfolders or top level? With subfolders of
> inbox I had to have at least two unread before it showed (2).

It happens in both cases.
For Gmail accounts, folders are subfolders of "[GMail]" folder.
(In reply to gene smith from comment #19)
> Also do you have any other "hidden" prefs set? You might try setting up a
> new test profile with just 1 account like just gmail and see if that works.
> (sorry for terse I'm on a phone)

What do you mean with "hidden" prefs?
Is there a way to completely reset preferences, without reinstalling Thunderbird e reconfiguring all the accounts (I have several, with tons of emails to synchronize)
Am I the only one experiencing this strange behaviour?
I don't think so...
During these years, I've been searching the web for this problem, I spent countless hours, read hundreds of posts, articles, kb, but none of these solutions worked.

For example: https://bugzilla.mozilla.org/show_bug.cgi?id=481845

This report is about the same problem I'm experiencing, but the solution didn't work for me.
(In reply to Erik from comment #21)
> 
> What do you mean with "hidden" prefs?

Anything you set via config editor that is not part of the normal tb gui configuration screen, such as mail.server.default.check_all_folders_for_new

> Is there a way to completely reset preferences, without reinstalling
> Thunderbird e reconfiguring all the accounts (I have several, with tons of
> emails to synchronize)

If you start tb from a command line with "thunderbird --ProfileManager" it should come up with a screen to create a new profile, e.g., call it "test". Then start with that profile and set-up your gmail account. That should run with "clean" configuration (no "hidden" prefs set). You don't have to uninstall and re-install tb. Then test the problem again with only the gmail account in the test profile. You can delete the "test" profile when you are done with it via the same --ProfileManager gui.
(In reply to Erik from comment #22)
> Am I the only one experiencing this strange behaviour?
> I don't think so...
> 
> For example: https://bugzilla.mozilla.org/show_bug.cgi?id=481845
> 
> This report is about the same problem I'm experiencing, but the solution
> didn't work for me.

A property for each folder that says check this folder on each "get new mail" activity!. But it didn't help for you when selected. I saw reference to this in another bug report but I didn't know about this. I will check the code to see what this really does. Thanks for pointing me to this!
On yahoo account, setting the destination folder to "...always check this folder" in folder properties, I see the count increment when the get new mail timer expires (set to 2m) or when I click get new mail. Without this checked it doesn't increment (unless I select the destination folder) since I now have mail.server.default.check_all_folders_for_new set to false. Since yahoo doesn't support IDLE that should be out of the picture.

At this point, since everything seems to work for me I can only think of 2 reasons it doesn't work for you:

1. I am on linux and you are on windows and apple. Unlikely since linux probably get much less testing.
2. Something in your default profile is messing something up.
or may 3. I am storing messages locally (default) and not just at the imap servers. Don't know which sync method you are using.
(In reply to gene smith from comment #25)

> 2. Something in your default profile is messing something up.

Maybe. I can try to start Thunderbird with --ProfileManager option and create a new profile.

> or may 3. I am storing messages locally (default) and not just at the imap
> servers. Don't know which sync method you are using.

Mmmh, this is interesting. I should check this. Maybe I choosed to not store messages locally. Tomorrow I will check as soon as I enter my office.

Thanks everyone for your time.
(In reply to Erik from comment #26)
> 
> Mmmh, this is interesting. I should check this. Maybe I choosed to not store
> messages locally. Tomorrow I will check as soon as I enter my office.
> 

I set my yahoo account to keep messages only at the server (Sync & storage), shutdown tb and then deleted all my yahoo folders at
/home/gene/.thunderbird/v2u6h75w.default/ImapMail/imap.mail.yahoo.com/ .

On restart, yahoo account is re-downloaded (now headers only) and folder structure re-appears. Destination folder still has "check for new mail" set in properties. Sent new email to account from other device and unread count bumped after 2 min with no clicking required (also still see a notification pop-up). So, at least for me, this sync setting has no effect. 

I haven't asked, but do you see a new mail received notification pop-up even when the unread count doesn't increment? (Not sure if this is relevant.)

One more thing. The Bug 481845 that you pointed out talks about IDLE. I forgot that IDLE is not really global for the server but is mailbox (a.k.a., folder) specific. Tb makes (by default) 5 connection to a server for up to 5 mboxes at a time (advanced server cached connections setting -- how is yours set?). So if your destination folder is not among the 5, IDLE probably will have no effect and won't detect new email for those folders. In that case, only the get new message timer or button will cause the a check for new message at the destination mbox (provided that "check for new mail" folder property is set or mail.server.default.check_all_folders_for_new is true).
I have new folders f1-f5 on outlook account. With idle, timed-checks and check at startup disabled, I see periodic check for *most* folders occurring every 5-10 min. Not sure why. However, only status for folders f2, f3 and f5 are checked. f1 and f4 are not checked and new messages to them are not detected. So seeing strange things.
Also, with idle, only 4 folders are monitored for changes. The 5th connection seems to be for other purposes like checking for new emails in un-monitored folders. 
Anyhow, things seems a bit confused here now. Will look again tomorrow...
Thank you Gene Smith for all your time and for all the tests you made, I really appreciate.

Some hour ago I completely uninstalled Thunderbird from my Windows 7 pc. I also deleted my profile folders and a bunch of registry keys. Then, I reinstalled Thunderbird 52.3.0 (32 bit), configuring all my accounts.

The only option I changed was "mail.server.default.check_all_folders_for_new", set to true. Stop.
I didn't change any check on any folder, didn't change options about IDLE, sync, etc. Nothing.

At the moment, Thunderbird seems to behave as expected. Every message is delivered in its folder and the unread count changes accordingly, without the need to click on the folder. It works with GMail and non-GMail accounts.

One note: today I talked with a couple of colleagues (we're in an IT department, as developers), both using Thunderbird. They have the same problem as mine. I suggested to completely reinstall Thunderbird.
Glad to hear it works. Still curious. How many accounts do you run. Also, how many folders in each account and how many of those do you monitor for new mail. -gene ( on phone)
(In reply to gene smith from comment #30)
> How many accounts do you run. Also,
> how many folders in each account and how many of those do you monitor for
> new mail. -gene ( on phone)

6 accounts; from a minimum of 10 to a maximum of 60 folders; I monitor all of them for new mails: some account is checked every minute, others every 10 minutes.
Unfortunately, after one day, this problem happened again.

At 08:15 I clicked on a subfolder, the unread counted updated from none to (1), then Thunderbird showed an e-mail received at 08:15 (so, 10 minutes before - Thunderbird checks that account for new e-mails every minute).

One note: yesterday I had to leave Thunderbird opened for hours, because it had do synchronize all my messages on all accounts (tens of thousands).

Today, all accounts are synchronized, but the problem happened again.
After some hour using Thunderbird, I noticed that its behaviour is random, unpredictable.
Sometimes I have to click on the folder to see the counter updating, sometimes it updates by itself.
Even if I click on "Get new messages" button, the folder doesn't update. As soon as I click on it, the counter updates.
I know it's not a technical explanation, but I can't find any pattern or any reason why it behaves this way.

I'm sorry to say this, but this is so frustrating.
Yes I think there is a random component here but not sure what it is. Still looking. Do you know if problem only on servers that support IDle? Since you are checking for new quite often disable IDle might help. -g, on phone
Fyi on my outlook account with many folders all are checked for new at timer or get new mail with that pref set. Need to now look with IDle enabled.
(In reply to gene smith from comment #34)
> Yes I think there is a random component here but not sure what it is. Still
> looking. Do you know if problem only on servers that support IDle? Since you
> are checking for new quite often disable IDle might help. -g, on phone

I really don't know. I have 2 accounts on a Courier IMAP server; the others are GMail.
About Courier, I just read an interesting fact: http://kb.mozillazine.org/IMAP:_advanced_account_configuration

I tried to lower the value from 5 to 2, let's see if something change.
Observations with IDLE enabled but check for new mail at start-up and check for new mail on timer both disabled; also mail.server.default.check_all_folders_for_new is set true in config editor but no individual mbox selected to check via the properties menu:
Tb chooses 5 mboxes to connect and SELECT and then does IDLE on them. One is alway INBOX and the other are randomly selected but seem to exclude "well known and typically unmonitored for new mail" mboxes such as Junk, Deleted and Sent. These connection also do STATUS on the mboxes in a somewhat random seeming way. So at each SELECTed mbox, IDLE is terminated, one or more STATUS is done on other mboxes and then IDLE reinstated at each of the 5 SELECTed mboxes. At startup it appears that STATUS *does* occur on every mbox.

After about 20m with connections quiet and nothing changing at outlook server (5 connection in IDLE state) outlook does a tcp disconnect (FIN) on all connections and tb ACKs the disconnect but does not re-establish the connection (SYN). Therefore, after 20 min, IDLE has no effect (no TCP connections) so new/unread mail will not be detected until "get messages" is clicked which re-establishes all the connections and SELECTed mbox IDLE states (with a different random set of 5 mboxes) and does STATUS again. But sometimes after 20m outlook sends an imap BYE and more gracefully terminates the tcp connection instead of just tcp FIN but tb still doesn't automatically restore the 5 connection. Perhaps this is intentional. After being disconnected for, maybe, 1 hour or so it appeared that just moving the mouse over tb caused the 5 connections to be re-established (need to re-visit this). Anyhow, this probably has no effect on the reported problem.

My outlook account (actually only used for tb testing) has these folder (mboxes) that I created: 2 subfolder of Inbox, 1 subfolder of Drafts and 5 top level folders (f1-f5). There also existing Sent, Junk, Deleted, Notes and Outbox that outlook created and all folders are subscribed.

I tested this by monitoring the network activity with wireshark and using stunnel to "remove" the TLS/SSL encryption. I also monitor with tb IMAP logging but can get a more exact picture of what is exactly happening network wise using wireshark's IMAP protocol interpreter. Then on another device with tb running and looking at another account with many archived and marked as unread messages, I copied an unread messages to a folder of my outlook account on that device. I then watched the device with wireshark and tb under test as each message was copied to each outlook folder. So far I see absolutely no problem. When the message is copied to one of the 5 folders that are in IDLE state the unread count bumps up immediately and automatically. On the others, when I click "Get Messages" button at the top the unread count for the non-IDLE folders increments. The only folders that require that I click on them to see the updated unread count were Junk and Deleted since tb skips these even when pref "check all folders" is set, which seems reasonable (maybe setting the "check this folder" property will still force a check on these but I didn't try it).

Anyhow, I will let tb sit idle for a while and try this test again on each folder and see if I see any problems.

P/S: While doing the above test, on the tb under test I was constantly selected on the outlook account Inbox folder except when I clicked Junk and Deleted as mentioned. There are two other non-outlook accounts present in the tb profile I am using but not testing with wireshark. Also, just to be sure, when you click the "Get Messages" button at the top be sure to use the "down arrow" to the right and click "get all new messages" so it applies it to all your accounts (at least I think that's what it does); in any case, your get mail timer on each account should also cause the check for accounts that have a timer enabled.
On mbox f1 an earlier SELECT returned 1 SEEN and 24 UNSEEN, total 25. Then IDLE was done on this mbox. I then copied an unread email from another account to this mbox using the other tb instance on a different device. This caused IDLE to return 1 RECENT and 25 EXISTS and an untagged FETCH response "* 25 FETCH (FLAGS (\Recent))". I expected tb to increment the unread (unseen) count for folder f1 from 24 to 25. Instead nothing happened even after clicking of "Get Messages". It appears that outlook is confused as to the number messages in f1 mbox since it thinks the 25th is RECENT and the total is still 25 when it should be 26.

A while later I clicked "Get Messages" again and the correct unread count appeared at f1. At this point, tb had reconfigured the 5 connections so the f1 was no longer in the IDLE category. Instead a simple STATUS request was done that returned 25 UNSEEN, 1 RECENT and total 26 and then tb's unread count went to 25 as I originally expected.

At least in this case, it appears tb is just reacting reasonably to the what appears to be incorrect information from the imap server.

(In reply to Erik from comment #36)
> (In reply to gene smith from comment #34)
> > Yes I think there is a random component here but not sure what it is. Still
> > looking. Do you know if problem only on servers that support IDle? Since you
> > are checking for new quite often disable IDle might help. -g, on phone
> 
> I really don't know. I have 2 accounts on a Courier IMAP server; the others
> are GMail.
> About Courier, I just read an interesting fact:
> http://kb.mozillazine.org/IMAP:_advanced_account_configuration
> 
> I tried to lower the value from 5 to 2, let's see if something change.

I somewhat doubt that will matter. Since you have so many folders in each account and you have the timers to check for new emails set short I don't think using IDLE will buy you anything. IDLE can only occur on a connection/mbox per account (5 or down to 2 like you are trying) and with many folder and short timers, the IDLE is being interrupt constantly to do STATUS on the various other folders so it might just confuse things (maybe like I see above). I know gmail supports IDLE and Courier probably does so I would just switch it off for every account and see if that helps. Meanwhile, I'm continuing to look...
Thanks for your constant support, Gene Smith.
I'll keep on monitoring Thunderbird to see if anything changes.
See no tb problems when testing with outlook account so am now testing gmail account with several extra folders. So far I see no problems...
Hi, sorry to bother you, but this bug is happening again.

After a few weeks during which I had no problems, now my folders doesn't update: I have to click on each folder to see that I've new e-mails inside it and to see that the counter has changed.

Meanwhile, Thunderbird versione has updated to 52.5.0.

Sorry to say this, but it's really really *frustrating*.
Sorry it's taken a while to get back to you on this. Been very busy with other bugs. 
Can we verify a few settings again that might be helpful?

1. Do you have the folder property "when getting new messages for this account, always check this folder" set for each folder you are interested in?

2. If your check for new mail timer is set short (maybe <= 3 min.) I had recommended that you turn off IDLE. Has this been done?

3. There is also the global mail.server.default.check_all_folders_for_new. If this is set true you might try setting it false and going though each "interesting" folder property and setting them as described in item 1 above. This way TB can concentrated its energy on just the folders you care about seeing updated.

4. If you are still using IDLE, you might try increasing the number of "cached" connection to match the number of interesting mailboxes. It defaults to 5. Not sure how high you can go before something breaks (maybe the imap server will object if you have too many?).

When I last tested this it appeared to be working. However, since I don't run in this mode with day to day usage, there may still be issues, as you are seeing.

Also, possibly an IMAP log would be helpful. You could send the log that occurs when you click 'get new mail' and you don't see the proper counts. When you attach the log, write in which mailbox doesn't have the right count and what you think it should be. Then I might be able to correlate this in the log. The info on how to record a log is here: 
https://wiki.mozilla.org/MailNews:Logging
Reviewing my Comment 37, I found that moving the mouse over the tb (folders?) may be enough to cause an update or at least a re-connect. Seems kind of unexpected but have you tried that?
Hi Gene Smith,

1. "When getting new messages for this account, always check this folder" has been set for ANY folder

2. I'll try this option (disable IDLE). My timer is set to 1 min.

3. mail.server.default.check_all_folders_for_new is true


Thank you
(In reply to gene smith from comment #43)
> Reviewing my Comment 37, I found that moving the mouse over the tb
> (folders?) may be enough to cause an update or at least a re-connect. Seems
> kind of unexpected but have you tried that?

Uhm, to me this doesn't make too much sense, because Thunderbird is always open, but in background, so I can't mouse over its window.
I also tried to disable IDLE, but had no luck.

Simply, Thunderbird doesn't behave as expected.

I'm sorry to insist, but I think there is some bug. I can't believe a normal-skilled Thunderbird user has to deal with all these options, tweaks and tests to simply receive e-mails in the right folders.

4 months ago I wrote, in this thread, that after reinstalling Thunderbird everything was going well.
Now I'm experiencing again the same problems.
Maybe this issue is related to the number of messages / the size of the local folders of each accounts.
Again, sorry for the slow response. Since I tried to duplicate your problem several months ago and couldn't, I didn't know what else to suggest. Still, I haven't forgotten about your issue.  But today, while looking at another problem, I found something else that might help. 

http://kb.mozillazine.org/Offline_folders

This doesn't sound like it is relevant but it mentions another option that might help (mail.imap.use_status_for_biff). I will go through this list from comment 42 and update it and discuss this option more below.

(In reply to gene smith from comment #42)
> 
> 1. Do you have the folder property "when getting new messages for this
> account, always check this folder" set for each folder you are interested in?

Yes, I think you should set each folder you want checked for new mail and uncheck those you don't.
 
> 
> 2. If your check for new mail timer is set short (maybe <= 3 min.) I had
> recommended that you turn off IDLE. Has this been done?

I think you can enable IDLE again. Keep your check for new message time at whatever you want but I would recommend the default of at least 10m (see below).

> 
> 3. There is also the global mail.server.default.check_all_folders_for_new.
> If this is set true you might try setting it false and going though each
> "interesting" folder property and setting them as described in item 1 above.
> This way TB can concentrated its energy on just the folders you care about
> seeing updated.

Yes, I think setting this default (false) is best. You don't want to have tb waste time checking trash, junk, drafts etc.

> 
> 4. If you are still using IDLE, you might try increasing the number of
> "cached" connection to match the number of interesting mailboxes. It
> defaults to 5. Not sure how high you can go before something breaks (maybe
> the imap server will object if you have too many?).

No, set this back to default of 5 is probably best.

The new thing to try in the config editor is 

mail.imap.use_status_for_biff = false

What this does is actually go through all your folders and do an imap SELECT command on each of the folders you have configured to check for new mail (called "biff", no idea why). The default method is to use imap STATUS command. The SELECT imap command may cause a bit more server and network activity than STATUS so that's why I recommend making your "check for new messages" time as large as possible (e.g., 10m) but this may not be necessary.

> 
> Also, possibly an IMAP log would be helpful. You could send the log that
> occurs when you click 'get new mail' and you don't see the proper counts.
> When you attach the log, write in which mailbox doesn't have the right count
> and what you think it should be. Then I might be able to correlate this in
> the log. The info on how to record a log is here: 
> https://wiki.mozilla.org/MailNews:Logging

I guess the log is still a possibility if the mail.imap.use_status_for_biff=false doesn't help.
Well, I just now looked at the duplicate bug 1284471 above. It also mentions the "biff/select" option as possibly helpful. However, the reference the reporter show seems to have the setting described wrong:

This

http://kb.mozillazine.org/Checking_for_new_messages_in_other_folders_-_Thunderbird

says set it true. mail.imap.use_status_for_biff must be set false to force a SELECT. The default value true causes STATUS to be used. I have verified this by looking at the network with wireshark.
P/S: When you change setting with the config editor or even with the user settings and options screens, it is best to restart tb to insure that the new parameters take effect. (You probably already know this.)

(In reply to gene smith from comment #6)

To fix this there is a "pref" that can be set to cause all folders to be
checked for new email and not just inbox. It is at Preferences(or
Options)/Advanced/Config Editor... In "about:config" widget, search for
"all_folders" and set "mail.server.default.check_all_folders_for_new" to
true by double-clicking on it and restart tb. This fixes yahoo for me so now
I see the unread count go up when a new filtered message arrives in the
destination folder.

There doesn't seem to be a UI for setting this for individual
accounts/servers so it's either all or none (*). This feature was added at
Bug 289208 but the UI for making it server specific was never done it seems.
Since it will check every folder this might slow down tb some but probably
not noticeable on a fast connection.

mail.server.serverX.check_all_folders_for_new [auto, no, yes, no-auto, yes-auto].

I've always understood that check_all_folders_for_new is not a performance option. I've never tested myself. Perhaps this will not be slow when there are just a few folders in a few accounts. But what about other cases?

OS: Unspecified → All

Wayne, As a minimum the option to check all folders for new mail will send a NOOP imap command to every folder in imap SELECTED state and fetch \recent flags for all others. If you check for new mail every minute and you have 100's of folders then it might cause a slowdown glitch especially if you are on a slow network (e.g., dialup, if that even still exists).

By the way, yesterday I sent an email to this bug's reporter Erik asking if he is still using TB and if he still sees the reported problem.

Looking at the code, the check for new email does recursive calls and loops though all the folders for an account so if you have 100's of folders the recursion would be 100's deep calling the same function over and over. It may be possible that if an error occurs, say at the 50th folder, that the remaining folders wouldn't be checked. But I've never observed a problem occurring.

Here's the function that calls itself recursively:

https://searchfox.org/comm-central/rev/ceb267da9a00fed5d541a1ae2314c62094b7cd1b/mailnews/imap/src/nsImapIncomingServer.cpp#3021

Perhaps any performance issue could be mitigated by imposing a reasonable minimum for the mail check interval. IMO 1 minute is an absurd minimum. It has caused problems elsewhere, for example bug 531455 and bug 512677

Do you also see this issue when using versoin 68?

Whiteboard: [closeme 2020-01-04

(In reply to Wayne Mery (:wsmwk) from comment #54)

Do you also see this issue when using versoin 68?

Yes, sometimes I still encounter this issue. Just tried on Thunderbird 68.3.1 (Windows)

Gene, bug 422729 has fewer comments and bug 422729 comment 15 summarized.
Should we dupe this bug to it, or is there something different here that warrants keeping this bug open?

Status: UNCONFIRMED → NEW
Component: Untriaged → Networking: IMAP
Ever confirmed: true
Flags: needinfo?(gds)
Product: Thunderbird → MailNews Core
Summary: Unread messages count not updated in folders when receiving new e-mail → Unread messages count not updated in folders when receiving new e-mail involving server-side filters
Whiteboard: [closeme 2020-01-04 → [dupme?]
Version: 52 Branch → 52

Another ~20 bugs https://mzl.la/34UOkE4

(In reply to Wayne Mery (:wsmwk) from comment #56)

Gene, bug 422729 has fewer comments and bug 422729 comment 15 summarized.
Should we dupe this bug to it, or is there something different here that warrants keeping this bug open?

Yes, I think this is actually a dup of that older bug. However, don't want to give reporter Erik the impression that this bug is actually fixed since it will then say RESOLVED as a duplicate of the older bug.
Anyhow, I need to research more why the "check this folder for new" setting doesn't seem to always work in detecting new mail when check for new mail is clicked.

Erik, at what level in the folder hierarchy are the problem folders? Maybe nested subfolders aren't being checked all the time?

Flags: needinfo?(gds)

I'm not sure if bug 1593611 is relevant to Erik's observation, but when account uses oauth2, like gmail, when you check for new messages in ALL accounts (shift-f5) the oauth2 based accounts are skipped as described here and in other comments: bug 1593611 comment 7. If the oauth2 based account is checked individually it works OK.

I am now on 72.0b2 and (still) see this issue (sub-folders not being updated). I have been following this thread (and related) and tried the 4 combinations of settings for "mail.server.default.check_all_folders_for_new" and "mail.imap.use_status_for_biff" with “when getting new messages for this account, always check this folder" enabled - without success. Quitting/restarting TB seems to reliably fetch new email again in all subfolders, it seems. So, it looks like after some time, TB (in combination with my yahoo/gmail accounts and about ~15 subscribed sub-folders) gets into a state, where at least certain sub-folders are not updated anymore. I tried to reduce my setup to a single yahoo account and still see the issue happening. Happy to evaluate any further debug suggestions.

Michael, thanks for the info. Just so I completely understand your issue, are you saying server filters are delivering emails to ~15 non-inbox yahoo folders and you don't see,

  1. New unread count in bold for folders with new message, or
  2. No notification pop-up when new mails in those folders, or
  3. Both?

I'm pretty sure you mean just 1. but want to make sure.

Also, what do you do that cause tb to see new messages? I assume you are just clicking the "get messages" button or waiting for the get new messages timer to expire. (Only inbox can always detect new messages on its own, without user or timer intervention, using imap IDLE.)

Yes, on the ~15 non-inbox server side filters.

I don't see (1) - I don't see unread counts in bold and neither unread messages in the respective folders.
I am not sure about (2) right now. I have TB notifications completely disabled.

I don't think that "Get Messages" or timer expiration solves my issue. Let me double-check though on if "Get Messages" helps to be sure. The only thing that seems to work reliably is to close (Command-Q) / restart of TB. This seems to fairly quickly make a pass through all folders and uncover all unread things. Inbox - as you mentioned - seems to reliably detect new messages and seems not affected by this issue.

Let me know, if there is anything you want me to try for a couple days and report back. I have next week off and super excited to help getting to bottom of this one.

On a side note: my reason for disabling TB notifications is that I get lot of (somewhat transient) errors for my Yahoo imap accounts for authentication issues. I believe this is a known (TB? or Yahoo?) issue in conjunction with Yahoo requiring use of "Allow apps that use less secure sign in", which does not seem to work well (for me at least). I assume this is a separate issue / topic and not related to me not getting new messages reliably.

If you still have your profile with just a single yahoo account you could try this.

1 Configure the server setting so new mail NOT checked at startup, and
2 timer to check for new is disabled, and
3. disable immediate notification of new mail.
4. Select a folder that has the least messages and you don't expect new mail to appear in it.
5. Also see last paragraph below for more details.

Then shutdown tb and restart from command line with IMAP:5,timestamp logging enabled as described here: https://wiki.mozilla.org/MailNews:Logging

On restart, tb should just make one connection to yahoo to the folder (mailbox) of the folder in step 4 above and will show the messages in that folder. Let tb just sit for at least 1/2 hour doing nothing so yahoo times out the connection. Then somehow cause new messages to occur in several of the 15 yahoo folders, maybe using another client like tb on another computer. Once the messages are processed at the yahoo server, on the tb under test, click the "get messages" button. Do the new unread number of message on the appropriate folders show in bold?

Now shutdown tb and attach the resulting log using the link above. Also, tell me which yahoo folder name and how many you expected to see new mail in and those that you didn't (if any).

Note also (and I'm sure you know this) that for any of the non-inbox folders to detect new mail they must have property “when getting new messages for this account, always check this folder" set. Also when doing this test make sure "mail.server.default.check_all_folders_for_new" and "mail.imap.use_status_for_biff" are set back to default values using config editor (false and true respectively).

Flags: needinfo?(michaelsincoming-thunderbird)

Thanks for the detailed info. Ok, all set up following 1~5 above. To do so, I set up a fresh user account on my Mac and created a new TB profile over there with just one yahoo account in it. That one only has 5 sub-folders + inbox, but I assume this is an ok start. I am pointing to a sub-folder, where I don't expect messages and will generate some traffic in the others soon, as you suggested. Note that since is a new TB profile, it is syncing all messages right now from that yahoo account. I will let that run through and play with it after that.

Flags: needinfo?(michaelsincoming-thunderbird)

I have uploaded the log. Looking at the log file, before I started the test, activity stopped around 14:01. Then at ~14:25 I clicked on "Get All New Messages", while still on the folder with no activity / no new unread message. Nothing happened, no unread messages where shown. I expected to see 1 unread email in each folder except for "Indoor'. A few mins later I clicked on a specific folder (which I know had an unread message), and then the unread message was shown by TB. Note that you will see a lot of "authlogin failed" errors in the log. As I mentioned above, I am struggling with yahoo these days. I hope it is not muddying the waters with this issue. Since I can see the new unread messages, once I click on a specific folder, I assume it is not the root-cause for the unread messages issue. But let me know, if you think otherwise.

Attached file michael_log_1 —

Sorry for delay in responding. Been trying to update h/w in my main system and not having lots of good luck. Anyhow, thanks for the log. Here is a rough summary of the log:

14:00
Tb produces select immo url
Tb makes connection
Selects immo
Tb produces select indoor url
Tb makes another connection
Selects indoor
(This looks OK. But not sure why immo and indoor are both selected unless you click on both folders.)

14:25
Tb produces select Inbox url
Tb produces folderstatus url for tadpoles
Tb sends status request for tadpoles but yahoo drops connection.
Tb produces folderstatus url for chargepoint
Tb tries to make 3 connections that seem to be dropped by yahoo or yahoo refuses authentication.
Tb gives up on attempt to obtain status to detect new messages.

14:28
Tb produces select incoming url
Tb makes new connection
Tb selects incoming folder and obtains needed info (flags etc.)
I don't see any new headers or message body fetched here for folder incoming.

Yes, it looks like at 14:25 tb tries to get status for "Tadpoles" folder and yahoo drops and then refuses further connections for other folders. Looks like when tb tries to make multiple parallel connections when checking for new mail in multiple folders, yahoo doesn't like it and drops them or won't authenticate them. You might try reducing the number of "cached" connection that tb makes using the "advanced" server setting. It defaults to 5 so you might even try to reduce this to 1 and see if yahoo remains happier.
But I need to look closer at this so I'm only suggesting this as a temporary work-around, not as a complete solution.

Setting cached connection = 1 seems to help indeed in my case. I did a few runs with it and in each case, TB was able to detect unread messages in all folders. I believe you are also correct that there does not seem authentication issue with my yahoo account.

Having said that, with cached connection = 1, I think I saw one case of some sort of deadlock, where TB would not see any new unread messages at all, not even in inbox. I recovered by setting cached connection back to 5. It may also have been related to the fact that I had another TB instance running under my normal (non-test) account still, which I quit out as well. Not 100% sure, but something funky was going on.

I will keep testing with cached connection = 1 in my normal account and see if it helps. Note that I am pretty sure that I tried that setting a while ago (since it was floating around as suggestion as well) and came to the conclusion that it did not help resolve all my issues, but I am taking a fresh look with it again.

It seems to work better after running "cached connection = 1" in my normal account. I will keep evaluating and report back, if I see anything odd. I am curious as to what the downside of this setting is? Is there anything to look out for?

In terms of bugs, does it make sense to open a separate bug for "cached connections = 5" causing issues with at least my Yahoo imap, if it is not specific to this thread's issue? Do you think this is a general issue with sub-folders and yahoo or specific to using TB/Yahoo's "less-secure" app support mode for clients such as TB.

BTW, is there a thread/update on what the support model for TB & yahoo/imap will be, once Yahoo is phasing out its "less-secure" apps support, i.e. from Yahoo: "Allow apps that use less secure sign in (This option will be going away on March 2nd, 2020)".

Thanks for testing this. I need to test it with yahoo and see what happens myself once my setup here stabilizes again. Don't know that we need another bug report regarding 5 connections with yahoo.
Not sure, but I thought I had a full oauth2 setup with yahoo but got rid of it to test another problem and now have it set back to "use less secure app". I need to check this too.
Having just one connection would theoretically cause more activity when you move from folder to folders. But if yahoo refuses multiple connections or re-connections then the benefit is reduced and could causes a failure to detect mail in non-inbox folders.

Ok. Just double-checked, and I am actually on oauth2 as well. At some point, I wasn't sure if it that was supposed to be working between TB/yahoo based on some thread comments I read, but I remember now that I tried oauth2 a few weeks ago again (as part of my debug effort for my auth issues) and seems to be working for me at this point. Pls disregard my comments regarding less-secure apps support above.

So, I will keep using this configuration for the next week and report back if I see anything odd with it. But it looks like "cached connections = 1" addresses my auth issues and maybe with it, the issue I was facing with unread emails not showing up consistently in sub-folders. For completeness, my config is as follows:

  • "cached connections = 1"
  • “when getting new messages for this account, always check this folder" for the relevant sub-folder
  • "mail.server.default.check_all_folders_for_new" = false
  • "mail.imap.use_status_for_biff" = true

And I should add:

  • Check or new messages every "3" minutes
  • "Allow immediate server notifications when new messages arrive" checked
Blocks: 201332

dup bug 422729 to this one?

Flags: needinfo?(gds)
Whiteboard: [dupme?]

(In reply to Wayne Mery (:wsmwk) from comment #74)

dup bug 422729 to this one?

Yes, that sounds good. I.e., mark bug 422729 a duplicate of this bug. That way this bug, which has the most recent activity, remains open and does not get set to resolved.

Flags: needinfo?(gds)
Severity: normal → S4

I'm pretty sure I've finally found the problem reported 4 long years ago by Erik.

Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: