428266 - keep IMAP folders up-to-date using the STATUS command

Status: RESOLVED FIXED

(Thunderbird :: Preferences, defect, P2)

Description

[Nikolay Shopik]

I believe setting mail.check_all_imap_folders_for_new should by enabled by default if we are now considering bug 385502. Offline won't work correctly if TB don't check all folders. I see no reason to have this disabled, in 95% cases it should be ENABLED and in other 5% you can disable it if you need such behavior.

Comment 1

[Magnus Melin [:mkmelin]]

There are good reasons not to check all folders by default, and many people certainly wouldn't want this. Even with my fairly modest 50ish folder structure it would be slow, but I wouldn't expect hundreds of folders to be very uncommon. Checking all of those would be slow, and greatly increase server loads.

Comment 2

[Nikolay Shopik]

That's one of examples of why people (regular) don't like use email client and relay on web interface. People don't care about server load, they just won't read their mail, and if they don't get something important in time, this frustrate them. And start thinking this email client is "broken". Even when I work on GRPS(EDGE) connection I still want know if there new mail in other folders.
IMAP is all about new experience if you compare it to POP3 and you know "no pain - no gain" ;) this will cost server resources.

Comment 3

[Jay Moran (jaym/lazvon)]

Another reason NOT to do this, is that at least currently it results in having to enter your IMAP password twice. At least with Microsoft Exchange IMAP servers. Needs to wait most like before checking the other folders after the INBOX.

Comment 4

[Nikolay Shopik]

Even if its true that's RFE should depends on such bug before enabling it but not be stoper for it. And I not seen any problems with 2 exchange servers using IMAP by myself.

Comment 5

[Jay Moran (jaym/lazvon)]

I personally think the best thing is to just leave the default as it is, and expose the config setting in the Preferences.

For my problem, there is a single Exchange Server, not two, with five folders other than Inbox. I enabled this setting last week when I read your request. I enabled it one morning, ran all day with it. Didn't realize the next day when I logged in why I was suddenly being prompted twice for my IMAP password. Last night I finally realized it was enabling check all folders, and posted here.

You may not think this is a stopper, but it certainly appears to be one for me and that bug should be fixed first. I'll file it I guess. I'll post back here and recommend that this Feature Request, DEPENDS on that Bug.

Comment 6

[Jay Moran (jaym/lazvon)]

I've created Bug 444999 describing the multiple password prompts. I believe this Feature Request were it to be accepted (which I'm not in agreement with) that Bug 444999 would definitely need to be fixed first to not create a negative user experience. Though, with folks using GMAIL IMAP these days and having lots of tags that show up as IMAP folders... if Thunderbird adopted the default behavior of checking all folders... performance could be severely impacted for the end user as all of those tags/folders are cycled through... and it would add quite a load to GMAIL IMAP as well.

Comment 7

[Nikolay Shopik]

To make less overhead also mail.imap.use_status_for_biff = FALSE must be set. In same time this will raise question of compatibility with server. This is debatable for using prefs by standards or support broken implementations.

Comment 8

[Brian Kennelly]

The STATUS command uses less overhead than SELECT, so setting mail.imap.use_status_for_biff to FALSE increases the overhead.  

FWIW, I disagree with this request.  The default should remain to check only the Inbox and selected folders, but "Check All" should be exposed in the Advanced server preferences.

Comment 9

[Nikolay Shopik]

I've remembered why I prefer mail.imap.use_status_for_biff=FALSE, because when I have autosync_offline_stores enabled TB doesn't download message to offline store until I open this folder.
Probably we need more intelligent way to handle this? Like if STATUS says there new message we should download this folder. If we have bug on that please point on that one while I'm searching it.

Comment 10

[Brian Kennelly]

The closest I can find is 231207.  I think that, when STATUS indicates a change in the message counts, the folder should be updated.  That gives the benefit of more efficient biff processing, but would keep the folder summaries up to date for searches and, with autosync, would download the new message bodies.

Comment 11

[David :Bienvenu]

I think this is the wrong thing to do, by default. Power users who are much more likely to want this can change the pref.

Comment 12

[David Ascher (:davida)]

Interesting, I was about to comment that only power users will have enough folders that they _wouldn't_ want this.  Interesting difference in perspective =)

Comment 13

[David :Bienvenu]

I think it would be nice to occasionally update other folders but I would not do it with the same frequency that I update the inbox

Comment 14

[David Ascher (:davida)]

Can you explain why?  

I certainly would agree w/ not wanting each folder to bing! on new mail, because of mailing lists, etc., but I can't think of a user-centric reason why users wouldn't want the folders to always be as up to date as possible.  

(I can think of reasons why ISPs wouldn't, but I don't think that's central to our UX choices)

Comment 15

[David Ascher (:davida)]

FWIW, Mail.app seems to do what this bug suggests, best I can figure out.

(also, if we don't do it, we should certainly make it easier than it is now IMO)

Comment 16

[David :Bienvenu]

Most users don't get mail delivered to imap folders other than their inbox, so once they're up and running, the normal way mail gets put in other folders is via filters or the user moving messages, both of which we know about and can update the other folders. The inbox is where new mail generally comes in. It's true that if you access the account from an other machine, then you need to update the other machine, but you don't need to do it at the check for new mail interval. And I would argue that server-side filters are a power user feature.

I'm all for keeping all the folders up to date, but I would not treat every folder as if it was the inbox and equally likely to receive new mail as the inbox. And even if we use the STATUS command, it does not scale well once the user has a moderate number of folders. And the STATUS command has issues, which is why most people who turn this pref on also turn on the pref NOT to use the STATUS command, which means we will SELECT each folder, which is expensive on the client, the network, and the server...

It's not just ISP's we have to worry about. I can imagine a university setting, where IT departments might be reluctant to recommend Thunderbird if it's going to hit their imap servers hard.

We can try turning on the STATUS command for every folder by default with biff, but I'd prefer updating other folders a bit less frequently than the INBOX. If we do want to turn it on, we should do it for b1 and see what the response is...

Comment 17

[David Ascher (:davida)]

Sounds good.  I can certainly believe that there should be a different frequency to hitting non-inbox server.  All I want to avoid is a default configuration where users feel that the only way to find out if they have stuff to read is to select each folder by hand, especially when the competition seems quite a bit more "eager" to show up-to-date information.

Comment 18

[David Ascher (:davida)]

Adjusting flags as per comment #16, with the understanding that comment #0 isn't what we're talking about doing at this point.

Comment 19

[Steve Simms]

Bug 278539 is related to the idea of checking various folders at different frequencies.

Comment 20

[David :Bienvenu]

(In reply to comment #17)
> Sounds good.  I can certainly believe that there should be a different
> frequency to hitting non-inbox server.  All I want to avoid is a default
> configuration where users feel that the only way to find out if they have stuff
> to read is to select each folder by hand, especially when the competition seems
> quite a bit more "eager" to show up-to-date information.
> 
I agree with that 100%

Comment 21

[Nikolay Shopik]

I though we are updating folders when server tell us todo so using IDLE command don't we? Except case when you are starting up TB and select all folders.

Comment 22

[David Ascher (:davida)]

Switching for b1 flags to target milestones, to avoid flag churn.

Comment 23

[Nikolay Shopik]

Just note, currently when STATUS detect new message its not download it immediately until you SELECT folder. mail.server.default.autosync_offline_stores is enabled. Only workaround is use mail.imap.use_status_for_biff=false.

Comment 24

[David :Bienvenu]

(In reply to comment #21)
> I though we are updating folders when server tell us todo so using IDLE command
> don't we? Except case when you are starting up TB and select all folders.
> 
We only use IDLE on folders the user has opened, and we don't keep more than 5 connections open, so it's not a general solution (plus many servers don't support IDLE)

Comment 25

[Lorenzo Mancini]

Bug #289208 has a patch for adding a per-server "mail.check_all_imap_folders_for_new" preference and exposing it in the GUI. Not sure if you're still interested in this bug's original purpose (see comment #0), anyway.

Comment 26

[Dan Mosedale (:dmosedale, :dmose)]

The better faster IMAP stuff that will be landing for Tb3 should make this mostly irrelevant; removing the blocking flag.

Comment 27

[Bob Lord]

(In reply to comment #26)
> The better faster IMAP stuff that will be landing for Tb3 should make this
> mostly irrelevant; removing the blocking flag.

What's the bug number for that work?

Comment 28

[Nikolay Shopik]

> What's the bug number for that work?

https://wiki.mozilla.org/MailNews:Better_Faster_IMAP_Plan

Comment 29

[Brian Kennelly]

(In reply to comment #26)
> The better faster IMAP stuff that will be landing for Tb3 should make this
> mostly irrelevant; removing the blocking flag.

In what sense does that work make this irrelevant?
It seems to me that this bug is a critical part of that work.  


(In reply to comment #28)
> > What's the bug number for that work?
> 
> https://wiki.mozilla.org/MailNews:Better_Faster_IMAP_Plan

How do we contribute?  That page lists as a requirement that messages should be stored offline.  that is definitely *not* desirable, and must remain a user option.

Also, from that page:
When new messages arrive, TB should download message headers immediately and should start downloading the bodies 

I agree that it would be beneficial to check all folders for new or deleted messages, and to synchronise the headers,and *optionally* the bodies, in changed folders.  This bug addresses the first part of that design.  Automatically updating the changed folders starts with identifying them.

Comment 30

[Bob Lord]

(In reply to comment #28)
> > What's the bug number for that work?
> 
> https://wiki.mozilla.org/MailNews:Better_Faster_IMAP_Plan

Thanks for the link. That's very helpful.  But I'm also looking for a tracking bug so I know when the feature lands.

Comment 31

[Bryan Clark (DevTools PM) [:clarkbw]]

You're looking for Bug 436615

Comment 32

[Dan Mosedale (:dmosedale, :dmose)]

(In reply to comment #29)
> How do we contribute?

Posting in the mozilla.dev.apps.thunderbird or in appropriate bugs are good choices.

>  That page lists as a requirement that messages should be
> stored offline.  that is definitely *not* desirable, and must remain a user
> option.

It will be optional; however, it will be turned on by default.

> I agree that it would be beneficial to check all folders for new or deleted
> messages, and to synchronise the headers,and *optionally* the bodies, in
> changed folders.  This bug addresses the first part of that design. 
> Automatically updating the changed folders starts with identifying them.

My understanding is that it's likely to be about the same cost to just select those folders to do that test.  Perhaps bienvenu or emre can chime in with more details...

Comment 33

[Magnus Melin [:mkmelin]]

(In reply to comment #26)
> The better faster IMAP stuff that will be landing for Tb3 should make this
> mostly irrelevant; removing the blocking flag.

Isn't this actually more less a pre-requisite for the better faster IMAP stuff?

Comment 34

[Dan Mosedale (:dmosedale, :dmose)]

Magnus: see the tail end of comment 32.

Comment 35

[David :Bienvenu]

Yes, the better faster imap stuff should be using STATUS instead of SELECT - STATUS may not be "free" on the server, but it's way cheaper in terms of the network bandwidth required for larger folders.  Somehow, my saying that STATUS wasn't always cheap has been conflated with saying it's just as expensive as SELECT. The IMAP RFC just warns that STATUS may have to do a similar amount of work on the server. Basically, the auto sync code should be doing STATUS commands to see if the number of messages in a folder has changed, and only if so, doing a SELECT and full header flag sync. There may be a few edge cases where STATUS makes it look like the messages in a folder haven't changed, but it's well worth the savings.

The one case where we shouldn't be doing a STATUS but rather do a SELECT directly is when we know there are pending messages in an imap folder, e.g., we move/copied a message into the folder. In that case, we know we should do a full select.

This is definitely something I'd like to fix before we ship.

Comment 36

[David :Bienvenu]

Attachment: wip

this changes imap auto sync to use STATUS to determine if a folder has changed, and then UPDATE to get new headers, if it has. I just got this working, so I'm sure there are a few issues. I'd like to take advantage of auto sync working with the activity manager in order to better tell that this is doing the right thing...

Comment 37

[David :Bienvenu]

this has the potential to lower the amount of network traffic auto sync takes up by a large amount, so I'm going to try hard to drive this into b3. It also will help with the file handles and .msf files left open, I think.

Comment 38

[David :Bienvenu]

Attachment: more wip

starting to actually work, as can be seen by the fact that the autosync activity manager code is increasingly unhappy, which it is whenever we decide we *don't* need to select a folder. The goal is to only select an imap folder for auto sync when it has changed from the last time we selected it.

It turns out that we weren't paying attention to UIDNEXT responses from anything but STATUS commands (e.g., SELECT), so I fixed that. It also turns out that servers don't always return UIDNEXT from SELECT commands, but we should not overwrite our previous UIDNEXT cached value in that case. And it turns out that ReadDBFolderInfo was returning an error when it had already been initialized, which it should not, afaict.

Next up, make auto sync activity manager happy.

Comment 39

[David :Bienvenu]

Attachment: proposed fix for auto sync

This makes the imap auto sync code use the IMAP STATUS command to see if a folder has changed, and if it has, then we update the folder. This seems to be working well for me, except for a few gmail folders which say one thing for STATUS and an other for SELECT, and I've pinged a contact at Google about that.

Comment 40

[Brian Kennelly]

The messages counts on STATUS and SELECT may not match on Gmail, if some of the messages are Chats.  Chats are not available through IMAP, and are not counted by the SELECT command.  But, they are counted by the STATUS command.  It is a known bug.

Comment 41

[David :Bienvenu]

thx Brian, that's what's going on.

Comment 42

[Mark Banner (:standard8)]

Comment on attachment 364732 [details] [diff] [review]
proposed fix for auto sync

>+  /** Message body download in progress */
>+  const long stDownloadInProgress= 3;

nit: space before =

>+  /**
>+   * If server supports UIDNEXT, we store the result here.
>+   **/
>+  attribute long nextUID;

nit: I think we can just use the 3-slash comment form for this.

>+  /**
>+   * @{
>+   * These are used to access the response to the STATUS or SELECT command.
>+   * The counts include deleted messages, or headers we haven't downloaded yet.
>+   **/

nit: only one star nencessary on the last line.

> NS_IMETHODIMP nsAutoSyncState::OnStopRunningUrl(nsIURI* aUrl, nsresult aExitCode)
>-{ 
>+{
>+
>   nsresult rv;

nit: we don't need the additional blank line.

>-  
>+  nsCOMPtr<nsIAutoSyncManager> autoSyncMgr = do_GetService(NS_AUTOSYNCMANAGER_CONTRACTID, &rv);
>+  nsCOMPtr<nsIUrlListener> autoSyncMgrListener = do_QueryInterface(autoSyncMgr, &rv);
>+  NS_ENSURE_SUCCESS(rv, rv);

nit: autoSyncMgr isn't being checked for success.

Do we need to worry about the google issue? Will it just cause us to issue status followed by select and then find we've not got as many as we thought?

Comment 43

[David :Bienvenu]

I'm not worried about the GMail issue - when we issue the select we just use the values we get from the Select response for the message counts. We'll do slightly more work for those folders than we did before, but we'll do a lot less work for all other folders. The GMail developers are aware of the issue

Comment 44

[David :Bienvenu]

Attachment: fix addressing Standard8's comments

address Standard8's comments, carry forward Neil's sr

Comment 45

[David :Bienvenu]

fix checked in - this probably isn't exactly what the reporter wanted since it's autosync that's causing us to keep the imap folders up to date, but autosync is on by default, and that should be good enough (especially since we're talking about default behavior, not capabilities).

Comment 46

[Nikolay Shopik]

This is better than nothing, also serve as helper for sync. How often we are sending STATUS command? Usually it issued when pressing "get mail" or when "check for messages every n minutes" enabled is that all?

Comment 47

[David :Bienvenu]

we now do STATUS from auto sync, which won't happen more frequently than the biff interval, or every 10 minutes, whichever is greater, if I remember correctly.

Comment 48

[Thomas Stache]

Shredder isn't discovering new mail in subfolders for me anymore, while at other times random folders appear bold without new mails in them. Naturally I blame this change for it, due to it's description. I use the big German provider GMX, fwiw. 

The folder setup is:
Account
├- Inbox
└┬ Mailinglists
 ├- Sports         <- these are not updated
 └- Pug            <-

1) Are there already known fallout bugs filed?
2) Can I identify my ISPs IMAP server software from Thunderbird, to find out what it supports?
3) Is there a pref to switch back to the more expensive checking mechanism?

Comment 49

[Thomas Stache]

Attachment: Screenshot of false-positive "new" folders

Promptly ran into the described issue with false-positive "new" state on random folders.

Comment 50

[Nikolay Shopik]

I've seen this too few times per week.

Comment 51

[Mark Banner (:standard8)]

Thomas, please file your issues in a new bug, then we can track them properly.

Comment 52

[Thomas Stache]

(In reply to comment #48)
> ... while at other times random folders appear bold without new mails in them.

This is already filed as bug 482754. Should be confirmed.

The other issue sounds similar to Magnus' description of bug 483859.