Query IMAP folders other than INBOX for new msgs

RESOLVED FIXED in mozilla1.1alpha

Status

P3
normal
RESOLVED FIXED
19 years ago
7 years ago

People

(Reporter: BenB, Assigned: emaijala+moz)

Tracking

(Blocks: 1 bug, {qawanted})

Trunk
mozilla1.1alpha
qawanted
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: Backend pref for checking all(!) subscribed folders checked in; new patch with ability to select folders waiting for review, URL)

Attachments

(3 attachments, 14 obsolete attachments)

10.86 KB, patch
Details | Diff | Splinter Review
10.40 KB, patch
Bienvenu
: review+
Details | Diff | Splinter Review
20.72 KB, image/jpeg
Details
(Reporter)

Description

19 years ago
If filtering is implemented IMAP-server-based, the users has to manually check
all possible folders for new msgs. This makes server-based filtering nearly
useless.

Phil Peterson wrote:
> The problem with that is that we don't poll all the IMAP folders for new
> messages (only the INBOX).

Updated

19 years ago
Assignee: phil → nobody
Summary: [RFE] Query all IMAP folders for new msgs → [HELP WANTED] Query all IMAP folders for new msgs
Whiteboard: [HELP WANTED]

Comment 1

19 years ago
Add to the [HELP WANTED] list. cc'ing mscott and bienvenu for IMAP guidance.
Also related, if you were going to make up a more generic biff, is bug 11043

Updated

19 years ago
Keywords: helpwanted

Updated

19 years ago
Summary: [HELP WANTED] Query all IMAP folders for new msgs → Query all IMAP folders for new msgs
Whiteboard: [HELP WANTED]
(Reporter)

Comment 2

18 years ago
*** Bug 38954 has been marked as a duplicate of this bug. ***
(Reporter)

Comment 3

18 years ago
Some notes from the dup bug, plus my own ones:
- It has to scale.
Some user said he has 100 folders and Outlook needs 20 minutes via a WAN,
probably because it establishs a new connection for each folder. I currently
have >360 mail folders on my IMAP server (on a LAN), so I have an interest in
performance as well :). It would be acceptable, maybe even prefereed, to be able
to specify which folders to check. The user should have the ability to turn this
feature off (check INBOX only).
- Trash
Don't biff for new/unseen mail in Trash.
(Reporter)

Comment 4

18 years ago
Bug 19254 is related.
Summary: Query all IMAP folders for new msgs → Query IMAP folders other than INBOX for new msgs

Comment 5

18 years ago
isn't that what subscribed folder are for?
I am working on a fix for this. I don't know how to assign the bug to myself.

I can implement a simple fix first, to check all folders except trash for new
messages, because this won't change any UI. We can see what the performance
looks like, and then talk about adding some UI to specify which folders should
be checked if necessary (if Subscribe isn't the right thing.)

To reproduce without server filtering:

Open the mailnews client and connect to an IMAP server.
Send a new message to yourself.
While the client is still open, use Outlook to connect to the same IMAP server
and move the new message to some folder.
Close Outlook.
Press GetMsg button on the mailnews client.
The folder where you moved the message to should indicate that it has a new
message in it.
(Reporter)

Comment 7

18 years ago
> I can implement a simple fix first, to check all folders except trash for new
> messages

Please make sure that there is at least a backend pref (a pref without UI) to
disable it.
Assignee: nobody → smeredith
(Reporter)

Comment 8

18 years ago
> isn't that what subscribed folder are for?

No. Subscribed folders are the only ones which are accessible at all. As I said,
I have almost 400 personal folders, and I am "subscribed" to most of them.

Comment 9

18 years ago
we're not going to turn this on by default (way way too expensive, and 99% of
people don't need it), so it would seem like the hidden backend pref would be to
turn this behaviour on.

Comment 10

18 years ago
We need QA to test this hidden pref and to write up test cases, run performance
measurements, etc. Putting qawanted.
Keywords: qawanted
Created attachment 32913 [details] [diff] [review]
Proposed patch. Checks all folders if backend pref set.
I have attached a proposed patch. This code looks for a backend pref (not sure
what this should be named--using mail.check_all_imap_folders_for_new right now)
and if it finds it, checks all folders except trash for new messages. I'm also
not sure how to get the new pref inserted into the documentation.
Status: NEW → ASSIGNED

Comment 13

18 years ago
Why are you removing the following lines? I went to some trouble to put them in!

-    nsCOMPtr<nsIImapIncomingServer> imapServer;
-    nsresult rv = GetImapIncomingServer(getter_AddRefs(imapServer));
 
-    if (NS_SUCCEEDED(rv) && imapServer)
-      imapServer->GetDownloadBodiesOnGetNewMail(&m_downloadingFolderForOfflineUse);

I'm not crazy about running all these urls simultaneously, instead of chaining
them. Have you tried this with a large number of folders? e.g. 20 or 30?

More comments later.
I will put those lines back in.

Not sure what you mean by chaining the URLs. If you can point me to an example,
I'll check it out. I've tried with more than 100 folders and it seems pretty fast.

Comment 15

18 years ago
could you attach a protocol log of such a session? I want to make sure we're not
opening 100 connections. Here's how to generate a log:

http://www.mozilla.org/quality/mailnews.html
Created attachment 32924 [details]
IMAP log of checking for new messages with > 100 folders.
Created attachment 32927 [details]
Retry: IMAP log of checking for new messages with > 100 folders.
Created attachment 32986 [details] [diff] [review]
Same patch as last time with the missing lines added back in.

Comment 19

18 years ago
I'm not familiar with this pref branch stuff. Is that new and improved, better
than just getting the pref service and calling GetBoolPref like this:

nsCOMPtr<nsIPref> prefs;
prefs = do_GetService( NS_PREF_CONTRACTID, &rv); 
  if (NS_SUCCEEDED(rv) && prefs) 
  {

In any case, you should use do_GetService instead of NS_WITH_SERVICE (which has
been deprecated)

Comment 20

18 years ago
From the log it seems that the proposed patch will go and SELECT each mailbox.
Wouldn't it be leaner if it just sent a STATUS for each mailbox?

Comment 21

18 years ago
On some servers, yes, that would be faster and easier on the servers. It's what
the old Update message counts command did, IIRC. However, it has the drawback of
not accounting for deleted but not expunged messages. (If we're just interested
in dealing with server-side filters, that's not much of a problem) It would also
be harder to code (or an excellent learning experience, depending on how you
look at it :-) )
nsIPrefBranch (and nsIPrefService) replace nsIPref which has been deprecated.
Steve's patch is "doing the right thing" as far as preferences goes.

Comment 23

18 years ago
yeah, I figured, but my do_GetService comment still applies.
Yep, that's actually my fault... I just found out this morning that
NS_WITH_SERVICE was depricated. :(
Created attachment 33110 [details] [diff] [review]
Same patch with do_GetService() instead of NS_WITH_SERVICE.

Comment 26

18 years ago
looks ok, except for one thing that I should have noticed before -
GetAllNewMessagesForAllFolders should be a method of nsIImapIncomingServer, not
nsImapMailFolder. Does that make sense? You really want to ask the server to get
all messages for all its folders, not an imap folder, or the root folder.
I think it's cleaner as a helper function to the existing GetNewMessages()
instead of a method of any class, so I've made that change and am attaching a
new patch.
Created attachment 33427 [details] [diff] [review]
Same patch, but changed from a method to a helper function.

Comment 29

18 years ago
think about it in an object-oriented sense. Why would you ask an imap folder to
get all messages in all folders? That's something the server object should do.
Created attachment 33601 [details] [diff] [review]
Same patch but with new method moved to nsImapIncomingServer class.

Comment 31

18 years ago
*** Bug 65567 has been marked as a duplicate of this bug. ***
*** Bug 76688 has been marked as a duplicate of this bug. ***

Updated

18 years ago
QA Contact: lchiang → huang

Comment 33

18 years ago
*** Bug 86564 has been marked as a duplicate of this bug. ***

Comment 34

18 years ago
r=bienvenu
Added patch and review keywords. Looking for sr.
Keywords: helpwanted → patch, review

Comment 36

18 years ago
sr=mscott
Checked in a fix. For now, if you want to use the feature, you have to set
mail.check_all_imap_folders_for_new to true in prefs.js. Bug 91246 is to surface
this to the UI.
Status: ASSIGNED → RESOLVED
Last Resolved: 18 years ago
Keywords: patch, review
Resolution: --- → FIXED

Comment 38

18 years ago
When I enable this property in 2001072006, mail gets very flaky (inbox filters
fail to work, I can't read messages when I select them, compacting the inbox
stops).  There always seem to be multiple UW IMAP daemons running.  If I disable
the property, mail behaves well again.

Should this be entered as a new bug, or this one reopened?

Comment 39

18 years ago
I'll re-open this. I found a problem or two with this patch that might be
related to your problem.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
I am investigating.
Status: REOPENED → ASSIGNED
I'll just toss this out for discussion: does it make sense to people that the 
GetMsg button in mail should behave more like it does in news? That is, 
whatever mail folder is highlighted is the folder that gets checked for new 
mail, including any subfolders. And if the server itself is highlight, all 
folders are checked for new mail. Biff would check all folders.

Comment 42

18 years ago
De-lurking here...

I'm voting for this bug so that I can configure Mozilla to check for new mail in
all IMAP folders.  This is primarily how I need biff to work, but I'm currently
using 4.7 in POP mode until this gets put into Mozilla (don't ask!), and I often
hit "Get new mail" to download all my new mail for all folders.  I would like to
see that behavior preserved in Mozilla.

I'd be willing to accept something esoteric as a compromise, like shift-clicking
GetMsg gets mail for all folders, while plain-clicking behaves as Steve
described.  That, or yet another prefs.js item.
I think if you say to Mozilla: "Hey Moz, check my acounts for new emails",
Mozilla should check all acounts and all subfolders.

Optionally, you can have right-button menus for saying:
"Hey Moz, check only this acount" or
"Hey Moz, check only this folder and his subfolders"

When this is fixed, maybe will also include a window with a progress
bar indicating how many folder are being checked and how log will
it take.

Oliver

Comment 44

18 years ago
Steve,

That sounds great :-) what's Biff?
Of course, in an ideal world there'd be a pane somewhere like the 'select 
folders for offline download' one which you could use to specify what folders 
on what servers you want to check for mail when you hit 'Get Messages'...

cheers,

Chris
(Reporter)

Comment 45

18 years ago
Biff is the new mail notifier.

Steve, do you have an figures on the resources (time, server thread on UW-IMAP)
needed per folder? As I said, I have almost 400 folders and counting, and I am
scared to even *try* the pref.
No, I don't have such figures. However, you might want to wait--I'm changing the
way this works so that the server doesn't get bombed with so many simultanious
requests. Seems that some servers don't like that much. I'm going to do the
requests one at a time.
(Reporter)

Comment 47

18 years ago
If it isn't too much work, it'd be nice to make it a backend-pref. For servers
that do like it (both the software and the admin), it'll improve speed quite a
lot, I guess.

Comment 48

18 years ago
I hope that when you change it to one request at a time you can still avoid 
having a spearate login for each folder. That's what kills the performance in 
other clients that update the counts for all folders, and would make this 
impractical for reasonably large numbers of folders.

Comment 49

18 years ago
I lean slightly towards having a back-end pref to enable simultaneous requests. 
I plan to use a private IMAP server on my local machine, so I'm not too worried
about load or bandwidth issues.  Parallel or serial probably won't make much of
a difference in my personal setup -- I just like to see lots of configuration
options.  :)
Created attachment 43285 [details] [diff] [review]
New code--chain the URLs instead of running them simultanious.
So I've redone this, chaining the URLs instead of submitting them all at once.
The first command has to complete before the next is sent. It doesn't take much
longer--maybe 10 seconds total to check 100 folders on our Netscape mail server.
It doesn't authenticate to the server for every folder.
(Reporter)

Comment 52

18 years ago
diff -b -u -r1.25 nsIImapIncomingServer.idl
-  void GetNewMessagesAllFolders(in nsIMsgFolder aRootFolder, in nsIMsgWindow
aWindow); 
+  void SetupCheckAllFoldersForNew(); 
+  void GetNewMessagesNextFolder(in nsIMsgWindow aWindow);


Isn't the fetch strategy an implementation detail? Can you reuse the old interface?

Also, the interface will break, if you implement the "Get only selected folders"
as I and others suggested.
(Reporter)

Comment 53

18 years ago
I shortly discussed my last comment with Stephen. Please ignore it.

Comment 54

18 years ago
could you attach a -uw diff so I don't see the whitespace changes. Also, you
removed the #ifdef DEBUG_bienvenu around an assertion - I'd rather that was left
in. Also, you added a switch statement that just has a break, and that should
have been handled by the default case just fine, I believe.

+        case nsIImapUrl::nsImapSelectFolder:
+            break;

Also, going backwards through that array might be a bit surprising for users,
unless the array was backwards to start with. In other words, I think the users
would expect the counts to get updated from the top down.

More comments after you do a -uw diff.
Created attachment 43454 [details] [diff] [review]
Patch with bienvenu's suggestions.
I've made the diff with the -w, but it doesn't help much. One function is being
entirely replaced with another with a few of the same lines as the old one, so
it's really hard to read as a diff. 
Keywords: patch, review

Comment 57

18 years ago
any update on the patch...?
I think the last patch I attached is better--waiting for feedback from the
mailnews guys.
Keywords: nsenterprise
Target Milestone: --- → mozilla0.9.4

Comment 59

18 years ago
I've read through the comments posted for this bug, but haven't gotten a good
sense what the status of this bug is.  But, that's ok, too.  I just wanted to
mention on thing about Maildir format IMAP servers - contrary to MBOX style
servers, Maildir's have three folders: cur,new,tmp (I'm not positive on the last
one, but the last one isn't important for this message).  In any case - it would
seem that checking the 'new' folder for each IMAP Maildir might be pretty fast,
compared w/ MBOX style mail folders.  Disclaimer: I'm not an expert on IMAP
protocols, I'm not sure what all is involved here.  Just wanted to toss this in.
 I know when I was setting up my mail server w/ qmail & courier-imap (both using
Maildir) one of the things it mentioned was performance benefits of not having
to parse through the entire MBOX to find a new message.

Maybe this is all server side and it doesn't matter to Mozilla, but if it does -
I figure I ought to bring it up!

Btw, is this usable now?  I've been kind of waiting to do server-side filtering
myself because of the lack of this capability.
-Brice

Comment 60

18 years ago
I've added this in prefs "mail.check_all_imap_folders_for_new to true" and its 
been working fine until the latest nightly build for Solaris 2.6 where it no 
longer works :-(. The last Solaris build was 8/17 - any updates coming soon?

Updated

18 years ago
Keywords: nsenterprise → nsenterprise+

Comment 61

18 years ago
Hello,

Just tried this option in my prefs but unfortunatly this has spawned another bug
for me.

My IMAP server does not support "folders that messages AND subfolders" but is
does support "folders that messages OR subfolders" and I have plenty set up.

Whenever I click on a folder that has subfolders (i.e. no messages - just used
for catogorisation/filing) it says "The command did not succeed. The mail server
responded: SELECT failed: Can't open xxxxxx: not a selectable mailbox." where
xxx is the name of a folder. This is fine and I would expect this error to be
thrown as the mailbox does not contain any messages.

Unfortunatly, when the check_all_imap_folders_for_new is set, every time my mail
is checked, it displays quite a few of these error message one after the other!

This is very annoying and requires quite a few clicks to get rid of them all!

Is there a hidden pref to disable this message completely (or at least during a
check all routine). If not can someone add one?

Many thanks

Comment 62

18 years ago
Please disregard the last comment.

It was my fault.

My .mailboxlist file was incorrect (through no fault of mozilla's) and did not
contain delimiting forward slashes at the end of folder names.

Sorry for the confusion.

Comment 63

18 years ago
*** Bug 97717 has been marked as a duplicate of this bug. ***

Comment 64

18 years ago
*** Bug 96944 has been marked as a duplicate of this bug. ***

Comment 65

18 years ago
*** Bug 93493 has been marked as a duplicate of this bug. ***

Updated

18 years ago
Blocks: 91246

Comment 66

18 years ago
*** Bug 92690 has been marked as a duplicate of this bug. ***

Comment 67

18 years ago
*** Bug 87977 has been marked as a duplicate of this bug. ***

Comment 68

18 years ago
I am wondering to know what's the differences between this bug & bug 18266?

Comment 69

18 years ago
Oops! I mean "I am wondering to know what's the differences between this bug &
bug 57660".....

Comment 70

18 years ago
Bienvenu, perhaps you could comment on the 'new' (one month old, really) patch? 

Thanks.

Comment 71

18 years ago
I'm wondering if there's any chance of being able to select this
behaviour on a per-folder basis?

I have 150 folders in a directory hierarchy, often accessed over IMAP
via a relatively slow link. When I turn on this feature mozilla becomes
more or less unusable, "freezing" up for minutes on end every time it
checks for new mail.

I use procmail to filter my mail, and "new" mail will only arrive in
around 10 of my folders. The other 140 will never have anything arrive
in them unless I file it there so it's insane for mozilla to check them
all.

I'd like to be able to select the folders that mozilla checks for new
mail in addition to INBOX.

Comment 72

18 years ago
I'm sure I've been begging for just this functionality for qutie a while now ;-)

Something like the dialog that lets you choose which folders are downloaded for 
offline reading would be great...
(Reporter)

Comment 73

18 years ago
> I mean "I am wondering to know what's the differences between this bug &
> bug 57660".....

The other bug is about Mozilla's (i.e. client-side) filters, esp with POP
accounts. This bug is about asking the *IMAP server*, which folders other than
INBOX have new mails (so server-side filters, which are unrelated to Mozilla,
can be used more nicely).
0.9.4 is out the door
Target Milestone: mozilla0.9.4 → mozilla0.9.5

Comment 75

18 years ago
Is this a dup?  Jeff Hooker says it works on the 09/17/01 build. Can someone
please verify?  Thanks. Making nsenterprise-
Keywords: nsenterprise+ → nsenterprise-

Comment 76

18 years ago
> Is this a dup?  Jeff Hooker says it works on the 09/17/01 build. Can someone
please verify? 

Well it sorta work if you edit the prefs file. But it slow like h-ll and it 
doesnt work "right". First it looks like it downloads all headers and not just 
check the status of the boxes! Then it check for "Unseen" mail and NOT "Recent" 
mail as it should be. (I dont want to know how many mail i havent read, i want 
to know how many NEW mail there are since the last check) And it only works if 
you manually check (Get new mail). I really want to see this one gets fixed, 
Mozilla Mail is useless for me if i doesnt work. 
(Reporter)

Comment 77

18 years ago
> First it looks like it downloads all headers and not just check the status of
> the boxes!

It downloads all headers of all folders??

Comment 78

18 years ago
> Then it check for "Unseen" mail and NOT "Recent" 
> mail as it should be. (I dont want to know how many mail i havent read, i want 
> to know how many NEW mail there are since the last check) 

My opinion that the folder status of Unread and read should remain as is. The
'unread' message count gets updated everytime there is new mail. I prefer that
it tell me the number of unread messages, not just what has recently arrived.
Just knowing what recently arrived will falsely give me impression that I less
unread mail then I actually have.

> And it only works if you manually check (Get new mail).

In server settings I have set "check for new messages every XX minutes. This
works great as I never need to use the "Get Mail" button.

Overall, this enhancement has been working great for me!

Comment 79

18 years ago
> It downloads all headers of all folders??

No, only subscribed folders I think.

Regarding the Unseen/Recent, the best would off course, to have both! Or a 
simple way to choose the one you like.

Comment 80

18 years ago
Sorry it took so long to get back to you - you should try pinging me if it takes
me more than a couple days to respond - it means I forgot.

Tabs seem messed up in this area:

+        // Take the last folder off the array and check for new messages on it. 
+        // The size of this array is the total number of mail folders on the
+        // server. Should be a resonable size: a few hundred at most. 
+
	PRUint32 count = 0;
+
	rv = m_foldersToCheckForNew->Count(&count);
+
	NS_ASSERTION(NS_SUCCEEDED(rv), "Count failed");
+        NS_ASSERTION(count > 0, "If count is now 0, m_foldersToCheckForNew
should be have been cleared.");

I don't like adding NS_ERROR_FAILURE's since it's a meaningless error code. You
can use NS_ERROR_NULL_POINTER in +nsImapIncomingServer::SetupCheckAllFoldersForNew()

and you can propagate the actual rv here:
+        if (NS_FAILED(rv) || (count <= 0))
+            return NS_ERROR_FAILURE;

also, shouldn't you clear m_foldersToCheckForNew in the above error case?

Also, this comment is not sufficient and a bit misleading:
+              // Call back to the server so it can chain to next next folder 
+              // to check for new messages.
Where you say server, I think you should say nsImapIncomingServer so that it's a
little clearer you're not referring to the actual server, but a c++ object.
Also, this should read something like : "if we're checking for new messages in
all folders, allow the server to get messages for the next folder in the list"
or soemthing like that. It should be clearer that this is the exception case,
not the normal case. It would be better if we did this with a url listener in
the one case where we are doing this chaining, instead of changing the main
control path for all folder selects. I'll try to look at the code to explain
better how that might be done.



, and in
(Reporter)

Comment 81

18 years ago
> > It downloads all headers of all folders??

> No, only subscribed folders I think.

Duh! This means that it will download 50,000 headers or so in my case.

(Sorry for just complaining and not contributing.)

Comment 82

18 years ago
>> No, only subscribed folders I think.
>Duh! This means that it will download 50,000 headers or so in my case.

Yeah, having it just suck down all mail from all folders is counter-productive.
 For anyone on less than 100Mbps it will suck for almost all situations.  And
even on 100Mbps, with my IMAP mail tree, it'll suck.

This is why I hope you add a flag to the IMAP tree class that defines wether
it's polled or not, and have it default to off.  Add a way of flipping it to the
properties.  That way, when I open my client or it otherwise calls the poll
member, it'll check for mail in all flagged folders (you could also have INBOX
respect this, and have INBOX default to true -- preserving old behaviour for
those who don't change it, but enchancing the semantics).

OK, I believe this issue is bigger than I thought, and it's going to go beyond 
my understanding of the mail/news subsystem to fix it. I would like to punt, 
and hand it back over to the mail/news guys. Hopefully you guys can use my work 
as a starting point.
Assignee: smeredith → mscott
Status: ASSIGNED → NEW
QA Contact: huang → esther

Comment 84

18 years ago
*** Bug 100557 has been marked as a duplicate of this bug. ***

Comment 85

18 years ago
That being said, *being able* to suck down all the headers for all folders on
command is actually a very useful feature.  That way you can prime your header
caches when plugged into a fast network (for mobile systems) or overnight etc.
(for systems with a slow, but nailed-up connection.)

Updated

18 years ago
QA Contact: esther → huang

Comment 86

18 years ago
Would it be possible to extend the offline sync usage?  Maybe have a copy of the
code which does regular polling, or extend the offline behaviour to always be
synced up?

Comment 87

18 years ago
Do you mean instead of checking this code in? I think that would be wasted
effort because you'd end up with the functionality of this code, which most
people agree is not ideal. I think we need one or two things - the ability to
selectively control which folders get checked for new messages at the least, and
perhaps the ability to check for new messages w/o downloading all the headers.
The way the latter would work might be something like this:
In the first pass, do an examine/select of a folder, and remember the total, and
unseen counts for a folder. In the subsequent passes, do the same and compare
the values. If the unseen counts have gone up, then there have been new messages
put in the folder. There are some tricky aspects to this - because by default,
we don't show you deleted messages, but the imap server counts them, we can't
know if we're really going to show/download a new header. Also, we wouldn't want
to do this for the inbox for similar reasons, since we filter messages out of
the inbox ourselves but leave them there unread but deleted. And, the UNSEEN
count is not really that reliable. Also, if the user selects a folder, causing
us to download all the headers, we should reset our counts. That's all a little
involved, but I'm just writing it down.
Status: NEW → RESOLVED
Last Resolved: 18 years ago18 years ago
Resolution: --- → FIXED
(Reporter)

Comment 88

18 years ago
bienvenu, you marked this fixed. I don't think it should be, because, although
the current implementation is probably a good start, the state is so that most
people have no use for it as-is. As the original reporter, I REOPEN it until the
most urgent problems are resolved. I wrote that in the Status Whiteboard.

> I think we need one or two things - the ability to
> selectively control which folders get checked for new messages at the least,
> and perhaps the ability to check for new messages w/o downloading all the
> headers.

Agreed. I filed bug 101219 about the latter. I think the former should be
implemented before this bug can be considered fixed.

Some other possible remaining problems (not necessarily blockers for this bug),
collecting from comments:
- I think that we should take a serious look at the stress this feature puts on
servers before we activate UI for it. IMAP clients are known to stress servers a
lot and this fact blocks IMAP deployance (e.g. at ISPs and webmail providers)
seriously. We should take care not to make it unnecessariy worse. What is the
state now? We sequentially check folders, right? We use only one login? IF so,
that seems to be OK.
- If the .mailboxlist is malformed (I vagely remember that MSOE causes this),
lots of error msgs come up. This would be fixed, if the user manually selects
folders to be checked. It isn't, if we keep an option which checks all folders.
- If Mailnews starts up and "Get new mail each x mins" is not checked, the user
has to manually click Get Mail to get mail in all folders. Would be annoying.
- If all folders are checked, don't check Trash. (Don't know, if that is fixed,
nobody commented on it.)
- Did I forget/miss something?

Finally, thanks to Steve (and his employer) for implementing a large part of
this bug!
Blocks: 101219
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Whiteboard: Backend pref for checking all(!) subscribed folders checked in; need to make checked folders selectable

Comment 89

18 years ago
I marked it assigned, not fixed. (perhaps there's a browser bug). So calm down!

Comment 90

18 years ago
Regarding "the ability to check for new messages w/o downloading all the
headers." -- I thought Mozilla did this already when I select a folder it has
been to before, because it keeps a header index.  When there are new headers, it
quickly retrieves the new ones and adds them to the index...

Surely this new check code doesn't just redownload ALL headers, right?

IMO: whatever code currently checks for mail in INBOX should be exetended to
check for all members of the tree(s) which have a flag.  By default, INBOX gets
the flag (and it's off elsewhere).  That way the current get now mail code is
unchanged to the outside, although the tree elements are extended and the
get_new code has an extra loop (perhaps sub-member for tree parsing?).


Comment 91

18 years ago
It only downloads the headers it hasn't downloaded before. The code basically
simulates the user clicking on each folder. I hope that was clear to everyone
who commented. If not, I'm sorry.

Comment 92

18 years ago
It was not clear to me that this patch downloads all new headers for all the 
folders that are being checked. Outlook Express has the behavior that I want: 
It goes through all subscribed folders using only one login to the IMAP server, 
and updates the unread message count displayed in the folder list.

That is fast enough even with lots of folders and a slow connection. That's the 
minimum functionality that makes me continue to use OE, necessary because I 
filter to IMAP folders using procmail on the server.

What I don't know, because I haven't yet seen a mail client do it, is how fast 
downloading all new headers would be if it is done with just one login. If that 
is what this patch does it might be acceptible, but if it is too slow you 
should drop the download of the headers. Outlook Express will not download the 
new headers without a separate login to the server for each folder, and that 
definitely is too slow to be practical. It might turn out to be fast enough to 
download the new headers when the check reveals that a folder has some new 
headers if the whole thing is done with just one login.
(Reporter)

Comment 93

18 years ago
Please move the discussion of 'download all heasers' vs. 'update msg counts
only' to bug 101219.

Comment 94

18 years ago
So this bug is ONLY about getting headers for folders other than INBOX.

Ok, so what we want then is a patch which allows Moz to ALSO get all new headers
in subfolders with a new msg count (see bug 101219 about how it finds which
subfolders have new msgs).

That way, when we mark folders (once bug 101219 functionality is in) to be
monitored for new msg counts, we can have this patch suck down all the new
headers.  The functionality in 101219 will let us pick which folders we monitor
(so we don't suck down headers in archive folders which are seldom touched
except for searches).  Accurate summary?
(Reporter)

Comment 95

18 years ago
> So this bug is ONLY about getting headers for folders other than INBOX.

No.

What I need is that the msg *count* in the folder pane is updated. Other people
might need the biff to notify about new msgs.

If/how/when new headers are downloaded is topic of bug 101219 and no more topic
of this bug.
*** Bug 57952 has been marked as a duplicate of this bug. ***
0.9.5 is out the door. bumping up the TM one notch
Target Milestone: mozilla0.9.5 → mozilla0.9.6

Comment 98

17 years ago
The current snapshot of Evolution (mail client in Gnome, from ximian.com) seems 
to check for new messages in IMAP folders reliably, in case it helps to look at 
how they do it.

I just changed IMAP servers to Courier IMAP (under RedHat 7.1) and the current 
patch here ends to kill my connection. I suspect that generated multiple 
threads to check folders is a mistake: It is easy for server defences against 
DOS attacks to trigger if it sees too many probes of the same port from the 
same ip address. Based on the performance I see in Outlook Express (which does 
this exactly right if you don't set it to download all headers when it checks 
for new message counts) and in Evolution, as long as you can use a single login 
to the IMAP server to handle all folders, performance will be fast enough to 
run the check in a single thread.

Comment 99

17 years ago
If you really want to know how it should work take a look at Cyrusofts Mulberry 
(runs on Win, Linux and Mac) Ive got two different accounts and about 10-15 
subfolders on each account who can have new mail in them. In some of them there 
are often 50-200 new mails every day. Mulberry takes about 5-10 sec. to check 
and report how many recent, unseen and total mails there are on both accounts 
and all subfolders and this is on an ISDN 64Kb connection. Mozilla, with this 
patch, can take up to 3-7 minutes! 
Of course Mulberry only starts to download headers after I choose to open one 
of the folders, and then it has a feature to only download the last X messages, 
very nice when you're on an slow connection.

Comment 100

17 years ago
reassigning to bienvenu and moving to 1.0.
Assignee: mscott → bienvenu
Status: REOPENED → NEW
Target Milestone: mozilla0.9.6 → mozilla1.0

Comment 101

17 years ago
*** Bug 104958 has been marked as a duplicate of this bug. ***

Comment 102

17 years ago
*** Bug 107358 has been marked as a duplicate of this bug. ***

Comment 103

17 years ago
One thought: each IMAP folder has "get status" properties for it (there is
already an option for offline reading anyway). You could specify does current
folder message count get updated at all (when pressing Get New Messages) or does
it have its own update interval.

There is no point updating automagically ALL IMAP folders because some of them
may get new messages only when user is sending messages manually into it. (Mail
archives for example).

If the mail filtering is done at the server side, I'd like to see general
discussion mailing list message count updated once a day and other folders more
often.

Looks like possibility to overconfigure the messenger but I assume that the user
who knows how to enable IMAP server side mail filtering is smart enough to
configure each folder separately and this way the server polling overhead is
still kept minimal.

Headers will be downloaded when user clicks on one folder as it works now.

I remember that StarOffice 5.2 IMAP client was built up this way.

(Sorry If I got confused about differences between 18266/101219 ;))

Updated

17 years ago
Target Milestone: mozilla1.0 → mozilla1.2
*** Bug 83855 has been marked as a duplicate of this bug. ***

Comment 105

17 years ago
*** Bug 116891 has been marked as a duplicate of this bug. ***

Comment 106

17 years ago
I tried to enable this feature on build 0.9.7, both under Linux and Win2K. I
have been running into some trouble though under both platforms. I'm not really
sure what is going on but I'll try to explain. 

Under Win2K I enable both this feature and using offline mode (through Offline &
Disk Space, Make the messages in my inbox available when I'm working offline).
After downloading my messages I quit out of mozilla, the GUI disappered but the
mozilla process didn't die. After killing this process and restarting the
Mozilla me messages where now not synced with the headers. I suspect BUG
http://bugzilla.mozilla.org/show_bug.cgi?id=50155 is causing this to happen.
This may or may not have anything to do with this feature, the reason I suspect
this is due to my testing with Linux.

Under Linux I enabled the same features. I now frequenltly see crasches when
staring the Mail Client (3 out of 4 times approx). I have reported those
crasches back through the talkback client, ID TB110854Q for example. Switching
offline mode off still caused the craches to appear. Turning the
check_all_imap_folders feature off did however cause mozilla to behave again.

I realize this is not much help, though I can't reproduce the steps and cause
things to fail every time, only most of the time... Maybe the incident ID will
provide some more information

Comment 107

17 years ago
*** Bug 110953 has been marked as a duplicate of this bug. ***

Comment 108

17 years ago
The last patch for this bug is dated 7/24/01, but discussion reveals that the
patch is not necessarily ideal. Also, comments indicate that this functionality
crashes recent builds. My question: has there been any recent progress on this
bug (maybe under a different bug #) recently? This functionality is actually
quite basic, for providing a 'complete' IMAP client implementation, in my
opinion. From what I've gathered in the comments and through personal
experience, there are hardly any other IMAP clients that do NOT implement this
feature. Personally, its keeping me from implementing server-side filtering, a
desperately desired feature, so that I don't have to move all my filter rules if
I need to use a different mail client (on a different machine).

Comment 109

17 years ago
Yes, I'm interesting in resolving this bug too. I have only 15 imap folders for
now (yes, they counting, but not very fast) and this option works good for me.
But I agree - this option must handle many imap folders reliably and
efficiently. May be as a good starting point to implement addition per-folder
setting (that allows to automatically check only mentioned folders). 
This suggestion was discussed in this tread and I dont realize the reason why
this theme was disappeared. 

Comment 110

17 years ago
Another question - why "Target Milestone" was set to mozilla1.2
I think many people cant use IMAP if this feature didnt implemented well.

Comment 111

17 years ago
> Another question - why "Target Milestone" was set to mozilla1.2
> I think many people cant use IMAP if this feature didnt implemented well.

Yes! Including me! Mozilla Mail is worthless to me if this does'nt get fixed!
I think its a shame that Netscape/Mozilla has more or less dropped it once 
very good IMAP support (when 4,5 came out it was one of the best, but there 
has been no progress since then) Please fix this before 1.0.

Comment 112

17 years ago
>Another question - why "Target Milestone" was set to mozilla1.2
>I think many people cant use IMAP if this feature didnt implemented well.

I disagree. I need this implemented, too, but there are many more critical
things to do, mostly things that make people dump the whole thing even before
they realize this 'bug'.

Comment 113

17 years ago
The lack of this feature almost prevent to use server-side IMAP filtres

The main benefit from using IMAP: your'e always stay in sync with all of your
email stuff. If you have server-side filters, you REALLY can use your mail
account wherewer you are. With client-side filters much of IMAP goodness just go
away and there almost no reason to prefer IMAP over POP.

Conclusion - without this function was correctrly implemented there almost NO
USE of ian at all (in contrast to POP)

This is my IMHO

Comment 114

17 years ago
------- Additional Comments From yak@jet.msk.su  2002-02-07 01:26 -------
The lack of this feature almost prevent to use server-side IMAP filtres

The main benefit from using IMAP: your'e always stay in sync with all of your
email stuff. If you have server-side filters, you REALLY can use your mail
account wherewer you are. With client-side filters much of IMAP goodness just go
away and there almost no reason to prefer IMAP over POP.


This is perfectly right.

But it works for me since 0.9.6 I think

add
user_pref("mail.check_all_imap_folders_for_new", true);
in pref.js

then every # min it automaticaly checks my 38 folders
that I filter with procmail.

All except the Trash, it was a choice from the guy who made
the patch, but I dont see why, because I'd like the trash to
be checked as well, I have some certainly badly written
rules for spam that directs some mail to trash...)

It's true that a "per folder" choice would be excellent...

-- 
bye everyone

          Olivier.

Comment 115

17 years ago
>add
>user_pref("mail.check_all_imap_folders_for_new", true);
>in pref.js

I just downloaded the latest nightly (for windows) and I don't see pref.js

Do I put it in mailnews.js? And is this still a valid entry?

Comment 116

17 years ago
This from the Mozilla Mail-News Prefs web page:

"You should not be editing your prefs.js by hand. If you feel you need to, its
probably a bug on our part, and you should log a bug."
Charles: Czo really did mean prefs.js ;). It's not in your Mozilla/bin directory
-- rather, it's probably in /Documents and Settings/Profiles/[Windows
username]/Mozilla/Profiles/[Mozilla profilename]/[salted directory name]/prefs.js

Some of that path is from memory, so if you can't find prefs.js, just do a
"Search" in Explorer for "prefs.js". 

And, regarding the warning: that's just saying that editing should only be done
by advanced users. You are allowed to ignore it, if you know what you're doing :).

Comment 118

17 years ago
Isn't userprefs.js the preferred place for such things?

Anyway, to get back on topic, I'm eagerly awaiting a proper fix for this bug.  I
have about 200 folders for various mailing lists, and yaz is right that IMAP is
about the same as POP without server-side filters.

Of course, server-side filters don't update the index files of each client, so
the mail I read on my work machine doesn't get marked as read on my home
machine.  What'd be really neat if each folder's index file were stored on the
server as well.  That way, I can read my filtered mail, and see which messages
I've read, from any Mozilla client.  Sweet.
(Reporter)

Comment 119

17 years ago
re prefs.js, please read an *arbitary* Netscape FAQ, for any version.

> mail I read on my work machine doesn't get marked as read on my home machine

marcarc, the read status of individual messages should be stored on the IMAP
server and honored by the clients. If not, that's a bug in the client or server.
Anyways, it's offtopic here, because this is about folders, not messages.

Comment 120

17 years ago
> marcarc, the read status of individual messages should be stored on the IMAP
> server and honored by the clients. If not, that's a bug in the client or server.
> Anyways, it's offtopic here, because this is about folders, not messages.

I don't think its offtopic, this is an other reason why I use imap...
It works for me : I use mozilla 0.9.7 (sometimes outlook express)
and both theese imap client honor the read status and the flag message.

marcarc, your problem might come from the server you use...
mine is IMAP4rev1 (UW 2000.287), so ask your sysadmin

btw, my pref.js is located in :
C:\Documents and Settings\czo\Application Data\Mozilla\Profiles\czo

bye.

Comment 121

17 years ago
This behaviour is preventing me from using Mozilla Mail as my email 
application. My email gets sorted server-side by a set of filters into various 
subfolders, and it's of little use to me if I can't tell when I have new email 
in these subfolders without manually having to go through and browse each 
folder one after another looking for new messages.

Given that Mozilla insists on removing my default mail application association 
each time it starts (does anyone know the bug # for this?), I thought I might 
consider moving to Mozilla Mail to alleviate this problem. Sadly, I can't do 
this while this bug still stands, and Mozilla refuses to leave my default mail 
client association alone, causing quite some distress.
Antony: You are aware of the pref, right? (see also comment #114)

PS When you comment in a bug, be sure to add yourself to the CC list, otherwise
you don't see any of the reponses :).

Comment 123

17 years ago
For what it's worth, I decided to try out the pref option mentioned in this bug
report.  I'm using Mozilla 0.9.6 on Linux RedHat 6.2 (0.9.7 and .8 have been
unusable for me, probably because my setup is so old -- I can't easily update
it, however.)  Unfortunately the performance is unacceptable, as it immediately
went to download all the headers for all the messages -- but apparently did so
in a way that the server did not handle well.  After having had Mozilla saturate
my 384 kbps DSL line for about four hours, and receiving uncountable cryptic
error messages without having any way of knowing which folder they applied to, I
gave up and turned the option off.
(Assignee)

Comment 124

17 years ago
*** Bug 127867 has been marked as a duplicate of this bug. ***

Comment 125

17 years ago
*** Bug 124319 has been marked as a duplicate of this bug. ***

Comment 126

17 years ago
There are several things to do:

1. Change the current code with the new (not so new) patch from 07/24/01 so we
don't try to download all the folders at once. Because this makes this hidden
pref unusable with most IMAP servers which are guarded against DoS attacks.

2. Then we can move on to change the code to update only message counts and not
to download the new headers.

3. Then we can make some ui work to allow it to check only user-selected folders.

The first thing is almost done (maybe the patch is a little bit rotten) so why
not  finish it and check it in.

Comment 127

17 years ago
> Change the current code with the new (not so new) patch from 07/24/01 so we
> don't try to download all the folders at once. Because this makes this hidden
> pref unusable with most IMAP servers which are guarded against DoS attacks.

In BuildID 18266 I do not see the "donwload all at once" problem. It only
downloads as many as mail.server.servern.max_cached_connections at once at that
can be set as low as 2 without any problems...

Comment 128

17 years ago
I meant BuildID 2002030417, sorry - must have been half-asleep at the time.

Comment 129

17 years ago
OK, that's better than all at once, but it's not the best solution either. It
should check only one folder at time regardless the max_cached_connections pref.
Because this way as it is now you can't display message while the mozilla is
looking into the folders because it uses all allowed connections to server and
you need one to download the message. And the overload it makes on the server is
also unnecessary - it won't be any faster.

Comment 130

17 years ago
> you can't display message while the mozilla is
> looking into the folders because it uses all allowed connections to server and
> you need one to download the message

Not quite - as soon as one particular folder is done, the connection will free
up and Moxzilla will d/l the message before going to the next folder.

> And the overload it makes on the server is also unnecessary - it won't be any
faster.

This is definitely untrue. I ran Courier IMAP locally (e.g. on localhost). I
experimented with setting different number of max_cached_connections. If I
delete all the msf and start Mozilla (which means it now has to d/l all the
headers for about 20 folders), with max_cached_connections=5 it does it much
faster than with max_cached_connections=2.

IMHO, the only things that might be needed is to:
- Make sure Mozilla never really has more than max_cached_connections open.
- Have a UI for setting it.

Comment 131

17 years ago
*** Bug 134808 has been marked as a duplicate of this bug. ***

Comment 132

17 years ago
*** Bug 129754 has been marked as a duplicate of this bug. ***
*** Bug 138153 has been marked as a duplicate of this bug. ***
*** Bug 122732 has been marked as a duplicate of this bug. ***

Comment 135

17 years ago
*** Bug 139745 has been marked as a duplicate of this bug. ***

Comment 136

17 years ago
*** Bug 141509 has been marked as a duplicate of this bug. ***

Comment 137

17 years ago
*** Bug 140620 has been marked as a duplicate of this bug. ***

Comment 138

17 years ago
The IMAP protocol allows you to tell whether the folder has been 'marked' as
interesting by the server (because something has happenned on the server side).
Mozilla should automatically check the status of any folders chich as the result
of a "LIST" imap command return that they are \MARKED ...

Thus it wouldn't be that expensive at the start of each mozilla mail/news
session to check what the status of all subscribed folders are.

Comment 139

17 years ago
that's a good suggestion. There's no guarantee that any particular server will
set the /MARKED flag, but we should have an option to query MARKED folders for
new messages. However, we only LIST (actually, LSUB) folders at startup, or when
you expand and collapse the server hierarchy, so this isn't a complete solution.

Comment 140

17 years ago
I don't know about anybody else, but it would be a considerable improvement just
to get the old "Update Message Counts" functionality back from NS4. With that,
you can at least manually select the folders you are interested in and update
them quickly and easily.

Comment 141

17 years ago
jifl, for "quick and dirty" the 
user_pref("mail.check_all_imap_folders_for_new", true);
is already implemented (attchment 33601 was checked in - see comment 37) and
works reasonably good. This bug is currently only about making something better.

P.S. Would it make sense to close this bug and open a follow-up to make it clearer?

Comment 142

17 years ago
Yes I've tried that, but it is really quite slow at dealing with even a
comparatively small number of folders. A real killer in fact is if your server
(like UW IMAPD) exports other namespaces like #news into your IMAP view, at
which point enabling this option makes Mozilla want to check every "folder"...
in turn causing the IMAPD to go away and talk to the news server each time, etc.
If it just updated the message count it might be bearable, but downloading the
headers is just far too extreme.

As for the purpose of this bug, other bugs are being considered dups of this bug
because they are for the same _original_ problem. I'm suggesting another way to
work around the original problem, albeit not quite as good certainly. If this
bug is to be constrained to improvements to one particular patch which doesn't
look like is ever going to be implemented, then yes a follow up should be opened
IMHO.
(Assignee)

Comment 143

17 years ago
Created attachment 82423 [details] [diff] [review]
Proposed patch to make it folder specific (a checkbox in folder props)

This patch modifies the current functionality to make it check inbox and all
folders marked to be checked. A checkbox is added to the folder properties. The
old pref for checking all folders is not used anymore. 

Please note that this is my first patch with xul involved. All comments are
welcome.
Attachment #32913 - Attachment is obsolete: true
Attachment #32924 - Attachment is obsolete: true
Attachment #32927 - Attachment is obsolete: true
Attachment #32986 - Attachment is obsolete: true
Attachment #33110 - Attachment is obsolete: true
Attachment #33427 - Attachment is obsolete: true
Attachment #33601 - Attachment is obsolete: true
Attachment #43285 - Attachment is obsolete: true
Attachment #43454 - Attachment is obsolete: true

Comment 144

17 years ago
Ere, is there some way to keep something similar to old behavior to those who
want it? If I do want to have every single folder checked, do I really have to
go mark them all one-by-one (and remember to do it again when a new folder is
created!)?
(Assignee)

Comment 145

17 years ago
Sure, I'll change it so that you can override everything with the old pref.
(Assignee)

Comment 146

17 years ago
Created attachment 82442 [details] [diff] [review]
Patch v1.1 to make it folder specific (a checkbox in folder props)

This is mostly the same as the previous patch, but will check all folders if
mail.check_all_imap_folders_for_new is set.
Attachment #82423 - Attachment is obsolete: true
(Assignee)

Comment 147

17 years ago
Created attachment 82443 [details] [diff] [review]
Same patch without whitespace changes
(Assignee)

Comment 148

17 years ago
Created attachment 82698 [details] [diff] [review]
Patch v1.2 to make it folder specific (a checkbox in folder props)

Fixed flag comparison style and whitespace issues.
Attachment #82442 - Attachment is obsolete: true
Attachment #82443 - Attachment is obsolete: true
(Assignee)

Updated

17 years ago
Whiteboard: Backend pref for checking all(!) subscribed folders checked in; need to make checked folders selectable → Backend pref for checking all(!) subscribed folders checked in; new patch with ability to select folders waiting for review

Comment 149

17 years ago
In general, very cool.

what's this for? Timeouts like this make me nervous.

     if (!initialUri) 
     {
-        MsgGetMessagesForAllServers(defaultServer);
+        // We need to give it a little time to initialize folder list
+        setTimeout("MsgGetMessagesForAllServers(null)", 2000);

I don't like having this routine "GetNewMessagesAllFolders" usually just doing a
biff on the inbox - it's misleadingly named, or perhaps it shouldn't be called
for the inbox case. Does that make sense? Someone looking at the code later on
could get very confused.
(Assignee)

Comment 150

17 years ago
Timeout is there to give it time to load the folder list. Otherwise the folder
list  for the first server might not be complete at the time of initial mail check. 

There is no "just inbox" case anymore. We always need to enumerate all folders
and check the marked ones (or every one of them if that old pref is set).
Therefore I think the name isn't misleading, but I'm always open to suggestions
for a better name.

Comment 151

17 years ago
Without being involved in the development: This timeout doesn't seem a
particularly good idea to me. The proper solution would be to block until the
folder list is loaded. I don't know how difficult that is, but be aware that the
loading time can vary greatly depending on the connection and no. of folders -
not a good case for a fixed timeout.

Comment 152

17 years ago
for you, there's no "just inbox" case, but for at least 99.999% of users and
99.9% of developers, it's just the inbox. I'd really prefer the code the way it
was before for clarity's sake.

biff shouldn't fire before the folder list is built up. The folder list is built
up synchronously. If biff is firing before the folder list is created, that
should be fixed.
(Assignee)

Comment 153

17 years ago
But there is no way to know if there are other folders that should be checked
without enumerating them, and that's essentially what is being done in
GetNewMessagesAllFolders(). 

I'll have to investigate why the folder list isn't loaded before the first biff
fires. Should that be a part of this bug or should I create a new one?

Comment 154

17 years ago
I understand that - all I'm suggesting is that the code to check the inbox be
done where it is now, and the GetNewMessagesAllFolders could skip the inbox, and
be called something like CheckNewMessagesForNonInboxFolders. It might even be
worthwhile at some point to actually cache whether or not any non-inbox folders
need to be checked, in order to skip this method completely for the 99.99% case.
I'm suggesting this so that the code be more readable/maintainable. I know this
feature is very valuable for users who have server-side filters, but it's very rare.

I think the biff thing should be part of this bug, since it's relevant. I
imagine that we're adding the biff timer before we build up the sub-folder list,
but I would have thought this code was pretty much synchronous - i.e., we run
through the startup code, loading the accounts and servers and folders on all
the servers and along the way, set a timer, but the timer won't fire until the
startup code is done, since the timer fires on the ui thread. Now, if you don't
have .msf files for your imap folders, we definitely won't know about the imap
folders until the inbox has been loaded and we go through imap folder discovery,
but that's not what you're seeing, is it? The folder list gets build by calling
GetSubFolders() - where are you in the code that you need the sub-folders, but
don't have them yet? This might be trivially fixed by just calling GetSubFolders
there (emphasis on might - startup code can be tricky...)
(Assignee)

Comment 155

17 years ago
Thanks, I finally got your point. I will do the necessary changes. Btw, I
disagree on the percentage :) And there are cases where this feature is useful
even without server side filters. I don't have any server side filters, but I
have filters at my work machine and home machine (using the same server). I
often leave Mozilla running at work when I go home. I still want to see if there
are new messages at home also, but if the machine at work gets them first and
moves them from inbox, I wouldn't know it without manually checking the folders. 

Comment 156

17 years ago
I used to have the scenario you describe with home and work machines, so I
understand that (though I always did it by accident :-) and the server would
give me duplicate copies of each filtered message, so it's not something I would
ever do on purpose! ). There's also the scenario where you shut down your client
at work but want your counts updated at home. However, the better fix for that
would be to have an option to update all folders on startup, not on every biff.
I biff every minute, and I wouldn't want all my folders selected every minute
:-) As someone commented, having an update message counts command (with an
option to run it at startup), would be a much more server-friendly way of
dealing with the latter scenario. 
(Assignee)

Comment 157

17 years ago
Created attachment 82817 [details] [diff] [review]
Patch v1.3, changed as suggested

This one contains changes requested by David. The setTimeout() from initial
mail check is removed. There was no problem with loading the folder list (it
was my fault that I thought so). The problem was that
MsgGetMessagesForAllServers() bypassed the default server it was given. Now it
is passed null instead.
Attachment #82698 - Attachment is obsolete: true

Comment 158

17 years ago
>bienvenu@netscape.com  said:
> for you, there's no "just inbox" case, but for at least 99.999% of users and
99.9% of developers,

Whoa!! This explains why this bug have seen so little action!
99,9% is such an wrong figure thats its really embarrassing! 
I dont think I know anyone who doesnt use server-side filtering and shared 
folders and just have an look at the mozilla newsgroup, this feature is very 
often requested (nearly once a week!)
Big thanks to Ere Maijala for having an go at this bug which should have been 
fixed a long, long time ago! When can we see it checked in?

Comment 159

17 years ago
If it were truly 99.9%+ of people who don't want this, I don't think that this
would be the 17th highest voted bug and have 26 duplicates.

Comment 160

17 years ago
Everyone likes to think they're the norm, but believe me, it's a tiny
percentage. Or people just imagine the percentages to be what they want them to
be ( 25% of Americans think they're in the top 1% in terms of income. Now,
that's embarrasing! :-)) The percentage is higher among people actively involved
with Mozilla, but among actual users of Mozilla-derived products, I was being
generous saying it was 99.99% - it's  a highly vocal minority. I've been
spending much of my time getting shared folders to work at all in Mozilla, which
I'm sure you would agree is a pre-req for checking shared folders for new postings.

Comment 161

17 years ago
looks good in general.

could you attach a non-whitespace diff?

I don't think you want this change. This is for downloading message bodies, not
message headers, and is specific to the inbox.

+    // check if we should download message bodies because it's the inbox or folder
+    // that should be checked and the server is specified as one where we
+    // download msg bodies automatically.
     PRBool autoDownloadNewHeaders = PR_FALSE;
-    if (mFlags & MSG_FOLDER_FLAG_INBOX)
+    if (mFlags & (MSG_FOLDER_FLAG_INBOX | MSG_FOLDER_FLAG_CHECK_NEW))

Comment 162

17 years ago
>...but among actual users of Mozilla-derived products, I was being
>generous saying it was 99.99% - it's  a highly vocal minority...

There may be a little bit of truth to what you say, but I think it is a mistake 
to minimize this bug's importance using this derived number. As Dir of Admin 
Computer Services at a university, I can tell you that one of the obstacles to 
officially adopting Mozilla campus wide is this bug. We use server side 
filtering with folders, so this bug is of high interest to us.
(Assignee)

Comment 163

17 years ago
Created attachment 82827 [details] [diff] [review]
Patch v1.4, removed the wrong change

Removed the part spotted by David.
Attachment #82817 - Attachment is obsolete: true
(Assignee)

Comment 164

17 years ago
Created attachment 82828 [details] [diff] [review]
Same patch without the whitespaces

Comment 165

17 years ago
Please don't get me wrong. I'm glad this is getting done. I was merely asking
for a code change to make the general case in the code easier to read. I was
also trying, poorly, to say that we have very limited resources here, so we have
to prioritize our work on the stuff that's going to help the most people, like
fixing crashes. But the beauty of Mozilla is that if something is a higher
priority for you than it is for the general population, you can usually do
something about it in the code.

Comment 166

17 years ago
Comment on attachment 82828 [details] [diff] [review]
Same patch without the whitespaces

r=bienvenu, great, Ere, thx.
Attachment #82828 - Flags: review+
(Assignee)

Comment 167

17 years ago
Sr=, please?

Comment 168

17 years ago
Comment on attachment 82828 [details] [diff] [review]
Same patch without the whitespaces

sr=scc
Attachment #82828 - Flags: superreview+
(Assignee)

Comment 169

17 years ago
Checkin requested from timeless.

Comment 170

17 years ago
fix checked in
Assignee: bienvenu → ere
Target Milestone: mozilla1.2alpha → mozilla1.1alpha
sorry for jumping in after the fact.  the sr= request for this bug was in my 
queue

did the new UI get buy off from jglick?

I'll get a screen shot of the imap only checkbox for her. 
Created attachment 83900 [details]
screenshot

note to jglick:

this new checkbox only appears for IMAP folders, except for the Inbox.
(Assignee)

Comment 174

17 years ago
Pardon my ignorance. Afterwards it's easy to say that I should have asked
jglick, but I didn't :(

The only thing I could find about properties in the spec was "Each folder has an
associated Properties Dialog.". Few lines before that sentence there is one
missing image though. (nice thing that the anchor doesn't work in trunk build
20020515.)

Comment 175

17 years ago
Thanks for the fyi Seth. This addition is fine with me and I'll update the 
Folder spec, http://www.mozilla.org/mailnews/specs/folder/, to include it. 

Does this pref check the folder for new messages only at startup? Or whenever 
the Inbox is check for new messages (based on users setting for "check for new 
msgs" interval plus whenever they click "Get Msgs")?

Comment 176

17 years ago
the latter.

Comment 177

17 years ago
Is there any desire to word the pref so that its more clear how enabling this 
setting is different from not enabling it? Or why a user may or may not want to 
check this setting?
sounds like this bug is fixed, and jglick is trying to work out some better wording.

if so, we should mark it fixed and create a new bug about the warning.
s/about the warning/about the wording
updating url to point to the spec.

markin fixed, as this is fixed on the trunk.  we'll take the wording to another bug.
Status: NEW → RESOLVED
Last Resolved: 18 years ago17 years ago
Resolution: --- → FIXED

Comment 181

17 years ago
Question: while this seems to be working great, isn't Mozilla supposed to
trigger the "new mail detected" indicator in the status bar?  This subfolders
patch doesn't seem to provide that (only inbox new mail will trigger it).

Or do I need to file a separate bug?
(Assignee)

Comment 182

17 years ago
I think it's time for a new bug. Cc me if you create one.

Comment 183

17 years ago
is this fix on the branch? See bug 146934

Comment 184

17 years ago
no, it's trunk only.
(Assignee)

Comment 185

17 years ago
*** Bug 146934 has been marked as a duplicate of this bug. ***
(Assignee)

Comment 186

17 years ago
*** Bug 147338 has been marked as a duplicate of this bug. ***

Comment 187

17 years ago
*** Bug 95527 has been marked as a duplicate of this bug. ***
(Reporter)

Comment 188

17 years ago
*** Bug 120996 has been marked as a duplicate of this bug. ***

Comment 189

17 years ago
*** Bug 79288 has been marked as a duplicate of this bug. ***

Comment 190

17 years ago
In recent builds (tested so far using macosx build), this feature is no longer
working. All folders are checked the first time Mozilla opens, but not on any
mail checks thereafter. I've noticed some pretty severe problems with IMAP mail
in recent Mozilla builds. Often, I'll click on a folder and not be able to load
any messages therein (throbber never stops). Almost always, if I go to a folder
and hit the key-combo for mark all messages as read, Moz changes all the
messages from bold to plain and updates the folder window, but the IMAP server
is not actually informed of the message status (i.e., closing Moz and opening
another mail app will still show the messages as new).

Don't know what's added instability recently, but at the moment, mail/IMAP is
practically useless on my system. Some regression has certainly occurred since
1.0 seemed quite useable.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Comment 191

17 years ago
Brice, do you know what the first build was that you noticed had problems (not
just this bug, but any of the imap problems you described)? There were some
changes to the mac event loop handling that could have either caused or exposed
other problems. Also, are you using SSL? Finally, we might want to spin up a bug
for the general problems you're seeing, if there's not already a macosx bug for
them.
(Assignee)

Comment 192

17 years ago
FWIW, mail checking seems to work fine for me in trunk 20002081908 on WinXP.

Comment 193

17 years ago
this is probably a mac-only problem.

Comment 194

17 years ago
I wasn't updating on a nightly basis, so I can't pin-point exactly when this
took place, but its been a couple weeks. I thought (initially) that it was just
a bad nightly, but after waiting a few days and downloading more nightlies, it
became apparent that this wasn't the case.

I do use SSL for IMAP and I also use the Courier-IMAP server, both of which had
bugs that recently had fixes checked in - so changes there could be the culprit.

I need to run out right now, but I hope to do a few things soon: (a) try the
most recent nightly (currently on 2002081508), then step back, week by week,
then day by day, to figure out when these problems first started. I'm relatively
certain that 1.1b was still fine, and I'm 100% certain that 1.1a was fine, so
that should limit my backtracking.

Comment 195

17 years ago
I should note that I rolled back to 1.1b because the IMAP SSL support (against
Cyrus) in 2002080614 seemed to be broken.  I filed bug 161424 for it.  I use
Slackware Linux with the dailies.

The behaviour of sending via SMTP before attepting to write to the SENT folder
seems a little broken: one would assume that it would do the SMTP part last so
that it could more easily handle (without the dataloss of the sent copy and
dupes sent via SMTP) the filing of mails.

I'll file another bug for that, though :)

Comment 196

17 years ago
 bug 161424 has nothing to do with SSL or this bug (it has everything to do with
the sent folder on cyrus servers, and should be fixed on the trunk now)

Comment 197

17 years ago
I'm becoming less convinced that this "regression" is a result of code change in
Mozilla as that its an incompatibility with Jaguar (Mac OS X 10.2, build 6c115).
I'm going to finish up my testing in OS X 10.1.5 and see what kind of results I
see there. The reason I'm starting to think this is that it *seems* at least
*some* of the issues I mentioned, I experience going all the way back to 1.1a
and I KNOW I haven't been dealing with this that long ... I would have long
since reported it! :)

Comment 198

17 years ago
Haven't switched over to 10.1 yet, but the problems definitely seem related to
SSL. I removed my existing Moz profile to start "clean" and noticed I wasn't
having any problems until I turned on SSL. Turning off SSL/IMAP got me back in
business, turning it back on got me back in trouble.

I'll continue my testing on 10.1 and report back any findings I make. Cheers.

Comment 199

17 years ago
I have the same problem in Windows 2000 with 1.1beta. I also tried it with trunk
20002081908 which was noted to work in Comment #192 and it still doesn't work.
I'm using a Cryrus imap server on a RedHat 7.2 machine.

Comment 200

17 years ago
Please disregard my comments about this happening on Windows 2000. I made some
minor changes to my procmail config files and now I get notifications that mail
is in folders other than the inbox. This indicates that the problem was with the
imap server or procmail, not with Mozilla. Sorry about the confusion.

Updated

17 years ago
Blocks: 163993
(Assignee)

Comment 201

17 years ago
I'm re-closing this now (WFM). I'd suggest filing new bugs for any remaining
problems (and feel free to cc me, but bear in mind that I can't fix Mac specific
problems).
Status: REOPENED → RESOLVED
Last Resolved: 17 years ago17 years ago
Resolution: --- → FIXED

Comment 202

17 years ago
Well, I couldn't find an option to enable this for all folder (that is, the
option is there, mail.check_all_imap_folders_for_new, but I couldn't find a way
to toggle it through the preferences dialog, at least not in mozilla 1.1).
And the biff count of messages seems to be totally random (right now it's
telling me that there are 30 new messages and there is actually none - this is
under windows, under linux I don't pay much attention when mozilla beeps at me).
Maybe this is related to the use of mail.check_all_imap_folders_for_new.
Besides, I still find it wasteful to SELECT each folder instead of just STATUS
it (see my comment #20) but I won't complain too much on this issue ;-)
(Assignee)

Comment 203

17 years ago
Luca, I think you have valid points here, but they are out of the scope of this
bug. STATUS/SELECT is bug 101219, I think. There's no UI for
mail.check_all_imap_folders_for_new. And as far I can remember we report the
number of downloaded headers as the number of new messages (which is arguably
wrong). Bug 101219 might have a solution for that also.

Comment 204

17 years ago
Ok, you're right about bug 101219, and the UI for
mail.check_all_imap_folders_for_new should be bug 91246. What misled me is that
your patch is both for the implementation *and* the UI.
I will be posting a comment in bug 91246.
Now, what could be relevant to this bug is that
mail.check_all_imap_folders_for_new is global for all servers. Wouldn't a per
server setting be better?
*** Bug 130952 has been marked as a duplicate of this bug. ***
*** Bug 157881 has been marked as a duplicate of this bug. ***

Comment 207

16 years ago
*** Bug 199288 has been marked as a duplicate of this bug. ***

Comment 208

16 years ago
With mozilla 1.3 mail.check_all_imap_folders_for_new isn't used at startup, I
have to manually check for messages or wait the "check for new messages" time.
This didn't happen with 1.2.1.

Comment 209

16 years ago
I know Ere said to open new bugs for remaining problems, but my issue seems
related specifically to the implemented fix...

I'm using last night's trunk build on Linux:  Mozilla/5.0 (X11; U; Linux i686;
en-US; rv:1.4b) Gecko/20030516

Mozilla's memory usage explodes when I use mail.check_all_imap_folders_for_new.
 Before new mail is checked, Mozilla's footprint is about 30MB.  After checking
for new mail it's about 100-120MB.

Is this expected (in which case I'll open a new bug) or is there a problem
(leak?) with the current implementation?

I also saw these memory problems in 1.3.

(BTW, I also see luca's problem where new mail isn't checked at startup.  But
that seems more appropriate for a separate bug.)

Comment 210

16 years ago
Marc, how many folders do you have? And how big are they? The most likely cause
I can think of is that the .msf files for all of your folders are opened and in
memory, which will eat up a lot of memory. You should be able to tell which .msf
files mozilla has open after the get new mail check runs. If that turns out to
be the case, we need to make sure the .msf files are closed after get new mail
runs (or limit the number that are kept opening - if you have large folders,
opening and closing the .msf files could get expensive as well)

Comment 211

16 years ago
In any case, please do open a new bug.

Comment 212

16 years ago
I've opened bug 206502 about the memory issue, where I asked how I can tell
which .msf files Mozilla is using.
*** Bug 212056 has been marked as a duplicate of this bug. ***
 
 (In reply to comment #213) 
> *** Bug 212056 has bee 
>  ma 
> ked as a duplicate of this bug. *** 
 
Faced the Same Problem with Mozilla Mail and little investigation led me to 
resolve the problem. 
 
If a user has previously used Outlook or evolution, there are chances that he 
might expect the new messages to be highlighed by default, instead of going to 
the properties of each folder and marking it to check for new messages. 
 
Should not be checking for new messages in all IMAP folders be default or made 
an option? 
It is reasonable for a user to expect so. 
 
Thanks! 
Senthil 
Product: MailNews → Core

Updated

14 years ago
No longer blocks: 163993
Product: Core → MailNews Core

Comment 215

9 years ago
I'm using Thunderbird 3.0.1~hg20091208r4496 on Ubuntu 9.10 (from Mozilla
Daily PPA). This doesn't work as intended, new messages are only "discovered" when opening the folder manually (clicking on it). Should I file a new bug?
I am also facing this problem recently. I tried Thunderbird 2 and now 3. Issue is pretty specific. When a new mail arrives in Thunderbird, I won't get any popup window saying "new mail arrived" but when any new mail arrives in inbox, it pops up properly.
Further, I am working in an office where all Thunderbird users are facing this issue.
I have enabled the option "Check this folder for new messages" and also mail.check_all_imap_folders_for_new is enabled.
Please let me know if I should file a new bug.
(Reporter)

Comment 217

9 years ago
Shashikant Prabhakar, please file a new bug and include which server software you use and what exactly you do and which relevant account settings / prefs you use.

Comment 218

8 years ago
I use IMAP server based filtering into sub-folder of the Inbox.
This works as expected when thunderbird (3.1.5) is running:
I get notifications on new emails if the folder has the appropriate checkbox selected ("When getting new messages...").
But whenever I start-up thunderbird it checks ALL sub-folders, regardless of the checkbox state.

I've now set mail.startup.enabledMailCheckOnce to "false" in the preferences, which appears to be on default "true".
This get's rid of the problem, I hope without any side effects.
At least is still shows new messages in the Inbox immediately.

Updated

7 years ago
Duplicate of this bug: 61428
You need to log in before you can comment on or make changes to this bug.