Closed Bug 141369 Opened 18 years ago Closed 17 years ago
Support IMAP4 IDLE command (RFC 2177)
From Bugzilla Helper: User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.0rc1) Gecko/20020417 BuildID: 2002041711 Mozilla mail does not seem to use the IMAP IDLE extension specified by RFC 2177. Would be very nice to have and should not be too complicated to implement. Reproducible: Always Steps to Reproduce: 1. add imap account with an IDLE capable imap server (such as cyrus) 2. open inbox 3. send mail 4. wait Actual Results: nothing happens till the "check for new messeges every # minutes" time has passed. Expected Results: There should be a checkbox "Use Imap IDLE command" which causes mozilla mail to use it.
Sounds useful. I can't find any duplicates so I'm marking this NEW.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Summary tweaking. Before: why not RFC2177 ? After: Support IMAP4 IDLE command (RFC 2177)
Summary: why not RFC2177 ? → Support IMAP4 IDLE command (RFC 2177)
Two things to consider here: Biff Support, for this there is an interesting comment in Bug 18266 (comment 154) - direct link http://bugzilla.mozilla.org/show_bug.cgi?id=18266#c154 and whether to open an additional TCP connection which we poll at least every 29 minutes according to the RFC or change nsImapProtocol to leave the TCP connection open, if this is not already done (I don't know the code well enough yet, my feeling is the connection is closed). The latter is the preferred solution.
we do cache connections (i.e., keep them open). This would probably be a fun little project for a mozilla contributor interested in imap.
It's on my todo list, but I want to properly fix the namespace issues first.
FYI: Courier-IMAP 1.6 adds significant functionality to its IMAP IDLE implementation. On server platforms that have SGI's File Alteration Monitor installed (Linux, Irix, with partial xBSD support) IDLE generates real-time folder status updates (when multiple client sessions have the same folder selected, all clients are notified immediately when any client makes any changes to the folder, and immediately upon arrival of new mail). I scavenged an old laptop with MS OE 4.0, and opened a folder, then also logged in using pine. Even though pine doesn't do IDLE, I played with it, marking messages as deleted and undeleted, expunging the folder, and copying messages into this folder. On the other monitor, OE 4.0 obediently repeated everything I was doing in pine. Worked rather well. So, the scoreboard right now has two of the top three non-commercial IMAP servers offering value-added IMAP IDLE functionality.
Re: Comment #6, yes I upgraded my courier-imap server very recently and added in FAM support. Outlook Express works very well with this extension and I would love to see it in Mozilla Mail at some point! Also, just thought I'd point out that Exchange 5.5 Server (and 2000 Server, AFAIK) support this extension.
I would like to have this feature added also It seems that the person that this is assigned to is (gone) ??? and this has been sitting without any additional comments for quite a while.. It seems that Mozilla is one of the few Email clients that do not implement the IDLE functionality yet..
Add my vote for this... :) Moral support.
Could somebody with adequate permissions change the assignee of this bug. It using Scott MacGregor's old e-mail address. He now uses firstname.lastname@example.org. Thanks.
I think I'm more likely to do this at some point - I've looked into it a little. What we'd need to do is set a timer and occasionally poll the transport to see if data was available, when in Idle mode. If data was available, we'd have to parse it. Darin has pointed out that timers like this are death on laptop batteries, so we need a way to turn this off - I've suggested a laptop mode setting that various components that used timers could check...
Assignee: mscott → bienvenu
(In reply to comment #11) > What we'd need to do is set a timer and occasionally poll the transport to see > if data was available, when in Idle mode. Pardon my ignorance but I thought the purpose of IDLE mode is to open up a connection that let the IMAP server know the client could receive updates whenever they happen. I'm unfamiliar with the the inner workings of Mozilla's IMAP communications but is there a way to make this event driven so that the arrival of data after the IDLE command is sent would trigger the parsing activity?
the problem is that we use blocking reads in the imap code, i.e., we can only read data in blocking mode. (This is for performance reasons - it's literally twice as fast). But we can't just issue an OnIdle and then go into a blocking read, because we still need to be able to re-use the connection in case the user wants to do something like read a message in that folder...unless we can actually issue a command on a transport that's in a blocking read - I kinda doubt that, though. If we could change the transport into a non-blocking mode to do OnIdle, then we'd do that, but transports can't do that - as I understand it, there are race conditions when you switch, etc...
david: can we do this: after issuing an OnIdle, periodically have the IMAP thread check Available on the socket transport's nsIInputStream. if it returns non-zero, then begin a blocking read until all of the data has been consumed. if it returns zero, then put the IMAP thread to sleep until another timeout event occurs or until some other user initiated command is issued that requires the IMAP connection. currently, ImapThreadMainLoop waits on m_urlReadyToRunMonitor. we could signal that monitor if there is either an "url ready to run" or an OnIdle Available check to process. wouldn't that be do-able without much hassle?
Darin, yes, that's more or less what I had in mind. I was responding to http://bugzilla.mozilla.org/show_bug.cgi?id=141369#c12 - what Wayne was saying, I think, was that ideally, we'd respond to OnIdle data immediately, instead of when the next timeout fired. I'm not sure how frequently we'd wake up and poll the transport, but I was thinking every 10 or 15 seconds - if we did the old non-blocking reads, we'd handle the data on demand...
hmm.... you just gave me an interesting idea. it turns out that nsIAsyncInputStream::AsyncWait can be called on a _blocking_ input stream just as easily as it can be called on a non-blocking input stream. so, you could continue to open blocking input streams, but then instead of using a timeout, you could just call AsyncWait on the stream. it will notify you when there is something to read (or when the stream closes). from that notification you can do the blocking read. you do not even need an event queue to handle the AsyncWait notification... you will get called on the socket transport thread... from which point you can tickle that monitor that will cause a blocking read to happen. going this route uses poll instead of a timer and should be all around much better. now, hopefully i'm not overlooking something! ;-)
(In reply to comment #11) > parse it. Darin has pointed out that timers like this are death on laptop > batteries, so we need a way to turn this off - I've suggested a laptop mode > setting that various components that used timers could check... David: How do other mail clients with IDLE support handle laptops? (I ask since I don't recall seeing "laptop mode" in a mail client before, but I could be wrong.)
Darin, that sounds like it's worth a try - it also sounds like I don't need to resurrect all the async notification code, which would be nice. Alex, I don't know off the top of my head, but it rings a vague bell that some programs have a laptop mode. It might also be the case that there's some win32 api or some linux api that tells you when you're in battery preservation mode and that's what I'm thinking of...
this is going to be very tricky... when the imap main loop gets notified that it has to do something (will have to change the name of the url ready to run monitor), it will check if there's no url to run, and we're idle - if so, we must have been told there's data to read. So, we'll invoke the imap parser, and tell it to parse a single line. That will cause it to do a blocking read for a single line. Then, we'll need to check if there's more data to parse (either in the input stream or the input stream line buffer - the input stream line buffer consumes all data available in the input stream, so maybe just the input stream line buffer), and if there is more data, we'll need to invoke the parser again, until there's no more data.). Currently, the parser doesn't know how to read just one line for a single unsolicited response, and we need to avoid the situation where the parser ends up doing a blocking read when there's no more data to come, because we need to be able to end the IDLE command whenever the user wants to run a url with that connection. So we need to teach the parser how to read just one line, and to be very careful not to try to do a blocking read when there's no data ready...
There is blog post from one of Microsoft Entourage IMAP engineer: http://blogs.msdn.com/omars/archive/2004/02/19/76061.aspx As soon as this bug is closed it'll be great to post an update on the Mozilla Mail/Thunderbird in his blog entry as it got linked from MozillaZine http://www.mozillazine.org/talkback.html?article=4349 (the way I got here as well).
More trickiness - we probably don't want to immediately go into IDLE mode when a url finishes running in the INBOX because we might have another url waiting to run on the inbox, and we don't want to go into IDLE mode if we're just going to get out of it immediately. So I need to figure out if we can tell that there's a url waiting for a connection. We probably want to do IDLE for all idle connections, not just the INBOX connection, but the INBOX connection is the most important one. We could probably use the return value of nsImapIncomingServer::LoadNextQueuedUrl to know if we should go into idle mode. Except that there are some operations that chain urls - we'd need to make sure that the chaining prevents IDLE'ing. I'd really like to avoid using a timer to go into idle mode but I may end up going that way.
Status: NEW → ASSIGNED
This is a work-in-progress patch. It doesn't quite work, so don't try it! But it's most of the way there. The big problem is that connections get stuck after a couple operations, which I have to figure out. I also need to figure out if we want a per-server pref to control the use of IDLE (at first, we definitely will, in case there are problems). Do we have a pref UI for using the IDLE command or just use it if the server has it? For keeping connections alive, do we let the user set up biff to do that, or do we do it automatically? I probably also need to only go idle after a certain amount of inactive time, but I can do that later...
this seems to work well for me. I put all connections in the idle state when done with them. I immediately go IDLE when done with a url, unless we're going to run another URL immediately. the Check for new mail interval is orthogonal - we still run normal biffs even if IDLE is supported. This way, users can set the biff interval to whatever will keep the server connected, for most servers, 29 minutes. I had to diddle with the parser a little bit so that it knows to just parse a single line of idle response at a time, but it seems to be working fine.
Attachment #141982 - Attachment is obsolete: true
Comment on attachment 142178 [details] [diff] [review] proposed fix I had several comments I told david over AIM: 1) I think we will want UI in our account manager to allow laptop users to turn this off. I was searching through google to see if Outlook has the ability to turn it off but have not come up with anything yet. 2) I wonder if we will want to put a cap on the number of connections that can go into the IDLE state? 3) We should file a spin off bug to track having the client re-issue the IDLE command before the server times out. 4) Down the road, we may want to consider hooking up the idle state to a timer so we don't actually go into that state until the user has been idle for a certain period of time. Then again, going into the idle state may be cheap and we won't care.
Attachment #142178 - Flags: superreview?(mscott) → superreview+
On comment 24, item 1 perhaps Outlook might do it differently depending on the current power state of the machine? Perhaps sniffing traffic on Outlook when changing the power state of the macine... comparing a plugged in laptop to a battery mode laptop with a new session might function differntly?
On comment 24: 1- I don't mean to rehash things, but how bad is the IDLE command going to be on a laptop battery? Usually when I'm on battery, I'm either offline (so IDLE is moot) or I'm wireless, since if I'm close enought to be near an ethernet jack, usually power isn't too far off. The IDLE support will actually reduce the amount of traffic I send, since I can back off on the check new messages timer. Less traffic would seem to mean less power used by the wireless NIC. 4- Are we checking the status of the mailbox before we go into IDLE? IDLE will only show us what changes have occurred since the IDLE command started. If a mail message comes in or is modified outside of Mozilla after the last check but before we go in to IDLE, it won't show up until the messages are checked, right? On the other hand, polling the server before every IDLE may be too painful, especially for large mailboxes. Having to wait until the next scheduled poll may be acceptable, especially if the connection spends all it's time while not executing commands in an IDLE state, since that would make the window for unnoticed changes be the length of time to run the command. This would certainly be no worse than current behavior.
On comment 26: 1. The IDLE mode does not generate any server activity unless there are changes to the folder's contents. If the IDLE mode is implemented properly -- by waiting until the file descriptor is readable, instead of polling it -- the IDLE mode will not require any additional CPU. 4. Not true. If there are pending changes to the folder's status, a proper server IDLE implementation will immediately push those changes to the client as soon as the client enters the IDLE mode. To do otherwise will break the entire IMAP protocol itself.
(In reply to comment #24) > 2) I wonder if we will want to put a cap on the number of > connections that can go into the IDLE state? You mean connections to the same server right? Wouldn't it be a problem to have more than a single connection to the same server to be in IDLE state? More than one and we would receive and act on the same notifications multiple times.
Brodie, the notifications are on a per-folder basis. Each connection is for a particular folder, and we never have more than one connection for a particular folder.
fix checked in. bug 236584 spun off for the idle keep-alive issue.
Status: ASSIGNED → RESOLVED
Closed: 17 years ago
Resolution: --- → FIXED
*** Bug 46378 has been marked as a duplicate of this bug. ***
Hi Mozilla People, I've just been writing the code for the other end of this problem, in an IMAP server, and thought I'd pass on some comments. (At the moment, my server does asynchronous notification, but I can't find any clients to test it with!) I think that the IDLE extension is a bit of a red herring. Normal IMAP servers without any extensions can send "unilateral" data AT ANY TIME. (Whether "real" servers actual do or not I don't know, but to quote from the RFC: "A client MUST be prepared to accept any server response at all times.") The difficulty is that normally you (the client) don't know if you're going to get these unilateral updates for new messages and activity on other connections, so you have to poll by sending NOOPs. With the IDLE extension, you KNOW that the server will send you updates without having to poll for them. Good luck - I'm looking forward to seeing this in Thunderbird soon. --Phil.
Phil, this in the nightly/weekly thunderbird builds now, and will be in .6 when it comes out. You do have a point about being prepared to accept input at any time - we should be waiting for input with a quiescent connection, even if we're not in IDLE mode because the server doesn't support IDLE. For the most part, we'll get that input the next time we try to do something with the connection.
hmm, I would like to reopen this because for me it seems buggy: - the idle state does notify the server that changes to the SELECTED mailbox should be pushed to the client. - as you always cache a maximum of 5 connections to the IMAP server the limit for observed folders is 5? If so, this should be communicated somehow. I see no solution for this problem without establishing more connections to the IMAP-Server. If I understand something wrong here, please prove me wrong. But I use the IDLE feature and always have some folders which are not notified unless I use "Check for new messages every ...".
> hmm, I would like to reopen this because for me it seems buggy: this is as designed. If you want, you can go into your advanced imap server prefs and increase the number of connections we'll hold open, but it's not a nice thing to do to your server...The RFC for the IDLE command doesn't say that we should keep open a connection for every folder you open in the UI, which kinda sounds like what you want to happen. If I understand you correctly, I think your complaint really has to do with the IDLE command, and not Mozilla's implementation of it.
The mozilla implementation (or documentation) is not really obvious. mozilla claims that the IDLE command is supported. But for what? There is no information about which folders will be "idled". (INBOX seems to be always IDLED). One IDLE session can only serve one folder. One thing would be a configuration option for every folder (up to max. connections) if it should be served by an idling imap server. The others have to be polled then.
Every cached connection we have (up to the max, default 5) is left in the IDLE state, instead of just idle :-) (assuming the connection is in the selected state, which they almost always are). It generally works out that the last 5 folders you selected in the UI have the 5 cached connections, so the last 5 folders you selected are in the IDLE state - accounting for the fact that we use one of those cached connections for the INBOX.
OK, understood. But this is not what I've expected. One thing is left which could be discussed. If one of the last five selected folders is not marked as "Check for new messages" it doesn't make much sense to IDLE this folder. What do you mean?
that's true, we could simply not IDLE connection that aren't set to be checked for new messages, and just leave them in the selected state...I could be wrong, but I'm getting the impression that you don't want us to cache connections to folders that aren't set to be checked for new messages, so that all the cached connections can be busy doing IDLE, and that's not the way it works - we cache connections to folders because we think you might be using them in the UI again (so that frequently used folders tend to stay cached).
I'm not sure if I'm rehashing what has already been discussed, but one thing I see is the IDLE enhancement detected new mail in folders I have not marked "Check for new messages", but have made active once. Not good. For example, I use server side filters to deliver to 3 folders as so: Inbox |-List1 |-List2 |-Spam I have "Check for new messages" checked on List1 and List2, but not Spam - I only want to check it occasionaly. Yet, as soon as I check "Spam" once, IDLE enhancement acts on it until I re-start Mozilla. This should not be...
To comment #39: I have to admit that your argument is correct. So let me shortly describe what was my initial problem: I've read that IDLE is supported now, so I deactivated the "Check for new mail every ... minutes". (hey, the mails will get pushed to me). After some days using this I observed that some does never notify me about new messages. hmm? Strange! Now I do understand that IDLE is only done in a best effort base and always for every folder which was opened some time before. It doesn't IDLE after startup but only if I accessed them before. That are some things which are confusing if you don't know it. And IMHO this is the word of RFC2177 but not the meaning. I want the IDLE state to get new mails pushed for folders which have the "Check for new mail" flag set. I understand that it's almost impossible to get the caching and the IDLING together. So comment #40 is true and not really wanted. All IMHO. And please note that I really like mozilla/thunderbird as IMAP client. I just want to give constructive feedback.
To fill in the "What would OE do?" question (and what I think is a sensible compromise), OE has one IDLE connection polling the INBOX at all times, and one IDLE connection polling the currently displayed folder. Other folders are updated only when viewed, or when the "Check for New Mail" runs every x minutes.
It does seem like Mozilla should make sure INBOX never gets dropped from the IDLE list...
Obviously, the only sensible response to comment #42 is, "OE sucks".
David, I suppose having IDLE scroll up the list window when (a) you're on the last line of the list and (b) new mail is received would be out of scope here?
yeah - out of scopoe - Idle acts just like get new mail, or biff, in that respect, so it's not particularly an IDLE issue. Re a previous comment, since we cache a connection to the INBOX, we will have an IDLE connection to the INBOX.
You need to log in before you can comment on or make changes to this bug.