141369 - Support IMAP4 IDLE command (RFC 2177)

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, enhancement)

Description

[Jan Kuemmel]

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.0rc1)
Gecko/20020417
BuildID:    2002041711

Mozilla mail does not seem to use the IMAP IDLE extension specified by RFC 2177.
Would be very nice to have and should not be too complicated to implement. 

Reproducible: Always
Steps to Reproduce:
1. add imap account with an IDLE capable imap server (such as cyrus)
2. open inbox
3. send mail
4. wait


Actual Results:  nothing happens till the "check for new messeges every #
minutes" time has passed.

Expected Results:  There should be a checkbox "Use Imap IDLE command" which
causes mozilla mail to use it.

Comment 1

[Alex Bishop]

Sounds useful. I can't find any duplicates so I'm marking this NEW.

Comment 2

[Alex Bishop]

Summary tweaking.

Before: why not RFC2177 ?
After: Support IMAP4 IDLE command (RFC 2177)

Comment 3

[Michael Klose]

Two things to consider here:

Biff Support, for this there is an interesting comment in Bug 18266 (comment
154) - direct link http://bugzilla.mozilla.org/show_bug.cgi?id=18266#c154

and whether to open an additional TCP connection which we poll at least every 29
minutes according to the RFC or change nsImapProtocol to leave the TCP
connection open, if this is not already done (I don't know the code well enough
yet, my feeling is the connection is closed).
The latter is the preferred solution.

Comment 4

[David :Bienvenu]

we do cache connections (i.e., keep them open). This would probably be a fun
little project for a mozilla contributor interested in imap.

Comment 5

[Michael Klose]

It's on my todo list, but I want to properly fix the namespace issues first. 

Comment 6

[Sam Varshavchik]

FYI:

Courier-IMAP 1.6 adds significant functionality to its IMAP IDLE implementation.
 On server platforms that have SGI's File Alteration Monitor installed (Linux,
Irix, with partial xBSD support) IDLE generates real-time folder status updates
(when multiple client sessions have the same folder selected, all clients are
notified immediately when any client makes any changes to the folder, and
immediately upon arrival of new mail).

I scavenged an old laptop with MS OE 4.0, and opened a folder, then also logged
in using pine.  Even though pine doesn't do IDLE, I played with it, marking
messages as deleted and undeleted, expunging the folder, and copying messages
into this folder.  On the other monitor, OE 4.0 obediently repeated everything I
was doing in pine.  Worked rather well.

So, the scoreboard right now has two of the top three non-commercial IMAP
servers offering value-added IMAP IDLE functionality.

Comment 7

[Kerin Millar]

Re: Comment #6, yes I upgraded my courier-imap server very recently and added in
FAM support. Outlook Express works very well with this extension and I would
love to see it in Mozilla Mail at some point!

Also, just thought I'd point out that Exchange 5.5 Server (and 2000 Server,
AFAIK) support this extension.

Comment 8

[Richard Tresidder]

I would like to have this feature added also 

It seems that the person that this is assigned to is (gone) ???

and this has been sitting without any additional comments for quite a while..

It seems that Mozilla is one of the few Email clients that do not implement the
IDLE functionality yet..

Comment 9

[Fernando Cassia]

Add my vote for this... :)

Moral support.

Comment 10

[Wayne Menzie]

Could somebody with adequate permissions change the assignee of this bug.  It
using Scott MacGregor's old e-mail address.  He now uses [e-mail]. 
Thanks.

Comment 11

[David :Bienvenu]

I think I'm more likely to do this at some point - I've looked into it a little.
What we'd need to do is set a timer and occasionally poll the transport to see
if data was available, when in Idle mode. If data was available, we'd have to
parse it. Darin has pointed out that timers like this are death on laptop
batteries, so we need a way to turn this off - I've suggested a laptop mode
setting that various components that used timers could check...

Comment 12

[Wayne Menzie]

(In reply to comment #11)
> What we'd need to do is set a timer and occasionally poll the transport to see
> if data was available, when in Idle mode.

Pardon my ignorance but I thought the purpose of IDLE mode is to open up a
connection that let the IMAP server know the client could receive updates
whenever they happen.  I'm unfamiliar with the the inner workings of Mozilla's
IMAP communications but is there a way to make this event driven so that the
arrival of data after the IDLE command is sent would trigger the parsing activity?

Comment 13

[David :Bienvenu]

the problem is that we use blocking reads in the imap code, i.e., we can only
read data in blocking mode. (This is for performance reasons - it's literally
twice as fast). But we can't just issue an OnIdle and then go into a blocking
read, because we still need to be able to re-use the connection in case the user
wants to do something like read a message in that folder...unless we can
actually issue a command on a transport that's in a blocking read - I kinda
doubt that, though. If we could change the transport into a non-blocking mode to
do OnIdle, then we'd do that, but transports can't do that - as I understand it,
there are race conditions when you switch, etc...

Comment 14

[Darin Fisher]

david: can we do this:

after issuing an OnIdle, periodically have the IMAP thread check Available on
the socket transport's nsIInputStream.  if it returns non-zero, then begin a
blocking read until all of the data has been consumed.  if it returns zero, then
put the IMAP thread to sleep until another timeout event occurs or until some
other user initiated command is issued that requires the IMAP connection.

currently, ImapThreadMainLoop waits on m_urlReadyToRunMonitor.  we could signal
that monitor if there is either an "url ready to run" or an OnIdle Available
check to process.  wouldn't that be do-able without much hassle?

Comment 15

[David :Bienvenu]

Darin, yes, that's more or less what I had in mind. I was responding to
http://bugzilla.mozilla.org/show_bug.cgi?id=141369#c12 - what Wayne was saying,
I think, was that ideally, we'd respond to OnIdle data immediately, instead of
when the next timeout fired. I'm not sure how frequently we'd wake up and poll
the transport, but I was thinking every 10 or 15 seconds - if we did the old
non-blocking reads, we'd handle the data on demand...

Comment 16

[Darin Fisher]

hmm.... you just gave me an interesting idea.  it turns out that
nsIAsyncInputStream::AsyncWait can be called on a _blocking_ input stream just
as easily as it can be called on a non-blocking input stream.  so, you could
continue to open blocking input streams, but then instead of using a timeout,
you could just call AsyncWait on the stream.  it will notify you when there is
something to read (or when the stream closes).  from that notification you can
do the blocking read.  you do not even need an event queue to handle the
AsyncWait notification... you will get called on the socket transport thread...
from which point you can tickle that monitor that will cause a blocking read to
happen.

going this route uses poll instead of a timer and should be all around much
better.  now, hopefully i'm not overlooking something! ;-)

Comment 17

[Ashley Bischoff (blog at handcoding.com)]

(In reply to comment #11)
> parse it. Darin has pointed out that timers like this are death on laptop
> batteries, so we need a way to turn this off - I've suggested a laptop mode
> setting that various components that used timers could check...

David:

How do other mail clients with IDLE support handle laptops? (I ask since I don't
recall seeing "laptop mode" in a mail client before, but I could be wrong.)

Comment 18

[David :Bienvenu]

Darin, that sounds like it's worth a try - it also sounds like I don't need to
resurrect all the async notification code, which would be nice.

Alex, I don't know off the top of my head, but it rings a vague bell that some
programs have a laptop mode. It might also be the case that there's some win32
api or some linux api that tells you when you're in battery preservation mode
and that's what I'm thinking of...

Comment 19

[David :Bienvenu]

this is going to be very tricky...

when the imap main loop gets notified that it has to do something (will have to
change the name of the url ready to run monitor), it will check if there's no
url to run, and we're idle - if so, we must have been told there's data to read.
So, we'll invoke the imap parser, and tell it to parse a single line. That will
cause it to do a blocking read for a single line. Then, we'll need to check if
there's more data to parse (either in the input stream or the input stream line
buffer - the input stream line buffer consumes all data available in the input
stream, so maybe just the input stream line buffer), and if there is more data,
we'll need to invoke the parser again, until there's no more data.). 

Currently, the parser doesn't know how to read just one line for a single
unsolicited response, and we need to avoid the situation where the parser ends
up doing a blocking read when there's no more data to come, because we need to
be able to end the IDLE command whenever the user wants to run a url with that
connection. So we need to teach the parser how to read just one line, and to be
very careful not to try to do a blocking read when there's no data ready...

Comment 20

[Myroslav Opyr]

There is blog post from one of Microsoft Entourage IMAP engineer:
http://blogs.msdn.com/omars/archive/2004/02/19/76061.aspx 

As soon as this bug is closed it'll be great to post an update on the Mozilla
Mail/Thunderbird in his blog entry as it got linked from MozillaZine
http://www.mozillazine.org/talkback.html?article=4349 (the way I got here as well).

Comment 21

[David :Bienvenu]

More trickiness - we probably don't want to immediately go into IDLE mode when a
url finishes running in the INBOX because we might have another url waiting to
run on the inbox, and we don't want to go into IDLE mode if we're just going to
get out of it immediately. So I need to figure out if we can tell that there's a
url waiting for a connection. We probably want to do IDLE for all idle
connections, not just the INBOX connection, but the INBOX connection is the most
important one.

We could probably use the return value of
nsImapIncomingServer::LoadNextQueuedUrl to know if we should go into idle mode.
Except that there are some operations that chain urls - we'd need to make sure
that the chaining prevents IDLE'ing. I'd really like to avoid using a timer to
go into idle mode but I may end up going that way.

Comment 22

[David :Bienvenu]

Attachment: work in progress

This is a work-in-progress patch. It doesn't quite work, so don't try it! But
it's most of the way there. The big problem is that connections get stuck after
a couple operations, which I have to figure out. I also need to figure out if
we want a per-server pref to control the use of IDLE (at first, we definitely
will, in case there are problems). Do we have a pref UI for using the IDLE
command or just use it if the server has it? For keeping connections alive, do
we let the user set up biff to do that, or do we do it automatically? I
probably also need to only go idle after a certain amount of inactive time, but
I can do that later...

Comment 23

[David :Bienvenu]

Attachment: proposed fix

this seems to work well for me. I put all connections in the idle state when
done with them. I immediately go IDLE when done with a url, unless we're going
to run another URL immediately. the Check for new mail interval is orthogonal -
we still run normal biffs even if IDLE is supported. This way, users can set
the biff interval to whatever will keep the server connected, for most servers,
29 minutes.

I had to diddle with the parser a little bit so that it knows to just parse a
single line of idle response at a time, but it seems to be working fine.

Comment 24

[Scott MacGregor]

Comment on attachment 142178 [details] [diff] [review]
proposed fix

I had several comments I told david over AIM:

1) I think we will want UI in our account manager to allow laptop users to turn
this off. I was searching through google to see if Outlook has the ability to
turn it off but have not come up with anything yet.

2) I wonder if we will want to put a cap on the number of connections that can
go into the IDLE state? 

3) We should file a spin off bug to track having the client re-issue the IDLE
command before the server times out. 

4) Down the road, we may want to consider hooking up the idle state to a timer
so we don't actually go into that state until the user has been idle for a
certain period of time. Then again, going into the idle state may be cheap and
we won't care.

Comment 25

[Steve Wardell]

On comment 24, item 1 perhaps Outlook might do it differently depending on the
current power state of the machine? Perhaps sniffing traffic on Outlook when
changing the power state of the macine... comparing a plugged in laptop to a
battery mode laptop with a new session might function differntly?

Comment 26

[Bob Snyder]

On comment 24:
1- I don't mean to rehash things, but how bad is the IDLE command going to be on
a laptop battery? Usually when I'm on battery, I'm either offline (so IDLE is
moot) or I'm wireless, since if I'm close enought to be near an ethernet jack,
usually power isn't too far off. The IDLE support will actually reduce the
amount of traffic I send, since I can back off on the check new messages timer.
Less traffic would seem to mean less power used by the wireless NIC.

4- Are we checking the status of the mailbox before we go into IDLE? IDLE will
only show us what changes have occurred since the IDLE command started. If a
mail message comes in or is modified outside of Mozilla after the last check but
before we go in to IDLE, it won't show up until the messages are checked, right?

On the other hand, polling the server before every IDLE may be too painful,
especially for large mailboxes. Having to wait until the next scheduled poll may
be acceptable, especially if the connection spends all it's time while not
executing commands in an IDLE state, since that would make the window for
unnoticed changes be the length of time to run the command. This would certainly
be no worse than current behavior.

Comment 27

[Sam Varshavchik]

On comment 26:

1.  The IDLE mode does not generate any server activity unless there are changes
to the folder's contents.  If the IDLE mode is implemented properly -- by
waiting until the file descriptor is readable, instead of polling it -- the IDLE
mode will not require any additional CPU.

4.  Not true. If there are pending changes to the folder's status, a proper
server IDLE implementation will immediately push those changes to the client as
soon as the client enters the IDLE mode.  To do otherwise will break the entire
IMAP protocol itself.

Comment 28

[Brodie]

(In reply to comment #24)
> 2) I wonder if we will want to put a cap on the number of 
> connections that can go into the IDLE state? 

You mean connections to the same server right? Wouldn't it be a problem to have
more than a single connection to the same server to be in IDLE state? More than
one and we would receive and act on the same notifications multiple times.

Comment 29

[David :Bienvenu]

Brodie, the notifications are on a per-folder basis. Each connection is for a
particular folder, and we never have more than one connection for a particular
folder.

Comment 30

[David :Bienvenu]

fix checked in. bug 236584 spun off for the idle keep-alive issue.

Comment 31

[J Luh]

*** Bug 46378 has been marked as a duplicate of this bug. ***

Comment 32

[Phil Endecott]

Hi Mozilla People,

I've just been writing the code for the other end of this problem, in an IMAP
server, and thought I'd pass on some comments.  (At the moment, my server does
asynchronous notification, but I can't find any clients to test it with!)

I think that the IDLE extension is a bit of a red herring.  Normal  IMAP servers
without any extensions can send "unilateral" data AT ANY TIME.  (Whether "real"
servers actual do or not I don't know, but to quote from the RFC: "A client MUST
be prepared to accept any server response at all times.")

The difficulty is that normally you (the client) don't know if you're going to
get these unilateral updates for new messages and activity on other connections,
so you have to poll by sending NOOPs.  With the IDLE extension, you KNOW that
the server will send you updates without having to poll for them.

Good luck - I'm looking forward to seeing this in Thunderbird soon.

--Phil.

Comment 33

[David :Bienvenu]

Phil, this in the nightly/weekly thunderbird builds now, and will be in .6 when
it comes out.

You do have a point about being prepared to accept input at any time - we should
be waiting for input with a quiescent connection, even if we're not in IDLE mode
because the server doesn't support IDLE. For the most part, we'll get that input
the next time we try to do something with the connection.

Comment 34

[Wolfgang Rosenauer [:wolfiR]]

hmm, I would like to reopen this because for me it seems buggy:

- the idle state does notify the server that changes to the
  SELECTED mailbox should be pushed to the client.
- as you always cache a maximum of 5 connections to the IMAP server
  the limit for observed folders is 5?

If so, this should be communicated somehow.
I see no solution for this problem without establishing more connections
to the IMAP-Server.
If I understand something wrong here, please prove me wrong.
But I use the IDLE feature and always have some folders which are not
notified unless I use "Check for new messages every ...".

Comment 35

[David :Bienvenu]

> hmm, I would like to reopen this because for me it seems buggy:

this is as designed. If you want, you can go into your advanced imap server
prefs and increase the number of connections we'll hold open, but it's not a
nice thing to do to your server...The RFC for the IDLE command doesn't say that
we should keep open a connection for every folder you open in the UI, which
kinda sounds like what you want to happen. If I understand you correctly, I
think your complaint really has to do with the IDLE command, and not Mozilla's
implementation of it.

Comment 36

[Wolfgang Rosenauer [:wolfiR]]

The mozilla implementation (or documentation) is not really obvious.
mozilla claims that the IDLE command is supported. But for what?

There is no information about which folders will be "idled".
(INBOX seems to be always IDLED).
One IDLE session can only serve one folder.
One thing would be a configuration option for every folder (up to max.
connections) if it should be served by an idling imap server.
The others have to be polled then.

Comment 37

[David :Bienvenu]

Every cached connection we have (up to the max, default 5) is left in the IDLE
state, instead of just idle :-) (assuming the connection is in the selected
state, which they almost always are). It generally works out that the last 5
folders you selected in the UI have the 5 cached connections, so the last 5
folders you selected are in the IDLE state - accounting for the fact that we use
one of those cached connections for the INBOX. 

Comment 38

[Wolfgang Rosenauer [:wolfiR]]

OK, understood. But this is not what I've expected.
One thing is left which could be discussed.
If one of the last five selected folders is not marked as "Check for new messages"
it doesn't make much sense to IDLE this folder. What do you mean?

Comment 39

[David :Bienvenu]

that's true, we could simply not IDLE connection that aren't set to be checked
for new messages, and just leave them in the selected state...I could be wrong,
but I'm getting the impression that you don't want us to cache connections to
folders that aren't set to be checked for new messages, so that all the cached
connections can be busy doing IDLE, and that's not the way it works - we cache
connections to folders because we think you might be using them in the UI again
(so that frequently used folders tend to stay cached). 

Comment 40

[Jerry Amundson]

I'm not sure if I'm rehashing what has already been discussed, but one thing I
see is the IDLE enhancement detected new mail in folders I have not marked
"Check for new messages", but have made active once. Not good.

For example, I use server side filters to deliver to 3 folders as so:
Inbox
|-List1
|-List2
|-Spam

I have "Check for new messages" checked on List1 and List2, but not Spam - I
only want to check it occasionaly. Yet, as soon as I check "Spam" once, IDLE
enhancement acts on it until I re-start Mozilla. This should not be...

Comment 41

[Wolfgang Rosenauer [:wolfiR]]

To comment #39: I have to admit that your argument is correct. So let me shortly
describe what was my initial problem:
I've read that IDLE is supported now, so I deactivated the "Check for new mail
every ... minutes". (hey, the mails will get pushed to me).
After some days using this I observed that some does never notify me about new
messages. hmm? Strange!

Now I do understand that IDLE is only done in a best effort base and always for
every folder which was opened some time before. It doesn't IDLE after startup
but only if I accessed them before.

That are some things which are confusing if you don't know it.
And IMHO this is the word of RFC2177 but not the meaning. I want the IDLE
state to get new mails pushed for folders which have the "Check for new mail"
flag set. I understand that it's almost impossible to get the caching and the
IDLING together.
So comment #40 is true and not really wanted. All IMHO.
And please note that I really like mozilla/thunderbird as IMAP client. I just
want to give constructive feedback.

Comment 42

[Rod Begbie]

To fill in the "What would OE do?" question (and what I think is a sensible
compromise), OE has one IDLE connection polling the INBOX at all times, and one
IDLE connection polling the currently displayed folder.  Other folders are
updated only when viewed, or when the "Check for New Mail" runs every x minutes.

Comment 43

[Jay Levitt]

It does seem like Mozilla should make sure INBOX never gets dropped from the 
IDLE list...

Comment 44

[Jerry Amundson]

Obviously, the only sensible response to comment #42 is, "OE sucks".

Comment 45

[Jay Levitt]

David, I suppose having IDLE scroll up the list window when (a) you're on the 
last line of the list and (b) new mail is received would be out of scope here?

Comment 46

[David :Bienvenu]

yeah - out of scopoe - Idle acts just like get new mail, or biff, in that
respect, so it's not particularly an IDLE issue.

Re a previous comment, since we cache a connection to the INBOX, we will have an
IDLE connection to the INBOX.