IMAP IDLE does not reconnect after a server timeout drops the connection. TB stops getting new messages because new mail notification via IDLE won't come from server

NEW
Unassigned

Status

--
major
10 years ago
3 months ago

People

(Reporter: azverkan, Unassigned, NeedInfo)

Tracking

(Depends on: 1 bug, Blocks: 1 bug)

Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [set small mail.server.server#.timeout, if connection loss so frequently happens])

(Reporter)

Description

10 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.1b1) Gecko/20081007 Firefox/3.1b1
Build Identifier: 2.0.0.18 (20081105)

IMAP IDLE support is a nice improvement but there are some corner cases that are not handled well with server side timeouts and connection drops.

Reproducible: Always

Steps to Reproduce:
1) Make sure "Check for new messages every N minutes" is disabled
2) Connect to an IMAP server with IDLE enabled with a server side timeout enabled
3) Let the connection IDLE until the timeout is reached

or

1) Make sure "Check for new messages every N minutes" is disabled
2) Close the TCP connection or kill the IMAP server process to close the connection before the server timeout is reached

or

1) Make sure "Check for new messages every N minutes" is set to a large value like 20 minutes
2) Close the TCP connection or kill the IMAP server process to close the connection

Actual Results:  
No indication that the server connection is lost and no incoming emails

Expected Results:  
It's extremely confusing when your mail client stops receiving email for no apparent reason.  A GUI indication of whether the IDLE connection is alive or an auto-reconnect attempt seems like the best fixes.


Currently if you want to receive email immediately, it seems like you need to set the "Check for new messages every N minutes" to a small amount (like 1 minute) and hammer the server every minute instead of relying completely upon IMAP IDLE.  With those settings it will reconnect a minute after the connection drops, but it also restarts the IMAP IDLE state every minute as well.  It would be nice if there was a way to specify immediate or 15 second delayed reconnects along with a longer time period between IMAP IDLE refreshes.

At the very least, some documentation on all the various corner cases would be nice as there is currently no help button or tooltips on the Account Settings screen.

Comment 1

10 years ago
According RFC2177 we should reissue IDLE command every 29 minutes. Could you give a try latest trunk of TB3?
Version: unspecified → 2.0

Comment 2

10 years ago
see also 

 Bug 433006 -  IMAP Idle doesn't recognize lost connection after S3 (dupe?)
 Bug 373272 -  IMAP IDLE support broken/regression from 1.0.7 -> 1.1.x
 Bug 344205 -  Do not handle server returning NO to IMAP IDLE command
Component: General → Networking: IMAP
OS: Windows Server 2003 → All
Product: Thunderbird → Core
QA Contact: general → networking.imap
Hardware: PC → All
Version: 2.0 → 1.8 Branch

Comment 3

10 years ago
bug 433006 is somewhat related but not dupe.

Comment 4

10 years ago
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3pre) Gecko/20081230 Shredder/3.0b2pre - confirm
Version: 1.8 Branch → Trunk

Updated

10 years ago
Status: UNCONFIRMED → NEW
Ever confirmed: true
(Assignee)

Updated

10 years ago
Product: Core → MailNews Core

Comment 5

10 years ago
David,
I though TB reconnect after 29 minutes as per rfc if where no activity from server.
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1pre) Gecko/20090614 
Shredder/3.0b3pre still there.

Comment 6

9 years ago
(In reply to comment #5)
> David,
> I though TB reconnect after 29 minutes as per rfc if where no activity from
> server.
We don't automatically reconnect. Only if you have check for new mail set.

Comment 7

8 years ago
Relevant text from RFC 2177

   The server MAY consider a client inactive if it has an IDLE command
   running, and if such a server has an inactivity timeout it MAY log
   the client off implicitly at the end of its timeout period.  Because
   of that, clients using IDLE are advised to terminate the IDLE and
   re-issue it at least every 29 minutes to avoid being logged off.
   This still allows a client to receive immediate mailbox updates even
   though it need only "poll" at half hour intervals.

Comment 8

8 years ago
(In reply to comment #6)
> (In reply to comment #5)
> > David,
> > I though TB reconnect after 29 minutes as per rfc if where no activity from
> > server.
> We don't automatically reconnect. Only if you have check for new mail set.

so only scenario 3 of comment 0 may be an outright bug?

Comment 9

8 years ago
RFC 2177 is vague on the topic of what the client should be doing during periods of inactivity.

From RFC 3501: "The NOOP command can also be used to reset any inactivity autologout timer on the server."

According to our server debug logs, other clients (the ones that do IDLE well) issue the NOOP command fairly regularly (every 1 minute IIRC, but I can double check this if anyone is interested).

Updated

8 years ago
Severity: normal → major
Summary: IMAP IDLE does not reconnect after a server timeout drops the connection → IMAP IDLE does not reconnect after a server timeout drops the connection. TB stops getting new messages

Comment 10

8 years ago
Jesse, could you double check that?

(In reply to comment #9)
> According to our server debug logs, other clients (the ones that do IDLE well)
> issue the NOOP command fairly regularly (every 1 minute IIRC, but I can double
> check this if anyone is interested).

Comment 11

8 years ago
(In reply to comment #9)

> 
> According to our server debug logs, other clients (the ones that do IDLE well)
> issue the NOOP command fairly regularly (every 1 minute IIRC, but I can double
> check this if anyone is interested).

Doesn't that defeat the whole purpose of using IDLE?

Comment 12

8 years ago
> > According to our server debug logs, other clients (the ones that do IDLE well)
> > issue the NOOP command fairly regularly (every 1 minute IIRC, but I can double
> > check this if anyone is interested).

> Doesn't that defeat the whole purpose of using IDLE?

The whole purpose of IDLE is to allow the server to "transmit updates to the client in real time" so that the client does not "need to poll extremely often".

The idea that the client should never need to acknowledge that it's still listening to avoid server defined inactivity timeouts is not stated as a purpose of IDLE.  In fact, the RFC says the opposite:

   The server MAY consider a client inactive if it has an IDLE command
   running, and if such a server has an inactivity timeout it MAY log
   the client off implicitly at the end of its timeout period.  Because
   of that, clients using IDLE are advised to terminate the IDLE and
   re-issue it at least every 29 minutes to avoid being logged off.
   This still allows a client to receive immediate mailbox updates even
   though it need only "poll" at half hour intervals.

You may rightly question whether IDLE should or should not have been designed in this way.  I suspect that the reason is due to practicality and backwards compatibility.

Regardless, we are in a situation in which Microsoft Outlook can do IDLE better than Mozilla Thunderbird.  I think that, in and of itself, is a call to action to get IDLE working in Thunderbird. 

Fortunately, the recommendation offered by the spec, that clients must terminate and reissue IDLE  every 29 minutes does not appear to be necessary.  The other clients that I have observed merely send periodic NOOP commands to avoid server inactivity timeouts.  I think that most clients issue NOOPs every 10 minutes, but in theory it could be up to 29 minutes.  

I'll do some debugging...

Comment 13

8 years ago
every 1 minute is fairly different from every 10 minutes - my point was that if you issue a noop every one minute, you probably don't need idle.

Comment 14

8 years ago
> every 1 minute is fairly different from every 10 minutes - my point was that if
> you issue a noop every one minute, you probably don't need idle.

Agreed.  I think that my initial recollection of 1 minute was incorrect.  We were troubleshooting an Outlook problem last week, and I'm fairly certain it was issuing NOOP every 10 minutes.  I'll double check.

Comment 15

8 years ago
Here is what we have found (in regards to how often other clients issue NOOP commands during an IDLE session):

 * Outlook - every 10 minutes
 * Apple Mail - every 1 minute
 * K-9 Mail (Android) - every 20 minutes

Let me know if you would like us to see what any other clients do.

Comment 16

8 years ago
AFAIK, Apple Mail doesn't support IDLE until latest version, so this is probably left because historical behaviour.  K9 have config option how often to reissue IDLE, dunno if that affect NOOP too.

Comment 17

8 years ago
I'd like to take a moment to address the question of whether issuing noop every one minute defeats the purpose of idle.

The cost of noop is extremely minimal to the client, the server, and the bandwidth consumed.  It's literally telling the server to do nothing (other than to refresh the inactivity timeout).  

Issuing frequent noop commands is better than polling without idle.  That requires the client to make a new connection to the server, initiate SSL/TLS, login, lsub, list, select, myrights, getacl, getquotaroot, fetch (FLAGS).

I would suggest that the extreme case of issuing noop every minute (and hence keeping the session alive indefinitely) is much more efficient to the client, server, and bandwidth, than polling every 30 minutes.

Comment 18

8 years ago
You're right.  K9 does have a config option for "Refresh IDLE connection" that corresponds exactly with the observed noop frequency.  I'm unsure what the default setting is.

Comment 19

7 years ago
minor correction/comment: to issue a NOOP on the IDLE connection, you have to terminate the IDLE.  If the client sends anything other then "done" while in IDLE, the server responds with "xx BAD only DONE allowed after Idle" and then you are no longer in IDLE.  So just doing "done" and then IDLE again is all that is needed.

Comment 20

6 years ago
I can't see how this could fit into this bug, but I'll throw it out anyway ...

For you long time thunderbird or seamonkey users, (sorry, this goes way back) did you experience this problem as a regression, as reported in bug 373272?   – IMAP IDLE support broken/regression from 1.0.7 -> 1.1.x - bug 373272 comment 9 has code details.  the regression range is not refined, being grossly between gecko 1.8.0 and 1.8.1. which would be equivalent between TB1.5 and TB2.0

at that time, 2007-03-11, it is reported that TB 2.0 *works*, which is odd because SM doesn't


FWIW, newer IDLE bugs: https://bugzilla.mozilla.org/buglist.cgi?list_id=3660902;short_desc=idle;resolution=---;chfieldto=Now;query_format=advanced;chfield=[Bug%20creation];chfieldfrom=2008-12-09;short_desc_type=allwordssubstr;type0-0-0=nowords;value0-0-0=count%20counts;product=MailNews%20Core;product=Thunderbird
Blocks: 433006

Comment 21

6 years ago
I just wanted to mention that this occurs using Gmail servers via IMAP. My Android phone works perfectly, but Thunderbird drops out.

Comment 22

6 years ago
This is on:

TB 17.0.5
(K)ubuntu 12.10
KDE 4.9.5
(In reply to Sparhawk from comment #21)
> I just wanted to mention that this occurs using Gmail servers via IMAP. 
> My Android phone works perfectly, but Thunderbird drops out.

What kind of phenomenon/symptom do you call by your "this occurs" in the context?

Even if "phenomenon stated in bug summary of this bug" happend on a cached connection during IDLE, Tb tries to do at least one of next after the unexpected connection drop while IDLE, unless unwanted problem like spin loop due to connection drop unfortunately occurs.
(a) Re-initiate next IDLE interval according to following setting.
    mail.server.server2.timeout=29(less than 30 min. default=29 min)
    Sen DONE to end IDLE, wait for OK response
    => timeout occurs sooner or later
(b) Request on Mbox, what is selected at the cached connection, occurs.
    Send DONE to request other command, wait for OK response
    => timeout occurs sooner or later
(c) Request on Mbox, what is not selected at any connection, occurs,
    no free cached connection for the Mbox, a cached connection is used.
    Send DONE to request select command, wait for OK response
    => timeout occurs sooner or later
(d) STAT on Mbox, what is not selected at any connection, occurs,
    no free cached connection for the STAT, a cached connection is used.
    (STAT Mbox shouldn't used after SELECT Mbox).
    This occurs by new mail check.
    Send DONE to request select command, wait for OK response
    => timeout occurs sooner or later

Sparhawk, in your case, no such request and timeout occured?
If so, what kind of activity was done on IMAP connection(s) by Tb when your problem occured?

If all connection is lost(can be checked by "netstat /n /b /o" etc.) when your problem occured, was PC's IP address assigned by DHCP changed(may be checked by ipconfig)?
If such case, "Go Work Offline then Go back Work Online" is usually effective recovery procedure.
Was "Go Work Offline then Go back Work Online" useless in your case?

If you already know your problem is IDLE related problem, have you checked with "IDLE command use" disabled?
Please note that "IDLE command use" is optional, even if default of Tb is Enabled.
(In reply to Sparhawk from comment #21)
> I just wanted to mention that this occurs using Gmail servers via IMAP. 
> My Android phone works perfectly, but Thunderbird drops out.

If you are talking about other than "new mail" and "expunged mail"(if Gmail IMAP, mail flagged \Deleted is automatically expunged if auro-expunging=On), if you are talking about existent mail's status such as Tag of Tb(flag/keyword in IMAP), Read/Unread status, Starrred status etc., please don't confuse this bug and phenomnon of bug 693204 / bug 512745, please.

Comment 25

6 years ago
I'm specifically talking about new mail here, but thanks for the links to the other bugs (which are related to my secondary concerns).

The phenomenon that I was referring to was the OP. That TB stops getting new messages in the circumstances in the OP. In my case, my connection was not lost, nor my IP address changed.

I'm aware that there are some workarounds, but certainly there is still a bug, and I just thought to mention that I see this bug with my configuration too. I've not waited (up to) 29 minutes to check if TB reconnects, but IMO it should re-connect within a few minutes. (And especially compared to the immediate response that I see on my Android phone.)
(In reply to Sparhawk from comment #25)
> I'm aware that there are some workarounds, but certainly there is still a bug, (snip)

"Go Work Offline, go back Work Online" is never workaround. It's a kind of problem isolation work.   

> The phenomenon that I was referring to was the OP. 
> That TB stops getting new messages in the circumstances in the OP.
> In my case, my connection was not lost, nor my IP address changed.

Sorry, I can't understand your "OP". What do you call by "OP"?

If all 5 cached connections are kept, and if Tb's states of any connection is "IDLE was sent and unsol response for IDLE was returned", status is following;
- server is ready to send unsol reponse for new mail etc.
  server is ready to receive DONE to terminate IDLE.
- Tb is ready to receive unsol response for new mail etc.
  Tb is ready to send DONE command to terminate IDLE in order to;
  - reinitiate next IDLE interval
  - send IMAP command for selected Mbox at the connection
  - send SELECT for Mbox which is not selected at any connection.
    If Mbox expanding, LIST "%/%" etc. may preceed.
  - send STATUS command for new mail check which is not selected at
    any connection
  - send LSUB command to show subscription list

If more than 5 Mbox is opened, state of at least one Mbox is "not selected at any connection", so upon next click of the Mbox, Tb trie to do following if connection's status is "idling".
- DONE, OK response, SELECT Mbox
If SELECT is issued for Mbox open, Tb requests "uid 1:* fetch flags". So, new mails in an Mbox is know by Tb sooner or later.
Exception is;
- Inbox, when max cached connections > 2
- Trash, when max cached connections > 3
Connections are reserved for Inbox and Trash.
If Inbox, connections is reserved, and Inbox is always selected at the connection, and "uid <highestUID+1>:* fetch flags" is used for new mail check. If scheduling of Biff is somehow stopped even though the connection is not lost, new mail check of Inbox may not be executed.

Do you enable automatic new mail check for Inbox only?

If yes, do you see your problem with following?
- Max cached connections = 5 or appropriate one
  Enable new mail check of several Mboxes
- Max cached connections = 1
  Switch Mbox at folder pane, including Mbox, Trash
  (a workaround of bug 693204 / bug 512745)

IIRC, new mail alert is not shown in Tb for mail of \Recent flag is not set and \Seen flag is set.
If other client such as Android Phone accessed Inbox before Tb accesses, \Recent flag is automatically reset by IMAP server.

Does your Android Phone surely keep "\Seen flag is not set" state of new mails by his new mail check?
(In reply to Sparhawk from comment #25)

FYI.
Majority of phenmena around "IDLE, and connection loss or hybernation" is phenomenon like bug 498054.
If IMAP server is not well configured, phenomenon like bug 344205 may occur. In this case, the connection may not be reused by Tb. If so, if bug 344205 occurs on all not-closed cached connections, Tb perhaps can do nothing, thus "go Work Offline, go back Work Online" will be nneded to rcover. Needless to say, this is not applicable to Gmail IMAP, so ths case is irrelevant to your case.
(In reply to Sparhawk from comment #25)
> That TB stops getting new messages in the circumstances in the OP.
> In my case, my connection was not
> lost, nor my IP address changed.

Even if "TB stops getting new messages" is written in bug sumary, as written in comment #0, this bug is for following.
- Periodical new mail check is not requested, per concept of IDLE
- New mail notification is requiired for Inbox or Inbox+few Mbox only
  (because limitation in number of cached connections,)
  (IDLE for many Mbox is imposibble utill RFC 5465 will be supported.)
  (See bug 479133.)
- New mail detection relies on noification from server via IDLE.
In this case, if connection is lost during IDLE, Tb currently doesn't re-request IDLE automatically. So, new mail notification from server via IDLE won't work if connection loss during IDLE happens.

This is apparenly different from your case - Connection is never lost in your case.
Summary: IMAP IDLE does not reconnect after a server timeout drops the connection. TB stops getting new messages → IMAP IDLE does not reconnect after a server timeout drops the connection. TB stops getting new messages because new mail noificaion via IDLE won't come from server

Comment 29

5 years ago
Sorry for the long delay in replying. I actually suspect that it's working fine now, actually! I'm not sure if my local configuration changed, or an update fixed it for me. I'll post back if I come across this problem again. Thanks for the posts.

Comment 30

5 years ago
Just to add another data point here regarding the "1 minute NOOP"s and IDLE.

We are "fortunate" enough to have an ISP which drops "idle" (in the dictionary definition sense) TCP connections after 5 minutes of inactivity. This is particularly annoying with e.g. SSH connections, but fortunately we can set "ServerAliveInterval 60" in the ssh client config to send "pings" every minute which ensures the TCP connection is "used" and thus spared the reaping process.

Sadly, when dealing with my IMAP IDLE, this 5 minute killing basically stops me checking my mail for about 24 minutes (29 minutes - the ISP's 5 minutes) as the open IDLE connection reports no new mail and there seems to be no way to really, honestly, definitely, actually *check* for new mail (other than going offline and back on again).

Sending a NOOP every minute (or some configurable value) would certainly work around this problem for me. I appreciate some people think this defeats the purpose of IDLE but it really doesn't - it's just a simple ping to keep the connection alive, not to actually do anything related to the notification of new mails.

Updated

4 years ago
Summary: IMAP IDLE does not reconnect after a server timeout drops the connection. TB stops getting new messages because new mail noificaion via IDLE won't come from server → IMAP IDLE does not reconnect after a server timeout drops the connection. TB stops getting new messages because new mail notification via IDLE won't come from server

Comment 31

4 years ago
(In reply to Colin Guthrie from comment #30)
> Just to add another data point here regarding the "1 minute NOOP"s and IDLE.
> 
> We are "fortunate" enough to have an ISP which drops "idle" (in the
> dictionary definition sense) TCP connections after 5 minutes of inactivity.
> This is particularly annoying with e.g. SSH connections, but fortunately we
> can set "ServerAliveInterval 60" in the ssh client config to send "pings"
> every minute which ensures the TCP connection is "used" and thus spared the
> reaping process.
> 
> Sadly, when dealing with my IMAP IDLE, this 5 minute killing basically stops
> me checking my mail for about 24 minutes (29 minutes - the ISP's 5 minutes)
> as the open IDLE connection reports no new mail and there seems to be no way
> to really, honestly, definitely, actually *check* for new mail (other than
> going offline and back on again).
> 
> Sending a NOOP every minute (or some configurable value) would certainly
> work around this problem for me. I appreciate some people think this defeats
> the purpose of IDLE but it really doesn't - it's just a simple ping to keep
> the connection alive, not to actually do anything related to the
> notification of new mails.

You appear to be reporting a different concern than this bug addresses.

Please create a feature request bug seperate from this one which requests a NOOP feature.
(In reply to Colin Guthrie from comment #30)
> We are "fortunate" enough to have an ISP which drops "idle" (in the dictionary definition sense) 
> TCP connections after 5 minutes of inactivity
> 
> Sadly, when dealing with my IMAP IDLE, this 5 minute killing basically stops
> me checking my mail for about 24 minutes (29 minutes - the ISP's 5 minutes)
> as the open IDLE connection reports no new mail and there seems to be no way
> to really, honestly, definitely, actually *check* for new mail (other than
> going offline and back on again).

If so, following is useful for you, isn't it?
   mail.server.server#.timeout = 5
Tb sends DONE after 5 minutes, then issues IDLE again. So timeout is detected within 5 minutes.

Following may help your case.
   Disable IDLE command use.
   check_all_imap_folders_for_new = true
      using [x] Check new messages every NN minutes, with appropriate NN(10 to 20 may be adequate).
   mail.server.default.use_condstore = false (See bug 912216 for reason why disable this option)
If FolderX is not selected at any cached connection, Tb issues "STATUS FolderX" every NN minutes,
and issues "select FolderX, uid fetch 1:* Flags()" if new mail is detected.
If FolderX is already selected at a cached connection, Tb issues "uid fetch HighestUID:* Flags()"" every NN minutes.
This is simple thechnique which is used when IDLE command is not supported.
If your server kills effectiveness of IDLE command, I think "stop using IDLE" is a best solution.

Please note that this bug is for issue in Tb when "accidental connection loss while IDLE" with "IMAP server who correctly/properly/validly supports IDLE with IDLE time limit=29 minutes<30 minutes".
Whiteboard: [set small mail.server.server#.timeout, if connection loss so frequently happens]

Updated

3 years ago
See Also: → bug 344205

Comment 33

2 years ago
(In reply to WADA from comment #32)

> If so, following is useful for you, isn't it?
>    mail.server.server#.timeout = 5
> Tb sends DONE after 5 minutes, then issues IDLE again. So timeout is
> detected within 5 minutes.

That's not how server#.timeout works for me. It's not issuing any commands automatically.

Let's say I set the timeout to 3 minutes. Looking at the IMAP log, nothing happens after 3 minutes of inactivity. However, once there *is* an interaction with the server beyond that point, TB issues DONE and performs logout+login. Normally (when the timeout period has not elapsed) it would just try to reuse the connection without the logout+login dance.

Can someone please confirm if this is indeed the design of server#.timeout or if I'm seeing a different bug?

Comment 34

2 years ago
bug 344205 comment 11 points out relevant code, though it's not clear that resolving that is sufficient.

Christian, is this an area on which you can comment?
Depends on: 344205
Flags: needinfo?(cristiklein)
See Also: bug 344205

Comment 35

3 months ago
I think this bug is still present and relates to the discussion in bug 1470448 comment 58 and later. As a test, I set my gmail account to use idle and never poll for new email at startup or on a timed basis. I remain selected on my default non-gmail inbox after starting tb. I can see there are imap connection to gmail and the default account using lsof. After 30 minutes the gmail connection goes to CLOSE-WAIT state since gmail (or maybe tb) has disconnected and tb never terminates idle and restarts idle at the 29m point like it should. It also never automatically re-established the connection. A test message sent after the 30m to the gmail account doesn't increase the unread count until the gmail inbox is clicked.

This also somewhat confirms other reports that new mail unread counts on mailboxes doesn't sometimes increase until the folder is clicked. (However, I think those reports also used polling so maybe not a valid confirmation.)

There does exist code that checks a 29m inactivity timeout,
https://dxr.mozilla.org/comm-central/source/mailnews/imap/src/nsImapIncomingServer.cpp#677
but it doesn't seem to restart the idle command but just drops the connection for the mailbox. Only clicking on the mailbox brings a connection back (and restarts idle mode).

Of course, if you do configure tb to poll for mail within the 30m window (default 10m), idle does get restored and the connection is maintained so you should still get "immediate" notification.
You need to log in before you can comment on or make changes to this bug.