468490 - IMAP IDLE does not reconnect after a server timeout drops the connection. TB stops getting new messages because new mail notification via IDLE won't come from server

Status: RESOLVED WONTFIX

(MailNews Core :: Networking: IMAP, defect)

Description

[Brandon Ehle]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.1b1) Gecko/20081007 Firefox/3.1b1
Build Identifier: 2.0.0.18 (20081105)

IMAP IDLE support is a nice improvement but there are some corner cases that are not handled well with server side timeouts and connection drops.

Reproducible: Always

Steps to Reproduce:
1) Make sure "Check for new messages every N minutes" is disabled
2) Connect to an IMAP server with IDLE enabled with a server side timeout enabled
3) Let the connection IDLE until the timeout is reached

or

1) Make sure "Check for new messages every N minutes" is disabled
2) Close the TCP connection or kill the IMAP server process to close the connection before the server timeout is reached

or

1) Make sure "Check for new messages every N minutes" is set to a large value like 20 minutes
2) Close the TCP connection or kill the IMAP server process to close the connection

Actual Results:  
No indication that the server connection is lost and no incoming emails

Expected Results:  
It's extremely confusing when your mail client stops receiving email for no apparent reason.  A GUI indication of whether the IDLE connection is alive or an auto-reconnect attempt seems like the best fixes.


Currently if you want to receive email immediately, it seems like you need to set the "Check for new messages every N minutes" to a small amount (like 1 minute) and hammer the server every minute instead of relying completely upon IMAP IDLE.  With those settings it will reconnect a minute after the connection drops, but it also restarts the IMAP IDLE state every minute as well.  It would be nice if there was a way to specify immediate or 15 second delayed reconnects along with a longer time period between IMAP IDLE refreshes.

At the very least, some documentation on all the various corner cases would be nice as there is currently no help button or tooltips on the Account Settings screen.

Comment 1

[Nikolay Shopik]

According RFC2177 we should reissue IDLE command every 29 minutes. Could you give a try latest trunk of TB3?

Comment 2

[Wayne Mery (:wsmwk)]

see also 

 Bug 433006 -  IMAP Idle doesn't recognize lost connection after S3 (dupe?)
 Bug 373272 -  IMAP IDLE support broken/regression from 1.0.7 -> 1.1.x
 Bug 344205 -  Do not handle server returning NO to IMAP IDLE command

Comment 3

[Nikolay Shopik]

bug 433006 is somewhat related but not dupe.

Comment 4

[Nikolay Shopik]

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b3pre) Gecko/20081230 Shredder/3.0b2pre - confirm

Comment 5

[Nikolay Shopik]

David,
I though TB reconnect after 29 minutes as per rfc if where no activity from server.
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1pre) Gecko/20090614 
Shredder/3.0b3pre still there.

Comment 6

[David :Bienvenu]

(In reply to comment #5)
> David,
> I though TB reconnect after 29 minutes as per rfc if where no activity from
> server.
We don't automatically reconnect. Only if you have check for new mail set.

Comment 7

[Jesse Thompson]

Relevant text from RFC 2177

   The server MAY consider a client inactive if it has an IDLE command
   running, and if such a server has an inactivity timeout it MAY log
   the client off implicitly at the end of its timeout period.  Because
   of that, clients using IDLE are advised to terminate the IDLE and
   re-issue it at least every 29 minutes to avoid being logged off.
   This still allows a client to receive immediate mailbox updates even
   though it need only "poll" at half hour intervals.

Comment 8

[Wayne Mery (:wsmwk)]

(In reply to comment #6)
> (In reply to comment #5)
> > David,
> > I though TB reconnect after 29 minutes as per rfc if where no activity from
> > server.
> We don't automatically reconnect. Only if you have check for new mail set.

so only scenario 3 of comment 0 may be an outright bug?

Comment 9

[Jesse Thompson]

RFC 2177 is vague on the topic of what the client should be doing during periods of inactivity.

From RFC 3501: "The NOOP command can also be used to reset any inactivity autologout timer on the server."

According to our server debug logs, other clients (the ones that do IDLE well) issue the NOOP command fairly regularly (every 1 minute IIRC, but I can double check this if anyone is interested).

Comment 10

[Wayne Mery (:wsmwk)]

Jesse, could you double check that?

(In reply to comment #9)
> According to our server debug logs, other clients (the ones that do IDLE well)
> issue the NOOP command fairly regularly (every 1 minute IIRC, but I can double
> check this if anyone is interested).

Comment 11

[David :Bienvenu]

(In reply to comment #9)

> 
> According to our server debug logs, other clients (the ones that do IDLE well)
> issue the NOOP command fairly regularly (every 1 minute IIRC, but I can double
> check this if anyone is interested).

Doesn't that defeat the whole purpose of using IDLE?

Comment 12

[Jesse Thompson]

> > According to our server debug logs, other clients (the ones that do IDLE well)
> > issue the NOOP command fairly regularly (every 1 minute IIRC, but I can double
> > check this if anyone is interested).

> Doesn't that defeat the whole purpose of using IDLE?

The whole purpose of IDLE is to allow the server to "transmit updates to the client in real time" so that the client does not "need to poll extremely often".

The idea that the client should never need to acknowledge that it's still listening to avoid server defined inactivity timeouts is not stated as a purpose of IDLE.  In fact, the RFC says the opposite:

   The server MAY consider a client inactive if it has an IDLE command
   running, and if such a server has an inactivity timeout it MAY log
   the client off implicitly at the end of its timeout period.  Because
   of that, clients using IDLE are advised to terminate the IDLE and
   re-issue it at least every 29 minutes to avoid being logged off.
   This still allows a client to receive immediate mailbox updates even
   though it need only "poll" at half hour intervals.

You may rightly question whether IDLE should or should not have been designed in this way.  I suspect that the reason is due to practicality and backwards compatibility.

Regardless, we are in a situation in which Microsoft Outlook can do IDLE better than Mozilla Thunderbird.  I think that, in and of itself, is a call to action to get IDLE working in Thunderbird. 

Fortunately, the recommendation offered by the spec, that clients must terminate and reissue IDLE  every 29 minutes does not appear to be necessary.  The other clients that I have observed merely send periodic NOOP commands to avoid server inactivity timeouts.  I think that most clients issue NOOPs every 10 minutes, but in theory it could be up to 29 minutes.  

I'll do some debugging...

Comment 13

[David :Bienvenu]

every 1 minute is fairly different from every 10 minutes - my point was that if you issue a noop every one minute, you probably don't need idle.

Comment 14

[Jesse Thompson]

> every 1 minute is fairly different from every 10 minutes - my point was that if
> you issue a noop every one minute, you probably don't need idle.

Agreed.  I think that my initial recollection of 1 minute was incorrect.  We were troubleshooting an Outlook problem last week, and I'm fairly certain it was issuing NOOP every 10 minutes.  I'll double check.

Comment 15

[Jesse Thompson]

Here is what we have found (in regards to how often other clients issue NOOP commands during an IDLE session):

 * Outlook - every 10 minutes
 * Apple Mail - every 1 minute
 * K-9 Mail (Android) - every 20 minutes

Let me know if you would like us to see what any other clients do.

Comment 16

[Nikolay Shopik]

AFAIK, Apple Mail doesn't support IDLE until latest version, so this is probably left because historical behaviour.  K9 have config option how often to reissue IDLE, dunno if that affect NOOP too.

Comment 17

[Jesse Thompson]

I'd like to take a moment to address the question of whether issuing noop every one minute defeats the purpose of idle.

The cost of noop is extremely minimal to the client, the server, and the bandwidth consumed.  It's literally telling the server to do nothing (other than to refresh the inactivity timeout).  

Issuing frequent noop commands is better than polling without idle.  That requires the client to make a new connection to the server, initiate SSL/TLS, login, lsub, list, select, myrights, getacl, getquotaroot, fetch (FLAGS).

I would suggest that the extreme case of issuing noop every minute (and hence keeping the session alive indefinitely) is much more efficient to the client, server, and bandwidth, than polling every 30 minutes.

Comment 18

[Jesse Thompson]

You're right.  K9 does have a config option for "Refresh IDLE connection" that corresponds exactly with the observed noop frequency.  I'm unsure what the default setting is.

Comment 19

[Kelly Caudill]

minor correction/comment: to issue a NOOP on the IDLE connection, you have to terminate the IDLE.  If the client sends anything other then "done" while in IDLE, the server responds with "xx BAD only DONE allowed after Idle" and then you are no longer in IDLE.  So just doing "done" and then IDLE again is all that is needed.

Comment 20

[Wayne Mery (:wsmwk)]

I can't see how this could fit into this bug, but I'll throw it out anyway ...

For you long time thunderbird or seamonkey users, (sorry, this goes way back) did you experience this problem as a regression, as reported in bug 373272?   – IMAP IDLE support broken/regression from 1.0.7 -> 1.1.x - bug 373272 comment 9 has code details.  the regression range is not refined, being grossly between gecko 1.8.0 and 1.8.1. which would be equivalent between TB1.5 and TB2.0

at that time, 2007-03-11, it is reported that TB 2.0 *works*, which is odd because SM doesn't


FWIW, newer IDLE bugs: https://bugzilla.mozilla.org/buglist.cgi?list_id=3660902;short_desc=idle;resolution=---;chfieldto=Now;query_format=advanced;chfield=[Bug%20creation];chfieldfrom=2008-12-09;short_desc_type=allwordssubstr;type0-0-0=nowords;value0-0-0=count%20counts;product=MailNews%20Core;product=Thunderbird

Comment 21

[Sparhawk]

I just wanted to mention that this occurs using Gmail servers via IMAP. My Android phone works perfectly, but Thunderbird drops out.

Comment 22

[Sparhawk]

This is on:

TB 17.0.5
(K)ubuntu 12.10
KDE 4.9.5

Comment 23

[WADA:World Anti-bad-Duping Agency]

(In reply to Sparhawk from comment #21)
> I just wanted to mention that this occurs using Gmail servers via IMAP. 
> My Android phone works perfectly, but Thunderbird drops out.

What kind of phenomenon/symptom do you call by your "this occurs" in the context?

Even if "phenomenon stated in bug summary of this bug" happend on a cached connection during IDLE, Tb tries to do at least one of next after the unexpected connection drop while IDLE, unless unwanted problem like spin loop due to connection drop unfortunately occurs.
(a) Re-initiate next IDLE interval according to following setting.
    mail.server.server2.timeout=29(less than 30 min. default=29 min)
    Sen DONE to end IDLE, wait for OK response
    => timeout occurs sooner or later
(b) Request on Mbox, what is selected at the cached connection, occurs.
    Send DONE to request other command, wait for OK response
    => timeout occurs sooner or later
(c) Request on Mbox, what is not selected at any connection, occurs,
    no free cached connection for the Mbox, a cached connection is used.
    Send DONE to request select command, wait for OK response
    => timeout occurs sooner or later
(d) STAT on Mbox, what is not selected at any connection, occurs,
    no free cached connection for the STAT, a cached connection is used.
    (STAT Mbox shouldn't used after SELECT Mbox).
    This occurs by new mail check.
    Send DONE to request select command, wait for OK response
    => timeout occurs sooner or later

Sparhawk, in your case, no such request and timeout occured?
If so, what kind of activity was done on IMAP connection(s) by Tb when your problem occured?

If all connection is lost(can be checked by "netstat /n /b /o" etc.) when your problem occured, was PC's IP address assigned by DHCP changed(may be checked by ipconfig)?
If such case, "Go Work Offline then Go back Work Online" is usually effective recovery procedure.
Was "Go Work Offline then Go back Work Online" useless in your case?

If you already know your problem is IDLE related problem, have you checked with "IDLE command use" disabled?
Please note that "IDLE command use" is optional, even if default of Tb is Enabled.

Comment 24

[WADA:World Anti-bad-Duping Agency]

(In reply to Sparhawk from comment #21)
> I just wanted to mention that this occurs using Gmail servers via IMAP. 
> My Android phone works perfectly, but Thunderbird drops out.

If you are talking about other than "new mail" and "expunged mail"(if Gmail IMAP, mail flagged \Deleted is automatically expunged if auro-expunging=On), if you are talking about existent mail's status such as Tag of Tb(flag/keyword in IMAP), Read/Unread status, Starrred status etc., please don't confuse this bug and phenomnon of bug 693204 / bug 512745, please.

Comment 25

[Sparhawk]

I'm specifically talking about new mail here, but thanks for the links to the other bugs (which are related to my secondary concerns).

The phenomenon that I was referring to was the OP. That TB stops getting new messages in the circumstances in the OP. In my case, my connection was not lost, nor my IP address changed.

I'm aware that there are some workarounds, but certainly there is still a bug, and I just thought to mention that I see this bug with my configuration too. I've not waited (up to) 29 minutes to check if TB reconnects, but IMO it should re-connect within a few minutes. (And especially compared to the immediate response that I see on my Android phone.)

Comment 26

[WADA:World Anti-bad-Duping Agency]

(In reply to Sparhawk from comment #25)
> I'm aware that there are some workarounds, but certainly there is still a bug, (snip)

"Go Work Offline, go back Work Online" is never workaround. It's a kind of problem isolation work.   

> The phenomenon that I was referring to was the OP. 
> That TB stops getting new messages in the circumstances in the OP.
> In my case, my connection was not lost, nor my IP address changed.

Sorry, I can't understand your "OP". What do you call by "OP"?

If all 5 cached connections are kept, and if Tb's states of any connection is "IDLE was sent and unsol response for IDLE was returned", status is following;
- server is ready to send unsol reponse for new mail etc.
  server is ready to receive DONE to terminate IDLE.
- Tb is ready to receive unsol response for new mail etc.
  Tb is ready to send DONE command to terminate IDLE in order to;
  - reinitiate next IDLE interval
  - send IMAP command for selected Mbox at the connection
  - send SELECT for Mbox which is not selected at any connection.
    If Mbox expanding, LIST "%/%" etc. may preceed.
  - send STATUS command for new mail check which is not selected at
    any connection
  - send LSUB command to show subscription list

If more than 5 Mbox is opened, state of at least one Mbox is "not selected at any connection", so upon next click of the Mbox, Tb trie to do following if connection's status is "idling".
- DONE, OK response, SELECT Mbox
If SELECT is issued for Mbox open, Tb requests "uid 1:* fetch flags". So, new mails in an Mbox is know by Tb sooner or later.
Exception is;
- Inbox, when max cached connections > 2
- Trash, when max cached connections > 3
Connections are reserved for Inbox and Trash.
If Inbox, connections is reserved, and Inbox is always selected at the connection, and "uid <highestUID+1>:* fetch flags" is used for new mail check. If scheduling of Biff is somehow stopped even though the connection is not lost, new mail check of Inbox may not be executed.

Do you enable automatic new mail check for Inbox only?

If yes, do you see your problem with following?
- Max cached connections = 5 or appropriate one
  Enable new mail check of several Mboxes
- Max cached connections = 1
  Switch Mbox at folder pane, including Mbox, Trash
  (a workaround of bug 693204 / bug 512745)

IIRC, new mail alert is not shown in Tb for mail of \Recent flag is not set and \Seen flag is set.
If other client such as Android Phone accessed Inbox before Tb accesses, \Recent flag is automatically reset by IMAP server.

Does your Android Phone surely keep "\Seen flag is not set" state of new mails by his new mail check?

Comment 27

[WADA:World Anti-bad-Duping Agency]

(In reply to Sparhawk from comment #25)

FYI.
Majority of phenmena around "IDLE, and connection loss or hybernation" is phenomenon like bug 498054.
If IMAP server is not well configured, phenomenon like bug 344205 may occur. In this case, the connection may not be reused by Tb. If so, if bug 344205 occurs on all not-closed cached connections, Tb perhaps can do nothing, thus "go Work Offline, go back Work Online" will be nneded to rcover. Needless to say, this is not applicable to Gmail IMAP, so ths case is irrelevant to your case.

Comment 28

[WADA:World Anti-bad-Duping Agency]

(In reply to Sparhawk from comment #25)
> That TB stops getting new messages in the circumstances in the OP.
> In my case, my connection was not
> lost, nor my IP address changed.

Even if "TB stops getting new messages" is written in bug sumary, as written in comment #0, this bug is for following.
- Periodical new mail check is not requested, per concept of IDLE
- New mail notification is requiired for Inbox or Inbox+few Mbox only
  (because limitation in number of cached connections,)
  (IDLE for many Mbox is imposibble utill RFC 5465 will be supported.)
  (See bug 479133.)
- New mail detection relies on noification from server via IDLE.
In this case, if connection is lost during IDLE, Tb currently doesn't re-request IDLE automatically. So, new mail notification from server via IDLE won't work if connection loss during IDLE happens.

This is apparenly different from your case - Connection is never lost in your case.

Comment 29

[Sparhawk]

Sorry for the long delay in replying. I actually suspect that it's working fine now, actually! I'm not sure if my local configuration changed, or an update fixed it for me. I'll post back if I come across this problem again. Thanks for the posts.

Comment 30

[Colin Guthrie]

Just to add another data point here regarding the "1 minute NOOP"s and IDLE.

We are "fortunate" enough to have an ISP which drops "idle" (in the dictionary definition sense) TCP connections after 5 minutes of inactivity. This is particularly annoying with e.g. SSH connections, but fortunately we can set "ServerAliveInterval 60" in the ssh client config to send "pings" every minute which ensures the TCP connection is "used" and thus spared the reaping process.

Sadly, when dealing with my IMAP IDLE, this 5 minute killing basically stops me checking my mail for about 24 minutes (29 minutes - the ISP's 5 minutes) as the open IDLE connection reports no new mail and there seems to be no way to really, honestly, definitely, actually *check* for new mail (other than going offline and back on again).

Sending a NOOP every minute (or some configurable value) would certainly work around this problem for me. I appreciate some people think this defeats the purpose of IDLE but it really doesn't - it's just a simple ping to keep the connection alive, not to actually do anything related to the notification of new mails.

Comment 31

[Bear]

(In reply to Colin Guthrie from comment #30)
> Just to add another data point here regarding the "1 minute NOOP"s and IDLE.
> 
> We are "fortunate" enough to have an ISP which drops "idle" (in the
> dictionary definition sense) TCP connections after 5 minutes of inactivity.
> This is particularly annoying with e.g. SSH connections, but fortunately we
> can set "ServerAliveInterval 60" in the ssh client config to send "pings"
> every minute which ensures the TCP connection is "used" and thus spared the
> reaping process.
> 
> Sadly, when dealing with my IMAP IDLE, this 5 minute killing basically stops
> me checking my mail for about 24 minutes (29 minutes - the ISP's 5 minutes)
> as the open IDLE connection reports no new mail and there seems to be no way
> to really, honestly, definitely, actually *check* for new mail (other than
> going offline and back on again).
> 
> Sending a NOOP every minute (or some configurable value) would certainly
> work around this problem for me. I appreciate some people think this defeats
> the purpose of IDLE but it really doesn't - it's just a simple ping to keep
> the connection alive, not to actually do anything related to the
> notification of new mails.

You appear to be reporting a different concern than this bug addresses.

Please create a feature request bug seperate from this one which requests a NOOP feature.

Comment 32

[WADA:World Anti-bad-Duping Agency]

(In reply to Colin Guthrie from comment #30)
> We are "fortunate" enough to have an ISP which drops "idle" (in the dictionary definition sense) 
> TCP connections after 5 minutes of inactivity
> 
> Sadly, when dealing with my IMAP IDLE, this 5 minute killing basically stops
> me checking my mail for about 24 minutes (29 minutes - the ISP's 5 minutes)
> as the open IDLE connection reports no new mail and there seems to be no way
> to really, honestly, definitely, actually *check* for new mail (other than
> going offline and back on again).

If so, following is useful for you, isn't it?
   mail.server.server#.timeout = 5
Tb sends DONE after 5 minutes, then issues IDLE again. So timeout is detected within 5 minutes.

Following may help your case.
   Disable IDLE command use.
   check_all_imap_folders_for_new = true
      using [x] Check new messages every NN minutes, with appropriate NN(10 to 20 may be adequate).
   mail.server.default.use_condstore = false (See bug 912216 for reason why disable this option)
If FolderX is not selected at any cached connection, Tb issues "STATUS FolderX" every NN minutes,
and issues "select FolderX, uid fetch 1:* Flags()" if new mail is detected.
If FolderX is already selected at a cached connection, Tb issues "uid fetch HighestUID:* Flags()"" every NN minutes.
This is simple thechnique which is used when IDLE command is not supported.
If your server kills effectiveness of IDLE command, I think "stop using IDLE" is a best solution.

Please note that this bug is for issue in Tb when "accidental connection loss while IDLE" with "IMAP server who correctly/properly/validly supports IDLE with IDLE time limit=29 minutes<30 minutes".

Comment 33

[bintoro]

(In reply to WADA from comment #32)

> If so, following is useful for you, isn't it?
>    mail.server.server#.timeout = 5
> Tb sends DONE after 5 minutes, then issues IDLE again. So timeout is
> detected within 5 minutes.

That's not how server#.timeout works for me. It's not issuing any commands automatically.

Let's say I set the timeout to 3 minutes. Looking at the IMAP log, nothing happens after 3 minutes of inactivity. However, once there *is* an interaction with the server beyond that point, TB issues DONE and performs logout+login. Normally (when the timeout period has not elapsed) it would just try to reuse the connection without the logout+login dance.

Can someone please confirm if this is indeed the design of server#.timeout or if I'm seeing a different bug?

Comment 34

[Wayne Mery (:wsmwk)]

bug 344205 comment 11 points out relevant code, though it's not clear that resolving that is sufficient.

Christian, is this an area on which you can comment?

Comment 35

[gene smith]

I think this bug is still present and relates to the discussion in bug 1470448 comment 58 and later. As a test, I set my gmail account to use idle and never poll for new email at startup or on a timed basis. I remain selected on my default non-gmail inbox after starting tb. I can see there are imap connection to gmail and the default account using lsof. After 30 minutes the gmail connection goes to CLOSE-WAIT state since gmail (or maybe tb) has disconnected and tb never terminates idle and restarts idle at the 29m point like it should. It also never automatically re-established the connection. A test message sent after the 30m to the gmail account doesn't increase the unread count until the gmail inbox is clicked.

This also somewhat confirms other reports that new mail unread counts on mailboxes doesn't sometimes increase until the folder is clicked. (However, I think those reports also used polling so maybe not a valid confirmation.)

There does exist code that checks a 29m inactivity timeout,
https://dxr.mozilla.org/comm-central/source/mailnews/imap/src/nsImapIncomingServer.cpp#677
but it doesn't seem to restart the idle command but just drops the connection for the mailbox. Only clicking on the mailbox brings a connection back (and restarts idle mode).

Of course, if you do configure tb to poll for mail within the 30m window (default 10m), idle does get restored and the connection is maintained so you should still get "immediate" notification.

Comment 36

[gene smith]

Attachment: t.gds.patch

Here's a 1st cut fix for this bug (still has printfs and commented out alternatives). The existing 29 second timeout parameter is basically useless and never seems to do anything, even if the connection is totally idle for hours. 

This now checks in the main imap task loop each time it wakes up (every minute) for how long the IDLE state has been present and after the user configured X seconds (default 29) it exits IDLE, sends a NOOP and returns to IDLE. This keeps servers that disconnect idle connections after 30 min (e.g., gmail) happy without requiring that any "check new messages every X min" polling be configured.

Some servers may disconnect sooner than 29m so the pref can still be reduce accordingly. For unspecified reasons the original code sets a user configuration of more than 29m back to 29m. So that makes it impossible to set the timeout higher than 29m. This doesn't seem right but I haven't changed that yet.

Comment 37

[Mike Cloaked]

Excellent to see this being moved forward.

Comment 38

[Wayne Mery (:wsmwk)]

bug 344205 has since been closed, fixed by Gene in 2018-11-09. But Christian seems gone.

Gene, Is this patch still worth persuing as a general solution?

Comment 39

[Wayne Mery (:wsmwk)]

More IDLE bugs https://mzl.la/3I1pNVh

Comment 40

[Wayne Mery (:wsmwk)]

Bugs with sleep, wake, hibernate in summary - https://mzl.la/3Wpo405

Comment 41

[gene smith]

(In reply to Wayne Mery (:wsmwk) from comment #38)

bug 344205 has since been closed, fixed by Gene in 2018-11-09. But Christian seems gone.

Gene, Is this patch still worth persuing as a general solution?

I tried testing this again with what I think is the worst case server for this (yahoo) which drops inactive connection after 5 minutes even though the rfc says it should hold the connection for 30m. Anyhow, if I redo yahoo's idle every 4 minutes to avoid yahoo's drop, it works for exactly 60 minutes but then yahoo still shuts down the connection . So the algorithm I originally propose for this bug (comment 36) doesn't fix the problem completely for yahoo since it still stops idle by dropping the connection after 60 minutes if nothing received. This is with no biff time enabled.
But if I set the biff time to 60m and redo yahoo's idle every 4 minutes, this revives the connection to Inbox and continues on and the idle redo works for another hour then yahoo drops the connection and biff re-establishes it, etc.

After a lot of testing and waiting (it takes an hour for yahoo to flag an error) I did get it to come back with biff disabled as long as I don't sleep the computer for more than 5 minutes (yahoo's timeout). If I sleep or hibernate more than 5 minutes, there is nothing to automatically cause the connection to inbox to come back without a biff event like a folderstatus URL -- an automatic re- connect to inbox doesn't occur without biff after wakeup. Autosync can cause a re-connect to inbox (if it has new mail and if you've been sleeping for at least the biff time) but unless you are using offline store autosync won't occur either. If after wakeup you select another folder and then go back to Inbox it will re-connect to inbox; also "get mail" button will. So some maybe non-intuitive user actions are needed to restore idle after wakeup without a fairly short biff time set. But imap idle is not really intended to eliminate or replace biff so I suppose this is useful and fills in a gap in the TB imap design.

I'll submit a new formal patch shortly.

I need to re-test this but probably just setting the diff time to 4 minutes would do the same thing with maybe a bit less network activity (and no code changes).
I haven't tested this with a 30m timeout server like dovecot, but for those servers just setting the biff time to 29 minutes would probably solve the problem too, at least for Inbox. But if user wants to check other folders for new mail using biff, the 29 minute setting might be unacceptably long since IDLE isn't automatically enabled for them, only for Inbox.

Comment 42

[gene smith]

Attachment: idle-reissue.diff

Here's a completed possible change to support the timed idle reissue. I added a new advanced server setting for the reissue timer defaulting to 29 seconds as specified in the RFC which sets the value of the currently existing preference mail.server.serverX.timeout. The allowed range on the parameter is 0 to 1440 minutes (where 1440 minutes is 24 hours). A value of zero disables the idle reissue and for new account setups it defaults to 29 minutes.

For change to the parameter to take effect an IMAP url must occur for the account such as a user click of Get Messages button or opening a new message. A TB restart will also cause a new idle reissue time setting to take effect.

As I mentioned before, yahoo (with it's very short 5 minute connection timeout) requires that the idle reissue time be set to 4 minutes. This keeps the connection alive for 1 hour but then yahoo still disconnects. Then the connection is re-established and the reissues continue.

Other servers like dovecot keep the connection for 30 minutes as required by the RFC. So the default idle reissue parameter of 29 minutes keeps the connection alive.

However, in both cases it appears that the biff timer can have the same effect.

In the case of short timeout yahoo, with idle reissue disabled and biff timer set to 4 minutes, you have basically the same result. In that case idle is stopped and a noop is sent and then idle is reissued. So the only real advantage of using the proposed idle reissue timer is that the noop is not sent.

In the case of servers that follow the RFC and keep the connection for at least 30 minutes, setting the biff timer to 29 minutes has the same effect except for the added noop. The connection is maintained indefinitely.

This proposed feature seems to me to only possibly be useful to user that prefer to not set a biff time and rely on imap IDLE or the "Get Mail" button to bring in new mail. So for that small set of users, I would say just set your biff timer to the longest possible server-dependent value to keep the idling connection alive.

For now, I'm just attaching/archiving my work on this rather than submitting a formal patch. (Maybe I'm missing something here and this really is useful?)

Comment 43

[gene smith]

Going ahead and closing this as WONTFIX.