1177624 - After some period of time (random), doesn't check for new mail on imap account

Status: RESOLVED INCOMPLETE

(MailNews Core :: Networking: IMAP, defect)

Description

[ilya-valeev]

Attachment: bug.txt

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0
Build ID: 20150514102509

Steps to reproduce:

Launch and configure Thunderbird.


Actual results:

After some period of time (random) it doesnt check mail. No new notifications. Mail window looks like no new mail arrived. Mail server supports IDLE, but Thunderbird stops check it. Pressings "Receive" button doesnt take effect.
Thunderbird check mail ONLY after I choose mail directory in main window.


Expected results:

Expect it would check mail.
Mail hosted by Yandex. Size - 3.2 Gb. Tried to turn off all extensions, to create new profile.
Im using Thunderbird from 31.6.0 to 38.0.1, bug is still here.
Tried on Windows 7 x64 (Thunderbird is x32), Debian 8 x64 (Thunderbird x64), 2 computers and 3 internet providers.
It is not Yandex problem: K-9 Mail receives mail perfect.

Comment 1

[Wayne Mery (:wsmwk)]

Version 31 is out since December 2014. Did you have the problem since, or before then?
And it's happening on all OS?

Comment 2

[ilya-valeev]

Before then I did not use Thunderbird.
Since Thunderbird 38 problem appears much seldom, but still remains.
What info can I provide to help sovle this issue?

Comment 3

[ilya-valeev]

And yes, it happening on all OS

Comment 4

[Wayne Mery (:wsmwk)]

The time frame and versions do not match the issues of bug 1196662 as a potential cause, but let's see what comes out of that

Comment 5

[WADA:World Anti-bad-Duping Agency]

(In reply to ilya-valeev from comment #0)
> Mail server supports IDLE, but Thunderbird stops check it.
> No new notifications.

If server supports CONDSTORE and CODSTORE support in Tb is enabled, followig problem can occur.
   When new mails are deected via IDLE,
   new mails are normally fetched and is shown in Tb's Thread pane,
   but new mail notification is not shown.
   (bug 885220, not Gmail IMAP specific issue)
Please surely rule out this problem by disabling Tb's CONDSTORE support.

> After some period of time (random) it doesnt check mail.

Inbox? or other than Inbox?

If other than Inbox, periodicaal new mail check by Biff is not invoked unless the mail folder is included in "folders to check new mail"(Folder Properties).
What is your new mail check interval setting? What is Folder Properties setting?

If other than Inbox, and if periodical new mail check is disaabed for an imap Mbox, "new mail detection via IDLE" is effective only when the imap Mbox is selected at a cached connection.
Default of "max cached connections" = 5 in Tb(max can not be Infinite), and Tb reserves two connections for Inbox and Trash.
So, "IDLE for other than Inbox, Trash" is limited to up to 3 Mboxes which are currently selected at cached connections.
  If you open MboxA, MboxB, MboxC, MboxD in this order, following occurs.
  Click MboxA -> Connection-3 : Select MboxA, fetch, IDLE for MboxA
  Click MboxB -> Connection-4 : Select MboxB, fetch, IDLE for MboxB
  Click MboxC -> Connection-5 : Select MboxC, fetch, IDLE for MboxC
  Click MboxD -> Connection-3 : Select MboxD -> MboxA is deselected -> fetch, IDLE for MboxD
                 So, "new mail detection via IDLE of MboxA" is now impossible.
  Click MboxA -> At one of cached connections, Select MboxA/fetch/IDLE for MboxA is issued.
                 So new mails in MboxA is shown.

> Thunderbird check mail ONLY after I choose mail directory in main window.

Is above phenomenon?

Comment 6

[Kent James (:rkent)]

Don't we have CONDSTORE disabled by default now?

Comment 7

[WADA:World Anti-bad-Duping Agency]

(In reply to Kent James (:rkent) from comment #6)
> Don't we have CONDSTORE disabled by default now?

Patch for bug 912216 is effective from Tb 36, but any user can change to mail.server.default.use_condstore=true and can set mail.server.server#.use_condstore=true.

Comment 8

[ilya-valeev]

mail.server.default.use_condstore=false, mail.server.server#.use_condstore (for any server from 1 to 7) doesnt exist

>Inbox? or other than Inbox?
Mostly other than inbox, but once inbox doesnt checked too.

>"folders to check new mail"(Folder Properties)
Enabled for any folder I need.

>What is your new mail check interval setting?
For now is 5 minutes. Tried 1, 5, 10 min with same result

>Default of "max cached connections" = 5
I set it to 15 (there are 6 folders I want to check)
Mail doesnt checked even if check interval passed. Once I didnt know that have new mail for 2 days.

Comment 9

[WADA:World Anti-bad-Duping Agency]

(In reply to ilya-valeev from comment #8)
> Mail doesnt checked even if check interval passed. Once I didnt know that have new mail for 2 days.

With which Thunderbird version? No hibernation during the "2 days"?
If you already upgraded to Tb 38.2.0, please downgrade to Tb 38.1.0, in order to surely tule out problem of bug 1196662 which is regression in Tb 38.2.0.

(In reply to ilya-valeev from comment #0)
> It is not Yandex problem: K-9 Mail receives mail perfect.

Does it mean that your IMAP Mbox is shared with K-3 Mail client?
Is new mails in IMAP Mbox fetched by K-3 Mail before Tb fetches?
I yes, \Recent flag is removed upon fetch by K-3, so Tb can't know about \Recet flag.
Is new mails in IMAP Mbox changed to Read state by K-3 Mail?
If yes, new mail notification is not shown by Tb when new mail is fetched by Tb, because "new mail for Biff of Tb" is "mail which has larger UID than Tb already knows" && "mail which doesn't have \Seen flag".
And, "Unread mail count of MboxA at Folder pane of Tb" is not modified because of "no unread mail is added".

See phenomenon for next operation.
1. At Mbomx in Tb, call MboxA, , Number of mails in MboxA = 2, HighestUID=aaa
   (show Order Received column. Value is UID if IMAP box).
   Keep MboxA selected at Folder pane in order to see Thread Pane display of MboxA. 
2. Using K-3, copy an Read mail with Important tag(Red color) to the MboxA.
   UID=bbb(larger than aaa). Number of mails in MboxA is now 3.
3. When IDLE command is used by Tb for the MboxA, mail of UID=bbb is notified from server.
   But This is done by "* 3 EXISTS" etc. (message count related data)
4. Because number of mails in MboxA is increased(known by "* 3 EXISTS" response), Tb knows about new mail,
   So Tb issues "uid fetch aaa+1:*", and UID=bbb is fetched automatically,
   and is shown at Thread pane of Tb with red color.
5. However, mail of UID=bbb is Read state.
   So, message count of MboxA is usually not updated at Folder pane of Tb,
   because "message count at Folder pane in Tb" is "Unread mail count only" by default in Tb.
   (To show total count or size at Folder pane, "Extra Folder column" addon is currently needed.)
   And, new mail alert is not shown because of "no new unread mail".
6. If you copy Unread state mail at step 2, you can know about new mail in MboxA
   via updated Unread mail count of MboxA at Folder pane of Tb,
   even when MboxA is not kept selected in Tb's main window at step 1.
 
Are you talking about this phenomenon?

Comment 10

[WADA:World Anti-bad-Duping Agency]

If "new mail" you call is "mail without \Seen flag"(==Read state mail, not Unread, maarked as Read by other client), and if MboxA is not selected at a cached connection, when new mail check is invoked for the Mbox by Biff, different situation can occur.
Note: "MboxA is not selected at a cached connection" can occur even after you explicitly opened MboxA once.
      MboxA is clicked and accessed and selected at a cached connection => Click other Mbox
      => after a while, the cached connection is used for other Mbox and other Mbox is selected.
      This can occur if auto-sync for offline-use is used, because auto-sync also uses cached connections. 

Call HighestUID which Tb knows=aaa(NextUID is aaa+1)
In this case, "new mail check by Biff" issues "STATUS MboxA (UIDNEXT MESSAGES UNSEEN RECENT)" at conection for Inbox. STATUS response is string like : * STATUS "MboxA (MESSAGES 5 RECENT 0 UIDNEXT aaa+1 UNSEEN 0).
Because UNSEEN count=0, or because number of USEEN mails iby STATUS==number of UNSEEN mail which Tb knows, this situatin=="no new mail", so Tb doesn't invoke uid:NextUID:*.

When Read State mails are copied to MboxA by other mail client before STATUS by Tb, or when new mails are added to the MboxA and te new mails are already read by other IMAP client, "UIDNEXT=nnn in STATUS response" is greater than NextUID(==aaa+1).
I believe Tb should fetch mails by Select MBOXA + "uid fetch 1:*" or "uid fetch NextUID_which_Tb_knows:+" in this situation, because larger UIDNEXT value meand that "UID which Tb doesn't know" is newly held at server.
However, Tb does do nothing in this case until MboxA is clicked again.
(Note: If MboxA is selected status at a cached connection,                 )
(      new mail check is executed by "uid fetch NextUID_which_Tb_knows:*", )
(      so newly added UIDs are fetched.                                    )            

Are you talking about this phenomenon?

Comment 11

[ilya-valeev]

>With which Thunderbird version?
Branch 31 (dont remember version accurately)

>No hibernation during the "2 days"?
Yes, without hibernation. If "hibernation" and "sleep mode" is the same, Tb work doesnt depend on hibernation in my case.

>Does it mean that your IMAP Mbox is shared with K-3 Mail client?
Yes. K-9 receives mail through IMAP from same mailbox.

>Is new mails in IMAP Mbox fetched by K-3 Mail before Tb fetches?
If Tb work correctly, sometimes Tb receives mail first, sometimes K-9. But they both receives it and both shows notification.

>Is new mails in IMAP Mbox changed to Read state by K-3 Mail?
No. I did not readed it with K-9.

>Are you talking about this phenomenon?
No, this is not what I am talking about.

>If "new mail" you call is "mail without \Seen flag"(==Read state mail
Tb doesnt show notification if I have a new unseen unread mail. Message count doesnt increases if I have unseen mail.
K-9 knows I receive mail, Tb not. Not everytime, but it happens. When it happens, I did not unlock my phone and did not read this mail, I am just hear notification about new mail. Tb in this case behaves like nothing happend.
Thats what I am talking about.

Comment 12

[WADA:World Anti-bad-Duping Agency]

I see.

Which phenomenon are you talking about?

(a) New Unread mail in an IMAP Mbox(other than Inbox) is not detected/fetched by Thunderbird.
    So Unread count of the Mbox is never updated at folder pane of Thunderbird.
(b) New Unread mail in an IMAP Mbox(other than Inbox) is properly detected/fetched by Thunderbird.
    And Unread count of the IMAP Mbox is correctly updated at folder pane.
    However, new mail alert(toaster popup) is not shown.

I couldn't observe (a). I could observe (b) only(checked with Tb 38.2.0)

1. No mail in MboxA, Offline-use=Off, Included in new mail check,
   IDLE command use is disabled, new mail check interval=1 minute.
   Go Work Offline, Select account at folder pane
   (Note: MboxA is not explicitly clicked at folder pane in this test until step 6)
2. At other IMAP client, copy an Unread mail to MboxA. Don't change Unread state of any mail.
3. At Thunderbird, Go Work Online, Click Inbox at folder pane.
   => at a cached connection, Select Inbox, "uid fetch 1:* Flags"
      STATUS MboxA => 1 EXISTS 1 UNREAD is returned
      At other cached connection, Select MboxA, uid fetch 1:* Flgs, and new unread mail is fetched
   => Unread count of MboxA is updated to 1
   => New mail alert(toaster popup) is not shown
   => Move mouse on MboxA of folder pane(mouseover) => Subject of newly fetched mails is shown like Tooltip
.(mouseover at MboxA)
4. At other IMAP client, copy an Unread mail to MboxA. Don't chnge Unread state.
   => Upon each new mail check interval, at cached connection where MboxA is selected,
      uid fetch NextUID:* Flags is issued and new unread mail is fetched,
      and Unread count of MboxA is incremented.
      But New mail alert(toaster popup) is not shown.
      Move mouse on MboxA of folder pane(mouseover) => Subject of newly fetched mails is shown like Tooltip
5. Repeaat step 4 multiple times. Result is same as step 4.
   Move mouse on MboxA of folder pane(mouseover)
   => Subject of newly fetched mails is shown like Tooltip.
After it,
6. Click MboxA => "New Mail" icon is shown for each new mail at Thread pane.
7. Click other Mbox and cluck MboxA again at folder pane
   => "New Mail" icon of each mail is cleared at Thread pane.
      This is because "New Mail" icon is shown for "new mail after last folder open" only.

If this kind of phenomenon, following your bug summary is absolutely invalid.
> Bug summary: After some period of time (random), doesn't check for new mail on imap account

Trigger of "new mail alert by Biff via toaster popup at right/bottom corner of Desktop" is "new mail in Inbox"?
IIRC, as far as CONDSTORE is disabled, when "new mail in other than Inbox" is detected via IDLE, "new mail alert via toaster popup" was shown by old Thunderbird  in test for problem of bug 885220.

Comment 13

[ilya-valeev]

>(a) New Unread mail in an IMAP Mbox(other than Inbox) is not detected/fetched by Thunderbird.
>    So Unread count of the Mbox is never updated at folder pane of Thunderbird.
Yes. I talked about this kind of bug.

>Trigger of "new mail alert by Biff via toaster popup at right/bottom corner of Desktop" is "new mail in Inbox"?
This is indication of new mail too. In my case Tb even dont update count of unread messages at folder pane.

Comment 14

[WADA:World Anti-bad-Duping Agency]

(In reply to ilya-valeev from comment #13)
> >(a) New Unread mail in an IMAP Mbox(other than Inbox) is not detected/fetched by Thunderbird.
> >    So Unread count of the Mbox is never updated at folder pane of Thunderbird.
> Yes. I talked about this kind of bug.

i see.

> In my case Tb even dont update count of unread messages at folder pane.

Is this aplicable to newly arrived Unread mails in Inbox?
I yes, it indicates that existence of new/unreaad mails is not notified via IDLE from you IMAP server.
  - You increased max cached connections to 15
  - IDLE command is supported by server, and IDLE command use is enaabled in Tb. 
  - "Check new messages upon startup and every NNN minutes" is enabled in Tb.
  - CONDSTORE support is disabled in Tb.
  So, Tb does do:
  1. After startup, Tb opens a connection, login, "uid fetch 1:* Flags", and issues IDLE
  2. If new/unread mail is saved in Inbox by server before next new mail check interval occurs, 
     following unsol response should be returned from server via IDLE.
       nnn EXISTS mmm UNSEEN (ppp RECENT if fetched by other client)
     where nnn && mmm is greater than count which Tb knows, ppp is positive if mail is not fetched by other. 
     in this case, Unread count of Inbx shuld be increased at Folder pane,
     even if new mail alert is somehow not shown.  
  3. If next new mail check occurs just before that server notifies about new mail via IDLE,
     Tb issues DONE in order to terminate IDLE state and to send IMAP commands, 
     then "uid fetch NextUID:* Flags, and UID/Flag of new/unread mails should be returned from server
     because UID of new/unread mails is grater thaan Highest UID which Tb alredy know.

Your case sounds one of next.
(a) If Tb 38.2 and phenomenon after hibernation or sleep, problem of bug 1196662 occurs.
    In tis case, Tb can't access to server due to bug unless go Work Offline+go Work Online works well.
(b) Connection loss while IDLE state.
    => Server can't notify about new/unread mail via IDLE.
    => Tb can't know about new/unread mail via IDLE,
       no response is returned from server to DONE while IDLE state
(b-1) If Tb can detect connection error during IDLE normally(for example, by timeout),
      Tb can retry sending IMAP command to server, so new mail checking caan be continued.
(b-2) If Tb can not detect.process connection error during IDLE normally(for example, by timeout),
      Tb can retry sending IMAP command to server, so new mail checking may be stopped for a while.
      Because default of IDLE interval is 29 minutes(should be less than 30 minutes),
      Correct detection of "connection loss during IDLE" takes 30 minute in Tb.
      This is known issue.

Problem like (b-2)?

Is your network connection sufficiently stable?
If unreliable network like Wifi(not connected by cable), intermittent error can happen at an element in path between PC aand IMAP server.

When you see your problem, what occurs by "Go Work Offline, the go Work Online"?
Do you see your problem with "IDLE command support disabled in Tb"?
Do you see your problem with "IDLE command support enabled in Tb" + mail.server.server#.timeout=29->1 or 3?

Comment 15

[ilya-valeev]

>Is this aplicable to newly arrived Unread mails in Inbox?
I saw this just once. In other cases it is now inbox.

>Problem like (b-2)?
In my bug occurs, I can wait much more than 30 minutes.

>Is your network connection sufficiently stable?
Yes, very stable. Cable and good provider.

>what occurs by "Go Work Offline, the go Work Online"?
You mean if I change "Work Online" to "Work Offline" and reverse? I did not tried that. Next time when I see problem will try.

>Do you see your problem with "IDLE command support disabled in Tb"?
>Do you see your problem with "IDLE command support enabled in Tb" + mail.server.server#.timeout=29->1 or 3?
Ok, will try this too.
There is problem: since Tb 38 I see this bug ~ 1 time per month. So testing will take many time.

Comment 16

[WADA:World Anti-bad-Duping Agency]

(In reply to ilya-valeev from comment #15)
> >Is this aplicable to newly arrived Unread mails in Inbox?
> I saw this just once. In other cases it is now inbox.

Question about Inbox case.
When new/unread mails are added to Inbox at server but unread count is not incremented at Tb's Folder pane,
new mails are shown by click of Inbox at Folder pane(open Inbox folder)?

If so, suspect is "connection between Tb and server is somehow killed while IDLE state".
In this case, Tb tries to do "uid fetch NextUID:*" immediately upon Inbox click. So, if connection loss exists, Tb can etect timeout quickly(by mailnews.tcptimeout. default=100 sec or 100 msec), then Tb retries IMAP command after timeout.

Comment 17

[ilya-valeev]

Did not remember how it was been in case with inbox.

Comment 18

[WADA:World Anti-bad-Duping Agency]

I recommend you to get imap log with minimum configuration and see actual IMAP level flow.
1. Create new Tb profile and start with the new profile.
2. Disable Global Search and Indexer(Tools/Options/Advanced)
   Define the IMAP account only, 
   Disable Message Synchronization : at Synchronization&Disk Space, unckeck "Keep messages for ...",
   Max cached connections = 3
   IMAP delete model=Mark it as deleted (to prohibit trash folder access)
   Check new messages upon startup = checked or unchecked
   Check new messages every NN minutes = unchecked or appropriate value
   Include only one small folder(call pretty small MboxA) in new mail check(Folder Properties/General)
   Terminate Tb.
3. Enable NSPR logging and start Thunderbird with the new profile, with -no-remote.
   See Bug 402793Comment #28 for getting log.
   At Command prompt,
     SET NSPR_LOG_MODULES=imap:5,msgbiff:5
     <Tb's progrm directory>\thunderbird.exe -no-remote -p
   => Profile manager is invoked, select profile
   Access Inbox aand MboxA only. Never touch other folder. 
   After some tests, terminate Tb, keep backup of logfile, check log content using Text Editor.

Because -no-remote is used, you can run this Thunderbird instance concurrently with your daily use Thunderbird.
Because mail data is logged, log file size is usually pretty large.
To reduce log size, keep Inbox as small as possible.
(for example, move mails in Inbox to Inbox/Pending as many as possible, and do daily job with the Inbox/Pending)

Comment 19

[Wayne Mery (:wsmwk)]

Needs protocol log per comment 18

Comment 20

[ilya-valeev]

I created new profile and run tests as WADA described, but I can not catch the bug (because it does not affects often). I can enable logging for main TB account and put it here, but it will be big (TB can work stable for weeks and months). Should I do this?

Comment 21

[Wayne Mery (:wsmwk)]

(In reply to ilya-valeev from comment #20)
> I created new profile and run tests as WADA described, but I can not catch
> the bug (because it does not affects often). I can enable logging for main
> TB account and put it here, but it will be big (TB can work stable for weeks
> and months). Should I do this?

sure :)

Comment 22

[Phoenix]

Resolved per whiteboard