Closed Bug 483859 Opened 16 years ago Closed 16 years ago

"check this folder for new messages" doesn't kick in anymore (for new mail in several folders)

Categories

(MailNews Core :: Backend, defect)

Product:
MailNews Core
Component:
Backend
Platform:
x86
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME
Milestone:
Thunderbird 3.0b3

People

(Reporter: mkmelin, Assigned: Bienvenu)

References

Details

(Keywords: regression)

Attachments

(2 files)

IMAP log of new mail check by STATUS
54.40 KB, text/plain
Details
IMAP log where folder doesn't indicate new message until selected
207.84 KB, text/plain
Details
The folder setting "check this folder for new messages" doesn't work for startup anymore. And doesn't start to work after the "check for new" period either. Once you've visited the folder once it seems to be working. Not sure when this regressed, I think more than a week but less than a moth ago.
could be the changes I made to make autosync use the imap status command - or something else entirely.
Reposting my comment from bug 428266 comment 48: Shredder isn't discovering new mail in subfolders for me anymore. Naturally I blame this change for it, due to it's description. I use the big German provider GMX, fwiw. The folder setup is: Account ├- Inbox └┬ Mailinglists ├- Sports <- these are not updated └- Pug <- 1) Can I identify my ISPs IMAP server software from Thunderbird, to find out what it supports? 2) Is there a pref to switch back to the more expensive checking mechanism?
Setting to blocking for investigation at least. If we've broken this its going to hurt some users at least.
Assignee: nobody → bienvenu
Flags: blocking-thunderbird3+
Whiteboard: [needs investigation to confirm blocking status]
Target Milestone: --- → Thunderbird 3.0b3
I set "check this folder for new messages" for Test-004/X-01, and copied 2 mails to the folder by other client, and set "Unread" status. When restart of Tb(trunk 2009/3/07 build, tested with Gmail IMAP), Tb issued following STATUS command. > SendData: 7 STATUS "Test-004/X-01" (UIDNEXT MESSAGES UNSEEN RECENT) And my IMAP server returned nothing to it, because "RECENT" is not set in 2 unread mails I copied. I couldn't observe phenomenon of "doesn't kick in anymore" in bug summary. Tb apparently issued STATUS command. Magnus Melin & Thomas Stache, does your IMAP server return UIDs to this command? "RECENT" is better to be removed from STATUS command for new mail check?
Now how would I know!? See my question 1) in comment 2... I'd like to identify the server software and version to help out, but don't know how.
See following, please. > Getting NSPR Log : Bug 402793 Comment #1 > IMAP command/response : http://www.faqs.org/rfcs/rfc3501.html
Ok, managed to get a log. Mailnews is picky about the SET lines, though. I wrote set NSPR_LOG_MODULES = imap:5 and nothing happened. Tried set NSPR_LOG_MODULES=imap:5 and the log appeared. So, from the startup log: The GMX server greets with "OK GMX IMAP4 StreamProxy ready" Capabilities are "CAPABILITY IMAP4 IMAP4rev1 IDLE AUTH=LOGIN STARTTLS" Then there follow multiple(!) LSUB commands, multiple list "" "INBOX" list "" "Gelöscht" (which is my trash folder) create "Gelöscht" and then STATUS messages for many folder in the form status "Spamverdacht" (UIDNEXT MESSAGES UNSEEN RECENT) Response: STATUS "Spamverdacht" (MESSAGES 10 RECENT 1 UIDNEXT 18297 UNSEEN 1) I have yet to encounter the "false positive after scheduled update" unread case, though.
> Response: > STATUS "Spamverdacht" (MESSAGES 10 RECENT 1 UIDNEXT 18297 UNSEEN 1) No such response from Gmail IMAP in my case seems to be bug of Gmail IMAP... By the way, RFC 3501 says as follows. STATUS is better choice? > 6.3.10. STATUS Command >(snip) > Unlike the LIST command, the STATUS command is not guaranteed to > be fast in its response. >(snip) > Because the STATUS command is not guaranteed to be fast > in its results, clients SHOULD NOT expect to be able to > issue many consecutive STATUS commands and obtain > reasonable performance.
I don't believe IMAP logs will help us here: After startup the log shows: 164[5516180]: 7049800:imap.gmx.net:A:ProcessCurrentURL: entering 2164[5516180]: 7049800:imap.gmx.net:A:ProcessCurrentURL:imap://2567049@imap.gmx.net:993/folderstatus%3E/Notification: = currentUrl 2164[5516180]: 7049800:imap.gmx.net:A:SendData: 8 STATUS "Notification" (UIDNEXT MESSAGES UNSEEN RECENT) 2164[5516180]: ReadNextLine [stream=70892a8 nb=72 needmore=0] 2164[5516180]: 7049800:imap.gmx.net:A:CreateNewLineFromSocket: * STATUS "Notification" (MESSAGES 55 RECENT 11 UIDNEXT 9703 UNSEEN 11) 2164[5516180]: ReadNextLine [stream=70892a8 nb=16 needmore=0] 2164[5516180]: 7049800:imap.gmx.net:A:CreateNewLineFromSocket: 8 OK Completed Yet the folder "Notification" is not showing the "new" or "unseen" styles.
simple tests are working here - I copy an unread message into a non-inbox folder configured to check for new messages on one machine, and hit get new mail on an other machine. The non-inbox folder gets updated counts and a green arrow shows up. I'll keep trying to reproduce the issue.
my artificial tests are all working here. Basically, we ignore Recent responses from STATUS - what we check for is if the unread count is > than the last unread count we saw for the folder. We've always done that. One thing that has changed is that we persist the STATUS counts across sessions so that auto sync can tell if things have changed across sessions. What I would really need to see from a log is two STATUS commands on the same folder, but the second one shows more unread messages, and the user doesn't see the green arrow on the folder. If that happens, I would suspect that somehow TB has updated the counts so that we don't think the second status result is really a change. If we have a cached connection to the folder, we don't do STATUS - we do a NOOP, which should cause the server to tell us about new messages. One big thing that bug 428266 changed is that we're much more likely to cache connections to folders you click on - prior to that bug fix, autosync would kick in and select various random folders so there wouldn't be any correlation between cached connections and folders you had selected in the UI. mail.imap.use_status_for_biff is the pref to set to false to go back to use SELECT to check for new messages in folders.
(In reply to comment #12) Tested again with Tb trunk 2009/3/07 build on MS Win-XP SP3. (Test-A) Once folder with "check this folder for new messages" is clicked(opened) and unless connection is not closed by other activities, unread count was automatically updated upon next new mail check interval. (Note: Due to Gmail IMAP's bug, not all mails are marked as unread upon the new mail check. This is possibly one of causes of no response to STATUS by Gmail IMAP.) (Test-B) When mail.server.serverX.download_on_biff=false is set, new mail count of the folder was updated automatically.
The key seems to be that you have to get new mail in several folders at once. My case is that my +mozilla mail are filtered server side into a mozilla folder, which is set to check for new mails. If I send only a +mozilla mail, restart, and receive the message - then it's noticed. If I send one +mozilla mail, and one other mail that will just be in the inbox, restart - then only the inbox shows to have new messages. The activity manager doesn't show anything about the mozilla folder either, before I select the folder.
Summary: "check this folder for new messages" doesn't kick in anymore → "check this folder for new messages" doesn't kick in anymore (for new mail in several folders)
(In reply to comment #14) > The key seems to be that you have to get new mail in several folders at once. This isn't the case for me. I get the new mail highlight on folders even when there are no new messages in any folders.
(In reply to comment #15) > I get the new mail highlight on folders even when there are no new messages in any folders. This phenomenon is already reported to Bug 482754. See Dependency Tree for Bug 428266.
No longer blocks: 428266
Another data point... From the log directly after startup: 8976[51d52c0]: 7368c00:imap.gmx.net:A:ProcessCurrentURL: entering 8976[51d52c0]: 7368c00:imap.gmx.net:A:ProcessCurrentURL:imap://2567049@imap.gmx.net:993/folderstatus%3E/Notification: = currentUrl 8976[51d52c0]: 7368c00:imap.gmx.net:A:SendData: 8 STATUS "Notification" (UIDNEXT MESSAGES UNSEEN RECENT) 8976[51d52c0]: ReadNextLine [stream=73565c8 nb=70 needmore=0] 8976[51d52c0]: 7368c00:imap.gmx.net:A:CreateNewLineFromSocket: * STATUS "Notification" (MESSAGES 14 RECENT 0 UIDNEXT 9733 UNSEEN 1) Still, the folder is not highlighted/bold. Why is "UNSEEN 1" not reflected in the UI?
I set TB to check all folders for new, and tried this with deeply nested sub-folders, and all my new messages showed up, along with bolded folders, both on startup, and on check for new mail. So I'm not sure what's going on for people who this isn't working for. Perhaps instead of tiny snippets of logs, someone could post a more complete log that I could look at for clues.
I could do that (tomorrow morning), but since the logs are cleared at startup they won't contain information about the previous session and the state that Thunderbird thinks it is in: comment 12: "Basically, we ignore Recent responses from STATUS - what we check for is if the unread count is > than the last unread count we saw for the folder. We've always done that. One thing that has changed is that we persist the STATUS counts across sessions so that auto sync can tell if things have changed across sessions."
Tested with 2009/3/21 build on MS Win-XP SP2. > Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b4pre) Gecko/20090321 Shredder/3.0b3pre Gmail IMAP is used. (Gmail IMAP returned correct response to STATUS. Sorry for previous wrong comment.) - Gloda=Disabled, Auto-Sync of IMAP account=Disabled - mail.server.serverX.download_on_biff=true - Tested folder : Test-004/X-01 offline use=OFF, Check this folder for new messages=ON, Never open this folder manually 1. Seamonkey : Copy first mail to X-01, set Unread status by Seamonkey 2. Restart Thunderbird. Never touch Test-004/X-01 folder. * STATUS "Test-004/X-01" (MESSAGES 1 RECENT 0 UIDNEXT 26 UNSEEN 1) => New mail count=1 at folder pane 3. Seamonney : Copy second mail to X-01, set Unread, change first mail to Read status. 4. * STATUS "Test-004/X-01" (MESSAGES 2 RECENT 0 UIDNEXT 27 UNSEEN 1) => Still new mail count=1 at folder pane 5. Seamonney : Copy third mail to X-01, set Unread, change first mail to Unread status. 6. * STATUS "Test-004/X-01" (MESSAGES 3 RECENT 0 UIDNEXT 28 UNSEEN 3) => New mail count=3 at folder pane New mail count at folder pane was successfully updated after STATUS command by 2009/3/21 build. Because server response is not changed, problem with 2009/3/07 build(Comment #4) seems to have been "folder pane refresh" problem. Note: If multiple client environment, following issue occurs. If some new mails are read by other client(s), newer mails are not detected until UNSEEN count exceeds previous value, when mail folder is not opened.
Here is my log from just now. 1. I started Shredder, one new mail displayed in INBOX. No other folder is bold. 2. Selected "Notification" folder, and voila!, there is a new mail (subject "[Bug 483859] "check this folder for new messages"...") 3. shutdown. Upon next startup I ran into bug 482754, almost all folders where bold, but without a message count. I'll attach a second log over there.
(In reply to comment #22) Thomas Stache, which build did you use in your latest test? Tb nightly latest-trunk?
(In reply to comment #23) > (In reply to comment #22) > Thomas Stache, which build did you use in your latest test? Tb nightly > latest-trunk? Looks like I missed to update yesterday, but I'm using the comm-central/mozilla-1.9.1 builds: Mozilla/5.0 (Windows; U; Windows NT 6.0; de-DE; rv:1.9.1b4pre) Gecko/20090322 Lightning/1.0pre Shredder/3.0b3pre
I may be seeing a failure to update IMAP boxes outside the context of Gmail. will attach log in a while.
can you try today's build? I checked in a fix for bug 482754, which was in some ways the opposite bug, but perhaps its better today.
Whiteboard: [needs investigation to confirm blocking status] → [may be fixed]
As for the non-google problem, it's cleared up.
Sorry, I still get this.
Whiteboard: [may be fixed]
Magnus, does c#14 still describe what you're seeing? Can you attach a protocol log, preferrably of a session where you do get new mail and don't get any mail, and then get the two messages as you described? That would tell me if the STATUS command is getting executed and working...
I tried to capture this, but seems I can't reproduce anymore. Things have definitely improved and I think what I saw yesterday is just that it doesn't update the count directly but only after the messages have been downloaded, and that just took long enough for me to think it wasn't doing anything at all. Does that make sense? Could we update the folder counts earlier like we used to? The only interesting thing in the log is that for a particular folder this entry keeps turning up: failed creating protocol instance to play queued url:imap://me@myserver:993/listfolder>/tmp/spam/test1
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → WORKSFORME
(In reply to comment #30) > it doesn't update the count directly but only after the messages have been > downloaded, and that just took long enough for me to think it wasn't doing > anything at all. Does that make sense? Could we update the folder counts > earlier like we used to? It doesn't make sense, no. We're not updating the folder pane counts any differently than we were, at least not deliberately. In your log, do you see us downloading the headers for the folder w/ new messages before you see the counts updated in the folder pane? > The only interesting thing in the log is that for a particular folder this > entry keeps turning up: > > failed creating protocol instance to play queued > url:imap://me@myserver:993/listfolder>/tmp/spam/test1 I bet that folder doesn't exist, and we're trying to LIST it to check. Why we would do that repeatedly is odd (on startup, I can see).
I just ran into this, my notorious "Notification" folder was not showing new messages in the folder tree, although 3 messages (roughly 3, 2 and 1 hours old) were there when I opened it manually. Sadly, I'm running without logging today.
FWIW, I saw the bad counts again one last time after picking up the fix for bug 482754. After the next full sync, everything has been fine since. I chalked it up to the message counts being locally cached from the last time I'd been online (when it was screwed up), and not changing until some activity happened on that folder to cause it to update (either a full sync, or a new message arriving in that folder, or something along those lines).
Gah, I still keep seeing this on startup a few times a week :( It's a lot better than the initial situation, and if I leave it running until next biff after startup it will update the counts now. Have an imap log for it, but it's 2.3M and I don't see anything of much interest.
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Magnus, do you think this is still a blocker? I assume you have server side filters running? Does your log show us trying to STATUS the folders in question when it didn't update the counts? And does mousing over the folder change the display counts, or is the only fix waiting for the next biff? I was going to ask sid if he was interested in digging into this, but I suspect you really need server-side filters to see this.
Are server-side filters really that uncommon? The web-mail providers I know have filters (Gmail, Hotmail, Yahoo Mail, GMX.de), and if you access such a server via IMAP from multiple machines (notebook + desktop/work + home), you need a good detection of new mail in subfolders, no?
I guess it's not serious enough to block now. Have to dig into it more sometimes later.
Flags: blocking-thunderbird3+ → blocking-thunderbird3-
I can confirm problem is still here. It's less frequent than before patch but still happens for me once or twice per day. Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b5pre) Gecko/20090517 Shredder/3.0b3pre
I seem to be able to work around this bug by disabling use of the IDLE command in the "advanced" server account settings. I also turned off IDLE support on my Cyrus (2.3.14) IMAP server -- something which I could do because I happen to run my own IMAP server. But I suspect turning off IDLE in the Thunderbird account settings is probably sufficient.
I haven't seen this for a while now. Rich: are you running the latest nightlies?
Sorry, no. And I'm not likely to have time right now to keep up with nightly builds. I'm afraid someone else will need to check to see if disabling IDLE makes a difference in the latest build. The best I can offer is that perhaps my experience could point to the IDLE code as a possible source of the problem, or as a place to confirm that the problem has or hasn't already been fixed.
Marking WFM then.
Status: REOPENED → RESOLVED
Closed: 16 years ago → 16 years ago
Resolution: --- → WORKSFORME
Somebody could re-open this issue ? I confirm this bug with Cyrus-imapd 2.2.13 and Thunderbird 2009-10-07-03-comm-1.9.1/thunderbird-3.0pre.en-US.linux-i686 On Cyrus server, some sieve scripts move the e-mail in a specific sub-folder of INBOX when it arrives. I've tested Cyrus with Kmail and Clawsmail, it works. I can provide some logs and I can test with nightly builds. Moreover, I can provide some test mail accounts.
Please file a new bug with details.
After a discussion with Cyrus users: http://lists.andrew.cmu.edu/pipermail/info-cyrus/2009-October/031825.html I've found the solution, you can close completely this issue. The problem is a "bad" configuration in Cyrus-IMAPd, you must use idled daemon to have the notifications in Thunderbird. For other people who could have the same problem on Debian, you must uncomment this line in /etc/cyrus.conf: idled cmd="idled" And you must change in /etc/imapd.conf: idlemethod: idled Don't forget to restart cyrus. Information from README.Debian.gz: 4. Configurable idled support. Cyrus IMAPd supports three options of using IDLE in IMAP sessions. The first option is not to support IDLE at all. The second is to use internal polling in the IMAP daemon. The third option is to use an external daemon, idled. Upstream only supports configuration of this during compilation, Debian however includes a patch which makes this runtime-configurable. Please set the 'idlemethod' imapd.conf option according to your needs and enable idled in cyrus.conf if you want to use it.
Please open again. Problem still exists on Linux with same mail server. Windos XP worked fine. (TB 24.1.1 and UW-Imap 2007e)
(In reply to xpert-reactos from comment #47) > Please open again. Problem still exists on Linux with same mail server. > Windos XP worked fine. > (TB 24.1.1 and UW-Imap 2007e) This bug won't be reopened. see comment 45, etc. But before you create a new bug report, I suggest you post in support forum https://support.mozillamessaging.com/en-US/kb/ask
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: