613353 - Does not check for new IMAP mail under nested folders

Status: NEW

(MailNews Core :: Networking: IMAP, defect)

Description

[Edward Lam]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 ( .NET CLR 3.5.30729; .NET4.0E)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6

I have a nested IMAP folder like so: Inbox > Clients > XXX
There's an IMAP server side filtering rule that automatically directs email into INBOX.Clients.XXX when it matches particular address domains. For some reason, Thunderbird never notices when there is new email in that folder. I have double-checked that both the Clients and Clients.XXX folders have "When getting new messages for this account, always check this folder" are already enabled. 

It's not a email fetch delay either because when I click explicitly on "Get Mail", I see new email arriving in other folders the use similar filtering except that the folder is a direct descendant of Inbox. Thunderbird seems to just never for check new email nested one folder level down. The new email I expect is never seen until I actually click on the nested folder. In some cases, I have waited over 4 HOURS before giving up and clicking on the nested folder.


Reproducible: Always

Steps to Reproduce:
1. Create IMAP folder structure and filtering as described in details.
2. Send email to yourself
3. Notice that via web mail, your email has arrived but Thunderbird doesn't notice that there is new email. 
Actual Results:  
No new email is found in nested folder.

Expected Results:  
New email is found in nested folder.

Comment 1

[WADA:World Anti-bad-Duping Agency]

Folder view of "All Folders"? Or "Unified Folders" view?
I assume "All Folders" view.

What is your "Check for new message upon startup/Every NN minutes" settings of Server Settings?

> Steps to Reproduce:
> 2. Send email to yourself
> 3. Notice that via web mail, your email has arrived
>    but Thunderbird doesn't notice that there is new email. 

Did you wait for NNN minutes which is set in "Check for new messages every NNN miutes" after step 2, if you didn't do manual Get Messages operation?

Is "mail filtered by server to INBOX.Clients.XXX" kept as Unread at server?
If no, mail count of INBOX.Clients.XXX at folder pane may not be changed even after new mail is detected by periodical new mail check, because shown "mail count" at folder pane is "unread mail count" only after Tb 3.0.    
I think new mail alert is shown in this case, because existence of new mail itself is detected by new mail check(UID FETCH highest-UID:*, or STATUS), although I'm not sure.

Comment 2

[Edward Lam]

(In reply to comment #1)
> Folder view of "All Folders"? Or "Unified Folders" view?
> I assume "All Folders" view.

I assume "All Folders" view because I don't know what "Unified Folders" view is. It's whatever the default Thunderbird behaviour is. In my folders pane, I see two accounts separately: my IMAP account, followed by "Local Folders".

> What is your "Check for new message upon startup/Every NN minutes" settings of
> Server Settings?

Again, it's the default setting: "Check for new messages every 10 minutes". I also have "Check for new messages at startup" enabled.

I should mention that this is a very recently recreated profile from scratch as well using Thunderbird 3.1.X.


> > Steps to Reproduce:
> > 2. Send email to yourself
> > 3. Notice that via web mail, your email has arrived
> >    but Thunderbird doesn't notice that there is new email. 
> 
> Did you wait for NNN minutes which is set in "Check for new messages every NNN
> miutes" after step 2, if you didn't do manual Get Messages operation?

Yes, as I noted above, I waited for over 4 hours before I checked webmail to see if it was Thunderbird screwing up. I also mentioned above that I continued to receive new emails during those 4 hours in folders directly underneath Inbox.

> Is "mail filtered by server to INBOX.Clients.XXX" kept as Unread at server?

Yes, it is. When I checked via webmail, the email was marked as unread.

==============

NEW INFO:
Odd, I just went to check webmail thinking that I would do another test only to find that I have some unread mail even under INBOX.YYY whereas Thunderbird still hasn't noticed it yet. This is plausible given that 10 minutes might not have elapsed yet since Thunderbird last checked. However, I now manually click on "Get Mail" and it did NOT find the new email at all. It only noticed the new email AFTER I RESTARTED Thunderbird.

Could this be related to the large number of leaf IMAP folders that I have. Conservatively, I estimate that I have at least over 100 IMAP folders.

Comment 3

[Edward Lam]

I should stress the fact that Thunderbird 2 was *MUCH* more stable and I never experienced any such problems. My email setup has been unchanged for years. I have on my plate a slew of hard to reproduce hangings of Thunderbird 3 (which did not exist in Thunderbird 2) that I'd log if I could.

Comment 4

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #3)

Can you get IMAP log, check IMAP level flow, and attach log file to this bug if(and only if) Tb's fault is seen in log file?
(please never paste long data to bug, please, please).
> https://wiki.mozilla.org/MailNews:Logging
> Win sample: SET NSPR_LOG_MODULES=timestamp,imap:5
Replace sensitive data in log file before open the log file to public, please.
Please show "Order Received" column which is UID of mail if IMAP folder, during your test.

Your mail addr is [e-mail]. IMAP server of Yahoo!?

Comment 5

[WADA:World Anti-bad-Duping Agency]

What is set in mail.server.default.use_condstore by Tb 2?
(Tools/Options/Advanced/General, Config Editor)
Tb 3's default of mail.server.default.use_condstore is true. If Tb2's default is fault, it may be reason why no problem with Tb 2.

Comment 6

[Edward Lam]

I was having some problems reproducing myself but after about an hour of testing, I've got a new test procedure that reproduces 99% of the time. I have confirmed that the bug reproduces regardless of whether mail.server.default.use_condstore is true or false. ie. mail.server.default.use_condstore setting DOES NOT MATTER.

New Steps to Reproduce:

0. Create IMAP folder structure and filtering as described in details.
1. Set account settings to Check for new messages every 1 minutes so that we can more easily reproduce the bug.
2. Click on SUBFOLDER of INBOX. eg. INBOX.Foo
3. MINIMIZE Thunderbird
4. Send email to yourself from a different mail account (eg. yahoo) such that it will be filtered into INBOX.Clients.XXX. Call this email A.
5. Send email to yourself from ANOTHER different mail account (eg. gmail) such that it will be arrive into INBOX (the default). Call this email B.
6. Since Thunderbird is minimized, wait for ~1-2 minutes until the pop up alert comes up saying that email B has arrived.
7. Wait another ~1-2 minutes to make sure that email A should have already arrived by now.
8. Restore the Thunderbird window. Notice that Thunderbird only says that email A has arrived. BUG!
9. Click on "Get Mail" on Thunderbird. It STILL has not noticed that email A has arrived.
10. Confirm via webmail (ie. different IMAP client) that email A has actually already arrived. NOTE: Do NOT start up any other IMAP client until this step to rule out the possibility that it is interfering with Thunderbird. Quit webmail.
11. Finally, the only way for Thunderbird to notice that email A has arrived is by clicking on INBOX.Clients.XXX.
11. If this procedure did not reproduce. Restart Thunderbird and repeat from step 2. I have not found a case where it did not reproduce the second time if it did not reproduce the first time.

Comment 7

[Edward Lam]

(In reply to comment #4)
> Can you get IMAP log, check IMAP level flow, and attach log file to this bug
> if(and only if) Tb's fault is seen in log file?
> (please never paste long data to bug, please, please).
> > https://wiki.mozilla.org/MailNews:Logging
> > Win sample: SET NSPR_LOG_MODULES=timestamp,imap:5
> Replace sensitive data in log file before open the log file to public, please.
> Please show "Order Received" column which is UID of mail if IMAP folder, 
> during your test.

I've got plenty of logs now but I'm not sure I understand how to filter the lines so that it is useful to you. Can you give me some grep patterns to use? 

I tried a grep "STATUS.*INBOX.Clients.XXX" and it never actually appears in my log whereas there are plenty of other folders that were checked for STATUS.

> Your mail addr is [e-mail]. IMAP server of Yahoo!?

The IMAP account is a company imap server. It is NOT yahoo/gmail/etc.

Comment 8

[WADA:World Anti-bad-Duping Agency]

If step 3, "3. MINIMIZE Thunderbird", is skipped, does problem occur?
Bug 581707? Is there any message relevant to new message check in Activity Manager? Is there ay error of STATUS command in IMAP log?

Comment 9

[Edward Lam]

(In reply to comment #8)
> If step 3, "3. MINIMIZE Thunderbird", is skipped, does problem occur?
> Bug 581707? Is there any message relevant to new message check in Activity
> Manager? 

If we do NOT minimize Thunderbird, then the bug does not seem to reproduce. Looking at bug 581707, it does look similar except that it was reportedly fixed in 3.1.2 whereas I'm running the latest 3.1.6. 

> Is there ay error of STATUS command in IMAP log?

Here's something that might be relevant that I found near the end of the log where the bug is occurring:

2010-11-23 14:24:06.300000 UTC - 3704[551e140]: 54da800:imap.HOSTNAME.com:A:CreateNewLineFromSocket: * STATUS INBOX.Clients (MESSAGES 0 RECENT 0 UIDNEXT 3 UNSEEN 0)

2010-11-23 14:24:06.300000 UTC - 3704[551e140]: ReadNextLine [stream=6418a28 nb=17 needmore=0]

2010-11-23 14:24:06.300000 UTC - 3704[551e140]: 54da800:imap.HOSTNAME.com:A:CreateNewLineFromSocket: 23 OK Completed

2010-11-23 14:24:06.300000 UTC - 2292[551eb40]: ReadNextLine [stream=6419068 nb=10 needmore=0]

2010-11-23 14:24:06.300000 UTC - 2292[551eb40]: 5a1e000:imap.HOSTNAME.com:S-INBOX.Commits 12_0:CreateNewLineFromSocket: + idling

2010-11-23 14:24:06.300000 UTC - 0[2c2b140]: queuing url:imap://[e-mail]:143/customKeywords>UID>.INBOX>21848>NonJunk>

2010-11-23 14:24:06.300000 UTC - 0[2c2b140]: considering playing queued url:imap://[e-mail]:143/customKeywords>UID>.INBOX>21848>NonJunk>

2010-11-23 14:24:06.300000 UTC - 0[2c2b140]: creating protocol instance to play queued url:imap://[e-mail]:143/customKeywords>UID>.INBOX>21848>NonJunk>

2010-11-23 14:24:06.300000 UTC - 0[2c2b140]: failed creating protocol instance to play queued url:imap://[e-mail]:143/customKeywords>UID>.INBOX>21848>NonJunk>

2010-11-23 14:24:06.300000 UTC - 2292[551eb40]: 5a1e000:imap.HOSTNAME.com:S-INBOX.Commits 12_0:SendData: DONE

Comment 10

[Edward Lam]

I just realized that I also have the "Manually sort folders" 0.6.6 extension installed. I had to install it after recreating my TB 3 profile so that I could resort my IMAP folders. I've redone the tests again with the extension disabled and the bug is still reproducing. I've noticed though that even after disabling the extension, my folders still remain sorted in the way I like. Perhaps this permanent resorting of my folders in TB 3 is part of the problem?

Comment 11

[Edward Lam]

PS. I'm also managed to get into a state where the new email would be noticed in INBOX.Clients.XXX but NOT in INBOX while doing these tests. But after restarting Thunderbird again, the bug began reproducing the same way again.

Comment 12

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #9)
> Looking at bug 581707, it does look similar except that it was reportedly
> fixed in 3.1.2 whereas I'm running the latest 3.1.6. 

Bug 581707 was fixed by Thunderbird 3.3a1(see Target Milestone: field). "bugzilla: approval‑thunderbird3.1.2+" of attached patch doesn't always mean the patch was really landed on Thunderbird 3.1.2...

> > Is there ay error of STATUS command in IMAP log?
> 2010-11-23 14:24:06.300000 UTC - 0[2c2b140]: failed creating protocol instance
> to play queued
> url:imap://[e-mail]:143/customKeywords>UID>.INBOX>21848>NonJunk>

This error seems equivalent to "error response to STATUS command for INBOX.21848.NonJunk" in bug 581707 for Tb. If comparison as character in ascii, INBOX.21848.NonJunk < INBOX.Clients.XXX.

> > If step 3, "3. MINIMIZE Thunderbird", is skipped, does problem occur?
> If we do NOT minimize Thunderbird, then the bug does not seem to reproduce.

It sounds new problem of "failed creating protocol instance to play queued" on a folder upon new mail check while minimized, which produces same problem as bug 581707.
Error happened on next URL.
> url:imap://[e-mail]:143/customKeywords>UID>.INBOX>21848>NonJunk>
It seems relevant to message filter for a customKeywords whose action is "move/copy to NonJunk folder" or "Mark as NonJunk".
Is there relevant message filter rule? 

(In reply to comment #10)
> I've redone the tests again with the extension disabled and the bug is still reproducing.
> I've noticed though that even after disabling the extension, my folders still remain sorted in the way I like.

Can you check with new profile, with the relevant IMAP account only defined?
Disable Tools/Options/Advanced/General, "Global Search and Indexer", when you test, to avoid interfere by indexing for Global Search(for whole/body text search.)

I guess the add-on is relevant only to which folder is affected by error on a folder, because problem of bug 581707 occurs on folders whose name is larger in alphabet order than folder name on which the eroor happened. However, problem by add-on should be ruled out first.

Comment 13

[Edward Lam]

(In reply to comment #12)
> Bug 581707 was fixed by Thunderbird 3.3a1(see Target Milestone: field).
> "bugzilla: approval‑thunderbird3.1.2+" of attached patch doesn't always mean
> the patch was really landed on Thunderbird 3.1.2.

How do I get a development build of Thunderbird to test to see if the bug has been fixed?

> This error seems equivalent to "error response to STATUS command for
> INBOX.21848.NonJunk" in bug 581707 for Tb. If comparison as character in 
> ascii, INBOX.21848.NonJunk < INBOX.Clients.XXX.

Since ASCII '2' is less than ASCII 'C', it is true that "INBOX.21848.NonJunk" < "INBOX.Clients.XXX"

> Error happened on next URL.
> > url:imap://[e-mail]:143/customKeywords>UID>.INBOX>21848>NonJunk>
> It seems relevant to message filter for a customKeywords whose action is
> "move/copy to NonJunk folder" or "Mark as NonJunk".
> Is there relevant message filter rule? 

I thought that was INBOX>21848>NonJunk> was some Thunderbird internal thing? I have no such folder. As for message filters, I only have 1 message filter in Thunderbird which was automatically created when I enabled "Trust junk mail headers set by: SpamAssasin" in the Account Settings > Junk Settings page. All my real mail filtering is done on the server side. The only other setting I changed here was to move messages when marked as junk to INBOX.Spam (on the server).

> Can you check with new profile, with the relevant IMAP account only defined?

I give that a try and report back tomorrow. Thanks for helping me out!

Comment 14

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #13)
> How do I get a development build of Thunderbird to test to see if the bug has been fixed?

Use Tb trunk win32.zip build, please. You can test by Unzip only. 
> http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-comm-central/
Please don't forget to keep backup of current profile, if you check trunk nightly build with your current profile.
I you have POP3 account, and if you check with new or different profile, please don't forget to enable "Leave Messages on Server" and to disable all options for automatic mail deletion.

> I thought that was INBOX>21848>NonJunk> was some Thunderbird internal thing?

I think so too. I guess; At Inbox, mail of UID=21848 was considered as NonJunk(I guess by header set by SpamAssin at server), Tb tried to store flag of \NonJunk and/or remove \Junk flag, and somehow the error happened. And, which folder is affected depends on internal URL on that the error happened.
It may be contention among (a) "fetch headers for other new mails in Inbox" and (b) "download by auto-sync for Inbox" and (c) "uid 21848 store flag \NonJunk at Inbox".
If such contention, "Minimized Tb" may be a reason why such contention happened. As dispatch priority is lowered if minimized, all of (a), (b), (c) takes longer than usual. 

Show "Order Received" column(UID of mail if IMAP). Does mail of UID=21848 exist in Inbox?
If yes, what kind of mail headers are used for the mail?
If yes, can you reproduce the error by "Copy the mail to Inbox by other mail client and mark it as Unread"? (other mail client can be; Tb started by thunderbird.exe -no-remote -p "other_profile", SeaMonkey, Outlook Express, Web Mail, Tb on other PC, ...)

Comment 15

[WADA:World Anti-bad-Duping Agency]

Bug 581707 was already fixed by Tb 3.1.5, but fix(do not stop new mail check) was for "error response to IMAP command" case only.

Example.
(1) AA, AA/AAAA exist. New mail check is executed for both folder.
(2) Delete AA at server(delete Gmail Label of AA) while Tb is running. 
(3) Upon next check, error occurs. This error is informed at Activity Manager.
> 18 STATUS "AA" (UIDNEXT MESSAGES UNSEEN RECENT) 
> 18 NO Invalid folder: AA (Failure)
Before fix, new mail check for AA/AAAA and higher was not executed.
After fix(Tb 3.1.5), new mail check for AA/AAAA and higher was executed.

Following is a part of patch for Bug 581707.
>        // if we get an error running the url, it's better
>        // not to chain the next url.
> -      if (NS_FAILED(exitCode))
> +      if (NS_FAILED(exitCode) && exitCode != NS_MSG_ERROR_IMAP_COMMAND_FAILED)
>          m_foldersToStat.Clear();
After patch,
  If error response to IMAP command, Tb doesn't stop new mail check by STATUS.

There is no need to check with trunk nightly in order to see Bug 581707 was relevant or not.

Comment 16

[WADA:World Anti-bad-Duping Agency]

Attachment: IMAP log, example of "failed creating protocol instance to play queued url:imap://..." in ordinal situation

FYI.
"failed creating protocol instance to play queued url:imap://..." itself was observed in ordinal situation.

In my test, settings was next.
  max cached connections=3, with IDLE enabled.
  Inbox is opened.
  Folder of offline-use=on : "Test-Ofline-1" only.
  "Check for new message" is enabled for some folders,
  and I created Inbox/Sub, Inbox/Sub/SubSub, and enabled new mail check for
  Inbox/Sub and Inbox/Sub/SubSub.
Afer that, Inbox, "Test-Ofline-1, and Inbox/Sub was selected at each cached connection.
When each new mail check interval, "failed creating protocol instance to play queued url:imap://..." was always logged for Inbox/Sub, and "noop" was issued at a cached connection where Inbox/Sub is selected.
(I checked with DebugView, SET NSPR_LOG_FILE=WinDebug)

It may be current implementation - try to select folder at new cached connection first upon new mail check if folder is not opened, and if it failed because folder is already selected at a cached connection, execute new mail check operation at the cached connection and issues "noop" if IDLE is enabled.

Comment 17

[WADA:World Anti-bad-Duping Agency]

Attachment: IMAP log, with SpamAssassinYes only is enabled in message filter

FYI.

IMAP log with next conditions:
- Tb 3.1.5 on Win-XP.
- Only 1 message filter is active which was automatically created
  when I enabled "Trust junk mail headers set by: SpamAssasin".
  Move target folder is root level Junk.
  (Gmail IMAP, so "move to [Gmail]/Spam" is avoided)
- 1 small text/plain mail is sent to me.
- Tb is not minimized, and the mail is downloaded by new mail check.
- Inbox : offline-use=off

(a) Following log is seen. It seems attempt of fetch for Junk move.
> failed creating protocol instance to play queued url:imap://[e-mail]@imap.gmail.com:993/fetch>UID>/INBOX>11935
It's finaly successful, and fetch is executed. 
> playing queued url:imap://[e-mail]@imap.gmail.com:993/fetch>UID>/INBOX>11935

Following log in your case is not seen in my case.
> failed creating protocol instance to play queued
> url:imap://[e-mail]:143/customKeywords>UID>.INBOX>21848>NonJunk>
It seems log when header by SpamAssain exists.

"failed creating protocol instance..." itself seems simply indicator of "waiting for resource" and is irrelevant to your problem, if it's finaly successful by re-queueing of the request.

(b) uid fetch 11935 body[] was issued twice.
It may be due to offline-use=off and current spec of Disk Cache/Memory Cache, but it may be similar issue to Bug 609683. 
> Bug 609683 double messages ending up in imap offline store

Comment 18

[WADA:World Anti-bad-Duping Agency]

Woops. Correction.

Following log lines existed in my log after second "uid fetch 11935 ... body[] ...". It was for set-as-NonJunk step of "Adaptive Junk Filtering" when a mail  in IMAP Inbox is determined as "Not Junk". And, it was finally successful by re-queueing.

> 00000247	26.97630501	[2812] 0[192c140]: queuing url:imap://[e-mail]@imap.gmail.com:993/customKeywords>UID>/INBOX>11935>NonJunk>	
> 00000248	26.97637749	[2812] 0[192c140]: considering playing queued url:imap://[e-mail]@imap.gmail.com:993/customKeywords>UID>/INBOX>11935>NonJunk>	
> 00000249	26.97644997	[2812] 0[192c140]: creating protocol instance to play queued url:imap://[e-mail]@imap.gmail.com:993/customKeywords>UID>/INBOX>11935>NonJunk>	
> 00000250	26.97670364	[2812] 0[192c140]: failed creating protocol instance to play queued url:imap://[e-mail]@imap.gmail.com:993/customKeywords>UID>/INBOX>11935>NonJunk>	
> 00000251	27.05378342	[2812] 0[192c140]: considering playing queued url:imap://[e-mail]@imap.gmail.com:993/customKeywords>UID>/INBOX>11935>NonJunk>	
> 00000252	27.05385017	[2812] 0[192c140]: creating protocol instance to play queued url:imap://[e-mail]@imap.gmail.com:993/customKeywords>UID>/INBOX>11935>NonJunk>	
> 00000253	27.05416679	[2812] 0[192c140]: playing queued url:imap://[e-mail]@imap.gmail.com:993/customKeywords>UID>/INBOX>11935>NonJunk>	
> 00000254	27.05495834	[2812] 3892[4e5b640]: 7b13000:imap.gmail.com:S-INBOX:ProcessCurrentURL: entering	
> 00000255	27.05504036	[2812] 3892[4e5b640]: 7b13000:imap.gmail.com:S-INBOX:ProcessCurrentURL:imap://[e-mail]:993/customKeywords%3EUID%3E/INBOX%3E11935%3ENonJunk%3E:  = currentUrl	
> 00000256	27.05533409	[2812] 3892[4e5b640]: 7b13000:imap.gmail.com:S-INBOX:SendData: 32 uid store 11935 +FLAGS (NonJunk)

Comment 19

[Edward Lam]

(In reply to comment #14)
> Show "Order Received" column(UID of mail if IMAP). Does mail of UID=21848 
> exist in Inbox?

Yes, it is the email sent from step 5 in comment #6.

> If yes, what kind of mail headers are used for the mail?

Return-Path: <[e-mail]>
Received: from MACHINENAME.HOSTNAME.com ([unix socket])
	by MACHINENAME.HOSTNAME.com (Cyrus v2.1.18-IPv6-Debian-2.1.18-5.1) with LMTP; Tue, 23 Nov 2010 09:24:14 -0500
X-Sieve: CMU Sieve 2.2
Received: by MACHINENAME.HOSTNAME.com (Postfix, from userid 65534)
	id E1798BBC00C; Tue, 23 Nov 2010 09:24:10 -0500 (EST)
X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on MACHINENAME.HOSTNAME.com
X-Spam-Level: 
X-Spam-Status: No, score=-2.1 required=4.2 tests=AWL,BAYES_00,HTML_MESSAGE
	autolearn=ham version=3.2.3
Received: from mail.HOSTNAME.com (mail.HOSTNAME.com [38.99.137.163])
	by MACHINENAME.HOSTNAME.com (Postfix) with ESMTP id CA270BBC00C
	for <[e-mail]>; Tue, 23 Nov 2010 09:24:07 -0500 (EST)
Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182])
	by mail.HOSTNAME.com (Postfix) with ESMTP id A06B25C012
	for <[e-mail]>; Tue, 23 Nov 2010 09:24:04 -0500 (EST)
Received: by yxh35 with SMTP id 35so10193479yxh.27
        for <[e-mail]>; Tue, 23 Nov 2010 06:24:04 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=WIzJAdU1NFycosAVhqm3RCTwpP7CTSFdPxuNNjXBZik=;
        b=JyB1qy/kOHwmzvYMOxm65+J7hTdLBzKTYptWdmgcWmc6I5jwid6jVztpkbi+23ebPl
         3pFuWVvNd/9qEaPrurhwQbK1t4/QwhtgOzUUk0kkdKrMFpZOPQVLHfKnqupUCRSKaIby
         X+1KSOH3SrkpwZZBMWFvyu1Dg787RMvwAw4oU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=CwOvzETEjOJy37t4o9JWYUDD4pGUafzoGLHtivy4lAzhtmzvZ2IdBqiR6a67mRxWfa
         qvrjPWAJBBBZJxs8rrC5zcimBZ4KpEeFVtdHMWFf16NQZcns3Mqpr2keM2fd55ruAByu
         YpUvr5wbrX9TJHjGtwhWQOjoClot5HmEVlYLc=
MIME-Version: 1.0
Received: by 10.100.105.16 with SMTP id d16mr5101274anc.219.1290522243712;
 Tue, 23 Nov 2010 06:24:03 -0800 (PST)
Received: by 10.101.1.6 with HTTP; Tue, 23 Nov 2010 06:24:03 -0800 (PST)
Date: Tue, 23 Nov 2010 09:24:03 -0500
Message-ID: <AANLkTinf8maT768ASWPdShU7=[e-mail]>
Subject: test 1 from gmail
From: USERNAME1_LABEL <[e-mail]>
To: USERNAME1_LABEL <[e-mail]>
Content-Type: multipart/alternative; boundary=0016e644cef4e9482a0495b91fb9
X-COMPANY-MailScanner: Found to be clean, Found to be clean
X-COMPANY-MailScanner-Information: [e-mail] 
X-MailScanner-From: [e-mail]

> If yes, can you reproduce the error by "Copy the mail to Inbox by other mail
> client and mark it as Unread"? (other mail client can be; Tb started by
> thunderbird.exe -no-remote -p "other_profile", SeaMonkey, Outlook Express, Web
> Mail, Tb on other PC, ...)

Since this email was sent during the test procedure, it is already within INBOX.
I tried but failed to reproduce the bug in a brand new profile. Then I went back to original profile where the bug was occurring and I could no longer reproduce either! :(

I then tried to reproduce by marking the UID=21848 email as unread first via separate IMAP client (webmail). This time the bug reproduced again.

Comment 20

[Edward Lam]

Leaving mail UID=21848 as unread, I'm still able to reproduce the bug even after I,
- Turn off "Trust junk mail headers set by: SpamAssassin"
- Turn off "Move new junk messages to: Other: Spam on [e-mail]"

I forget now, but I vaguely recall that Thunderbird 3 might have created Junk and NonJunk folders by default at one point? I think that I explicitly deleted them because I already have "Spam" and "Non-Spam" folders that go through server side spam filtering.

One further test: Turn OFF "Enable adaptive junk mail controls for this account". The bug still reproduces.

The only explanation I have right now is that the bug is related to the fact that I manually resorted my INBOX.Spam folder to occur BEFORE INBOX.Clients using the "Manually sort folders" Add-On. Note that I have since comment #10, I have had it disabled and the bug continues to occur.

I should also mention that in all of this testing, sometimes it takes multiple retries before the bug would re-occur.

Comment 21

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #20)
> The only explanation I have right now is that the bug is related to the fact
> that I manually resorted my INBOX.Spam folder to occur BEFORE INBOX.Clients
> using the "Manually sort folders" Add-On. Note that I have since comment #10,
> I have had it disabled and the bug continues to occur.

Sort order change is probably done by internal resource name change around mail folder(DbName shown as folder name at folder pane, associated file name, ...).

Associatd file name != folder name is easily seen by next test utilizing bug 520437.
1. Create folder AA. file of AA.msf is used.
2. Unsubscribe AA, subscribe AA again. AA-1.msf is used due to the bug.
   AA.msf may remain as garbage.
3. Create folder AA-1. AA-1-1.msf is used, as AA-1.msf already used.
4. Create folder AA-1-1. AA-1-1-1.msf is used, as AA-1-1.msf already used.
This situation doesn't produce problem usually, but I saw several problems like Tb's confusion due to mismatch between folder name at server and local file name, during intentional test for bugs which produces special situation.

If add-on used such technique, disable of add-on may not revert back changes done by add-on.
"Uninstall of add-on + restat Tb" reverts back folder sort order to initial order.
How about "unsubscribe Inbox.Spam and subscribe Inbox.Spam again"?
(note: bug 520437 happens)
How about "delete panacea.dat + restart of Tb"?
("folder cache" data is saved in panacea.dat. This is simplest recovery procedure from mismatch between folder name and file name produced by bug 520437, although it'll produce re-download of all mails in related folders.)

In any check, keep backup of profile directory, please, unless your purpose of this bug is only to get out from funny "new mail check of Inbox.Clint.XXX somehow stopped".

Even if problem on Inbox.Client.XXX is resolved by change back of sort order, automatic new mail check will probably be stopped at Inbox.SpamXXX or higher, if your problem is same phenomenon as Bug 581707 due to different problem on Inbox.Spam or something.
Is there any error around and after last succesful STATUS command for Inbox.Client.XXX in IMAP log?
If you need log analysis by other people, please attach log file to this bug, after replacing or removing sensitive data and removing irrelevant data(mail data itself is not needed in your case. exact folder name is not needed too, as far as consistency/structure/alphebetical-order of folder names in log is kept).

Comment 22

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #19)
> I then tried to reproduce by marking the UID=21848 email as unread first via
> separate IMAP client (webmail). This time the bug reproduced again.

What is difference between mail of UID=21848 and other mails?

> X-Spam-Level: 

No data in X-Spam-Level: header. Is this header seen in other mail?
There were problems when no data in To: or some other headers, for example, Tb missed Content-Type: header after To: of no data. I couldn't see such problem when To: of no data in recent test for other bug, but this kind of header may produce problem around header handling especially when add-on is relevant.

Does you server permit creation of folder lower than INBOX such as AA? Or any folder is created under INBOX?

By the way, "status-thunderbird3.1: .2fixed" was set in Bug 581707. It means bug was surely fixed by Tb 3.1.2, although it's "patch is landed on Tb 3.1.2" and it's not always "VERIFIED with Tb 3.1.2". Sorry for misleading.

Comment 23

[Wayne Mery (:wsmwk)]

wada, Edward is no longer ablle to test. do you want to keep the bug open?

Comment 24

[gene smith]

Sounds like a very common complaint where new mail is not delivered or notified in various subfolders. I guess typical users only expect to see new mail in INBOX so this doesn't affect everybody.