3.0.6 update broke IMAP folder new message notification on some but not all IMAP folders (if STATUS error occurs on a folder, Tb doesn't check new mail of other folders)

RESOLVED FIXED in Thunderbird 3.3a1

Status

MailNews Core
Networking: IMAP
--
major
RESOLVED FIXED
7 years ago
7 years ago

People

(Reporter: Stan Hoeppner, Assigned: Bienvenu)

Tracking

1.9.1 Branch
Thunderbird 3.3a1
x86
Windows 2000
Dependency tree / graph
Bug Flags:
in-litmus +

Firefox Tracking Flags

(blocking-thunderbird3.1 .2+, thunderbird3.1 .2-fixed, blocking-thunderbird3.0 -)

Details

(Whiteboard: [needs bug 403603 back-porting to comm-1.9.1])

Attachments

(2 attachments, 2 obsolete attachments)

(Reporter)

Description

7 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.2.7) Gecko/20100713 Firefox/3.6.7
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.1.11) Gecko/20100711 Thunderbird/3.0.6

Postfix 2.5.5 receives email and pipes to Dovecot 1.2.11 IMAP server.  Dovecot Sieve plugin sorts new mail into IMAP folders, heretofore called "sort folders".  Mail not meeting sort criteria is dumped into INBOX folder.  Neither Postfix nor Dovecot has been updated nor config changes made in months.  

Before the 3.0.6 update of Jul 20, new message notification worked for new emails in INBOX and all of the "sort folders", each of whose "Check this folder for new messages" box was checked in TBird.  

After the 3.0.6 update, new mail notification only seems to work on INBOX and a subset of "sort folders".  It appears the only "sort folders" which still show new message notification are those whose name begin with "1-".  I'm not sure if this "1-" folder naming is relevant but since it appears these are the only folders where notification still works that I should relay this information just in case.

In an attempt to fix the problem, via about:config I set
mail.check_all_imap_folders_for_new = TRUE
and restarted TBird.  This did not fix the problem.  I created a fresh profile and that did not fix the problem either.

When I click on a "sort folder" which does have new mail in it:

1.  Folder name is now highlighted bold
2.  A bold number in parentheses shows the count of new emails in folder
3.  Audible notification occurs

Thus, when _manually selecting_ each "sort folder" where auto notification is broken, but which does contain new mail, TBird performs all of the "result" actions of finding new mail in a folder.

This seems to demonstrate that TBird 3.0.6 is simply not auto-checking these folders any longer for new mail, whereas as 3.0.5 and previous versions did.



Reproducible: Always

Steps to Reproduce:
AFAICT, there are no steps to reproduce the problem as it is a background processing task that has been broken between 3.0.5 and 3.0.6 causing said problem.
Actual Results:  
Again, there are no user steps/actions to perform to reproduce the problem.  It seems pretty clear that a code or other change in the 3.0.6 update broke previously working functionality.

Expected Results:  
IMAP folder new message notification should have continued to work in 3.0.6 as it did in previous versions.
(Reporter)

Updated

7 years ago
Version: unspecified → 3.0
Problemm like bug 576708 comment #20?
Is there error message in Activity Manager?
Component: General → Networking: IMAP
Product: Thunderbird → MailNews Core
QA Contact: general → networking.imap
Version: 3.0 → 1.9.1 Branch
(Reporter)

Comment 2

7 years ago
(In reply to comment #1)
> Problemm like bug 576708 comment #20?
> Is there error message in Activity Manager?

Thanks for the quick response WADA.  Yes.  Every 1 minute, which is the interval I configured in account settings, I see:

"The current command did not succeed.  The mail server responded:[CANNOT] Mailbox isn't selectable: Friends."

The first entry is today at 1:40am, when I launched TB, and last at 3:39am when I deselected the Friends folder.  What file do I look at to see when these errors first started, i.e. logs going back to Jul 20 before the update?

This is interesting.  "Friends" is a folder in the main tree that only contains subfolders due to mbox mailbox storage format on the IMAP server.  I had selected "Friends" for new mail checking when I implemented Dovecot Sieve scripts for server side mail sorting.  It was at this same time I selected many other folders for new mail checking.  "Friends" is the only main tree folder in which I have subfolders configured for new mail checking as well, two of them.  I wasn't sure at the time if I needed to check the box on the parent folder as well as the subfolders, so I went ahead and checked all 3.  Having that parent folder selected for new mail checking is apparently what caused the problem after the update.  TTBOMK, everything worked fine until the update to 3.0.6 on the Jul 20th.  I didn't notice the problem until a few days afterward because I don't receive mail from those senders on a daily or regular basis, and/or it is mail I don't look for/at on a regular basis.

Due to the failure when checking the "Friends" folder for new mail, it appears TB stopped checking the remainder of the configured folders in the account tree for new mail.  Since breaking it did not check the two subfolders within "Friends", nor the three folders further down the main folder tree, a total of 5 folders that were marked for new message checking were no longer checked.

As I mentioned, after digesting/analyzing the Account Manager log error, I've since unchecked the box for the "Friends" folder.  Since then I've not received any mail into the two "Friends" subfolders, but I have received mail into the 3 folders in the main tree that had been broken, and notification is now working on those 3 folders once again.  I'm assuming since 5 folders were broken simultaneously that it is the same issue causing all 5.  As 3 are now working again, I'll assume the other two are as well, and it's just a matter of time until mail arrives from those two senders to confirm it.

It appears my problem is solved for now, but the bug still exists.  An error returned when checking for new mail on one unselectable folder shouldn't stop the new mail checks on the remaining folders selected for such, including any subfolders of the folder which returned the error.
(In reply to comment #2)
> As I mentioned, after digesting/analyzing the Account Manager log error,
> I've since unchecked the box for the "Friends" folder.
> Since then I've not received any mail into the two "Friends" subfolders,
> but I have received mail into the 3 folders in the main tree
> that had been broken, and notification is now working on those 3 folders once again.

What action do you mean by "unchecked the box for the Friends folder"?
You unchecked "[ ] Include this folder for new mail check" or similar settings at Folder Properties/General of the Friends folder?
Is the the Friends folder shown in grayed-out/in Italic at folder pane?
No error at Activity Manager after your action?

If so, phenomenon/problem of next itself is same as Bug 574521.
> The mail server responded:[CANNOT] Mailbox isn't selectable: Friends."
If same problem as as bug 576708, unsubscribe is needed to avoid similar error upon folder click, instead of only disabling new mail check of the folder.
See bugs listed in dependency tree for bug 577343, please.
As Tb 3.1.0 had problem of bug 577343(already fixed by Tb 3.1.1), error in your case, Bug 574521, bug 576708 etc. are exposed to Tb 3.1.0 users.

In any case of Bug 574521 and bug 576708, problem of bug 576708 comment #20 occurs. AFAIK, your bug is first bug report at B.M.O for problem of bug 576708 comment #20 which I happend to see during duplication test of some bugs.
(Reporter)

Comment 4

7 years ago
(In reply to comment #1)
> Problemm like bug 576708 comment #20?
> Is there error message in Activity Manager?

I just read #20 on 576708 and would like to add the following related to how folder names contribute to my problem.  Here is my folder tree.  Those with an * are those whose "Check this folder for new messages" box is checked when right clicking the folder and selecting "Properties...", henceforth called simply "selected":

*1-Dad-Mom
*1-Debian-Users
*1-Linux-IDE
*1-Postfix-Users
*1-Roundcube
*1-Samba
*1-Spam-l
*1-XFS
Alif-OBFUSCATED
Archived
Dotster
*Friends (8 subfolders)
	..
	*Fred
	*Mike
	..
Legacy   (9 subfolders)
        ..
        ..
*Postmaster
Recent-Spam
*snugmonster
*SpamTrap

When the main tree Friends folder is "selected", notification breaks for Friends and every * folder below it.  Notification for everything above in the tree works.  It seems I can break and heal the problem simply by checking and unchecking the previously mentioned check box.
(Reporter)

Comment 5

7 years ago
(In reply to comment #3)
> (In reply to comment #2)
> > As I mentioned, after digesting/analyzing the Account Manager log error,
> > I've since unchecked the box for the "Friends" folder.
> > Since then I've not received any mail into the two "Friends" subfolders,
> > but I have received mail into the 3 folders in the main tree
> > that had been broken, and notification is now working on those 3 folders once again.
> 
> What action do you mean by "unchecked the box for the Friends folder"?
> You unchecked "[ ] Include this folder for new mail check" or similar settings
> at Folder Properties/General of the Friends folder?

Yes.

> Is the the Friends folder shown in grayed-out/in Italic at folder pane?

Yes.

> No error at Activity Manager after your action?

None.

> If so, phenomenon/problem of next itself is same as Bug 574521.
> > The mail server responded:[CANNOT] Mailbox isn't selectable: Friends."
> If same problem as as bug 576708, unsubscribe is needed to avoid similar error
> upon folder click, instead of only disabling new mail check of the folder.

I long ago had the folder click error you describe, but one of the updates fixed it.  I can't recall when or which one though.  It was many months ago.

> See bugs listed in dependency tree for bug 577343, please.
> As Tb 3.1.0 had problem of bug 577343(already fixed by Tb 3.1.1), error in your
> case, Bug 574521, bug 576708 etc. are exposed to Tb 3.1.0 users.
> 
> In any case of Bug 574521 and bug 576708, problem of bug 576708 comment #20
> occurs. AFAIK, your bug is first bug report at B.M.O for problem of bug 576708
> comment #20 which I happend to see during duplication test of some bugs.

How can I turn on the IMAP logging in TB like that shown in 574521?  Or would that even be helpful at this point?  I see no errors in the Dovecot log.
(In reply to comment #5)
> How can I turn on the IMAP logging in TB like that shown in 574521?

Have you read all comments in bugs I pointed in your this bug? Have you read document pointed by Bug 574521 comment #1 or Bug 576708 comment #2?

> Or would that even be helpful at this point?  I see no errors in the Dovecot log.

By your report and answers in this bug, I'm very confident that problem of bug 576708 comment #20 happened on you. However, real data such as IMAP log in your environment is very helpful for diagnosis of this bug, because such data will surely rule out our misunderstanding/misinterception about your comment/report.
Please replace sensitive data in log file such as mail addr, mail body text in log file, before open log data to public.
(Assignee)

Comment 7

7 years ago
Perhaps I broke this when fixing a crash in STATUS for imap folders, but that fix was in 3.0.5...
Blocks: 574521
Blocks: 576708
Confirming per report by bug opener and my bug 576708 comment #20.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Summary: 3.0.6 update broke IMAP folder new message notification on some but not all IMAP folders → 3.0.6 update broke IMAP folder new message notification on some but not all IMAP folders (if STATUS error occurs on a folder, Tb doesn't check new mail of other folders)
(Reporter)

Comment 9

7 years ago
Created attachment 460362 [details]
imap log
(In reply to comment #9)
> imap log

Server returns \Noselect for Friends correctly.
> SendData: 5 list "" "%"
> CreateNewLineFromSocket: * LIST (\Noselect \HasChildren) "/" "Friends"
Same case as Bug 574521. Tb issued STATUS even though folder of \Noselect.
blocking-thunderbird3.0: --- → ?
blocking-thunderbird3.1: --- → ?
Marking blocking for now, as I'd like David to have at least a brief investigation into the regression before the next releases.
Assignee: nobody → bienvenu
blocking-thunderbird3.0: ? → .7+
blocking-thunderbird3.1: ? → .3+
(Assignee)

Comment 12

7 years ago
I believe this is broken on the trunk as well (so presumably 3.1 as well)
(Assignee)

Comment 13

7 years ago
this is a regression from bug 550455, so it was in 3.0.5 as well.

      // if we get an error running the url, it's better
      // not to chain the next url.
      if (NS_FAILED(exitCode))
        m_foldersToStat.Clear();

these lines of code are most likely responsible. The particular error I was trying to deal with was a deadlock with certain kinds of errors. But a NO response from the IMAP server was not the kind of error I was trying to deal with. 

I can try to fix this so we don't STAT /noselect folders, and try to relax the failure check here as well.
(In reply to comment #13)
> I can try to fix this so we don't STAT /noselect folders, (snip)

(a) This bug's case and Bug 574521's case will be resolved by only it.
(b) Bug 576708's case won't be resolved by it because \Noselect is not returned to LSUB(\Noselect is probably returned to SELECT). Still workaround of "Unsubscribe" is required in this case, and "Unsubscribe" resolves the problem.
(c) Other case exists. As STATUS requires locking of mail folder at server, server may temporarily return NO. This case is not relieved by it. 
I think (b) and (c) is rare than (a) (Gmail IMAP==a). Resolving of (b)/(c) can be postponed, and can be separated bug.
(Assignee)

Comment 15

7 years ago
Created attachment 461023 [details] [diff] [review]
proposed fix

I've verified that this fixes the issue. A unit test will probably be laughably hard given the fake server, but I'll give it a try.
(Assignee)

Comment 16

7 years ago
(In reply to comment #14)
 
> (a) This bug's case and Bug 574521's case will be resolved by only it.
> (b) Bug 576708's case won't be resolved by it because \Noselect is not returned
> to LSUB(\Noselect is probably returned to SELECT). Still workaround of
> "Unsubscribe" is required in this case, and "Unsubscribe" resolves the problem.
> (c) Other case exists. As STATUS requires locking of mail folder at server,
> server may temporarily return NO. This case is not relieved by it. 
> I think (b) and (c) is rare than (a) (Gmail IMAP==a). Resolving of (b)/(c) can
> be postponed, and can be separated bug.

All true, but (b) and (c) are trivial to fix as well, so I've done that.
(Assignee)

Comment 17

7 years ago
Created attachment 461136 [details] [diff] [review]
unit test wip

this gets the test past the fake server screwiness, but doesn't work yet. I should be able to get it working tomorrow.
(Assignee)

Comment 18

7 years ago
fake server is being incredibly resistant to my attempts to make it fail on no select folders. I hope to have something by the end of the day.
(Assignee)

Comment 19

7 years ago
Created attachment 461398 [details] [diff] [review]
proposed fix

oy, ok, this should do it.

From looking at the code, the people affected by these bugs are:

1. users whose servers don't return /Noselect via LSUB - they will have all folders after STAT error folders not get checked for new messages (in both the check_all and check individual folders configurations).
2. Users whose servers do return /Noselect via LSUB, but who select individual folders (but not all) for checking for new messages even if they are /Noselect will have subsequent folders not get checked for new mail.

For 2), they can simply unselect the /Noselect folders for checking for new messages and get back to pre-regression state. For 1), there's no good remedy other than getting a build with this bug fixed.

In other words, in the check all state, we do respect /Noselect, but not in the individual check state. And in both cases, a STATUS error stops subsequent folders from getting checked.
Attachment #461023 - Attachment is obsolete: true
Attachment #461136 - Attachment is obsolete: true
Attachment #461398 - Flags: superreview?(bugzilla)
Attachment #461398 - Flags: review?(bugzilla)
blocking-thunderbird3.1: .3+ → .2+
Comment on attachment 461398 [details] [diff] [review]
proposed fix

r/sr/a=Standard8. Please land on comm-1.9.2 default and COMM1927_20100730_RELBRANCH
Attachment #461398 - Flags: superreview?(bugzilla)
Attachment #461398 - Flags: superreview+
Attachment #461398 - Flags: review?(bugzilla)
Attachment #461398 - Flags: review+
Attachment #461398 - Flags: approval-thunderbird3.1.2+
(Assignee)

Comment 21

7 years ago
fixed on trunk and 
comm-central 1.9.2 changeset:   5752:da1e3808c8dc 
and rel branch - 5751:cb11aee9da97
Status: NEW → RESOLVED
Last Resolved: 7 years ago
status-thunderbird3.1: --- → .2-fixed
Flags: in-litmus+
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 3.2a1
Comment on attachment 461398 [details] [diff] [review]
proposed fix

I believe we should take this on 3.0.7 as well.
Attachment #461398 - Flags: approval-thunderbird3.0.7+
Checked into 1.9.1 with minor bitrot fix:

http://hg.mozilla.org/releases/comm-1.9.1/rev/38ceeacc6141
status-thunderbird3.0: --- → .7-fixed

Comment 24

7 years ago
Either this or bug 581704 has turned all 1.9.1-based trees building nsImapIncomingServer.cpp red on all platforms. Please test the patch before checkin?
Backout out on 1.9.1 due to failure all-around in the push itself. 

http://hg.mozilla.org/releases/comm-1.9.1/rev/bde712b03c86
status-thunderbird3.0: .7-fixed → ---
Comment on attachment 461398 [details] [diff] [review]
proposed fix

I believe in its current state, this would need bug 403603 porting back as well (due to the use of NS_MSG_ERROR_IMAP_COMMAND_FAILED).
Attachment #461398 - Flags: approval-thunderbird3.0.7+ → approval-thunderbird3.0.7-
(Assignee)

Comment 27

7 years ago
(In reply to comment #26)
> Comment on attachment 461398 [details] [diff] [review]
> proposed fix
> 
> I believe in its current state, this would need bug 403603 porting back as well
> (due to the use of NS_MSG_ERROR_IMAP_COMMAND_FAILED).

Yes, that's right; and that part of the patch helps keep this change minimal, in the sense that we stop on other errors, like network errors.
Spoke to David over irc about the bustages, because of the need to back port bug 403603 to 1.9.1 as well, we're going to defer porting this bug until 3.0.8. This bug will be fixed in 3.1.3 however.
blocking-thunderbird3.0: .7+ → needed
Whiteboard: [needs bug 403603 back-porting to comm-1.9.1]
We unfortunately haven't had time to back-port the fixes, so if you're still seeing this bug in 3.0.x you'll need to upgrade to Thunderbird 3.1. That is the recommended route anyway because 3.0.x will no longer be supported at the end of the year.
blocking-thunderbird3.0: needed → -
FYI.
VERIFID with a Tb trunk build ater a while this bug was fixed, and,
VERIFID with Tb 3.1.5. see bug 613353 comment #15. I missed .2fixed... :-)

I don't think backport to 1.9.1 is mandatory, because;
- problem is not so critical(not crash, data loss etc.),
- work around of "kill new mail check option of problematic folder" exists,
  except when "hidden prefs for check new message of all folders" is mandatory
  in user's environment,
- user can know problematic folder by error message in Activity Manager window.
I believe "WONTFIX for Tb 3.0.x" is reasonable choice if support will end soon.
You need to log in before you can comment on or make changes to this bug.