317597 - uwimap IMAP Server - noselect folders not honored in folder list

Status: RESOLVED WORKSFORME

(MailNews Core :: Networking: IMAP, defect)

Description

[Michael Long]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.12) Gecko/20050915 Firefox/1.0.7
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.12) Gecko/20050915 Firefox/1.0.7

I am using uw imap

On the folder list on the left, I am able to click on a noselect folder, and get this error:

The current command did not succeed. The mail server responded: SELECT failed: Can't open Family: not a selectable mailbox.
(where Family is the name of the box I tried to open - which is a parent folder containing children)

I suspect the problem is this... RFC3501 specifies the flags are \Noselect and \Noinferiors, but UWIMAP returns \NoSelect and \NoInferiors (notice the capitalization)

So you just need to have thunderbird work with the non-standard behavior

* LIST (\NoSelect) "/" Family
* LIST (\NoInferiors \Marked) "/" Family/ChildFolder1
* LIST (\NoInferiors \Marked) "/" Family/ChildFolder2


Reproducible: Always

Steps to Reproduce:
Click on any noselect folder
Actual Results:  
Tries to select the folder

Expected Results:  
Clicking on the folder should never have sent any command to the IMAP server

Comment 1

[Ray Booysen]

TB development won't be concentrating on fixing a server that breaks the RFC.  The server should be fixed if this happens to be the case.

An IMAP log would be nice.
http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html

Here are the instructions.

Comment 2

[Michael Long]

Well I created a test account on the same server, and it appears to work fine. Created a new profile against my existing account, and it is broken there. I compared the LSUB and LIST responses and they look identical. So I have no idea why it is not working on my account. Unfortunately the debug log against my own account has too much personal data so I guess just close the bug and if I can ever reproduce on a test account I'll resubmit it

Comment 3

[Ray Booysen]

What personal data is it showing aside from folder names?

Comment 4

[Michael Long]

login information (passwords), subjects and content of emails

i could cut that stuff out without a problem, it just wouldn't be the complete log

Comment 5

[John Robinson]

I see something like this. I too use TB against uw-imapd.

In my folder list, some of my folder-containing folders get shown in grey italics, and all that happens is that you get an empty message list, as you'd expect. Some of them, however, are shown in black roman text, and I get the same error message as the OP when I click on them.

Exactly which folders are shown which way changes sometimes, but I can't say I've noted any pattern yet.

Comment 6

[John Robinson]

Attachment: screenshot of folder list

Shows some noselect folders as if they were selectable as referred to in my previous comment

Comment 7

[Insert Real Name]

(In reply to comment #6)
> Created an attachment (id=230377) [edit]
> screenshot of folder list
> 
> Shows some noselect folders as if they were selectable as referred to in my
> previous comment
> 
I can confirm the noselect behaviour discussed in this bug for Thunderbird 1.5.0.8 and previous versions in that series when using an uw-IMAP server (2004.357 cPanel in my case):
1) In the left-hand folder tree, folders containing other folders and/or mailboxes start an IMAP select operation when selected by the user; their names are ordinary or bold characters, depending on the read/unread status of mailboxes under them, and of course the select operation always fails.
2) In the subscription dialog tree, these same folders are shown in faded cursive characters and cannot be subscribed.
3) In the Search dialog, attempts to search in a folder with inferior mailboxes always throw up the annoying IMAP select failure dialog, which must be OK'ed before the search continues.

This has been going on for a long time; maybe making all Thunderbird interactions with IMAP flags case-insensitive and case-preserving might make a difference.

I'm sticking my neck out and asking that this bug's status be set to confirmed.

Comment 8

[John Robinson]

Attachment: IMAP protocol log

As requested. I've clicked on the Archive folder, which is a noselect folder which appears in roman text and which produced the error described, then expanded it and its Archive/ADC subfolder (also a noselect folder but which appears in italics as it should), and finally opened the Archive/ADC/ADC-to-200605 folder (which contains messages).

I've edited the log slightly, because it was too big for Bugzilla, by removing similar lines where all the messages in a folder appeared to be being checked. These removals are marked (in the 2 places I did them) with "[...many similar lines deleted...]"

Comment 9

[John Robinson]

Re comment #1: no of course TB development shouldn't concentrate on fixing broken servers, but if might be helpful if TB development aimed for full compatibility with the most common servers, even if those servers aren't perfect, and as long as it can be done without compromising TB's standards-compliance.

Comment 10

[Insert Real Name]

Attachment: IMAP Protocol Trace

I have edited this protocol trace to mask personal information and to snip a long LSUB list which also leaks personal information. I have left in the all-yawns Viagra spam.

Comment 11

[John Robinson]

OK now I'm worried. What "personal information" have I left in my trace (that isn't totally obvious or irrelevant anyway, like the name of my mail server, what I called my profile, or a few folder names which are necessary for the trace to make sense)?

Comment 12

[David :Bienvenu]

John, there's nothing too personal in your logs, just a bunch of folder names.

John, in imap sever settings, click the "advanced" button, have you UNchecked the check box that says "server supports folders that can contain messages and sub-folders"?

Comment 13

[David :Bienvenu]

for the "insert real name here" log, the server doesn't tell us that the folder is NOSELECT in its LSUB response. The issue here is that LIST returns /NOSELECT and LSUB does not - this pretty much defeats the purpose of using LSUB. You can turn off the use of LSUB if you want in the advanced imap server settings, though that makes us use the slower LIST command.

Comment 14

[Insert Real Name]

(In reply to comment #11)
> OK now I'm worried. What "personal information" have I left in my trace (that
> isn't totally obvious or irrelevant anyway, like the name of my mail server,
> what I called my profile, or a few folder names which are necessary for the
> trace to make sense)?
> 
Oh, sorry, I just have folders named after various people/organizations I correspond with--nothing more than that, but "personal" information in any case.

Comment 15

[Insert Real Name]

(In reply to comment #13)
> for the "insert real name here" log, the server doesn't tell us that the folder
> is NOSELECT in its LSUB response. The issue here is that LIST returns /NOSELECT
> and LSUB does not - this pretty much defeats the purpose of using LSUB. You can
> turn off the use of LSUB if you want in the advanced imap server settings,
> though that makes us use the slower LIST command.
> 
Thanks for explaining this--I set the mail account in Thunderbird to show all folders, not just subscribed, restarted Thunderbird, and lo-and-behold, clicking on a non-selectable folder did not cause an error message and searching messages--the essential thing about this bug--is no longer interrupted by the error message. It's an acceptable work-around.

Comment 16

[John Robinson]

Re comment #12: thanks for the reassurance. Yes, I have unchecked the "server supports folders that can contain messages and sub-folders" box in the advanced server settings.

You can see from the image I attached earlier that TB is doing the Right Thing for some folders, but the Wrong Thing for others; if I hadn't checked that box, I'd expect it to be doing the Wrong Thing always (but that'd be my fault for not asking it to do the Right Thing).

Comment 17

[John Robinson]

Just discovered a real downside with this bug: when folder-containing folders are shown wrongly, I can't create a new subfolder of them, not from the folder's own context menu, the account's context menu, nor after selecting the folder (and getting the IMAP error) the File menu.

Are we quite sure this has anything to do with uw-imap?

Anyway, please can the importance of this bug be elevated, as working round it requires me to ssh in to the IMAP server and make new folders by hand, which is both inconvenient for me and impossible for most people.

Comment 18

[David :Bienvenu]

Right, the UW imap server, when using the mailbox file format, can't have folders that both contain messages and have sub-folders - In order to create a folder that can have sub-folders, you need to give it a name with a trailing /, like a/, or create the sub-folder at the same time, e.g., create a/b (and 'a' can't already exist)

Comment 19

[John Robinson]

(In reply to comment #18)
> Right, the UW imap server, when using the mailbox file format, can't have
> folders that both contain messages and have sub-folders

This we know, and is why TB has the option in the advanced IMAP settings that you mentioned previously and that I have always had set since day 1.

> - In order to create a
> folder that can have sub-folders, you need to give it a name with a trailing /,
> like a/, or create the sub-folder at the same time, e.g., create a/b (and 'a'
> can't already exist)

That's nonsense. In order to create a folder with subfolders, you pick the "folders only" radio button, instead of the "messages only" radio button.

However, you have helped me with a workaround, in that where TB shows e.g. my Archive folder in black roman not grey italic, and clicking it causes the error message we know about, and doesn't let me create a sub-folder, I can still from the top level create a folder called Archive/subfolder and it gets created in the right place.

Comment 20

[John Robinson]

Just noticed Dovecot IMAP server now has a workaround for this Thunderbird bug: http://wiki.dovecot.org/MainConfig#line-373

Comment 21

[Josh Larios]

(In reply to comment #0)

> I suspect the problem is this... RFC3501 specifies the flags are \Noselect and
> \Noinferiors, but UWIMAP returns \NoSelect and \NoInferiors (notice the
> capitalization)

RFC3501 also states, in the formal syntax section:

        (1) Except as noted otherwise, all alphabetic characters
        are case-insensitive.  The use of upper or lower case
        characters to define token strings is for editorial clarity
        only.  Implementations MUST accept these strings in a
        case-insensitive fashion.

If the problem really is with capitalization, then the non-RFC-compliant component is TB, not UW-imapd.

Comment 22

[David :Bienvenu]

I'd be surprised if the issue wasn't that UW IMAP returns Noselect for the LIST flags, but doesn't return the same flags for LSUB.  That's allowed by the RFC, but I think UW IMAP is the only server that does that. The workaround is to tell TB not to use IMAP subscription.

Comment 23

[Wayne Mery (:wsmwk)]

Can you get an imap log so we can confirm whether the issue is comment 21 or 22?
Instructions at http://wiki.mozilla.org/MailNews:Logging

Comment 24

[Michael Long]

I've been running dovecot for quite a while now, so I no longer have any way to further assist with this bug report

Comment 25

[John Robinson]

I'm running Dovecot now too, but still using mbox folders, and even with its tb-extra-mailbox-sep workaround, I still have this problem. I'll attach an imap log.

Comment 26

[John Robinson]

Attachment: IMAP protocol log 2008-12-17, bzip2'ed

Recent IMAP protocol log, as requested in comment 23.

Comment 27

[John Robinson]

I've now tried unticking "show subscribed folders only", and after restarting TB all my noselect folders reappeared in grey italics, and didn't cause the error. Switching it on again, they're right back to black Roman and causing the error again (as per my recent IMAP protocol log). So is this because Dovecot too doesn't return NoSelect when using LSUB?

Comment 28

[Michael Long]

I'm using Maildirs with Dovecot and haven't had any problem. I actually have been able to put emails into all levels of the hierarchy (by accident actually) and it has worked fine. So maybe this is limited to mbox (which would seem to be the case, since the comments for tb-extra-mailbox-sep says it only applies to mbox)

Comment 29

[Timo Sirainen]

Note that in LSUB reply a \NoSelect means that the mailbox itself isn't subscribed, but that it has children that are subscribed. So there's really no good way for server to even provide the \NoSelect flag. Either you'd need to do both LIST+LSUB, or from Dovecot's point of view you could also switch to use LIST-EXTENDED (RFC 5258), where you could simply do:

a list (subscribed) "" *
* LIST (\Subscribed) "/" "INBOX"
* LIST (\Subscribed \NonExistent) "/" "test"
* LIST (\Subscribed \NoInferiors \Marked) "/" "Trash"
x OK List completed.

Comment 30

[David Lechner (:dlech)]

(In reply to Timo Sirainen from comment #29)
> Note that in LSUB reply a \NoSelect means that the mailbox itself isn't
> subscribed, but that it has children that are subscribed. So there's really
> no good way for server to even provide the \NoSelect flag. Either you'd need
> to do both LIST+LSUB, or from Dovecot's point of view you could also switch
> to use LIST-EXTENDED (RFC 5258), where you could simply do:

Thanks for the tip. I was looking into doing LSUB + LIST. It is doable, but presents a number of difficulties. I did not consider LIST-EXTEDNED. This would probably fix most cases.

So, we can mark this bug as depends on Bug 495318.

Comment 31

[John Robinson]

I am still having a problem with this with Thunderbird 15.0.1. However I have discovered (by looking at my .subscriptions file) that I am subscribed to some folder-only mailboxes, which show as selectable in TB and throw the select failed error, and I am not subscribed to some other folder-only mailboxes, which show correctly (grey/italics) as unselectable. I think I am subscribed to them because SquirrelMail likes to be subscribed to them (it doesn't show a folder-only mailbox if you don't subscribe to it, so you can't expand it to list its contents). Unsubscribing in SquirrelMail, followed by telling TB to show all folders, restarting it, telling TB to show only subscribed folders, and restarting it again, gets a folder-only mailbox back to being unselectable.

Is Squirrelmail wrong to want to subscribe to folder-only mailboxes, or is TB wrong not to cope with being subscribed to folder-only mailboxes?

Comment 32

[David Lechner (:dlech)]

(In reply to John Robinson from comment #31)
> Is Squirrelmail wrong to want to subscribe to folder-only mailboxes, or is
> TB wrong not to cope with being subscribed to folder-only mailboxes?

TB can't cope - with some servers anyway. This should be getting fixed in bug 799821 pending approval.

In the meantime the workarounds are:
1. Uncheck the 'Server supports folders that contain sub-folders and messages' option in Account Settings > Server Settings > Advanced... (button) > Advanced Account Settings
- OR -
2. Uncheck the 'Show only subscribed folders' setting in the same Advanced Account Settings dialog.

Comment 33

[Wayne Mery (:wsmwk)]

As I read the "solution" per comment 30, the intention (comment 32?) is this would be fixed by bug 799821 - which was closed with a patch in 2014.

So it is safe to also close this bug?

Comment 34

[gene smith]

Yes, I think it's OK to close this since LIST-EXTENDED cap makes it moot. But I doubt it will fix the UW server but, AFAIK, it's now used very little and replaced by Dovecot which does support LIST-EXTENDED.