850628 - Wrong creation of IMAP folder if set to "server supports folders that contain sub-folders and messages" is unchecked, i.e. contain "only folders". "NO CLIENT BUG DETECTED: subscribe of non-mailbox directory" for yahoo account

Status: RESOLVED WORKSFORME

(MailNews Core :: Networking: IMAP, defect)

Description

[Andrei Boros]

User Agent: Mozilla/5.0 (Windows NT 5.1; rv:19.0) Gecko/20100101 Firefox/19.0
Build ID: 20130215130331

Steps to reproduce:

I have several accounts on Linux machines. 
They are all configured to use IMAP and server side is imapd v4 from Berkeley. 

My client is configured in account settings that "server does not support folders containing both messages and subfolders". 

This issue happened since TB16, at least. 


Actual results:

Consider the above setup. 
Create a new folder, mark it as to contain "only folders".

On next run of TB, this folder is marked by UI as a folder containing messages as well. 
On server side, in .mailboxlist, the newly created folder does not have a trailing /
Earlier versions of TB created these folders properly.


Expected results:

A proper folder to contain only subfolders should be created in the IMAP space. 

This is definitely a TB issue, because I've been using these servers with exactly the same setup, software and version since TB was at version 2, and I am the administrator of these servers.

So far the only workaround was to manually edit on the server, via shell access, the .mailboxlist file and add the trailing / to these problem folders.

Comment 1

[Andrei Boros]

As evidence that this folder is created erroneously, the next time I start TB and click on that folder I get this error : 

Mail server responded : SELECT failed : Can't open [folder], not a selectable mailbox.

Comment 2

[Andrei Boros]

And given these conditions, TB UI won't let me create additional subfolders (containing messages) under this folder.

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to Andrei Boros from comment #0)
> On next run of TB, this folder is marked by UI as a folder containing
> messages as well. 

It may be one of following.
(A) Server doesn't return \Noselect for the Mbox correctly. It may be wrong \NoSelect instead of proper \Noselect problem.
(B) Server doesn't return \Noselect in LSUB response. Server returns \Noselect to LIST only.
Bug 534021 is an exaple of this kind of issue.

If (B), disabling "Show subscribed folders only" is a workaround. Try it, please.
In any case, IMAP log is helpful to know your server's response to IMAP LSUB/LIST command. See bug 402793 comment #28 for etting IMAP log.

> So far the only workaround was to manually edit on the server,
> via shell access, the .mailboxlist file and add the trailing / to
> these problem folders.

In protocol named IMAP, "/" is Mbox name delimiter, except when delimiter other than / is used by server.
Tb also uses the "/" as delimiter of Mbox hierarchy in internal URL like path representation. So, "/" in IMAP sometimes causes funny phenomenon.
And, some *nix IMAP server don't cleanly separate "delimiter of / in protocol named IMAP" and "delimiter of / in file path of OS's file system". This produced funny rule of "trailing / in create command == Mbox which can hold subfolders only, even though Mbox name shouldn't have the trailing /", in protocol named IMAP.
See bug 29926, bug 63038 etc., please.

In Tb, create request of "ABC/" is passed as-is to IMAP server. If IMAP server interprets it as "Mbox of subfolders only", server will create it. Because bug 301714 is already fixed, Tb correctly requests "subscribe ABC".

When ABC currently exists, even if \Noselect is not returned and even if server returns NO to "select ABC", I think create request of "ABC/XYZ" is accepted by server.
Try following.
- Select account, create folder,
  request folder name of ABC/XYZ (for mail only),
  or ABC/XYZ/ (for subfolder only).
Does it work?

Comment 4

[Wayne Mery (:wsmwk)]

Andrei, see question at end...
And is bug 833551 a duplicate?

(In reply to WADA from comment #3)
> (In reply to Andrei Boros from comment #0)
> > On next run of TB, this folder is marked by UI as a folder containing
> > messages as well. 
> 
> It may be one of following.
> (A) Server doesn't return \Noselect for the Mbox correctly. It may be wrong
> \NoSelect instead of proper \Noselect problem.
> (B) Server doesn't return \Noselect in LSUB response. Server returns
> \Noselect to LIST only.
> Bug 534021 is an exaple of this kind of issue.
> 
> If (B), disabling "Show subscribed folders only" is a workaround. Try it,
> please.
> In any case, IMAP log is helpful to know your server's response to IMAP
> LSUB/LIST command. See bug 402793 comment #28 for etting IMAP log.
> 
> > So far the only workaround was to manually edit on the server,
> > via shell access, the .mailboxlist file and add the trailing / to
> > these problem folders.
> 
> In protocol named IMAP, "/" is Mbox name delimiter, except when delimiter
> other than / is used by server.
> Tb also uses the "/" as delimiter of Mbox hierarchy in internal URL like
> path representation. So, "/" in IMAP sometimes causes funny phenomenon.
> And, some *nix IMAP server don't cleanly separate "delimiter of / in
> protocol named IMAP" and "delimiter of / in file path of OS's file system".
> This produced funny rule of "trailing / in create command == Mbox which can
> hold subfolders only, even though Mbox name shouldn't have the trailing /",
> in protocol named IMAP.
> See bug 29926, bug 63038 etc., please.
> 
> In Tb, create request of "ABC/" is passed as-is to IMAP server. If IMAP
> server interprets it as "Mbox of subfolders only", server will create it.
> Because bug 301714 is already fixed, Tb correctly requests "subscribe ABC".
> 
> When ABC currently exists, even if \Noselect is not returned and even if
> server returns NO to "select ABC", I think create request of "ABC/XYZ" is
> accepted by server.
> Try following.
> - Select account, create folder,
>   request folder name of ABC/XYZ (for mail only),
>   or ABC/XYZ/ (for subfolder only).
> Does it work?

Comment 5

[Andrei Boros]

will get back to you on tuesday. 

Will also make a tcpdump of the IMAP conversation.

Comment 6

[Wayne Mery (:wsmwk)]

Andrei, please use mozilla logging for your bug reports 
https://wiki.mozilla.org/MailNews:Logging
Thanks

Comment 7

[Andrei Boros]

Will return with detailed testcase between aug 1st and 10th. thank you.

Comment 8

[Andrei Boros]

https://bugzilla.mozilla.org/show_bug.cgi?id=833551 is a duplicate. 
My mistake.

Tested the following scenarion (without logging): 
Steps 1-3 starting from root of account
1) Create folder (mail only): A/B
2) Create folder (folders only): C/D/
3) Create folder (folders only): TEST5
4) Create subfolder under TEST5 (messages only): TEST6

All folders from steps 1-4 displayed properly while TB is still running. 
Folders from steps 1,4 work properly after TB restart
Folder D from step 2 is shown as mail only
Folder TEST5 from step 3 is shown as containing both messages and folders.

Comment 9

[Andrei Boros]

Attachment: IMAP log

IMAP log from tcpdump analyzed with wireshark and extracting the TCP stream (to show only the IMAP protocol), for my test just now. 

.mailboxlist file contains (for test folders): 

A/B
C/D
TEST5
TEST5/TEST6

.mailboxfile contains for properly working folders, created with previous versions of TB: 

ELECTRONIC/
ELECTRONIC/Arduino
ELECTRONIC/Raspberry_PI

imapd --version
* OK [CAPABILITY IMAP4REV1 I18NLEVEL=1 LITERAL+ SASL-IR LOGIN-REFERRALS STARTTLS] ******* IMAP4rev1 2007c.404 at Wed, 7 Aug 2013 10:40:55 +0300 (EEST)

PACKAGE NAME:     imapd-2.00-i486-1
PACKAGE DESCRIPTION:
imapd: imapd (IMAP4rev1 from alpine)
imapd:
imapd: /usr/sbin/ipop3d and /usr/sbin/imapd are servers supporting the POP3
imapd: and IMAP remote mail access protocols.  They allow users to download
imapd: mail from your Linux system for remote viewing.

Please note that only TB version has changed, IMAP server is the same. This is a Slackware 12.2. I experience the same errors with IMAP from a Slackware-14.0 (though imapd package seems to be the same version since there hasn't been a new release since)

Comment 11

[WADA:World Anti-bad-Duping Agency]

(In reply to Andrei Boros from comment #9)
> Created attachment 786781 [details]
> IMAP log

> 50 create "C/D/"
> 50 OK CREATE completed
> 51 subscribe "C/D"
> * NO CLIENT BUG DETECTED: subscribe of non-mailbox directory C/D
> 51 OK SUBSCRIBE completed
> 52 list "" "C/D"
> * LIST (\NoSelect \HasNoChildren) "/" C/D
> 52 OK LIST completed

> 54 create "TEST5/"
> 54 OK CREATE completed
> 55 subscribe "TEST5"
> * NO CLIENT BUG DETECTED: subscribe of non-mailbox directory TEST5
> 55 OK SUBSCRIBE completed
> 56 list "" "TEST5"
> * LIST (\NoSelect \HasNoChildren) "/" TEST5
> 56 OK LIST completed

Your server looks to reject "subscribe of subfolders only folder(==directory at server)", but looks to return "LIST response" as expected.

Do you uncheck "Server supports folders that have sub-folders and messages"?

With "Server supports folders that have sub-folders and messages" checked,
do you see C/D and TEST5 at Tb's Folder Pane with "Show subscrived folder only" Checked at Server Settings/Advanced?
How about after creation of C/D/XYZ or TEST5/XYZ(message only folder==perhaps file at server)?

With "Server supports folders that have sub-folders and messages" checked,
do you see C/D and TEST5 at Tb's Folder Pane with "Show subscrived folder only" Unchecked at Server Settings/Advanced?
How about after creation of C/D/XYZ or TEST5/XYZ(message only folder==perhaps file at server)?

Comment 12

[Andrei Boros]

"Server supports folders that have sub-folders and messages" is UNCHECKED.
(for all tests and for all accounts that experienced this problem).

Will get back with further tests tomorrow.

Comment 13

[Andrei Boros]

Attachment: folders-tb9.png

Folders as created by TB9

Comment 14

[Andrei Boros]

Attachment: folders-tb24.png

Folders as created by TB24, then closing TB24 and starting it again. 
This is what is displayed. 
This is an error.

Comment 15

[Andrei Boros]

Attachment: tb-imap-settings.png

IMAP advanced settings.

Comment 16

[Andrei Boros]

One thing let to another and  ... :-( 

Performing new tests as this: I'm using a VM with TB-9.0.1 installed. 
I have configured a test account, using IMAP. 
I am using wireshark to see the IMAP traffic. 

Server is Linux, imapd (IMAP4rev1 from alpine) (imapd-2.00)
Account is configured with "Server supports folders that have sub-folders and messages" is UNCHECKED.

When I create a folder that should contain ONLY folders, then a folder with only messages inside it, this is the IMAP conversation: 

18 OK IDLE completed
19 create "TEST1/"
19 OK CREATE completed
20 subscribe "TEST1/"
* NO CLIENT BUG DETECTED: subscribe of non-mailbox directory TEST1/
20 OK SUBSCRIBE completed
21 list "" "TEST1/"
* LIST (\NoSelect \HasNoChildren) "/" TEST1/
21 OK LIST completed
22 IDLE
+ Waiting for DONE
DONE
22 OK IDLE completed
23 create "TEST1/test11"
23 OK CREATE completed
24 subscribe "TEST1/test11"
24 OK SUBSCRIBE completed
25 list "" "TEST1/test11"
* LIST (\NoInferiors \UnMarked) "/" TEST1/test11
25 OK LIST completed
26 IDLE
+ Waiting for DONE
* OK Timeout in 30 minutes

-----------------------------------------------------------
On the server, relevant .mailboxlist contents: 

TEST1/
TEST1/test11
-----------------------------------------------------------
If I CHECK the "Server supports folders with subfolders" and I create a folder, when I try to create a subfolder inside it, the context menu does not offer the "new folder" option. 
The IMAP conversation to create the folders is this: 

10 create "TEST2"
10 OK CREATE completed
11 subscribe "TEST2"
11 OK SUBSCRIBE completed
12 list "" "TEST2"
* LIST (\NoInferiors \UnMarked) "/" TEST2
12 OK LIST completed


===========================================================
Install TB-24.1.1 (upgrade the TB-9 installation). Account setup is retained. 
Opening TB, checking the account settings, they are correct. 
"server supports folders with subfolders" UNCHECKED. 

Create a folder with folders only: IMAP conversation: 

DONE
8 OK IDLE completed
9 create "TEST24/"
9 OK CREATE completed
10 subscribe "TEST24"
* NO CLIENT BUG DETECTED: subscribe of non-mailbox directory TEST24
10 OK SUBSCRIBE completed
11 list "" "TEST24"
* LIST (\NoSelect \HasNoChildren) "/" TEST24
11 OK LIST completed
12 IDLE
+ Waiting for DONE
DONE
12 OK IDLE completed
13 create "TEST24/test22"
13 OK CREATE completed
14 subscribe "TEST24/test22"
14 OK SUBSCRIBE completed
15 list "" "TEST24/test22"
* LIST (\NoInferiors \UnMarked) "/" TEST24/test22
15 OK LIST completed
16 IDLE
+ Waiting for DONE

-------------------------------------------------------------------------
Contents of .mailboxlist on server: 

TEST24
TEST24/test22

-------------------------------------------------------------------------
At this point I close TB and restart it. 

Now I see a different folder layout, see the attachment.

Comment 17

[Andrei Boros]

Upon closer inspection I found the following difference in the IMAP logs: 

TB9: 
create "TEST1/"
subscribe "TEST1/"
list "" "TEST1/"

TB24: 
create "TEST24/"
subscribe "TEST24"
list "" "TEST24"

=================================================================
So the folder creation is done properly, but the subscribe part is not.

Comment 18

[Andrei Boros]

Bug still present in TB31. 

IMAP server's CAPABILITY response: 

* CAPABILITY IMAP4REV1 I18NLEVEL=1 LITERAL+ IDLE UIDPLUS NAMESPACE CHILDREN MAILBOX-REFERRALS BINARY UNSELECT ESEARCH WITHIN SCAN SORT THREAD=REFERENCES THREAD=ORDEREDSUBJECT MULTIAPPEND SASL-IR LOGIN-REFERRALS STARTTLS

Comment 19

[Wayne Mery (:wsmwk)]

we must have duplicates of this

Comment 20

[Wayne Mery (:wsmwk)]

Andrei, do you still see this issue?

Comment 21

[Andrei Boros]

Retested under the following conditions: 

Client: Thunderbird 52.3.0 (32-bit) / Win7
Server: imapd-2.11 (IMAP4rev1 from alpine)

Thunderbird client settings for this account, under account settings, server settings, Advanced:
"Server supports folders that contain sub-folders and messages" = UNCHECKED

Test procedure: 
- start TB
- go to email account and create a new folder TEST1 (containing only sub-folders)
- under TEST1, create folders test2 and test3 (containing only messages)

At this time TB shows TEST1 correctly, with italic and grey to indicate it does not support messages, only folders.
Attempts to copy messages to TEST1 fail.
Copying messages to test2 and test3 work as expected.

A quick check on the mailserver shows the contents of .mailboxlist:
=========================
TEST1
TEST1/test2
TEST1/test3
=========================

- exit TB
- wait a bit (takes some time for mine to exit and stop all background tasks)
- start TB

folder TEST1 is correctly marked as italic and grey. 
Attempts to copy messages to TEST1 fail.
Copying messages to test2 and test3 work as expected.

Contents of .mailboxlist on server is unchanged. 


------------------------------------------------
Note: TB versions that worked properly before this bug was reported (see description above) created folders that appeared in .mailboxlist as:
=================
TEST1/
=================

TB versions working improperly as this bug reported behaved correctly if .mailboxlist contained:
=================
TEST1/
=================
but themselves created such folders that appeared in .mailboxlist as:
=================
TEST1
=================
and these worked improperly as described above. 


-----------------------------------------------------
Apparently at this time, the TB version I tested works as expected, ie. the client issue has been fixed, and I say this because I know for certain that there has been no changed at the server side.