885162 - Nonexistent IMAP folders listed not just in the folder pane but also in a refreshed subscribe window without check box for (un)subscribe (After "LIST (subscribed)" support by Bug 495318, there is no way in Tb to unsubscribe \NonExistent \Subscribed Mbox)

Status: NEW

(MailNews Core :: Networking: IMAP, defect)

Description

[Eric Moore]

Attachment: IMAP log file using Thunderbird Daily 24.0a1 (2013-06-17)

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0 (Beta/Release)
Build ID: 20130511120803

Steps to reproduce:

I use both Thunderbird Daily and Thunderbird 17.0.6 with the same test profile under Windows 7. It has a single IMAP account with Fastmail.fm, which uses a Cyrus IMAP server. The config editor has not been used to modify any of the profiles settings, though offline folders and global search/indexing are disabled. 

The INBOX.Misc.DeveloperWorks , INBOX.Misc.Java.JavaRanch , INBOX.Misc.Java.Serverside, and INBOX.Misc.purchases child folders had been deleted several months ago using Thunderbird. They disappeared from the folder pane when I did that. I had emptied the trash and ran File -> Compact Folders and thought all traces of those folders were gone.

The test profile has that account configured to use "Move it to this folder - Trash" for "when I delete a message". My normal profile (which has the same problem) is configured to use "Just mark it as deleted" and "Clean up (expunge) inbox on exit". 

I recently expanded the Inbox.Misc folder hierarchy and noticed those folders listed, in grey italic letters. The folder pane lists those folders both with and without safe mode. The folders are empty, are listed in panacea.dat, and do not exist in webmail. 

If I run Thunderbird 17.0.6 (using the same profile) it does not list any of those deleted folders. None of those folders have mbox files (global search/indexing and offline folders are disabled). File -> Compact Folders does not remove those folders from the folder listing and has no effect on this problem. The grey italic folders are not listed in Message -> Copy To. If I try to drag and drop a message from the inbox to one of those grey italic folders that fails as the mouse cursor never changes from a circle with a slash through it. If I browse the folder hierarchy in the Subscribe window (both before and after refreshing it) it lists all of the grey italic folders. So it appears that only the folder pane and the list of subscribed folders know about those grey italic folders. 

A Cyrus IMAP server has no problem with folders that contain both child folders and messages. Several of the child folders have both messages and child folders. Fastmail IMAP accounts are configured to have all folders within the inbox folder hierarchy. You can not create a peer folder for the inbox like you can with Gmail. 

The sharing tab in the folder permissions normally says "You have the following permissions: Read, Write, Insert (Copy Into), Set Read/Unread State, Delete Messages, Expunge, Create Subfolder, Post". For the grey italic folders it says "You have the following permissions: Full Control". 

As an experiment I right clicked on "purchases", selected Delete, and confirmed that I wanted to delete the folder. It did. I also created a xyzzy child folder in Inbox.Misc, exited and restarted. That new child folder is empty but is displayed normally. I deleted the folder, emptied the trash, compacted the accounts folders, exited, and restarted. The deleted folder was not resurrected (its not in the folder pane etc.). Neither was "purchases".

I started Thunderbird (daily) with IMAP logging enabled, opened the subscription window for the inbox, refreshed it, closed it and then tried to open one of the grey italic folders.

Looking in the log file I noticed 3212[974df70]: 8988800:mail.messagingengine.com:A:SendData: 8 xlist "" "%.%" didn't return any of the grey italic folders but
3212[974df70]: 8988800:mail.messagingengine.com:A:SendData: 9 list (subscribed) "" "INBOX.*" did:

3212[974df70]: 8988800:mail.messagingengine.com:A:CreateNewLineFromSocket: * LIST (\NonExistent \Subscribed) "." INBOX.Misc.DeveloperWorks
3212[974df70]: 8988800:mail.messagingengine.com:A:CreateNewLineFromSocket: * LIST (\NonExistent \Subscribed) "." INBOX.Misc.Java.JavaRanch
3212[974df70]: 8988800:mail.messagingengine.com:A:CreateNewLineFromSocket: * LIST (\NonExistent \Subscribed) "." INBOX.Misc.Java.Serverside

later on the call to "discoverallandsubscribedboxes" (which I assume is from refreshing the subscribe window) also returned the last three lines.

No network calls were made by my attempt to open the non-existent developerworks folder.

My first thought was that daily is not paying attention to the \NonExistent flag while 17.0.6 does. However when I reran it using 17.0.6 the log file did not contain any \NonExistent flags or any of the folders that Daily displayed as grey italic folders.

Right after all of the XLIST commands Daily sends:
3212[974df70]: 8988800:mail.messagingengine.com:A:SendData: 9 list (subscribed) "" "INBOX.*"
this returns the data that seems to cause the problem

17.0.6 sends instead
3308[7c4f2e0]: c3f1800:mail.messagingengine.com:A:SendData: 9 lsub "" "INBOX.*"

According to RFC 3501 "The LIST command returns a subset of names from the complete set of all names available to the client." while "The LSUB command returns a subset of names from the set of names that the user has declared as being "active" or "subscribed"."

Could this be the reason for the difference in behavior? Its not clear to me what active really means. I'm wondering if my deleting a remote folder just caused it to be inactive rather than physically deleted on the IMAP server. 

I don't have Windows Mail configured but ran eM client 5.0 instead and verified it did not have this problem. 

I can reproduce this problem 100% of the time. It occurs both in this test profile and my normal profile (which has Fastmail, Lavabit, Gmail, Yahoo and Comcast IMAP accounts plus 2 POP accounts). However, I suspect that if somebody signs up for a free trial IMAP account with www.fastmail.fm that they will not be able to reproduce the problem. I have attached the IMAP log files for Thunderbird 24.0a1 and will attach the log file for 17.0.6 later on. 



Actual results:

It lists several folders previously deleted by Thunderbird (that no longer exist in webmail) in grey italic text in the folder pane. You can not open or use them. However, they are not just an artifact of the folder pane listing. If you open the subscribe window for the inbox, expand the folder listing, and refresh the list of folders it lists the deleted folders. 

The release build (Thunderbird 17.0.6) does not have this problem, while both 32 and 64 bit versions of the latest Thunderbird daily build do. 


Expected results:

It should not have listed the deleted folders either in the folder pane or in the subscribe window.

Comment 1

[Eric Moore]

Attachment: IMAP log file using Thunderbird 17.0.6

Comment 2

[Eric Moore]

I installed SeaMonkey 2.20a1 (build identifier 20130507003002) and had it import my test profile. It has the same problem as Thunderbird Daily (both in the folder pane and a refreshed Subscribe window)

Comment 3

[Sebastian Hagedorn]

I can't explain the difference in behavior between the two versions, but I can tell you that you are making a wrong assumption. With IMAP it is perfectly normal and allowed to be subscribed to a non-existent folder. You could argue that for normal usage a folder should be automatically unsubscribed when you delete it, but given a scenario where you deleted a folder without unsubscribing, showing it in the Subscribe window is actually correct behavior.

Comment 4

[Eric Moore]

What do you mean by non-existent? That "the mailbox name does not refer to a mailbox accessible to the client" or that it no longer physically exists?

When I delete other folders (purchase, xyzzy for example) they don't remain subscribed.

I deleted the folder "INBOX.Misc.Testing.a b c d e f g h i j k l m n o p q r s t u v w x y z" using 17.0.6. I checked the subscribe window and the folder was no longer listed. The subscribe window did show a new "a b c d e f g h i j k l m n o p q r s t u v w x y z" child folder within the Trash folder. (It is still configured to use "Move it to this folder - Trash" for "when I delete a message".)

It looks like it deleting it actually renamed it to be a child folder in Trash, the child folder in the trash folder was subscribed and then the original folder unsubscribed. 

6080[a51c1d0]: 3cbd800:mail.messagingengine.com:A:SendData: 6 rename "INBOX.Misc.Testing.a b c d e f g h i j k l m n o p q r s t u v w x y z" "INBOX.Trash.a b c d e f g h i j k l m n o p q r s t u v w x y z"

6080[a51c1d0]: ReadNextLine [stream=58bbb80 nb=148 needmore=0]
6080[a51c1d0]: 3cbd800:mail.messagingengine.com:A:CreateNewLineFromSocket: * OK rename INBOX.Misc.Testing.a b c d e f g h i j k l m n o p q r s t u v w x y z INBOX.Trash.a b c d e f g h i j k l m n o p q r s t u v w x y z

6080[a51c1d0]: ReadNextLine [stream=58bbb80 nb=16 needmore=0]
6080[a51c1d0]: 3cbd800:mail.messagingengine.com:A:CreateNewLineFromSocket: 6 OK Completed

6080[a51c1d0]: 3cbd800:mail.messagingengine.com:A:SendData: 7 subscribe "INBOX.Trash.a b c d e f g h i j k l m n o p q r s t u v w x y z"

6080[a51c1d0]: ReadNextLine [stream=58bbb80 nb=16 needmore=0]
6080[a51c1d0]: 3cbd800:mail.messagingengine.com:A:CreateNewLineFromSocket: 7 OK Completed

6080[a51c1d0]: 3cbd800:mail.messagingengine.com:A:SendData: 8 unsubscribe "INBOX.Misc.Testing.a b c d e f g h i j k l m n o p q r s t u v w x y z"

6080[a51c1d0]: ReadNextLine [stream=58bbb80 nb=16 needmore=0]
6080[a51c1d0]: 3cbd800:mail.messagingengine.com:A:CreateNewLineFromSocket: 8 OK Completed

I recreated the child folder, emptied the trash, compacted the accounts folders and ran daily. I changed it to use "Just mark it as deleted", deleted the same child folder, and then looked in the subscribed window. Neither the folder pane or the subscribe window mentions that folder. 

In this case it appears to delete the child folder and then unsubscribe it.

5908[8a8b750]: 8a97800:mail.messagingengine.com:A:SendData: 11 delete "INBOX.Misc.Testing.a b c d e f g h i j k l m n o p q r s t u v w x y z"

5908[8a8b750]: ReadNextLine [stream=9f069a0 nb=17 needmore=0]
5908[8a8b750]: 8a97800:mail.messagingengine.com:A:CreateNewLineFromSocket: 11 OK Completed

5908[8a8b750]: 8a97800:mail.messagingengine.com:A:SendData: 12 unsubscribe "INBOX.Misc.Testing.a b c d e f g h i j k l m n o p q r s t u v w x y z"

5908[8a8b750]: ReadNextLine [stream=9f069a0 nb=17 needmore=0]
5908[8a8b750]: 8a97800:mail.messagingengine.com:A:CreateNewLineFromSocket: 12 OK Completed

Comment 5

[WADA:World Anti-bad-Duping Agency]

(In reply to Eric Moore from comment #0)
> Right after all of the XLIST commands Daily sends:
> 3212[974df70]: 8988800:mail.messagingengine.com:A:SendData: 9 list
> (subscribed) "" "INBOX.*"
> this returns the data that seems to cause the problem
> 17.0.6 sends instead
> 3308[7c4f2e0]: c3f1800:mail.messagingengine.com:A:SendData: 9 lsub ""
> "INBOX.*"

> CAPABILITy response from your IMAP server.
> * CAPABILITY IMAP4 IMAP4rev1 LITERAL+ COMPRESS=DEFLATE ACL RIGHTS=kxte QUOTA
>   MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN
>   MULTIAPPEND BINARY SORT SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES
>   ANNOTATEMORE CATENATE CONDSTORE LIST-EXTENDED WITHIN QRESYNC SCAN XLIST
>   SPECIAL-USE CREATE-SPECIAL-USE DIGEST=SHA1 IDLE SASL-IR AUTH=PLAIN

I believe this is simply:
- Your IMAP server supports RFC 5258 LIST Command Extensions.
- Tb 17 doesn't support RFC 5258 LIST Command Extensions.
- Daily does support RFC 5258 LIST Command Extensions,
  and uses "LIST (subscribed)" instead of standard "LSUB".

> 3212[974df70]: 8988800:mail.messagingengine.com:A:SendData: 9 list
> (subscribed) "" "INBOX.*"
> 3212[974df70]: 8988800:mail.messagingengine.com:A:CreateNewLineFromSocket: *
> LIST (\NonExistent \Subscribed) "." INBOX.Misc.DeveloperWorks
> 3212[974df70]:

This is essentially same as LSUB and LSUB response.
- Only subscribed Mbox name is returned
- Because "Subscription list" is independent from "existence of Mbox",
  non-existent Mbox name is returned when the Mbox name exists in
  subscription list at server.

(Q1) Which option do you use?
  Server Settings/Advanced, [?] Show subscribed folders only
  (a) Checked
  (b) Unchecked
  (I guess (a), because LSUB is issued after XLIST by Tb 17)

(Q2) Is "Mbox of (\NonExistent \Subscribed)" removed from Tb's folder pane by "Unsubscribe" operation at Tb's subscription list panel?

Comment 6

[Eric Moore]

Answer 1: Show subscribed folders only is checked

Answer 2: I didn't have any mbox files due to "offline folders and global search/indexing are disabled". 

I created a "DeveloperWorks." text file in the same directory as its *.msf file, started daily and then noticed that there is no checkbox to check or uncheck next to DeveloperWorks in the subscribe window. I tried using the unsubscribe button but that had no effect on the mbox file or how the folder was listed in that window or the folder pane. 

"Because "Subscription list" is independent from "existence of Mbox", non-existent Mbox name is returned when the Mbox name exists in subscription list at server."

It seems more like its returning the mailbox name than the mbox name. Wouldn't it return the same thing if Thunderbird was configured to use maildir?

I'll read RFC 5258.

Comment 7

[WADA:World Anti-bad-Duping Agency]

(In reply to Eric Moore from comment #6)
> Answer 2: I didn't have any mbox files due to "offline folders and global search/indexing are disabled".
> I created a "DeveloperWorks." text file in the same directory as its
> *.msf file, (snip)
> Wouldn't it return the same thing if Thunderbird was configured to use maildir?

???

"." INBOX.Misc.DeveloperWorks means;
- Delimiter is "."(no quot)
- "Mail folder named DeveloperWorks under Misc under INBOX",
  which is sometimes written like "INBOX->Misc->DeveloperWorks",
  whose internal URL in Tb == "imap:// ... /INBOX/Misc/DeveloperWorks",
  is currently subscribed.
- Because Mbox of INBOX.Misc.DeveloperWorks doesn't exist at server,
  attribute of "\NonExistent" is returned from server.
Tb uses following file/directory for this Mbox.
- Inbox.msf (may be INBOX.msf)
- Inbox.sbd/Misc.msf
- Inbox.sbd/Misc.sbd/DeveloperWorks.msf
This is independent from BerkleyStore or MaildirStore, and same directory/file structure is used by Tb for both message store system. Difference is associated offline-store file;
- BerkleyStore : Inbox.sbd/Misc.sbd/DeveloperWorks (file)
- MaildirStore : Inbox.sbd/Misc.sbd/DeveloperWorks (directory)
                 Inbox.sbd/Misc.sbd/DeveloperWorks/cur/nnnnnnnn (file)
                 where nnnnnnnn is file name assigned to a mail

Comment 8

[WADA:World Anti-bad-Duping Agency]

FYI.
LIST-EXTENDED support was implemented by following bug and was new feature from Tb 19.
> Bug 495318 add support for LIST (SUBSCRIBED) part of IMAP LIST-EXTENDED command (RFC 5258)
Added support looks "list (subscribed)" use only.
Because \NonExistent is treated as \Noselect, folder should be shown in Italic at Tb's folder pane, and "select" command shouldn't be issued upon folder click at Tb's folder pane.
+      else if (!PL_strncasecmp(fNextToken, "\\NonExistent", 11))
+      {
+        boxSpec->mBoxFlags |= kNonExistent;
+        // RFC 5258 \NonExistent implies \Noselect
+        boxSpec->mBoxFlags |= kNoselect;
+      }

Comment 9

[WADA:World Anti-bad-Duping Agency]

rfc5258 : http://tools.ietf.org/html/rfc5258

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to Eric Moore from comment #6)
> started daily and then noticed that there is no checkbox to check or
> uncheck next to DeveloperWorks in the subscribe window.
> I tried using the unsubscribe button but that had no effect on the mbox file
> or how the folder was listed in that window or the folder pane. 

How about non-\NonExistent == Existent folder?
> * LIST (\Subscribed) "." INBOX.Misc.Books
No check box for INBOX.Misc.Books at subscribe window of Tb?

When \Noselect, folder is shown in Italic at folder pane(folder can hold sub folders, but can't hold messages), and Tb doesn't issue "select" command for this folder because this folder is "/Noselect == not selectable".
In Gmail IMAP, following can occur.
  "create Parent/Child" when Parent doesn't exist.
  Gmail generates Gmail Label of "Parent/Child".
  Gmail doesn't create Gmail Label of "Parent".
  Gmail Label corresponds to Mbox via IMAP.
  So, following occurs.
    "Parent" doesn't exist => not returned to LIST, LSUB.
    "Parent/Child" is returned to LIST, LSUB.
  Because folder named "Parent" is needed for Tb in order to access
  folder named "Parent/Child", Tb internally generates dummy "Parent",
  and adds \Noselect attribute to this dummy "Parent" in order
  not to issue "select" command to "Parent".
In this case, "Parent" is shown without check box at Tb's subscribe window.

I think "no check box for \NonExistent Mbox" is a result of "adding \Noselect attribute to \NonExistent Mbox".

Comment 11

[WADA:World Anti-bad-Duping Agency]

If there is no way to clean up "\NonExistent Mbox" in subscription list, following is needed.
(1) Start Tb with "Show only subscribed folders" unchecked.
    At Server Settings, select IMAP delete model of
    "Just mark it deleted", to stop "move to Trash" upon folder delete.
(2) Create Mbox of \NonExistent by Tb.
    => Tb issues "create XXX", "subscribe XXX"
(3) Delete Mbox of \NonExistent by Tb.
    => Tb issues "unsubscribe XXX", "delete XXX"

Comment 12

[WADA:World Anti-bad-Duping Agency]

(In reply to Eric Moore from comment #0)
> The INBOX.Misc.DeveloperWorks , INBOX.Misc.Java.JavaRanch ,
> INBOX.Misc.Java.Serverside, and INBOX.Misc.purchases
> child folders had been deleted several months ago using Thunderbird.

Your IMAP server doesn't retuen these Mboxes to LSUB command issued by Tb 17.
i.e.
Tb correctly issued UNSUBSCRIBE command to these Mboxes and issued DELETE command for these Mboxes, and your IMAP server doesn't return them to LSUB command as expected, as defined by protocol named IMAP.

"\NonExistent \Subscribed" response for these already-unsubscribed Mboxes is merely a bug of your IMAP server, isn't it?
Or "LIST (subscribed)" means "LIST all subscribed folders in the past" instead of "LIST currently subscribed folders"?

Comment 13

[Eric Moore]

Attachment: IMAP log file of \NonExistent \Subscribed folders being unsubscribed when deleted. That required mbox files be created first.

Comment 14

[Eric Moore]

Adding that log file seems to have lost the comment that I thought would be added at the same time.

I had created the three mbox files, selected "just mark it as removed", and deleted the folders using daily. That deleted the folders, unsubscribed them and deleted the associated mbox files. 

Question: "\NonExistent \Subscribed" response for these already-unsubscribed Mboxes is merely a bug of your IMAP server, isn't it?

Answer: No idea. I don't know as a fact that the earlier version of Thunderbird unsubscribed them, as expected. Tomorrow I'll try to figure out how to use OpenSSL to list the subscribed folders, delete a subscribed folder without unsubscribing it, and then see what the IMAP server returns when I try different types of List commands.

Comment 15

[Eric Moore]

The only other IMAP account I have that supports LIST-EXTENDED is Comcast (my ISP). I'll try to compare what it returns with the Fastmail Cyrus server.  I don't know what server Comcast uses (no mention in the log file).

Comment 16

[WADA:World Anti-bad-Duping Agency]

(In reply to Eric Moore from comment #13)
> Created attachment 765731 [details]
> IMAP log file of \NonExistent \Subscribed folders being unsubscribed when deleted.

In this IMAP log, your server doesn't return \NonExistent Mbox to "LIST (unsubscribe)" after unsubscribe by Tb.

What is difference between following two "delete folders by Tb"?
(i)  Your comment #0
> The INBOX.Misc.DeveloperWorks, ... child folders had been deleted
> several months ago using Thunderbird.
(ii) Your comment #13
> ... \NonExistent \Subscribed folders ... when deleted

"Just mark it as deleted"?
If so, "\NonExistent Mbox in LIST (unsubcribed)" may be phenomenon when delete folder is done by "rename xxx.yyy.zzz INBOX.Trash.zzz" with "Move to trash", because Tb always(with any IMAP delete model) issues unsubscribe for deleted Mbox, and because "no INBOX.Misc.DeveloperWorks in LUB response" indicates "Tb surely issued unsubscribe in the past".

Or, it may be phenomenon(your IMAP server's issue) when "delete or rename is requested first, unsubscribed second".
Sorry but I don't remember order of delete/rename and unsubscribe issued by Tb.

(In reply to Eric Moore from comment #14)
> Tomorrow I'll try to (snip) delete a subscribed folder without unsubscribing it, (snip)

If using Tb, it's almost impossible, because Tb always issues unsubscribe upon delete or rename folder, unless Tb's bug fortunately happens(for example, bug when special char in Mbox name).
And, if "delete a subscribed folder without unsubscribing it" is done by someone, the Mbox should be returned to both "LSUB" command and "LIST (subscribed)" command always.
In your case, problem is that "INBOX.Misc.DeveloperWorks with \NonExistet in LIST (subscribe) response" even though "no INBOX.Misc.DeveloperWorks in LUB response".
So, I don't think such test is needed for problem determination.

However, "\NonExistent \Subscribed in LIST (subcribe) response" is needed to test Tb's behavior on "\NonExistent \Subscribed" Mbox.
Find an easy way to force "\NonExistent \Subscribed in LIST (subcribe) response", please.

By the way, I also can check with IMAP server who supports LIST-EXTENDED(vmail.me), although I can't generate states of "\NonExistent \Subscribed in LIST (subcribe) response".

Comment 17

[WADA:World Anti-bad-Duping Agency]

FYI.
Following is problem in other project which is similar to our this bug, which was found by Google Search for "dovecot imap list subscribed namespace \Nonexistent".
> http://trac.roundcube.net/ticket/1486225

Comment 18

[gene smith]

(In reply to WADA:World Anti-bad-Duping Agency from comment #17)

FYI.
Following is problem in other project which is similar to our this bug,
which was found by Google Search for "dovecot imap list subscribed namespace
\Nonexistent".

http://trac.roundcube.net/ticket/1486225

Note: The info for that link is now on github: https://github.com/roundcube/roundcubemail/issues/2474