922250 - IMAP shared mailbox/folder support is broken in Thunderbird 24 (Dovecot server returns "\NonExistent" to "LIST (subscribed)" command used by Tb 24, despite that subfolder under PublicNamespace is existent & subscribed)

Status: RESOLVED WORKSFORME

(MailNews Core :: Networking: IMAP, defect)

Description

[David Epping]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/28.0.1500.71 Chrome/28.0.1500.71 Safari/537.36

Steps to reproduce:

My dovecot mailserver uses shared mailboxes to share some folders between users.

Upgrade from Thunderbird 17.0.8 to 24.0 on Windows XP and 17.0.5 to 24.0 on Ubuntu 12.04 breaks shared folder support.
Downgrading to for example TB 17 ESR fixes the problem again.

I don't have very much insight into TB and can not debug what's going on. I can provide much more detail on the mail server side if desired.

If it helps I can try more intermediate versions from http://ftp.mozilla.org/pub/mozilla.org/thunderbird/releases/


Actual results:

While regular private user folders are still operational the shared folders are greyed-out in TB 24. Also they don't show up in the list of "Copy to" folders any more. Mail can not be copied or moved to them and I believe the content is also no longer updated if it changes on the server. old mail that was there before the update can however still be viewed (local cache version?).
The subscription window still shows those folder, but checkboxes don't stay ticked.


Expected results:

I would expect no change in behavior between TB 17 and TB 24 with regards to shared IMAP folders.

Comment 1

[Emile Joubert]

Same problem after upgrading to 24.0. Also a Dovecot server: shared folders greyed out.
Attempting to recreate the profile doesn't help.

Comment 2

[WADA:World Anti-bad-Duping Agency]

(In reply to David Epping from comment #0)
> Expected results:
> I would expect no change in behavior between TB 17 and TB 24 with regards to
> shared IMAP folders.

As written in bug 918968, if your server has problem of bug 317597, bug 534021 on the shared IMAP folders, it's simply because behavior Tb 17 is special and behavior of Tb 24 is same as one before Tb 17.
  Workaround of bug 317597, bug 534021 etc. was tried in Tb 17 by patch for bug 799821,
  but the patch for bug 799821 was backed out in Tb 24 because problem of bug 858062 and
  bug 859269 occurred on some not-well-configured IMAP servers.

What is your following setting in Server Settings, Advanced?
(a) Show only subscribed folders
(b) Server supports folders that contain sub-folders and messages
Which case? 
 a=Checked/b=Checked, a=Checked/b=Unchecked, a=Unchecked/b=Checked, a=Unchecked/b=Unchecked

Do you see your problem in Tb 24 with a=Unchecked?

Does your server has problem like bug 317597, bug 534021?
(Read bugs well, get IMAP log and check log content by yourself using Text Editor)

Does problem like bug 918968 comment #13 occur with Tb 24 in your environment?
(see IMAP log attache by bug 918968 comment #10)

Comment 3

[David Epping]

Hello,

thank you for your detailed response. From the behavior described this bug could very well be a duplicate of bug 918968. I honestly looked and searched for existing bugs mentioning my problem but did not see/find that one.
However, the discussion in this thread seems to go into another direction than what my findings based on your input suggest.

With regards to your question: a) and b) have been checked when it was not working. Unchecking a) makes it work again. I don't think this problem exited prior to TB17, however. To the best of my knowledge TB 3.5 worked fine.


I looked into the bugs you referenced and they all seem to focus on the \NoSelect and \NoInferiors flags. Neither of those is returned by my server.
However, connecting to the server via telnet gives the following summarized results for LIST, LSUB, and LIST (subscribed). Notice that the server claims \nonexistent for all shared folder which it does not do for regular folders. A SELECT on a shared folder claimed to not exists succeeds however:

a2 LIST "" "*"
* LIST (\HasNoChildren) "." "INBOX"
* LIST (\HasNoChildren) "." "shared-rba.TFU"
a2 OK List completed.

a2 LSUB "" "*"
* LSUB () "." "INBOX"
* LSUB () "." "shared-rba.TFU"
a2 OK Lsub completed.

a2 LIST (subscribed) "" "*"
* LIST (\Subscribed) "." "INBOX"
* LIST (\Subscribed \NonExistent) "." "shared-rba.TFU"
a2 OK List completed.

a2 select "shared-rba.TFU"
* OK [CLOSED] Previous mailbox closed.
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
* 1500 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1286806646] UIDs valid
* OK [UIDNEXT 1505] Predicted next UID
* OK [HIGHESTMODSEQ 875] Highest
a2 OK [READ-WRITE] Select completed.


Looks like it is this bug in dovecot that TB did not care about previously (nor does any other client I tested):
http://www.dovecot.org/list/dovecot/2010-October/054372.html


I also looked into the commands and responses with a) checked and unchecked in the server log:
With "Show only subscribed folders" checked TB sends the LIST (subscribed) "" "*" command.
Without "Show only subscribed folders" TB sends LIST "" "%" command.

As shown in the telnet examples above the reply to the lest (subscribed) command contains the nonexistent flag and I guess that is what TB24 does not like.

Comment 4

[WADA:World Anti-bad-Duping Agency]

(In reply to David Epping from comment #3)
> I also looked into the commands and responses with a) checked and unchecked in the server log:
> With "Show only subscribed folders" checked TB sends the LIST (subscribed) "" "*" command.
> Without "Show only subscribed folders" TB sends LIST "" "%" command.

Thanks for presenting IMAP log.

Oh, not too old Dovecot. Dovecot with LIST-EXTENDED.
LIST-EXTENDED support on "LIST (subscribed)" is implemented by bug 495318, and is new feature from Tb 19. (See bug 813331 for other LIST-EXTENDED command support)

> a2 LIST (subscribed) "" "*"
> * LIST (\Subscribed) "." "INBOX"
> * LIST (\Subscribed \NonExistent) "." "shared-rba.TFU"
> a2 OK List completed.

Because "shared-rba" is not returned to LIST "subscribed", "shared-rba" is identical to actually non-existent Mbox for Tb with "Show only subscribed folders" checked.
Because \NonExistent is returned from your Dovecot server for "shared-rba.TFU"(TFU under shared-rba), "shared-rba.TFU" is actually non-existent Mbox for any IMAP client who trusts on LIST-EXTENDED in CAPABILITY response and who utilizes LIST subscribed command.
If "non-existent Mbox", IMAP client's correct behavior is "never issue SELECT command for the Mbox".

> Looks like it is this bug in dovecot that TB did not care about previously (nor does any other client I tested):
> http://www.dovecot.org/list/dovecot/2010-October/054372.html
It may be "not-so-well Dovecot configuration". No configuration option for LIST-EXTENDED?
If LIST-EXTENDED relevant bug of Dovecot, remove LIST-EXTENDED from CAPABILITY response.

Comment 5

[WADA:World Anti-bad-Duping Agency]

> a2 LIST (subscribed) "" "*"
> * LIST (\Subscribed \NonExistent) "." "shared-rba.TFU"
It may be intentional default in "Dovecot"+"LIST-Extended"+"shared(public) Namespace".
 Major purpose of "Mbox as shared namespace"(Mbox as PublicNamespace) is container of
 many shared Mboxes. In order to avoid "select command by IMAP clients for the Mbox",
 Dovecot may force \NonExistent by deault.

Comment 6

[WADA:World Anti-bad-Duping Agency]

FYI.
Tb side IMAP log is useful. See bug 402793 comment #28 for getting NSPR log. Following parameter is usually used for first IMAP problem analysis.
> timstamp,imap:5      (Termination of Tb is needed to force writing buffered data)
> timstamp,imap:5,sync (If you want to see log while Tb is running)

Comment 7

[WADA:World Anti-bad-Duping Agency]

FYI.
As I wrote in bug 885162 comment #8, \NonExistent in LIST (subscribed) response is treated as \NoSelect in LIST response by Tb, because \NonExistent implies \NoSelect.
Bug 885162 is for problem of;
  Because Tb doesn't show check box for \NoSelect folder at subscription list
  (File/Subscribe), check box in subscription list is not shown for folder of \NonExistent
  and \Subscribed in LIST (subscribed) response.
  So, there is no way to unsubscribe the \NonExistent and \Subscribed folder in Tb,
  when server supports LIST-EXTENDED and "Show only subscribed folders" is checked.

Comment 8

[WADA:World Anti-bad-Duping Agency]

FYI.
Following is Web documents found by Google Search for "dovecot imap listextended list subscribed namespace".
> http://wiki.dovecot.org/Namespaces#Namespace_settings
> http://wiki2.dovecot.org/Namespaces#Namespace_settings (Dovecot V2.x)
> http://wiki1.dovecot.org/Namespaces#Namespace_settings (Dovecot V1.x)
According to the documents, there is setting related to to namespace and LIST. This setting seems relevant to LIST (subscribed).
Needless to say, LISTEXTENDED extention can be disabled always.

Comment 9

[WADA:World Anti-bad-Duping Agency]

Following is problem in other project which is similar to our bug 885162, which was found by Google Search for "dovecot imap list subscribed namespace \Nonexistent".
> http://trac.roundcube.net/ticket/1486225
By http://trac.roundcube.net/ticket/1486225#comment:24, following document is pointed too.
> http://www.dovecot.org/list/dovecot/2010-October/054372.html
In that project, "map_force_lsub option" looks to have been implemented in 2010, in order to force LSUB command use instead of "LIST (subscribed)" because it's Dovecot's actual bug.
Following is also a document found by the Google Search.
> http://comments.gmane.org/gmane.mail.imap.dovecot/53235
In this document, following was written on 2010-10-26.
> Fixed: http://hg.dovecot.org/dovecot-2.0/rev/3e59dbe29f81
The Dovecot's bug looks already resolved by Dovecot V2 in 2010.

To David Epping(bug opener): 
When was Dovecot V2.0 released?
(According to http://www.dovecot.org/ , latest Dovecot seems v2.2.6 which was released on
Wed Sep 25 10:10:48 EEST 2013)
Which version of Dovecot is used by your server?

Comment 10

[WADA:World Anti-bad-Duping Agency]

Problem fixed by dovecot-2.0/rev/3e59dbe29f81 is;
  LIST "" "*" RETURN (STATUS (MESSAGES UNSEEN))
  => * LIST (\NonExistent) "/" "Public"
  LIST (SUBSCRIBED) "" "*" RETURN (STATUS (MESSAGES UNSEEN))
  or LSUB "*"
  => "Public" is not returned.
Problem of "\Nonexistent by LIST (subscribed) for any mbox under PublicNamespace" was generated by such change, or was exposed by such change?

Comment 11

[WADA:World Anti-bad-Duping Agency]

Followng is a document found by search for "dovecot bug imap list subscribed \nonexistent namespace"("bug" is added), and an entry seems relevant.
> http://markmail.org/browse/org.dovecot.dovecot-cvs/2012-04
>   dovecot-2.1: Maildir++: Return nonexistent subscribed mailbox wi...
> http://code.metager.de/source/history/dovecot/2.1/src/
>   14392:f03e81b702a8 04-Apr-2012 Timo Sirainen(==Author of source code)
>     Maildir++: Return nonexistent subscribed mailbox with nonexistent flag. e.g. LIST (SUBSCRIBED) "" *
Following is pointed.
> http://markmail.org/message/kvvcotcazneiwafo
>   details: http://hg.dovecot.org/dovecot-2.1/rev/f03e81b702a8
Change by f03e81b702a8 is;
  Add third "default_nonexistent" parameter to mailbox_list_subscriptions_fill().
  Pass default_nonexistent upon call of mailbox_list_subscriptions_fill().
  In mailbox_list_subscriptions_fill(), add following code,
     if (default_nonexistent) update_ctx.leaf_flags |= MAILBOX_NONEXISTENT;
  and recursively calls mailbox_list_subscriptions_fill() with default_nonexistent=FALSE
  for child mail boxes.
This seems a fix of problem we are experiencing.
(As usual in Mozilla family, it's pretty hard to know "what kind of bug was actually fixed by a patch"...)

David Epping(bug opener), do you see your problem with Dovecot 2.1 or newer?

Comment 12

[David Epping]

Hi,
I'm running Dovecot 1.2.9 (1:1.2.9-1ubuntu6.5 from Ubuntu Lucid).
It is very well possible this problem does not exist with Dovecot 2.1 and newer. This is a production company machine I can't just upgrade, I would have to install a new one. I'm currently short on time and don't know how long it will take to port the configuration. A client side experiment would certainly be quicker, but if this si not urgent I can schedule a new server installation for end of November.

Your conlusions sound like that is the problem. I don't see that I can set anything other than "subscriptions=no" in the server config if I want per-user subscriptions. Sending back nonexistent in that case can probably be seen as a server side bug that previously did not matter for TB.

Thanks for all your effort and let me know if you need logs, config files, or want me to test something (other than a new server setup).
Best regards, David

Comment 13

[WADA:World Anti-bad-Duping Agency]

Cause of "\NonExistent" seems "Mbox pointed by PublicNamespace doesn't exist, and is not subscribed".
If so, following may be a workaround of Dovecot's bug.
  PublicNamespace is perhaps "shared-rba." in your case.
  Create Mbox named "shared-rba", and add \NoSelect attribute.
  Subscribe Mbox named "shared-rba", if it's needed for workaround.
  Note: To subscribe using Tb, following is perhaps required.
        1. Without \NoSelect, access the Mbox with "Show only subscribed folders" disabled.
        2. Subscribe the Mbox using Tb.
        3. Terminate Tb, add \NoSelect attribute to the Mbox at server.
        4. Restart Tb with "Show only subscribed folders" enabled.
If you can guide users not to save mails in shared-rba, "Create Mbox named shared-rba without \NoSelect" can be checked with no Dovecot's configuration change.
If possible, please find simple/easy workaround with minimum setup change for opener of bug such as bug 918968, bug 926604, and for possible new dup bugs for same problem.

Anyway,
"version up of server software for production system" is very very tough work. Version up usually produces many unwanted problems due to version up, and applying hot fix for critical new problems is frequently required.
If LISTEXTENDED doesn't exist in CAPABILITY response, Tb 24 or newer doesn't use "LIST (subscribed)" and any Tb uses "LSUB *" command when "Show only subscribed folders" is checked. And, because Tb currently utilizes only "LIST (subscribed)" in LISTEXTENDED, "no LISTEXTENDED" won't produce any problem in Tb.
Please remove LISTEXTENDED from CAPABILITY response of current production system, if it won't produce problems in other mail clients.

Comment 14

[Mark]

Same problem here running dovecot 1.2.17. I logged the imap protocol and saw that initially, CAPABILITY is shown as 

IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE STARTTLS AUTH=PLAIN

but after successful logon it is shown as 

IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS

so I copied that long list and removed LIST-EXTENDED and pasted it into dovecot.conf like this:

# Override the IMAP CAPABILITY response.
imap_capability = IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN NAMESPACE UIDPLUS I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS

Now, the long version of CAPABILITY is shown initially and after logon, which I guess doesn't hurt anything. Also, the subscription problem is fixed. This was a fairly easy fix on a production server that should hold us until we get around to a full dovecot upgrade.

Thanks for the help,

Comment 15

[WADA:World Anti-bad-Duping Agency]

(In reply to Mark from comment #14)
> Now, the long version of CAPABILITY is shown initially and after logon, (snip)
No problem with "no STARTTLS in initial greeting or capability response before login"?

Comment 16

[Mark]

That's a good point. I've gone back and looked at the Tbird logs again and now I see that dovecot adds STARTTLS and AUTH=PLAIN to the end of the long string of capabilities in the initial greeting. They are not in the list of capabilities after successful login, which makes sense.

Comment 18

[WADA:World Anti-bad-Duping Agency]

(In reply to Mark from comment #14)
> CONDSTORE
Read bug 912216 and bug 885220, and remove CONDSTORE too if possible.

Comment 19

[Mark]

Thanks, I removed CONDSTORE too. I'm not sure yet, but I think that may have improved some other issues similar to those reported in 885220 we we seeing sometimes. I'll have to get some reports from my users.

Comment 20

[Wayne Mery (:wsmwk)]

This will impact some enterprise folk

Comment 21

[WADA:World Anti-bad-Duping Agency]

> Keywords: regression
This bug is not "regression". It's simply that Dovecot's old bug(known in 2010) was exposed by "LIST (subscribed)" support which is introduced by Tb 19. And, close code should be "INVALID", unless this bug will be morphed to "enhancement request for torelance with Dovecot's bug".

Comment 22

[gene smith]

I run my test dovecot and zimbra servers with shared folders that works fine in tb. (It appears this is closed as a duplicate of another bug and that bug is also closed as a duplicate of this bug. Maybe both should be closed as INVALID?)