918968 - shared imaps folders only show synchronized contents but are readonly and not updated. solved by updating dovecot server

Status: RESOLVED INVALID

(MailNews Core :: Networking: IMAP, defect)

Description

[hirt]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/28.0.1500.71 Chrome/28.0.1500.71 Safari/537.36

Steps to reproduce:

Updated on Lubuntu 13.04 64 bit from 17 to 24


Actual results:

shared imap folders on our dovecot imap server are only readonly when trying to move a email into it. They are also not marked subscribed when going into subscribe dialog. The are also no longer updated such that might only the already synchronized local copy is shown. But no error message is shown that the folder is not accessible



Expected results:

Should have been no change to behaviour

Comment 1

[hirt]

Downgraded to Version 17.05 and the folders are working again as expected, but some collateral damage of extensions as enigmail and lightning

Comment 2

[WADA:World Anti-bad-Duping Agency]

(In reply to hirt from comment #0)
> They are also not marked subscribed when going into subscribe dialog.

Do you use Tb with "Show only subscribed folders" unchecked? Or with checked? (Server Settings/Advanced)
Do you see your problem with "Show only subscribed folders" checked(if unchecked) or unchecked(if checked)? (restart of Tb is required after setting change)

How about "Server supporsts folders that contain sub-folders and messages" setting?

> Actual results:
> shared imap folders on our dovecot imap server are only readonly when trying to move a email into it.

If \Noselect is returned for the "shared imap folders on your dovecot imap server", and if Tb 17 doesn't/can't detect \Noselect but Tb 24 detects \Noselect, it's pretty normal phenomenon, because Tb doesn't issue "select Mbox" when \Noselect according to request of "\Noselect".
See bug 317597, bug 534021, and bug 799821 for issues around "no required \Noinferior, \Noinferior from server".
Get IMAP log, and check IMAP log content by Text Editor. (see bug 402793 comment #28)
Is \Noselect, \Noinferior etc. returned as expected from your Dovecot server?

Comment 3

[hirt]

(In reply to WADA from comment #2)
> (In reply to hirt from comment #0)
> > They are also not marked subscribed when going into subscribe dialog.
> 
> Do you use Tb with "Show only subscribed folders" unchecked? Or with
> checked? (Server Settings/Advanced)
> Do you see your problem with "Show only subscribed folders" checked(if
> unchecked) or unchecked(if checked)? (restart of Tb is required after
> setting change)

We have the "Show only subscribed folders" checked. A test with unchecked setting can not be performed as the affected systems are downgraded in the meantime to V17.x

> 
> How about "Server supporsts folders that contain sub-folders and messages"
> setting?

It's on

> 
> > Actual results:
> > shared imap folders on our dovecot imap server are only readonly when trying to move a email into it.
> 
> If \Noselect is returned for the "shared imap folders on your dovecot imap
> server", and if Tb 17 doesn't/can't detect \Noselect but Tb 24 detects
> \Noselect, it's pretty normal phenomenon, because Tb doesn't issue "select
> Mbox" when \Noselect according to request of "\Noselect".
> See bug 317597, bug 534021, and bug 799821 for issues around "no required
> \Noinferior, \Noinferior from server".
> Get IMAP log, and check IMAP log content by Text Editor. (see bug 402793
> comment #28)
> Is \Noselect, \Noinferior etc. returned as expected from your Dovecot server?

The is no entry in the imap log. It seems that TB does not even send out the requests

Comment 4

[hirt]

Having one of my colleagues running into the bug we took the opportunity to test further:

If "Show only subscribed folders" is unchecked IMAP shared folder work.

This is even independent of the "Server supports folders that contain sub-folders
and messages" setting.

Comment 5

[hirt]

The error is not only on Linux but also on Windows both 32 and 64 Bit

Comment 6

[WADA:World Anti-bad-Duping Agency]

(In reply to hirt from comment #4)
> If "Show only subscribed folders" is unchecked IMAP shared folder work.
> This is even independent of the "Server supports folders that contain sub-folders and messages" setting.
This indicates;
1. Your Dovecot server has problem of bug 317597, bug 534021 in LSUB response.
2. Your Dovecot server correctly returns \Noselect, \Noinferiors to LIST command.
   (if not \Noselect, no \Noselect in LIST response as expected,)
   (and if \Noselect, \Noselect in LIST response as expected.   )
3. Tb 17 : bug 317597 and bug 534021 were worked around by patch of bug 799821,
           so "Show only subscrived folders" was usable.
4. Tb 24 : patch of bug 799821 was backed out, so bug 317597, bug 534021 started to occur
           again. So, "Show only subscrived folders" is not usable.
           Because your Dovecot server correctly returns \Noselect, \Noinferiors to LIST
           command, no problem occurs with "Show only subscrived folders" unchecked
           and no problem occurs regardless of "Server supports folders that contain
           sub-folders and messages" setting.

To confirm it, get IMAP log and check LIST response, SELECT response etc., please.

Comment 7

[Jeremy]

I confirm that this bug affects the Mac build as well. The workaround of unchecking "Show only subscribed folders" seems to work, but it is highly annoying when you have a gazillion folders in that namespace and only care about a few of them. I will be holding my users at v17 until this issue is resolved.

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to Jeremy from comment #7)
> The workaround of unchecking "Show only subscribed folders" seems to work,
> but it is highly annoying when you have a gazillion folders in that namespace and only care about a few of them.

Patch proposed in bug 799821 is a solution for your case, and the patch worked pretty well in your case, because your server fortunately don't have problem like bug 859269 and bug 897854(i.e. Server setup of your server is not garbled).
Watch bug 799821, please.

> I will be holding my users at v17 until this issue is resolved.

Another proper solution : 
  Dovecot server returns \Noselect, \Noinferiours in LSUB response,
  as majority of recent IMAP servers always do, and as Tb expects.
I believe "no \Noselect, no \Noinferiors, of LSUB response in Dovecot/UW-IMAP" is merely a result of default setting of Dovecot/UW-IMAP or sample setting of Dovecot/UW-IMAP package.

Comment 9

[WADA:World Anti-bad-Duping Agency]

Cause of problem may be opposite of bug 317597, bug 534021 in your case;
  For shared namespace folder,
    \Noselect exists in LSUB response.
    \Noselect doesn't exist in LIST response.
Tb 17:
  "Show only subscribed folders"==Checked :
    By patch of bug 799821, LIST "*" is issued before LSUB.
    No \Noselect in LIST response, then no \Noselect => Folder is writable.
  "Show only subscribed folders"==Unchecked :
    LIST "%/%" *" is used by Tb.
    No \Noselect in LIST response, then no \Noselect => Folder is writable.
Tb 24:
  "Show only subscribed folders"==Checked :
    Because patch of bug 799821 is backed out, LIST "*" is not issued before LSUB,
    as done before Tb 17.
    \Noselect in LSUB response, then \Noselect => Folder is read only.
  "Show only subscribed folders"==Unchecked :
    LIST "%/%" *" is used by Tb.
    No \Noselect in LIST response, then no \Noselect => Folder is writable.

If intention of Dovecot server administrator is "no \Noselect in LiST response", "\Noselect in LSUB response" is merely bad/wrong/broken server setups.

In contrast to it, "\Noselect in LIST response, but no \Noselect in LSUB response" case is due to fault in protocol named IMAP.
  Flag in LSUB response is not mandatory.
bug 317597, bug 534021 is for this case, and patch of bug 799821 tried to resolve problem in this case.
And, "LIST * before LSUB *" by patch of bug 799821 incidentally worked around "broken flag setting in LSUB response" case.

Comment 10

[hirt]

Attachment: Log of TB 17 with Dovecot working "show only subscribed folders"

(In reply to WADA from comment #6)

> 
> To confirm it, get IMAP log and check LIST response, SELECT response etc.,
> please.
Please check the attached IMAP log (shorted) from TB17 with dovecot for your questions.

Comment 12

[Martin]

(In reply to Jeremy from comment #7)
> The workaround of
> unchecking "Show only subscribed folders" seems to work, but it is highly
> annoying when you have a gazillion folders in that namespace and only care
> about a few of them.

similar scenario here for that workaround:
A shared folder where lots of new mails arrive all the time,
and most people don't want to subscribe to that folder.
So, it doesn't have to be gazillion folders to get a bit ugly.

Comment 13

[WADA:World Anti-bad-Duping Agency]

(In reply to hirt from comment #10)
> Created attachment 812473 [details]
> Log of TB 17 with Dovecot working "show only subscribed folders"

With Tb 17? If Tb 17, "LIST *" should be logged when "show only subscribed folders"=Enabled and "LSUB *" is used by Tb.

PublickNamespace(shared namespace) = "AoT/" and "AoT-Admin".
To "LSUB *", only following is returned from your server.
  INBOX, Junk Deleted, Templates
  AoT/EDV, 
  AoT/EDV/CRM,         AoT/EDV/CRM/Openerp, AoT/EDV/CRM/vtiger,
  AoT/EDV/General,
  AoT/EDV/Bad_E-Mails,
  AoT/EDV/Libreoffice,
  AoT/EDV/Wiki
  AoT/EDV/Website,     AoT/EDV/Website/Joomla1-5,
  AoT/EDV/Linux-client

Why　following can occur even though these Mboxes are not returned to "LSUB *"?
  select "AoT/Projects/Almatech/STIX-DEM/CPPA"
  select "AoT/CAD"
You set these Mboxes at Copies&Folders\Junk Settigs while you used Tb with "show only subscribed folders"=Disabled?
Why these Mboxes are "unsubscrbed status" even though you try to use Tb with "show only subscribed folders"=Enabled?

Which Mbox is folder of "folder is shared imaps folders only show synchronized contents but are readonly and not updated"?
All Mboxes in "LSUB *" response and above two Mboxes?

What is your "Server supporsts folders that contain sub-folders and messages" setting when the IMAP log was obtained?

Following is seen in your log. 
  - select "AoT^Projects^Almatech^STIX-DEM^Contract" (CONDSTORE)
    NO Mailbox doesn't exist: AoT^Projects^Almatech^STIX-DEM^Contract
  - STATUS "AoT^Projects"
    NO Mailbox doesn't exist: AoT^Projects
  - STATUS "AoT^Projects^STBL"
    NO Mailbox doesn't exist: AoT^Projects^STBL
This is known phenomenon. (see dependency tree for bug 124287). Phenomenon is "/ is escaped by ^".
Why does access to "AoT/Projects/Almatech/STIX-DEM/Contract", "AoT/Projects", "AoT/Projects/STBL" happen in your environment even though these are not returned in "LSUB *" response and "LIST *" is not issued by Tb?

Comment 14

[hirt]

Attachment: 2013-10-06 IMAP Settings

(In reply to WADA from comment #13)
> 
> With Tb 17? If Tb 17, "LIST *" should be logged when "show only subscribed
> folders"=Enabled and "LSUB *" is used by Tb.
Yes the log is with Tb17

> 
> PublickNamespace(shared namespace) = "AoT/" and "AoT-Admin".
> To "LSUB *", only following is returned from your server.
>   INBOX, Junk Deleted, Templates
>   AoT/EDV, 
>   AoT/EDV/CRM,         AoT/EDV/CRM/Openerp, AoT/EDV/CRM/vtiger,
>   AoT/EDV/General,
>   AoT/EDV/Bad_E-Mails,
>   AoT/EDV/Libreoffice,
>   AoT/EDV/Wiki
>   AoT/EDV/Website,     AoT/EDV/Website/Joomla1-5,
>   AoT/EDV/Linux-client
> 
> Why　following can occur even though these Mboxes are not returned to "LSUB
> *"?
>   select "AoT/Projects/Almatech/STIX-DEM/CPPA"
>   select "AoT/CAD"
> You set these Mboxes at Copies&Folders\Junk Settigs while you used Tb with
> "show only subscribed folders"=Disabled?
No enabled -> see screenshot

> Why these Mboxes are "unsubscrbed status" even though you try to use Tb with
> "show only subscribed folders"=Enabled?
TB17 does this also with Sent etc. special folders

> 
> Which Mbox is folder of "folder is shared imaps folders only show
> synchronized contents but are readonly and not updated"?
> All Mboxes in "LSUB *" response and above two Mboxes?
All shared folders (Aot, AoTAdmin) see attached screenshot


> 
> What is your "Server supporsts folders that contain sub-folders and
> messages" setting when the IMAP log was obtained?
On see screenshot

> 
> Following is seen in your log. 
>   - select "AoT^Projects^Almatech^STIX-DEM^Contract" (CONDSTORE)
>     NO Mailbox doesn't exist: AoT^Projects^Almatech^STIX-DEM^Contract
>   - STATUS "AoT^Projects"
>     NO Mailbox doesn't exist: AoT^Projects
>   - STATUS "AoT^Projects^STBL"
>     NO Mailbox doesn't exist: AoT^Projects^STBL
This is correct as only the subfolders exist. Does not influence this TB24 problem of 

> This is known phenomenon. (see dependency tree for bug 124287). Phenomenon
> is "/ is escaped by ^".
> Why does access to "AoT/Projects/Almatech/STIX-DEM/Contract",
> "AoT/Projects", "AoT/Projects/STBL" happen in your environment even though
> these are not returned in "LSUB *" response and "LIST *" is not issued by Tb?
See above

Comment 15

[WADA:World Anti-bad-Duping Agency]

(In reply to hirt from comment #14)
> Yes the log is with Tb17

The IMAP log is not entire log data from "start of Tb" to "terminate Tb"? (partial log data?)
Did you remove "LSUB *" responses for other Mboxes from attached log file?

> > Which Mbox is folder of "folder is shared imaps folders only show
> > synchronized contents but are readonly and not updated"?
> > All Mboxes in "LSUB *" response and above two Mboxes?
> All shared folders (Aot, AoTAdmin) see attached screenshot

Aha! Your "shared folders" mainly refers to "Mbox specified in Namespace" only.
i.e. Phenomenon is mainly that root level "Aot" is grayed out/shown in Italic at folder pane.
  
> Created attachment 813923 [details]
> 2013-10-06 IMAP Settings

As seen in your screen shot, "Aot/Incomming", which has subfolders, is also grayed out.
This is following for Tb, because "Show only subscribed folders" is checked.
- "Aot" doesn't exist (not returned to "LSUB *")
  "Aot/Something" exist (returned to "LSUB *")
  Because "Aot" doesn't exist(not returned to "LSUB *") but subfolder of "Aot" exists,
  Tb generates dummy "Aot" as place holder of subfolder of "Aot".
  Because "Aot" doesn't exist, "Aot" is same as "Mbox of \Noselect".
- "Aot/Incomming" doesn't exist (not returned to "LSUB *")
  "Aot/Incomming/Something" exist (returned to "LSUB *")
  Because "Aot/Incomming" doesn't exist(not returned to "LSUB *")
  but subfolder of "Aot/Incomming" exists,
  Tb generates dummy "Aot/Incomming" as place holder of subfolder of "Aot/Incomming".
  Because "Aot/Incomming" doesn't exists, "Aot/Incomming" is same as "Mbox of \Noselect".

Because you say "no problem even in Tb 24 if 'Show only subscribed folders' is disabled", your server perhaps returns "Aot" and "Aot/Incomming" to "LIST %", "LIST %/%", and "LIST *", and \Noselect perhaps doesn't exists in LIST response for "Aot" and "Aot/Incomming".

Because "Aot" and "Aot/Incomming" is not returned to "LSUB *" from your IMAP server, it's same as ""Aot and Aot/Incomming doesn't exist at server", when "Show only subscrbed folders" is enabled. ("Show only subscrbed folders" is designed for acessing subscribed folders only.)
If you want to access not-subscribed folder, you have to uncheck "Show only subscribed folder".
If you want to access currently not-subscribed folder with "Show only subscribed folder" checked, you have to suscribe the folder.

> All shared folders (Aot, AoTAdmin) see attached screenshot
AoTAdmin? Or AoT-Admin?
AoT-Admin nor AoTAdmin is not seen in your IMAP log and screen shot.
Is AoT-Admin or AoTAdmin shown at folder pane?

Is check box for "Aot" and "Aot/Incomming" shown at subscription list of Tb? (File/Subscribe)
If No in Tb 24 when "Show only subscribed folders" is enabled, can "Aot" and "Aot/Incomming" be subscribed by Tb 24 with "Show only subscribed folders" disabled?

FYI.
If "Aot" is subscribed, bug 517054 may occur(see bug 517054 comment #11).
"Aot is writable in Tb 17 in your case" may be following.
  Because "LIST *" is used in Tb 17 when "Show only subscribed folders" is checked,
  and because "Aot" is returned to LIST without \Noselect, "Aot" is writable.
  Because "Aot" is not returned to "LSUB *" in your case, bug 517054 doesn't occur.

Comment 16

[WADA:World Anti-bad-Duping Agency]

(In reply to hirt from comment #10)
> Log of TB 17 with Dovecot working "show only subscribed folders"
Your server supported LIST-EXTENDED.
> 1 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT SORT=DISPLAY
> THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN NAMESPACE UIDPLUS
> LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS] Logged in

As known by bug 922250, Tb 24 uses "LIST (subscribed)" instead of "LSUB *" when LIST-EXTENDED is returned in CAPABILITY response and when "Show only subscribed folders" is checked.
If your Dovecot server returns \Non-Existent to "LIST (subscribed)" for Mbox used as shared namespace(PublicNamespace), same problem as bug 922250 occurs in Tb 24.

Is your problem on "Aot" same as bug 922250?
How about your problem on "Aot/Incomming"?

Comment 17

[Wayne Mery (:wsmwk)]

Martin, hirt, can you have a look at comment 16 please, and offer your thoughts. Thanks.

Comment 18

[hirt]

Dear All,

We have solved the issue by updating dovecot and this have no possibility anymore to reproduce the error.

We must assume that comment 16 is correct. 

I propose to close this bug

Comment 19

[Martin]

dovecot here was version 1.2.15, updating to 2.2.13 solved it for us.

Comment 20

[Wayne Mery (:wsmwk)]

Martin, hirt,
Thanks for the new info!