529618 - When "show only subscribed folders" is disabled, Thunderbird does not show some child folders below the second level

Status: NEW

(Thunderbird :: Folder and Message Lists, defect)

Description

[Saul Tannenbaum]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5
Build Identifier: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.5) Gecko/20091112 Thunderbird/3.0

I use Thunderbird with "Show only subscribed folders" disabled. When I created a new profile to test Thunderbird 3 RC1 (build 1) and disabled subscriptions, some of my folders did not appear, specifically folders in the Archives/year/month hierarchy. While the year folder would appear, nothing below that appeared.

After verifying through a webmail interface that those folders did still exist, and some testing against another server that mirrors my message store, I determined that enabling "Show only subscribed folders" (these folders are all
marked as subscribed) made those folders and the messages in them visible.

If you enable "Show only subscribed folders", allow Thunderbird to populate its folder list, then disable "Show only subscribed folders", quit and rerun Thunderbird, the folders remain visible.


Reproducible: Always

Steps to Reproduce:
1. Disable "Show only subscribed folders"
2. Navigate into Archives/year/month hierarchy
3. 
Actual Results:  
Nothing below the Archives/year is displayed.

Expected Results:  
Folders and messages below the Archives/year/ should be displayed.

I cannot easily attach protocol logs because some of my folder names have peoples' names and would have to be sanitized. I've noted the following:

- Obviously in the difference cases, Thunderbird uses LIST in one and LSUB in the other.
- For LIST, Thunderbird does a LIST "" % and a LIST "" %/% but no lower in the hierarchy. While Thunderbird does traverse the hierarchy of folders and the
server returns "HASCHILDREN", Thunderbird shows no interest in descending further despite multiple clicks and folder expansion attempts.
- For LSUB, Thunderbird does a LSUB "" * which, because the folders are subscribed, returns the entire hierarchy.
- My speculation about why it works the third time is that the folders are already locally cached, and Thunderbird will generate the right IMAP even if it has no other reason to believe the folders exist.

This problem may  be limited to the Archives folders, but I haven't done enough testing to be sure.

I'm going to mark this as a MAJOR bug as it leads to apparent data loss. The folders/messages are, in fact, still there, but the appearance that your message
archives are gone is very disturbing. Since you can't get to your data without doing something counter-intuitive (restricting the folders that Thunderbird shows), this is at least Major, if not critical, IMHO. I'll leave it to the developers for more appropriate triage, either upwards or downwards.

Comment 1

[Philippp]

I have a very similar problem, on Windows XP, which persists since we started using Thunderbird here (TB 1.5.x I think):
Using TB to access IMAP folders on a local mail server (Kerio Mailserver 6.7.3), the subscribe dialog does not show all folders! Especially it misses folders in the second level hierarchy and below, but it is not 100% reproducible.

When I switch off the "show only subscribed folders", all folders get visible when I play around expanding/collasing/expanding the parent folders. Still, in the subscribe-to dialog (which for the user is obsolete in this case), some folders are missing! So obviously the subscribe dialog does not get the full hierarchy. Pressing "refresh" does not help.

It seems to happen more often if the folder names consist of only numbers, like 2009, or if the sub-sub folders also consist of only numbers.
(The folders do behave correctly in webmail, so I do not think that it is related to the server. I played around with the max. connections per server setting, but no difference.)

This is really stupid because it makes TBs subscription unusable to us - which would have been a major advantage over using the usual "office" software...

Comment 2

[Philippp]

Interesting finding:
This only happens when the folders have not been created by thunderbird.

I tried Outlook Express, and found something interesting:
In the subscribe dialog for IMAP folders, there are two tabs, called "all" and "visible". This "visible" property does _not_ reflect the subscription status of a folder! My problematic folders only show up in the "all" tab.
So obviously, there is some possiblity to make an imap folder "invisible" in aother way than by unsubscribing!? Once I subscribe to the "invisible" folders in Outlook Express, these folders also get visible in the thunderbird subscription dialog - even if I unsubscribe again!
So, Outlook Express seems to "unhide" these folders.

I cannot find any info on "hiding" folders in IMAP other than unsubscribing them.

Though, the problem does not occur with my provider, only with my local mailserver.

Saul, does the Outlook-Express trick also work for you?

Comment 3

[Ludovic Hirlimann [:Usul]]

(In reply to comment #2)
> Interesting finding:
> This only happens when the folders have not been created by thunderbird.
> 

Philipp, if you are using 3.0, now that you have found a way to reproduce, could you please take the time to reproduce your issue and create an imap log file as explained at https://wiki.mozilla.org/MailNews:Logging and attach the log file to this bug (you migth want to sanitize if before) ?

Comment 4

[Philippp]

Attachment: IMAP session: Subfolders missing in subscribe dialog

See the screenshot imap_folders_2010-01-18.gif for the real folder structure. The subfolders of the folder named "2" are missing in the subscribe dialog.

Comment 5

[Philippp]

Attachment: Screenshot from webmail, showing missing subfolders

Relates to the IMAP log.

Comment 6

[Philippp]

Hi Ludovic,

thanks for your answer - I have attached my log and a screenshot from webmail so that you can see the missing folders.
In fact, I had misread the title (I have "show only subscribed folders" enabled), but for me problem is independent from this setting, so perhaps we can leave this here until we get more insight.

Recording the log, I right-clicked on my IMAP folders and chose "subscribe";
in the subscribe dialog I clicked the "refresh" button;
then I subscribed to the folder named "2". All subfolders of this folder are missing in the folder tree of the subscribe dialog.
Then I openend the subscribe dialog a second time to see about any changes.


As this would be part of the exercise anyway (I guess), it would be very helpful to me if you can first tell me from the log whether it is a TB bug or whether my local IMAP server is causing the problem. (I can reproduce the issue on a second live server, but not on a test server.)

Comment 7

[Philippp]

Attachment: Screenshot: Subscribe dialog in thunderbird

Here is what the subscribe dialog is showing.

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
It may be already reported in other bug.
CHILDREN is supported but \HasChildren is not returned to LSUB. \HasChildren is returned to LIST only. (This is not RFC violation of IMAP server)
=> There is no way to subscribe subfolder by Tb.
   Tb should use \HasChildren/\HasNoChildren of LIST command.

(In reply to comment #4)
> IMAP session: Subfolders missing in subscribe dialog

capability response after sarttls:
> * CAPABILITY IMAP4 IMAP4rev1 IDLE ACL LITERAL+ UIDPLUS QUOTA ID SORT ANNOTATE
> ANNOTATEMORE STATUS-COUNTERS UNSELECT LISTEXT NAMESPACE XLIST AUTH=CRAM-MD5
> AUTH=PLAIN AUTH=LOGIN AUTH=DIGEST-MD5 AUTH=NTLM
CHILDREN is not supported, so \HasChildren/\HasNoChildren is not returned to LIST command. (needless to say, not for LSUB,XLIST too)
> RFC 3348 IMAP4 Child Mailbox Extension
> http://tools.ietf.org/html/rfc3348

Two problems.
(1) No \HasChildren in "LSUB *" response, , because no CHILDREN support.
    => No expand button at subscription list, unless already subscribed
       by other user or mail client.
       There is no way to subscribe third or deeper subfolder by Tb.
       Similar to comment #0 case.
(2) No \HasChildren in "LSUB" response, because no CHILDREN support.
    Only "LIST %" and "LIST %/%" is issued.
    => No expand button at second level folder?

Comment 9

[Philippp]

Big thanks to WADA!

(so if I understand correctly, this is a shortcoming of tb that only becomes visible because my imap server does not support the CHILDREN feature?)

Probably this similar to bug 529244 ?
https://bugzilla.mozilla.org/show_bug.cgi?id=529244

Comment 10

[WADA:World Anti-bad-Duping Agency]

Philippp, thanks for pointing bug 529244.
Bug I saw was bug 537298, and that bug is apparently dup of bug 529244.
Comment #0 is also dup of bug 529244(CHILDREN is supported).
However, your case(CHILDREN is not supported) is different, although similar problem to bug 529244 occurs.
In bug 529244 case(CHILDREN is supported), merge of \HasChildren/\HasNoChildren of LIST response to subscription list(LSUB * response) can be a solution.
But, LIST xxx/% before SELECT xxx(and/or LIST *) and merging LIST response to subscription list is required in your case(CHILDREN is not supported).

Comment 11

[Philippp]

Thanks WADA.
So probably the chances that my problem will get fixed soon are not so good, I guess.
Is it unusual that a server does not support CHILDREN?

Comment 12

[Philippp]

Attachment: IMAP Session where subscription works.

This log is from a session to a different server machine, but with the same IMAP server software. Here, Tb finds all subfolders.

Comment 13

[Philippp]

Sorry for geting on your nerves again - the strange thing is that on a test machine (same imap server!), the problem does not occur.

Can you please have a look at the log that I just posted (422355)?
As far as I could see, the server does not support CHILDREN either, but magically Tb seems to find all subfolders of the folder called "testnew".

Comment 14

[Stephen]

I think either the server support CHILDREN or not, tb still doesn't show 3rd level children.

I use server that has CHILDREN support. 3rd level doesn't show up. Unless you close TB, and open another client such as Mulberry (sorry to name this). Select the namespace, shared folder, then 3rd level children. Once you did this, the server will create all necessary shared folder for this level. Close Mulberry, then re-open TB. Now you can see it is there as subscribed.

Even for the level below the 3rd.

Comment 15

[Philippp]

That would be logical.
Still, when connecting to my test server (which runs the same server software), all subfolders (created by webmail) are visible in Tb - without the help of another client.
I dont understand this, thats why I recorded the second log. Maybe there is some other parameter that triggers or avoids the problem?

Comment 16

[Stephen]

Is it related to how TB finding the available shared folders which not yet subscribed ?

Could it be a separator problem ?

Comment 17

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #4)
> IMAP session: Subfolders missing in subscribe dialog (422171)

StartTLS, CRAM-MD5, XLIST in capability response, no CHILDREN,
XLIST command is issued by Tb.

(In reply to comment #12)
> Created an attachment  [details]
> IMAP Session where subscription works (422355)

authenticate plain, XLIST in capability response, no CHILDREN,
XLIST command is not issued by Tb.
Next log is seen.
> 28 list "" "testnew/0000/001/%"
> * LIST () "/" "testnew/0000/001/00"
> 28 OK LIST completed
> 29 list "" "testnew/0000/001/%/%"
> 29 OK LIST completed

Why XLIST is used in (422171) but is not used in (422355)?

Comment 18

[WADA:World Anti-bad-Duping Agency]

Philippp, "Show only subscribed folders" setting is same in both cases?
If yes, checked? Or not checked?

Comment 19

[Philippp]

I am 95% sure that the state of "Show only subscribed folders" is irrelevant for reproducing the problem. I will double-check this tomorrow (when I have access to the machine). Shall I post logs with the setting ON and OFF?

(My feeling is that the subscribe dialog does not make use of the folder hierarchy that is shown in the main window. When the problem occurs, subscribing a folder does not make any subfolders appear the next time the subscribe dialog is opened.)

Comment 20

[Philippp]

Attachment: IMAP session log, plus-signs appearing after collapse/expand of parent

What I did in the log:
Open TB, open "subscribe" dialog, expand folder PublicFolders/CallID
In the folder CallID, the subfolder test2 is visible, plus-sign missing.
Collapse/expand CallID -> plus-sign of test2 appears!
Click plus-sign of test2, subfolders 1000 and 2000 appear, plus-signs missing.
Collapse/expand test2 -> plus-signs of 1000 and 2000 appear.

(I did cut out a lot, hopefully this does not obstruct your analysis. I can post the full log if you need it.)

Comment 21

[Philippp]

Hi WADA,

I tested again (on Tb2 and Tb 3.0.1, seems identical).
I could not find any dependency on the "Show only subscribed folders" setting.

Though, I found something new in the subscribe dialog:
Originally (or after pressing "Refresh"), some subfolders are missing (as alreaday stated), and the corresponding parent folders do not show a plus-sign.
Now, if I collapse/expand the uppernext folder, the plus signs appear - sometimes instantly, sometimes it takes another collapse/expand cycle.
For the lower-next subfolders, the same happens.
Hitting the "refresh" button makes the subfolders and plus-signs of all the "flaky levels" disappear again.

The funny thing, after a refresh, if I collapse/expand again and the subfolders reappear, is that even the collapsed/expanded-state of any sub-subfolders is restored. Maybe Thunderbird is not using all the data it does have?

I also had a case where collapsing/expand the folder two levels above made the plus-signs appear.

Having subscribed to any subfolders previously seems to circumvent the problem.

(I found a message in the activities window saying something like "cannot access folder Public Folders/Public Folders" - perhaps there is some misunderstanding concerning my hierarchy?)

I posted a new log, https://bugzilla.mozilla.org/attachment.cgi?id=423147

Comment 22

[WADA:World Anti-bad-Duping Agency]

Philippp, your problem is sam as bug 514106.
  test case of bug 514106 : CHILDREN is supported, XLIST is supported.
               \HasChildren/\HasNoChildren is returned to LSUB/XLIST
  your case  : CHILDREN is not supported, so \HasChildren/\HasNoChildren
               is not returned to LIST, LSUB, and XLIST.
In both cases, LIST is issued correctly but expand button doesn't appear at subscription list, thus unable to subscribe child folders. Some tries of collapse/expand of higher folder displays the missing expand buttton.

Bug 529244, this bug(comment #0), Bug 537298 are for next case.
  CHILDREN is supported, XLIST is not supported.
  \HasChildren/\HasNoChildren is NOT returned to LSUB.
I dont't think the difference affects problem of bug 514106, because problem is in "subscription list refresh after correct LIST command". However, mixing of multiple cases in a bug makes problem anaysis hard(sometimes almost impossible). If you find new issues in your case, open separate bug, please.

Comment 23

[Philippp]

Thanks, WADA - youre obviously right, sorry for mixing this up.
(Finding the collapse/expand trick appeared as a workaround for my original problem, which did not show up clearly, once knowing the collapse/expand trick. So it might be that my original problem was the same - although, on the other hand, the bottom-most question of your comment 17 https://bugzilla.mozilla.org/show_bug.cgi?id=529618#c17 might remain? I dont know.)

Comment 24

[WADA:World Anti-bad-Duping Agency]

> Bug summar : When "show only subscribed folders" is disabled, (snip)

To Saul Tannenbaum(bug opener):
"show only subscribed folders" is checked? Or unchecked?

Comment 25

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #23)
> the bottom-most question of your comment 17
> https://bugzilla.mozilla.org/show_bug.cgi?id=529618#c17 might remain?

Yes. The question is one of reasons why I asked you about "Show only subscribed folders was checked or not" when you got IMAP log.

Comment 26

[Philippp]

Attachment: IMAP log with "show only subscribed folders" disabled

Comment 27

[Philippp]

Attachment: IMAP log with "show only subscribed folders" enabled

The option was checked for 422171, but I am sorry I do not remember for sure regarding the other logs. I recorded 2 new logs, one with "Show only subscribed
folders" enabled, one with that option disabled. I used different folders (3333 instead of 1111) in the second log, to sort out any influence of the local cache.

(Those 2 new logs are from my test machine, where I was not able to reproduce the original problem - so there might be some mysterious difference.)

Comment 28

[jnerin]

I'm using Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100216 Thunderbird/3.0.2

I have a server running Cyrus 2.3.14 and have just created with cyradm a folder like "Shared Folder/test/subfolder" (I have unixhierarchysep: on so the folder separator is "/"). I apply the following ACL:

lam "Shared Folder/test/subfolder"
anyone lrswipktecd

Note in this particular case the folder "Shared Folder/test" is not a mailbox and only contains a subfolder:
lm "Shared Folder/test"
Mailbox does not exist

But "Shared Folder/test/subfolder" is a full mailbox:

lm "Shared Folder/test/subfolder"
Shared Folder/test/subfolder (\HasNoChildren)  

I can see the mailbox Shared Folder/test/subfolder through webmail without problems, but I have had to use the expand/collapse trick in the comment #21 in order to be able to subscribe to it. I have not been previously subscribed to anything below Shared Folder/test as Shared Folder/test is a new folder.

I have "Show only subscribed folders" checked and I will post some excerpts of a 52Mb log I gathered in which I was not able to subscribe to the folder TB3 showed me the folder Shared Folder/test greyed (correctly as it's not possible to subscribe) and without the triangle to open it (incorrectly as it does have a child):

2010-03-01 18:52:13.722438 UTC - -1355809936[b0e7e690]: b0ed8800:mail2.leggio.es:NA:CreateNewLineFromSocket: 1 OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID LOGINDISABLED ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE CATENATE CONDSTORE SCAN IDLE LISTEXT LIST-SUBSCRIBED X-NETSCAPE URLAUTH] Success (tls protection)
[...]
2010-03-01 18:52:13.969772 UTC - -1384125584[b0e7f870]: ae008000:mail2.leggio.es:A:CreateNewLineFromSocket: * NAMESPACE (("INBOX/" "/")) (("user/" "/")) (("" "/"))
[...]
2010-03-01 18:53:15.250951 UTC - -1384125584[b0e7f870]: ae008000:mail2.leggio.es:A:CreateNewLineFromSocket: * LIST (\NonExistent \Noselect \HasChildren) "/" "Shared Folder/test"
[...]
2010-03-01 18:53:23.588933 UTC - -1384125584[b0e7f870]: ae008000:mail2.leggio.es:A:CreateNewLineFromSocket: * LIST (\HasNoChildren) "/" "Shared Folder/test/subfolder"

Comment 29

[WADA:World Anti-bad-Duping Agency]

Problem with "Show only subscribed folders" checked is already known issue and is reported to some bugs. It's being processed by bug 592633.

This bug is for problem with "Show only subscribed folders" disaled, because bug opener clearly states next in bug summary, 
> When "show only subscribed folders" is disabled, (snip)
and bug opener clearly states "problem only with Show only subscribed folders disaled" in original report of comment #0. 
 
Please don't add comment about phenomenon only with "Show only subscribed folders" checked(=enabled). Please don't hijack a bug by your "similar external symptom but with absolutely different condition"(different problem in many many cases). Please keep this bug for next only.
(a) problem only when "show only subscribed folders" is disabled(==Uncheced)   
(b) same problem both when "show only subscribed folders" is disabled   
    and when "show only subscribed folders" is enabled(==Checked)

To all comment posters on your problem:
You could reproduce original problem of this bug which is stated in comment #0 by bug opener?

Comment 30

[Bill McGonigle (not currently reading bugmail; please contact directly)]

Confirming.  When I switch this setting, the missing 3rd-level folders show up.  I'd rather not be dependent on subscription lists, so I was running with it off.

I needed to quit and re-launch for the change to work.  I tried first going offline then online, and I got:

"The operation failed because an other operation is using the folder. Please wait for that operation to finish and then try again."

When opening the subscription window, it just spun, never showed the folder list.  At that point, I quit and re-launched.

For others, the setting is located at:  "Tools -> Account Settings -> Offline & Disk space -> Offline -> Select folders for offline use"

Comment 31

[WADA:World Anti-bad-Duping Agency]

(In reply to Bill McGonigle from comment #30)
> When opening the subscription window, it just spun, never showed the folder
> list.  At that point, I quit and re-launched.

If offline mode, it's bug 534994.

Comment 32

[WADA:World Anti-bad-Duping Agency]

Going back to original problem of this bug.

(In reply to Saul Tannenbaum from comment #0)
> Gecko/20091112 Thunderbird/3.0
>(snip)
> Steps to Reproduce:
> 1. Disable "Show only subscribed folders"
> 2. Navigate into Archives/year/month hierarchy
> Actual Results:  
> Nothing below the Archives/year is displayed.
>(snip)
> - For LIST, Thunderbird does a LIST "" % and a LIST "" %/% but no lower in the hierarchy.

I could see it in Tb 8 with Yahoo! IMAP, but I couldn't see it in Tb 8 with Gmail IMAP. 
(1) At SeaMonkey 2.4.1, create folder hieararchy like X1, X1/X2, ... , X1/X2/X3/X4/X5/X6/X7/X8 under Yahoo! IMAP account and Gmail IMAP account.
(2) Gmail IMAP account, "Show only subscribed folders" is disabled, Tb 8.
    X1 is shown with expand button(+ button before folder name at folder pane).
    When + of X1 is clicked, X2 is shown with + button.
    Finally, when + of X7 is clicked, X8 is shown without + button.
(3) Yahoo! IMAP account. "Show only subscribed folders" is disabled, Tb 8.
    X1 is shown with + button. 
    When + of X1 is clicked, X2 is shown without + button.
    Subscrib shows all folder level and + button at any level, as expected.
    To see X3 folder under X2, one of next is required.
    a. Create search folder under X2 => + is shown, then X3 is shown by reexpand
       => X1\sbd\X2.sbd directory is created, and subfolder is created in it.
    b. Create IMAP   folder under X2 => + is shown, then X3 is shown by reexpand
       => X1\sbd\X2.sbd directory is created, and subfolder is created in it.
    c. Create X2.sbd directory manually, add "url=imap:// ... /X1/X2/dummy-vf"
       entry, and restart Tb => search folder of "vf" is created under X1/X2.
    After it, delete dummy search/IMAP folder.
    This is same for X3, X4, ..., X7.

(problem-1 seen in above test)
If SeaMonkey 2.4, expand button(+ button) is not automatically shown for X3 when X3 is created under X2. Collapse/re-expand of X2 is required to show + button of X3.
(problem-2 seen in above test)
Reason why no expand button until subfolder creation with Yahoo! IMAP seems that no folder list is returned from Yahoo! IMAP server to 'list "" "X1/%/%"' although folder list is norlly returned to "list X1/%".
After creation of subfolder under X2, Tb looks to issue "list X1/X2/%" and Yahoo! IMAP returns folder list as expected.

As seen in above, SeaMonkey has still folder expand button related problem, and Tb also had such problem. And as seen in Yahoo! IMAP case, servers may still have problem.

To Saul Tannenbaum(bug opener), original reporter of problem which occurs only with "Show only subscribed folders" disabled:

Do you still see your problem with newest Tb(currently Tb 8)?
If yes, can you attach IMAP log?

Comment 33

[WADA:World Anti-bad-Duping Agency]

Correction on observed phenomenon.
If checked with Tb's new profile, only first level of X1 is shown. No second level folder is shown, and no expand button is shown at root level folder.

Workaround was fortunatly found accidentally : Double click of IMAP folder.
When IMAP folder was double clicked at folder pane, expand button automatically appeared and subfolder was automatically shown when subfolder exists.