592633 - subscription manager don't show some subfolders

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Leo Unglaub]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 ( .NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.8) Gecko/20100802 Thunderbird/3.1.2

I try to subscripte some folders from another email account. I open the Subscription Manager (i am not sure if it's the real english title, but i have only the german version installed) and see the other mailboxes. I expand the mailbox [e-mail] and see the subfolders like inbox,sent,junk,projects. The projects folder has also subfolders. But i can't expand them. First, i thought it was a imap server fault, but i checked my config and tryed with 2 other imap clients. (opera mail and windows live mail) in this clients i see all subfolders and can subscripe to them. But not in Thunderbird.

If i subscripe in a other tool to one of this sub-subfolders and start thunderbird i see the folder, but can't change this in the subscription manager.

Summary: Thunderbird has some troubles in the subscription manager with subfolders of folders. Sometimes there are not listet.

Reproducible: Always

Comment 1

[Ludovic Hirlimann [:Usul]]

Can you provide an imap log (https://wiki.mozilla.org/MailNews:Logging) ?

Comment 2

[Leo Unglaub]

Attachment: imap:5 log file

Comment 3

[Leo Unglaub]

(In reply to comment #1)
> Can you provide an imap log (https://wiki.mozilla.org/MailNews:Logging) ?

hi, sure i can. For thunderbird i do everythink ;)
In the attatchment, the interesting part starts at line 1261. You see that thunderbird crawl throw every mailbox under "mitarbeiter/office/projekte". Every single mailbox is listet there, but in the GUI i can't  expand the "projekte" folder. 

I hope that helps you!
Thank you very much.
greetings from Austria
Leo

Comment 4

[Leo Unglaub]

Attachment: GIU error

Comment 5

[Leo Unglaub]

i also addet a screen from the GUI so you can see the problem.

Comment 6

[Ludovic Hirlimann [:Usul]]

(In reply to comment #2)
> Created attachment 471784 [details]
> imap:5 log file

I'm not seeing any errors here - seems imap figures out the proper hierarchy - but another pair of eyes might help.

Did you change anything on disk to where your files would be stored (ie is there a path in projecte that would be longer than 256 char ?)

Comment 7

[Leo Unglaub]

Hi,
my profile is stored in the default Thunderbird profile folder. (C:\Dokumente und Einstellungen\unglaub\Anwendungsdaten\Thunderbird\Profiles\uzngc86s.default) 

On the imap folder is no folder name longer than 255 chars. Only 2 of them have a space chars in the folder name. (i know this is bad, but try telling this your customers ;) )

The IMAP is working file, because 2 other MUA's have no problem with this account. I think in this case it must be a thunderbird issue. The imapd is a dovecot imapd from the debian backports project.

mail:/var/mail/e-c-o.at/office/maildir# dpkg -l |grep dovecot
ii  dovecot-common                          1:1.2.13-1~bpo50+1         secure mail server that supports mbox and maildir mailboxes
ii  dovecot-imapd                           1:1.2.13-1~bpo50+1         secure IMAP server that supports mbox and maildir mailboxes

Greetings Leo

Comment 8

[Leo Unglaub]

I tested a little bit. With a new and complete empty profile it works. Thunderbird show all sub-sub folders and i can subscripte to them. then i closed the MUA and restartet it again. Suddenly i have the old problem again.

I hope that helps you!
Greetings
Leo

Comment 9

[Ludovic Hirlimann [:Usul]]

(In reply to comment #8)
> I tested a little bit. With a new and complete empty profile it works.
> Thunderbird show all sub-sub folders and i can subscripte to them. then i
> closed the MUA and restartet it again. Suddenly i have the old problem again.

I think having imap log on first run and second run would help us figure out why it works on first run and not on the subsequent ones.

Comment 10

[Leo Unglaub]

Attachment: with a new profile

so, here is the imap:4 log with a empty profile. I disables the message indexing in this case, otherwise the log grows to 220mb ;)

Comment 11

[David :Bienvenu]

Leo, if you toggle the parent of the folder that incorrectly doesn't have twistie (i.e., collapse and expand it), does the child folder get a twistie?

Comment 12

[Patrick]

Hi everyone,

I have a very similar problem.

After moving the imap maildir from an courier imapserver to an dovecot imapserver, Thunderbird and SeaMonkey don´t list sub-sub-folders.

3668[4359d00]: ReadNextLine [stream=53857a8 nb=69 needmore=0]
3668[4359d00]: 53a1800:10.5.29.11:S-INBOX:CreateNewLineFromSocket: * LIST (\HasChildren) "." "INBOX.Kauf.erledigt.++ Primergy_RX100S2"

Thunderbird only shows INBOX.Kauf.erledigt

Comment 13

[David :Bienvenu]

The other question is, how long is the path to your user profile directory? Windows in particular has a limit to how long a path you can have, so if your profile dir has a long path, it limits how long the path to your imap folders can be.

Comment 14

[Patrick]

Hi everyone,

I have a very similar problem.

After moving the imap maildir from an courier imapserver to an dovecot imapserver, Thunderbird and SeaMonkey don´t list sub-sub-folders.

3668[4359d00]: ReadNextLine [stream=53857a8 nb=69 needmore=0]
3668[4359d00]: 53a1800:10.5.29.11:S-INBOX:CreateNewLineFromSocket: * LIST (\HasChildren) "." "INBOX.Kauf.erledigt.++ Primergy_RX100S2"

Thunderbird only shows INBOX.Kauf.erledigt

Comment 15

[Patrick]

Sorry for the double post.

The path to the profile direcotry hasn´t changed.
I just changed the server to connect to.

D:\Thunderbird Mails\patrick\ccuvqtlr.slt\ImapMail\10.5.29-1.11
D:\Thunderbird Mails\patrick\ccuvqtlr.slt\ImapMail\10.5.29-1.11\INBOX.sbd\Kauf.sbd\erledigt.sbd\++ Primergy_RX100S2.sbd

Comment 16

[William Blunn]

I have seen this problem manifest first-hand.

IMAP server has folders:

INBOX
foo
foo.bar
foo.bar.baz

(separator is ".")

All folders were created on the server side (not through Thunderbird).

Thunderbird can see INBOX, foo, and foo/bar but not foo/bar/baz.

Even going into the Subscribe dialog and clicking "Refresh" does not bring up foo/bar/baz.

If, however, we create a third-level folder through Thunderbird, e.g.

foo/bar/qux

this is fine, and Thunderbird can access it normally.

There is more. The server maintains a list of subscribed folders. If we have a look at this we can see that there is no subscription to foo/bar/baz. If we alter the server-side subscribed folders list, then re-start Thunderbird, this folder magically appears in Thunderbird.

If we run a manual IMAP session to the server, and do, for example:

b LIST "" "*"

then we see the folder foo/bar/baz is listed.

I just tried unsubscribing from all folders, then going back into the subscribe window, and lo, I can no longer see foo/bar/baz again.

Comment 17

[William Blunn]

My test environment:

Thunderbird 3.0.6 over Windows Vista 32-bit
Dovecot 2.0.1 over Ubuntu 10.04 64-bit

FTOAD, when I said that the folder "foo/bar/baz" is listed under IMAP, what I meant was "foo.bar.baz". Here is a log of the whole list request:

b list "" "*"
* LIST (\HasChildren) "." "foo"
* LIST (\HasChildren) "." "foo.bar"
* LIST (\HasNoChildren) "." "foo.bar.baz"
* LIST (\HasNoChildren) "." "Trash"
* LIST (\HasNoChildren) "." "INBOX"
b OK List completed.

I can also configure the server to use a slash "/" as the hierachy separator but with the same results; Thunderbird still can't see any unsubscribed folder at the third level.

Comment 18

[William Blunn]

Sorry noticed another mistake in what I wrote before. When I wrote

"If we alter the server-side subscribed folders list, then re-start Thunderbird, this folder magically appears in Thunderbird."

what I meant to write was

"If we alter the server-side subscribed folders list so that it includes the folder foo/bar/baz (which appears as foo.bar.baz), then re-start Thunderbird, this folder magically appears in Thunderbird."

Comment 19

[William Blunn]

Had a sniff round in the source code.

Source code inspection shows a potential bug.

We have a method:

void nsImapProtocol::DiscoverAllAndSubscribedBoxes()

Which can be seen here:

http://mxr.mozilla.org/comm-1.9.1/source/mailnews/imap/src/nsImapProtocol.cpp#6877

At line 6877 we see that this method is "used for subscribe pane".

We see in lines 6962..6976 that we appear to be doing IMAP along the lines of

LSUB "" "*"
LIST "" "%"
LIST "" "%.%"

The "*" pattern on the LSUB (which picks up only *subscribed* folders) will mean we get all subscribed folders.

But the LIST command (which picks up *unsubscribed* folders ) is only used with patterns "%" and "%.%", so we will only get the top two levels of unsubscribed folders.

Unless I'm missing something, that seems pretty obviously broken.

Comment 20

[David :Bienvenu]

We List two levels deep when you expand a sub-folder, and reporters have said expanding the parent doesn't fix the bug, so I don't think it's that simple.

Comment 21

[William Blunn]

OK. I turned on some logging and went into the subscribe pane. Again I could not get it to show folder foo/bar/baz.

But I've got a bit of log here which says that the server is returning this:

* LIST (\HasNoChildren) "." "foo.bar.baz"

to Thunderbird, but Thunderbird doesn't seem to be incorporating that information into its world view.

Log context:

624[8357d00]: 8ac6800:<myIMAPServer>:A:ProcessCurrentURL: entering
624[8357d00]: 8ac6800:<myIMAPServer>:A:ProcessCurrentURL:imap://bill@<myIMAPServer>:993/discoverchildren%3E.foo:  = currentUrl
624[8357d00]: 8ac6800:<myIMAPServer>:A:SendData: 9 list "" "foo.%"

624[8357d00]: ReadNextLine [stream=93940c8 nb=37 needmore=0]
624[8357d00]: 8ac6800:<myIMAPServer>:A:CreateNewLineFromSocket: * LIST (\HasChildren) "." "foo.bar"

624[8357d00]: ReadNextLine [stream=93940c8 nb=22 needmore=0]
624[8357d00]: 8ac6800:<myIMAPServer>:A:CreateNewLineFromSocket: 9 OK List completed.

624[8357d00]: 8ac6800:<myIMAPServer>:A:SendData: 10 list "" "foo.%.%"

624[8357d00]: ReadNextLine [stream=93940c8 nb=43 needmore=0]
624[8357d00]: 8ac6800:<myIMAPServer>:A:CreateNewLineFromSocket: * LIST (\HasNoChildren) "." "foo.bar.baz"

624[8357d00]: ReadNextLine [stream=93940c8 nb=23 needmore=0]
624[8357d00]: 8ac6800:<myIMAPServer>:A:CreateNewLineFromSocket: 10 OK List completed.

2856[8358ac0]: ImapThreadMainLoop entering [this=9a88800]

Comment 22

[William Blunn]

Would it help if I set up an account for you on my IMAP server, and set it up with the relevant folders?

Then you should be able to see the problem happening for yourself.

Comment 23

[David :Bienvenu]

(In reply to comment #22)
> Would it help if I set up an account for you on my IMAP server, and set it up
> with the relevant folders?
> 
> Then you should be able to see the problem happening for yourself.

That would help tremendously, yes.

Comment 24

[Patrick]

(In reply to comment #20)
> We List two levels deep when you expand a sub-folder, and reporters have said
> expanding the parent doesn't fix the bug, so I don't think it's that simple.

Isn´t it possible to release a nightly build with this LIST command changed to LIST "" "*"?

If it is that simple, we could save a lot of research-time

Comment 25

[Leo Unglaub]

Hi,
i tryed a lot of examples this night. I found some interresting. If i have the ug that i can't see the subfolders, i opened the mailbox in a different mail client and disables all subscriptions. Than i restart Thunderbird again and i had only my imap folders there, but the subscription manager works file if i have no other subscription.

I know it's a little bit confusing...
Greetigs Leo

Comment 26

[William Blunn]

> Isn´t it possible to release a nightly build with this LIST command changed
> to LIST "" "*"?
> 
> If it is that simple, we could save a lot of research-time

The problem with that is that my hypothesis that Thunderbird is only LISTing two levels deep is compromised because (a) David's message implied that the two levels are not from the root but from a point of activity, and (b) I have seen a three-level-deep server response in the logs.

So I don't think it is the case that Thunderbird is issuing weak commands; I think it is more in the interpretation.

Comment 27

[Leo Unglaub]

>The problem with that is that my hypothesis that Thunderbird is only LISTing
>two levels deep is compromised because (a) David's message implied that the two
>levels are not from the root but from a point of activity, and (b) I have seen
>a three-level-deep server response in the logs.
i don't think so, because with an empty profil i see all sub folders down to more thant 6 Folders. But after some time it down't work anymore.

Comment 28

[William Blunn]

More test results:

If I un-tick the box "Show only subscribed folders" (Account settings / Server Settings / Advanced), then the third-level folder (foo/bar/baz) appears (after a bit of kicking/restarting/expanding-collapsing-trees etc), and I can see into the third-level deep folder.

If I then re-tick the box "Show only subscribed folders", then the third-level folder is still there. But after re-starting Thunderbird and frobnicating the folder view a bit, the third-level folder (foo/bar/baz) disappears again.

Comment 29

[William Blunn]

David, I have created an account for you on my IMAP server. Will send login details to you directly by e-mail.

You can verify the mailboxes present on the server using a manual IMAP session:

---8<---cut-here---
openssl s_client -connect <servername>:993
a login <username> <password>
b list "" "*"
---8<---cut-here---

This should show mailboxes "foo", "foo.bar", and "foo.bar.baz" with the separator as ".".

If you then create an IMAP account on Thunderbird 3, you should then try to drill down to the folder "foo.bar.baz", which I believe should prove to be impossible (unless you un-tick the box "Show only subscribed folders").

Comment 30

[William Blunn]

> David, I have created an account for you on my IMAP server. Will send
> login details to you directly by e-mail.

David, I have now sent that message to the e-mail address which appears under your name over your comments on this bug.

To help you find the message, here are two of the header records:

Date: Thu, 09 Sep 2010 10:00:14 +0100
Subject: bugzilla.mozilla.org Bug 592633 - Test IMAP server

Comment 31

[David :Bienvenu]

Thx, William. I will try to recreate this today.

Comment 32

[David :Bienvenu]

Thx, William, I can see the problem with the test account. I believe it's a bug in the xul template builder, which doesn't seem to want to add the grand child to the UI, even though we're sending the notification. The RDF XUL Template builder code is not longer really owned, afaik, so I may need to hack around the bug in the subscribe code.

Comment 33

[William Blunn]

> I can see the problem with the test account.

Marvellous. I see that Status has now gone up to NEW.

I will probably need to take away the test server now because I want to assign that machine for production use.

It feels like this bug should manifest with any IMAP server which maintains a server-side mailbox subscription list.

But if you find that you need to be able to run tests on something which is suitably similar to my test system, I can provide some notes about what the test system was.

The test system was running Ubuntu 10.04 64-bit.

The IMAP server software was Dovecot 2.0.1, built from the Debian sources accessed via:

/etc/apt/sources.list.d/dovecot-2.0.list:

deb-src http://xi.rename-it.nl/debian/ testing-auto/dovecot-2.0 main

(This could all be set up inside a VMWare Player virtual machine if desired.)

Dovecot was configured to use mail stored in Maildir format using the vanilla Maildir configuration:

In /etc/dovecot/conf.d/10-mail.conf:

mail_location = maildir:~/Maildir

Also I configured it to auto-create mailboxes, in /etc/dovecot/conf.d/15-lda.conf:

lda_mailbox_autocreate = yes

Then, to create the test folders:

As a test user create stub messages and pipe them into the delivery program.

( echo 'Subject: test' ; echo 'test' ) | /usr/lib/dovecot/dovecot-lda
... Ditto ... | /usr/lib/dovecot/dovecot-lda -m 'foo'
... Ditto ... | /usr/lib/dovecot/dovecot-lda -m 'foo.bar'
... Ditto ... | /usr/lib/dovecot/dovecot-lda -m 'foo.bar.baz'

This then creates folders

INBOX, "foo", "foo.bar", "foo.bar.baz"

each containing a single test message.

This was effectively where David Bienvenu came in and was able to replicate the problem.

Comment 34

[William Blunn]

Oops. Missed a bit out of the stub message creation. We should have a blank line between the head and the body.

( echo 'Subject: test' ; echo ; echo 'test' ) | /usr/lib/dovecot/dovecot-lda

Comment 35

[David :Bienvenu]

Neil, this seems to be a xul template/tree builder issue. The subscribe datasource basically rdf asserts foo/bar, and then a little bit later, foo/bar/baz. But, when nsXULTemplateBuilder::UpdateResult tries to add baz, it notes that foo/bar is collapsed, and decides it doesn't have to do anything. This, coupled with the fact that nsXULTreeBuilder::IsContainer() decides that it knows that foo/bar is not a container and never asks again basically breaks this part of the subscribe UI.

If I change this line to if (PR_TRUE), everything works:
    if (iter->mContainerType == nsTreeRows::eContainerType_Unknown) {

though obviously that causes us to ask if a row is a container quite a bit more.

The other part of the picture is that I think we get a paint request in between discovering foo/bar and foo/bar/baz, and that paint call is what ultimately causes us to cache the container state of foo/bar.

I'm at a bit of a loss as to how to fix this. I believe this breaks various pieces of the SeaMonkey UI that still use xul template builders, especially with RDF. Any advice would be appreciated.

Comment 36

[Neil Deakin]

(In reply to comment #35)

> If I change this line to if (PR_TRUE), everything works:
>     if (iter->mContainerType == nsTreeRows::eContainerType_Unknown) {
> 
> though obviously that causes us to ask if a row is a container quite a bit
> more.

I'd just remove that optimization. The content builder asks every time for instance. Or do you think there might be a performance hit here?

Comment 37

[David :Bienvenu]

(In reply to comment #36)

> I'd just remove that optimization. The content builder asks every time for
> instance. Or do you think there might be a performance hit here?

It's not a performance hit that worries me at all for our uses, but I don't know what the other uses might be. I'm happy to propose that as a patch.

Comment 38

[WADA:World Anti-bad-Duping Agency]

David :Bienvenu, what is difference of this bug from already reported bug 514106? Although phenomenon of bug 514106 was found just after unsubscibe, it happens if a subfolder is not subscribed. And, "collapse and expand of parent folder" or "new Tb window opened via Open of account's/folder's context menu" shows children many times, but it's not always.

This bug is for no chance of display of not-shown children in subscription list by such operational workaround?

Comment 39

[David :Bienvenu]

WADA, bug 514106 could be the same cause.

Comment 40

[David :Bienvenu]

Attachment: proposed fix

Comment 43

[David :Bienvenu]

Comment on attachment 474043 [details] [diff] [review]
proposed fix

This will fix several bugs in Thunderbird and SeaMonkey, and I don't think it actually affects Firefox directly.

Comment 44

[David :Bienvenu]

Oh, and I should say, I did try server builds with this and they all passed.

Comment 46

[Rob Mueller]

We've just seen a user reporting this problem at Fastmail and have been able to reproduce it.

I think this is a duplicate of a bunch of other bugs.

https://bugzilla.mozilla.org/show_bug.cgi?id=529618
https://bugzilla.mozilla.org/show_bug.cgi?id=537298
https://bugzilla.mozilla.org/show_bug.cgi?id=529244

I think the comment at the end of 529244 is relevant as well.

---
> * LSUB () "/" INBOX/aaa
> * LIST (\HasChildren) "/" INBOX/aaa

CHILDREN extension is supported, but \HasChildren is returned only to LIST
command, not to LSUB command.
---

I think that may be related to the problem, because we also the same thing:

---
6 list "" "INBOX.%.%"
...
* LIST (\HasChildren) "." INBOX.Folder1.SubFolder1
...
10 lsub "" "*"
...
* LSUB () "." INBOX.Folder1.SubFolder1
---

In the folder list, SubFolder1 doesn't display as having any sub-folders.

I'm guessing what's happening is that the LSUB empty () response is overriding the earlier LIST (\HasChildren) response.

This is the wrong thing to do.

---
https://tools.ietf.org/html/rfc3348

   The \HasChildren and \HasNoChildren attributes might not be returned
   in response to a LSUB response.  Many servers maintain a simple
   mailbox subscription list that is not updated when the underlying
   mailbox structure is changed.  A client MUST NOT assume that
   hierarchy information will be maintained in the subscription list.
---

Can someone that know that area check the TB code, and see that if the CHILDREN extension is available, the attribute results of LSUB are NOT being used, they should be ignored.

Comment 47

[David :Bienvenu]

Pinging for 2.0 approval - this xul bug mainly affects thunderbird, since I don't think the xul template builder is used as intensely in Firefox...

Comment 48

[Mike Beltzner [:beltzner, not reading bugmail]]

Comment on attachment 474043 [details] [diff] [review]
proposed fix

a=beltzner, assuming you get a clean test run, either locally or when running through the tryserver

Comment 49

[David :Bienvenu]

(In reply to comment #48)

> a=beltzner, assuming you get a clean test run, either locally or when running
> through the tryserver

thx, yes, I ran through try server before requesting approval per https://bugzilla.mozilla.org/show_bug.cgi?id=592633#c44

Comment 50

[David :Bienvenu]

fixed on trunk.

Comment 51

[LegNeato]

Comment on attachment 474043 [details] [diff] [review]
proposed fix

a=LegNeato for 1.9.2.13.

Comment 52

[David :Bienvenu]

fixed for 1.9.2.13 - http://hg.mozilla.org/releases/mozilla-1.9.2/rev/16afadc0d3ba

Comment 54

[Charles]

What are chances for this fix showing up in 3.1.7? This is a really bad bug for people hitting it (and of course my boss is one of them)...

Comment 55

[Mark Banner (:standard8)]

(In reply to comment #54)
> What are chances for this fix showing up in 3.1.7? This is a really bad bug for
> people hitting it (and of course my boss is one of them)...

This is bug fixed in 3.1.7 (which will use gecko 1.9.2.13).

Comment 56

[Aleix Dorca]

Is it possible that this bug is still present in version 15?

I've recently moved to Dovecot from Exchange. The users that are now using TB have this problem: folders not subscribed over the second level don't appear on the subscription list. 

If the folder is subscribed via RoundCubeMail or any other IMAP client then the folder also appears on TB and on the Subscription List.

Comment 57

[[e-mail]]

Similar to the above commenter, I this this issue once more in Thunderbird 38.1.0. Only folders and subfolders are visible in the subscription dialog, even if sub-subfolders are present on the server. If an alternate mail client is used to subscribe to those sub-subfolders, they properly appear in both the folder pane and the subscription dialog.

Bug 1089351 appears to be another report of this regression.

Is it possible to re-open this bug or move work over to the above-mentioned one?

Comment 58

[Wayne Mery (:wsmwk)]

This original problem was resolved, so this bug won't be reopened.
It is always best to file new bugs rather than try to re-raise issues in FIXED bug reports.

Comment 59

[Kent James (:rkent)]

Looking at many of the comments here, there is an unresolved issue that was reopened recently here:

Bug 1244779 - nsImapProtocol::DiscoverAllAndSubscribedBoxes() lists only 2 levels of IMAP folders - causing "Subscribe" to ignore sub-(sub)-folders of shared IMAP folders