147528 - IMAP: Cannot access deeply-nested subfolders

Status: VERIFIED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Dennis Mezhevich]

We have a big subfolders tree on our enterprise MS Exchange 2000 Server. It's
look like as the following:

+--Inbox
|  ... ...
+--Public Folders
    +--NO WORK
    |   +----- Chat
    |   +----- Sport
    |   +----- Music
... +--Projects
        +-- E-Diagnostics
        +-- GEM
        +-- Spider
... etc

In the Mosilla mail I can see only folders just inside the "Public Folder" but
NOT their subfolders (Chat,Sport,Music,E-Diagnostics etc).

This problem does not occur if I create subfolders by myself (inside the "Inbox"
folder). But already created folders inside the "Public Folder" are not
accessible. These folders are not shown neither in the e-mail accounts pane nor
in the Subscribe dialog.
--------
Mozilla 1.0rc3 (BID: 2002052306), Windows NT 4.0 SP6a.

Comment 1

[Dennis Mezhevich]

The problem seems to be related to bugs #126647 and #111256

Comment 2

[Dennis Mezhevich]

And #89032  ? It seems this bug may be dublicate to 889032

Comment 3

[Henry Jia]

Can you give me an account to test? Thx.

CCing to myself.

Comment 4

[Dennis Mezhevich]

Attachment: Dump'ed traffic of the example IMAP account working

To demonstrate the bug I've sniffed all traffic between workstation with
Mozilla Mail and IMAP server. 
I performed the following steps:
1. Completly uninstal Mozilla, clean all directory with profiles and mail data
2. Install mozilla
3. Start sniffing
4. Run Mozilla Mail and create new IMAP account
5. Switch off the 'Show only subscribed folders' option
6. Open the account and browse folders
7. Exit mozilla completly (with Quick Launcher)
9. Stop sniffing

In the attached ZIP archive you can see two files - a binary traffic log (made
by tcpdump) and decripted text file with packet data in hex and ASCI.

Comment 5

[Dennis Mezhevich]

Sorry, but unfortunately I'm unable to give you a test account: I have no access
to externally-accessable IMAP server (our enterprise server can be accessed only
from inside the network). Instead of this I've created a test account on our
server and sniffed all traffic (see an attacment).
Sorry for redundant packets in the attachments, but I'm not shure which of the
packets may be dropped from the log without making the log useless.

If required I can perform any tests you can give me and send back another packet
log.

Comment 6

[Dennis Mezhevich]

I've found a way in which I can stable reproduce the problem.

1. I create folder named "Ôîëäåð ñ äëèííûì ðóññêèì èìåíåì" (In russian it means
"Folder with long russian name")
2. In the folder I create big subfolders tree (4-level deep)
3. For now all subfolders are accessible
4. Quit Mozilla completly (with switching off a Quick Launcher)
5. Start Mozilla Mail. For now I can see all my subfolders in unfolded tree, but
when I start browsing through these folders (clicking on them) they suddenly
disappears from folder tree pane.
6. Opening the "Subscribe" dialog window I can see all may folders, and all of
them are checked us "subscribed"
7. I switch off subscription for all my folders in the "Subscribe" dialog and
press "OK"
8. Then I cannot see my folders (excep of those direct inside the top-lebel
folder) neither in the folder-tree pane nor in the "Subscribe" dialog window tree.

I've repeated the test with different folders names (both russian and english)
the problem always persist as soon as the full folder "path" name grows enough.
With only english folders name you have to create MUCH MORE long folders names
or much more deep tree hierarchy. I think the difference present because of
russian folders name represents by long encoded strings.

It seems, that Mozilla have a limit for the overall (the folder name + all
folder's parents) folder name length.

Comment 7

[Dennis Mezhevich]

One more test.

In the MS Outloock I've created 41 nested folders, named from 000 to 040.
After this I've started Mozilla, run Mozilla mail and open "Subscribe" dialog.
In the subscribe dialog I've started browsing throught the folder.
When I've opened folder "030" I've seen, that Mozilla display the following:
...
 v 029
 |
 +---- v 030
       +------ > 031
       |
       +------ > 031
       |
       +------ > Calendar

You see - there are dublicated folders "031" and the wrong "Calendar" folder -
in fact the Calendar is placed on the top level next to "000" folder. 
After this the Mozilla hangs with 100% CPU loading. The only way to close
Mozilla in the state is using WinNT Task Manager, clicking "End Process" on
"mozilla.exe" process.


---
changing bug summary and severity.

Comment 8

[Bill Mason]

Bug 122845 may be related to or a dup of this bug.

Comment 9

[Dennis Mezhevich]

It seems, that there is a partial workaround of the bug in the Mozilla 1.1a
[BID:2002061104].

1. switch on "Show only subscribed folders" and exit mozilla.
2. use another mail reader (MS Outlook Express for an example) to subscribe to
the some folders, which are inaccessible in mozilla
3. Start mozilla and fold/unfold mail folders tree several times.

After doing this I can see/access some of my deep-nested folders. But some of
them are still inaccessible (but visible). If I click on some of such folders I
get a message: "The current command did not succeed. The mail server responded:"

In the subscribe dialog window I can see only those deep-nested folders, which
was subscribed in the another mail reader. Unsubscribed folders on the same
level are invisible.

Comment 10

[Todd Buiten]

I'm going to pile on here, although my problem is slightly different (but still
consistent with the title of this bug).  I'm running U-W 2000c, build 20020826
(release on XP), SSL, and I've turned off the "server supports folders that
contain sub-folders and messages" bit.  I've been unable to view just the
sub-subfolders and below on my machine.  They don't appear at all in the
subscription pane, even after a refresh.  I've had this problem before, and I've
taken care of it by collapsing and expanding the containing subfolder after a
refresh, but this strategy no longer appears to work.  If I turn off "Show only
subscribed folders" then I see the folders in the folder view, but still not in
the subscription pane.  I've check server privs to make sure that the server can
see folders and they're ok.

Comment 11

[Boris 'pi' Piwinger]

*** Bug 122845 has been marked as a duplicate of this bug. ***

Comment 12

[Kerin Millar]

*** Bug 172338 has been marked as a duplicate of this bug. ***

Comment 13

[Kerin Millar]

I installed the latest 1.2 alpha build in the hope that some progress had been
made on the issue. Having brought up the subscribe dialog box, it seemed that
the selector was able to display folders a few levels deeper in the heirarchy
than before. But my excitement was short lived because the process (in terms of
subscription and the proper displaying in the main window) still didn't work and
when I reconfigured from scratch I was unable to reproduce that behaviour which
was puzzling. I tried with and without the 'Show only subscribed folders'
setting, reconfiguring the account from scratch each time. So to all intents and
purposes, the issue seems to remain exactly as it is with the 1.1 build.

However, I experienced success using the approach recommended by Dennis
Mezhevich in comment #9; furthermore it was *not* necessary for me to
fold/unfold the folders. I used Outlook Express for the subscription process.
Mozilla Mail continues to operate correctly after exiting and re-loading so at
least I can actually use Mozilla as my mail client now.

I'm using Exchange 2000 SP2. I don't know if anyone is using SP3, but I am
planning to upgrade soon. Not that I expect that to make any difference.

If a developer wishes to have an IMAP account (I noticed that Henry Jia asked
for one) to play with then I will consider providing one, just contact me by
email if interested.

Comment 14

[Kerin Millar]

It occurred to me that Exchange Server supports protocol logging. The resultant
log files could be more digestable in terms of diagnosing any IMAP
implementation quirks than a TCP packet log. See:
http://support.microsoft.com/default.aspx?scid=/support/Exchange/Content/Whitepapers/intprot.asp
Perhaps I could go through a series of steps in Mail, then submit this data?

This probably isn't pertinent to the issue, but one thing I have discovered is
that Exchange has some kind of extension whereby Microsoft clients are informed
when a new message is available. For example Outlook Express will display a new
message immediately without it being necessary to poll the server. I don't know
the ins and outs of the RFC but I presume this is non-standard.

Comment 15

[Alessandro Oliveira]

When we were testing Mozilla and Cyrus, they were working very good with 10
users and about a hundred folders. But now I have about 50 users and 1500
folders and I expect an increase about 1500 folder per month.

Our folder hierarchy looks like the example below:
news
process
  |-A
  |-B
  |-C
  |-D
    |-DCustomer1
    |-DCustomer2
      |-2002-00123
      |-2002-00345
  |-E
  |-...
  |-Z

when I open the process folder, it shows folders from A..Z, and slowly puts the
arrows indicating they have sub-folders, then I start to open folders A..Z, and
it searches for sub-folders (with customer names), but when I am at folder C, it
stops to serach for sub-folders, like it has reached a kind of limit of open
sub-folders. 

In order to open the remaining folders I have to close the A..C folders, then
close the IMAP connection, reopen and start from C..Z, until the problem shows
up again.

I'm willing to create an account in my server to [e-mail] and
[e-mail] if you want to see whats happening here.




 I have about 1500 shared subfolders in my cyrus-imap server, 

Comment 16

[David :Bienvenu]

OK, the fix for this is to show newly discovered imap parent folders as
collapsed, not expanded. The backend code expects this, because it does issule a
list % and %.% when you expand sub-folders, which would be sufficient if we
showed newly discovered parent folders as collapsed, like we did in 4.x. It
would also be more convenient for users, I think, since they don't want every
folder expanded...

Comment 17

[David :Bienvenu]

Attachment: proposed fix

make new folders collapsed by default

Comment 18

[(not reading, please use [e-mail] instead)]

Comment on attachment 109360 [details] [diff] [review]
proposed fix

r/sr=sspitzer

Comment 19

[David :Bienvenu]

fix checked in. To verify this, you should start with a new profile directory.

Comment 20

[grylchan]

using commercial build
2003-01-06-08-trunk XP
2003-01-07-11-trunk Mac OS 10.1.2
2003-01-07-08-trunk Linux 2.2

Verified on the following servers:
 netscape msg server 6.0
 UW IMAP4rev1 v12.250
 CMU Cyrus Server 1.5.2
 MS Exchange Server 5.5

For a new profile only:
  -when you expand a parent folder, all children folders aren't expanded
   until you click on each children folder
  -children folders do have twisties next to them
  -when you collapse a parent and then expand it 'remembers'
   how you displayed the children folders (ie maybe I collapsed
   the children, maybe I didnt but it will remember the last state)
  -I tried nested folders 5 levels deep and no problems
  -when pref is checked 'show only subscribed folders'
    -that subscribed folders appear only
    -i can access/see nested subfolders with no problems
  -when pref is NOT checked 'show only subscribed folders'
    -that all folders appear (subscribe and unsubscribed)
    -i can access/see nested subfolders from subscribed/unsubscribed folders
     with no problems

marking as verified.

Comment 21

[grylchan]

*** Bug 162644 has been marked as a duplicate of this bug. ***

Comment 22

[grylchan]

*** Bug 178648 has been marked as a duplicate of this bug. ***

Comment 23

[Marc Branchaud]

Is there anything I can do to get the benefit of this fix _without_ wiping my
profile?

Comment 24

[Marc Branchaud]

OK, I tried using a new profile.  Sadly I'm still seeing nested folder problems.

I'm using last night's trunk build on Linux:  Mozilla/5.0 (X11; U; Linux i686;
en-US; rv:1.4b) Gecko/20030516

When mailnews starts up, before I enter my password, the folder display matches
the state I left it in when I last quit Mozilla.

After I enter my password, any nested folders get collapsed.  When I re-open
them, it's like Mozilla lost its .msf file for that folder: it needs to download
all the headers again and it's forgotten my sorting preferences.

Also, Mozilla still loses "twisties" on child folders.  I need to
collapse/expand the parent folder to get them back.  When I re-expand the parent
folder, Mozilla then displays collapsed/expanded children properly (although it
still doesn't have any .msf info, e.g. the total message counts aren't shown and
all headers are downloded if I view the folder).

"Show only subscribed folders" is _not_ checked.

I also had nested-folder problems in 1.3.  I think this bug is far from Fixed...

Comment 25

[Marc Branchaud]

This still does not work in 1.4.

Comment 26

[Marc Branchaud]

This is finally working for me in 1.7...