138759 - Deeply nested IMAP folders aren't seen by Mozilla until parent gets folded and unfolded

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Aleksander Adamowski]

Reproducible on: Linux trunk build 2002041921. Not reproducible on same win32
trunk build.

After an upgrade of our IMAP server from Netscape Messaging Server {some older
version} to Netscape Messaging Server 4.15 Patch 7 (built Sep 12 2001) some
metadata on the IMAP server was lost (e.g. e-mail marked as read were all marked
as new again).

I happen to have a structure of deeply nested folders (4 levels) on that server.
Level 4 subfolders on an IMAP server aren't seen by MailNews immediately after
startup and only their parents are visible.
Level 0 being the root account folder, level 1 - Inbox, Sent etc...
See attached screenshot of my folder structure.

I've tried removing localstore.rdf, XUL.mfasl, also tried removing the IMAP
account, involved entries in prefs.js (the ones starting with
"mail.server.serverX") and then re-adding the IMAP account, but the same
subfolders still aren't seen by Mozilla after MailNews startup.

Starting with a new profile, adding the IMAP account, everything works fine.

I suspect that there's some file in the profile directory that caches IMAP
folder structure and it isn't properly refreshed, I'd appreciate hints about
which file would that be. If I find that file, I'll attach it to this bug.

I'm attaching a screenshot showing my full folder structure in IMAP and an image
illustrating the whole process.

Comment 1

[Aleksander Adamowski]

Attachment: My folder structure on IMAP server

Comment 2

[Aleksander Adamowski]

Attachment: Illustration of whole process

This file is 160 KB big!

Comment 3

[Aleksander Adamowski]

Comment on attachment 80176 [details]
My folder structure on IMAP server

Green arrows show the folders whose subfolders aren't initially visible

Comment 4

[Craig Ringer]

Confirm for mozilla 0.9.9-4 on Debian woody/i386, and also moz.org's rc1 on same.
Same as original description _except_ problem still ocurrs with a clean new
account. IMAP server is MDaemon 3 for Windows NT.

Comment 5

[[e-mail]]

Confirm for Mozilla 1.0R1, 1.0R2, 1.0R3, and 1.1alpha on Windows 2000. IMAP
server is Cyrus. Also confirm that problem still ocurrs with a clean new account.

Comment 6

[Kenneth Porter]

Confirmed for Moz 1.1 on Win2k, using UW-IMAP server on Linux with host folder
set to ~/mail. The mbox file ~/mail/Lists/RPM is visible (as Lists/RPM in the
Moz tree view), and the gray directory ~/mail/Lists/Mail is visible, but the
mbox files within the latter directory are not, and no "+" is visible next to
the Mail subdirectory to expand it.

Comment 7

[Kenneth Porter]

Just build Moz from CVS on Red Hat 7.3, and found that I could enter a mailbox
as a URL:

imap://[e-mail]/Lists/Mail/UW-IMAP

Entering this URL, I get a popup asking me if I want to subscribe to
~ken/UW-IMAP. So it looks like maybe the folders are getting collapsed to a
single level somewhere.

Comment 8

[Greg Colyer]

Also happens with
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.2.1) Gecko/20021130

One consequence is that mail filters attempting to move messages to a deep
folder fail on startup. Workaround is to switch off "Check for new mail at
startup", then collapse and reopen the folder hierarchy involved before clicking
Get Msgs.

Comment 9

[hjp]

I think bug #167107, bug #144103, bug #131938 and bug #121104 are all duplicates
of this bug.

Comment 10

[Aleksander Adamowski]

*** Bug 167107 has been marked as a duplicate of this bug. ***

Comment 11

[Aleksander Adamowski]

*** Bug 144103 has been marked as a duplicate of this bug. ***

Comment 12

[Aleksander Adamowski]

*** Bug 121104 has been marked as a duplicate of this bug. ***

Comment 13

[Aleksander Adamowski]

*** Bug 131938 has been marked as a duplicate of this bug. ***

Comment 14

[u85929]

My comments on bug 121104 still apply.

Comment 15

[Aleksander Adamowski]

It seems unrelated to a particular server - UW-IMAP, Cyrus-IMAP and Courier-IMAP
sightings were reported in the dupes.

Comment 16

[u85929]

Aleksander, I propose marking this as Major severity because of two reasons:

1. the loss of functionality in the Message Filters. If I reconnect to the IMAP
server then message filters that send messages to folders 3 or more levels deep
get (very helpfully) turned off. Then I have to go through the whole Message
Filter dialog to re-enable them (which annoyingly returns me to the top of the
list of the filters every time I re-enable one).

2. saved messages, when downloaded offline, are discarded by the Moz Mail
browser for the folders it's "forgotten about" (again, three or more levels
deep). For archive-style folders with hundreds or thousands of messages, this is
a BIG hassle; after "reminding" Moz Mail about the deep folders, the messages
have to be re-downloaded. And there's no workaround for this.

Please also change Hardware and OS to all, as were the case on the "duplicates"
of this bug. I can certainly confirm this on Windows/PC.

Comment 17

[Jay Smith]

What is described here (bug 138759) *does* appear to be same as bug 167107.  

In bug 167107 I reported that the problem was happening against two IMAP systems
and running on i686-based servers ... the Mozilla is also on a i686-based server.

   a) Red Hat 6.0 with kernel 2.2.5-15 and considerable updates (notably glibc
1.2.3-1). UW IMAP ver 4.5 release 3 from Red Hat supplied RPM imap-4.5-3

   b) Red Hat 8.0 with kernel 2.4.18-18.8.0 and all but the latest updates via
RH up2date. UW IMAP vers 2001a release 15 from Red Hat supplied RPMs (from RH
install CD). 

Using...
   Mozilla 1.2.1 (Xft build) from Mozilla RPMs
   Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2) Gecko/20021202

Only the RH8 machine is still in service now.  We continue to have the problem
and it continues to cause serious extra work for us.

Also, when we manually resubscribe (to pick up the mailboxes that coworkers
created) it is a tremendously tedious process to make all those mailboxes
visible in the subscribe dialog (over 1000 mailboxes nested to 8 levels) -- and
takes over an HOUR PER USER to do this. 

Furthermore after doing that subscribing, it seems that *all* the
mailboxes/messages are set to unread.

I sure hope that this will be taken care of soon -- it is a huge pain and a
tremendous productivity killer.

Jay

Comment 18

[Aleksander Adamowski]

Ok, I support this.

BTW, for me the bug is only exhibited if I have "Show only subscribed folders"
in account's advanced options disabled.

When I turn "Show only subscribed folders" on and subscribe to all those
subfolders (lots of work, that's a fact) I'm running fine from now on (until I
turn "Show only subscribed folders" off again).

Comment 19

[Aleksander Adamowski]

Raising severity to Major.

Comment 20

[Jay Smith]

Hi Alexsander,

You are correct that the "can't see deep levels" bug seems to only occur when
"show only subscribed folders" is DISABLED. 

However, as you point out, the [thus required to workaround the bug] subscribing
process is a huge pain. This pain could be solved by a "easy" step -- which I
think is in other numbered bug report: Add a "open all" button to the subscribe
dialog.  This feature was present in NS 4.76 but was not included in Mozilla; I
believe somebody said that it was left out to simplify the interface --
apparently that person does not a lot of mailboxes.

Additionally, I suggest that you try to change your subscriptions (i.e.
unsubscribe to a couple mailboxes).  Let it take effect and then restart
Mozilla.  Take a look at the read/unread status of all your folders -- on mine
all the "read" status is lost and all are unread.  Since I have several
individual mailboxes with as many as 20,000 messages (archived listserve
postings) in each of them, this is a big deal for Mozilla to rebuild its message
count, etc. when I go into one of these mailboxes.

Jay

Comment 21

[u85929]

I can't confirm that using "Show only subscribed folders" is a workaround. I
have enabled it, and subscribed to all my folders, and I still see the problem.

And I guess this is Asa's way of saying "good luck, this is NEVER going to get
fixed", since she also shot this down as blocking 1.3.

---

Asa Dotzler <[e-mail]> has denied [e-mail]'s request
for blocking1.4a:

Bug 138759: Deeply nested IMAP folders aren't seen by Mozilla until parent gets
folded and unfolded
http://bugzilla.mozilla.org/show_bug.cgi?id=138759

---

Comment 22

[Jay Smith]

To:  [e-mail]

I am willing to bet that while you *think* you have subscribed to all your
folders you probably have not because of a bug in the subscription
dialog/visibility process.  When in the subscription dialog, you have to close
level 1 and reopen level 1 in order to see level 3.  Then close level 2 and
reopen level 2 in order to see level 4.  And so forth.  In the subscription
dialog, drill down through your subscribed folders and check to see if the
deepest ones are really subscribed.  I am willing to be that they are not.  Once
subscribed, I think you will see them visible in the main mail window.

Question: Excuse my "rudeness", but does anybody know if Asa uses IMAP and deep
folder levels in her work?  I can't imagine that anybody who has to use this on
a daily basis -- whose employment productivity depends upon it -- would think
that this is not important.

Jay

Comment 23

[Wenzhuo Zhang]

I recommended Mozilla/Netscape7 to a couple of friends. This bug hurts the
credibility of my recommendations. ;)

Comment 24

[Asa Dotzler [:asa]]

wenzhou, please don't set flags if you don't know how.

Comment 25

[Wenzhuo Zhang]

I intended to vote. I apologize for the mistake and for the 3 unnecessary
messages you've received.

Comment 26

[Ole-Henrik Edland]

My Additional Comment #16 on bug 121104 still apply.

This bug is making Mozilla Mail a seriously restrained/hampered IMAP-client. 
Most people have moved or seem to be moving from POP to IMAP. With this in mind, 
I can not understand why this bug are not resolved.

Is there a time table for the solution of this bug? This one is nearly a year 
old. Duplicate bug 121104 is more than a year old. Instead of resolving it, 
somebody tried to bury it twice!

Comment 27

[llanero]

Get any other mail app but mailnews, ie. evolution, you can see *all* the
folders, subscribe to the folders that are not showing in mailnews. Get back to
mozilla and you will see the folders that were hidden.

BTW: bug 131938 is older than this, why the hell must I change my vote?

Comment 28

[Lars Uffmann]

This is mainly to add /me to the Cc: List for 138759.

138759 needs to be resolved urgently, since it is a real showstopper when
migrating from NS 4.X to Mozilla as a whole, including mail&news.

I was assinged internal to take care of this problem when our PC-Support decided
it must be a server (courier-imap) problem, which it is definitly _not_.

Is there any way to prioritize the bug, besides voting? It is unresolved for at
least one year.

Lars

Comment 29

[Calvin Liu]

Lars doubt if this bug is similar to 200989.

For those who wants to reproduce this bug, please:
1. unix/linux
export NSPR_LOG_FILE=<a file that you want to log>
export NSPR_LOG_MODULES=IMAP:5
2. win
set NSPR_LOG_FILE=<a file that you want to log>
set NSPR_LOG_MODULES=IMAP:5
Then run mozilla mail.

Please attach the log file to this bug.

Please refer to bug 200989. I gave a patch there.

Calvin

Comment 30

[Lars Uffmann]

> Lars doubt if this bug is similar to 200989.

Actually, I have no doubt the two bugs are similar.
A misunderstanding? Anyway, Debug output will follow.

Lars

Comment 31

[Lars Uffmann]

Attachment: Trace of a MailNews Folder Exploration

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1) Gecko/20021130

Comment 32

[Lars Uffmann]

Attachment: Trace of a MailNews Folder Exploration with 1.4a

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4a) Gecko/20030401

Comment 33

[Calvin Liu]

Lars,
Thanks for the log.
I guess they are duplicated. Then could you try the patch of 200989?
Calvin

Comment 34

[Lars Uffmann]

Calvin,

The logs are from two versions, 1.2.1 and 1.4a.

I assume trying the patch means to build Mozilla from CVS. I personally have
currently no ressources to do this. But I asked a colleage inside our software
development team to set up the necessary infrastructure. This may take time,
though. Are there binaries I could test?

Lars

Comment 35

[Calum Mackay]

Calvin's patch, in comment #1 of bug #200989, certainly fixes the problem for
me, which suggests that these should have a dependency, if not a duplication.

Comment 36

[Aleksander Adamowski]

Adding bug 200989 to deps.

Comment 37

[Dirk Heinrichs]

I'm also hit by this bug. Server is MS Exchange.
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/20030808

Comment 38

[Kalin Kozhuharov]

Still present on 1.5 
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031018
built from source on Gentoo linux.

Comment 39

[David :Bienvenu]

fixed in 1.7a

Comment 40

[Asa Dotzler [:asa]]

*** Bug 146804 has been marked as a duplicate of this bug. ***

Comment 41

[Eric Vaandering (no email)]

I still see this bug on mozilla suite 1.7.8 and on Thunderbird 1.0.2 

Comment 42

[David :Bienvenu]

unless you can recreate this with a trunk build, please don't re-open it.

Comment 43

[Jay Smith]

The problem with this bug is that it seems that nobody who is in the loop with
the latest versions, etc., uses *deeply* nested Imap folders and thus it doesn't
rise to a level of concern.

In fact it seems that rather few people at all understand the concept or need
for *deeply* nested Imap folders.  I have mail boxes for thousands of clients,
projects, in multiple (but small) companies and divisions.  Keeping all that
sorted out properly is what keeps us sane.

I use Mozilla 1.7.3 on RedHat 8 and still have the problem, but I know that is
an old program at this point.

=============

Editorial: People like myself who have only moderate skills would be a *lot*
better able to contribute to resolving these issues/bugs if we did not need a
"secret decoder ring" to understand how to do things like compile the most
recent version for Red Hat 8, or to compile with "talkbalk" (I know what it is,
but have not been able to figure out how I would get/install/use it).  Yes, yes,
I understand that if I RTFM *after* reading a few books and taking a few
classes, I would be up to speed and could do all this stuff.  As it is, anytime
I want an updated Mozilla to run on my aged Red Hat 8, I have to hire somebody
at $75/hour for at least two or three hours to compile it, etc., etc.  The the
last time I did this (for the 1.7.3 that we currently run on RH8 with KDE and
Xprint), we spent an addition 10 billable hours -- before giving up -- trying to
get things like email printing to work (i.e. get *basic* headers to print; we
never were able to get the From header to print; we never did get the Date to
appear in Forwarded emails).  

Hey, I am not trying to be rude.  I am incredibly pleased with how well the
program works and what I am able to do with it.  I gaze in wonder upon the great
minds that have created it.  I just wish that I too could help to demonstrate
issues such as this bug in the latest version, without going broke trying to do it.

Jay