58691 - Cyrus & Courier IMAP: Should reflect the correct folders on the folder pane

Status: VERIFIED FIXED

(MailNews Core :: Networking: IMAP, defect, P3)

Description

[Karen Huang]

Used 1030-09-MN6 build

Should reflect the correct folders on the folder pane

1)Create new Cyrus IMAP mail profile and setup personal Namespace "Inbox." in 
the server Advanced account settings.
2)Login to this Cyrus IMAP mail account
Actual Results:
- After first launch, the folders under the Inbox seems not displaying all the 
folders completely (please see the following second attached screen shot)
- After second launch, the fodlers seems displaying the folders, but there are 
INBOX. display in front of some folders (please see the following third attached 
screen shot)
- It seems sometimes after expand/collapse the Inbox/folders will display 
different folders view on the folder pane.

Expected results: Compare with 4.x (please see the following first attached 
screen shot) , it should reflect the correct folders view on the folder pane.

Comment 1

[Karen Huang]

Attachment: Folder view of the The first launch for netscape 6.0 Cyrus IMAP

Comment 2

[Karen Huang]

Attachment: Folders view of the Second launch for Netscape6.0 Cyrus IMAP

Comment 3

[Karen Huang]

Attachment: Correct Folder view of Communicator 4.7 for Cyrus IMAP

Comment 4

[Karen Huang]

Adding regression and RTM for the keywords since not displaying folders 
completely is "data loss"....

Comment 5

[Scott MacGregor]

can you attach an imap log? thanks!

Comment 6

[selmer (gone)]

Is mucking with the namespace normal for Cyrus IMAP?  Does this step make this
case less frequent among real users?

Comment 7

[Scott MacGregor]

load balancing...Jeff, do you have time to take a look at this? 

Comment 8

[Scott MacGregor]

forgot to actually re-assign it! =)

Comment 9

[Karen Huang]

Attachment: Attach the IMAP log as request (for the first launch result)

Comment 10

[Karen Huang]

It seems that this is only occurring on the migrated Cyrus IMAP mail profile.

Comment 11

[Karen Huang]

Attachment: attach another screen shot - sometimes, the 2nd level subfolders disappear on the folder pane

Comment 12

[jefft]

Something is fishy here. Any idea when this starts regress?

Comment 13

[Phil Peterson]

Any status on this? Does this happen on all profiles? On all platforms?

Comment 14

[jefft]

This happens on all platforms I am afraid so. With 11-02-00's build the folders
view seems display right however in one case the selecting a folder failed with
not prefixing the namespace. In another word the problem seems intermittent. 

Comment 15

[jefft]

Adding a new accout or deleting all *.msf file can cause this to happen easily.
Along wiht not setting flags correctly as stated in bug 58686, new folders added
through the folder discovery did not being added correctly to the rdf tree
(folder pane view). Shutdown and then relauch the app prior login in you can see
that there are multiple folders listed under the INBOX prefixed with INBOX.____.
After login in correct folder view shows. However, selecting those folders other
than INBOX gets "folder does not exist" error message from the server. Very bad.

Comment 16

[selmer (gone)]

As in 58686, could this be eliminated by disabling the personal namespace?

Comment 17

[David :Bienvenu]

Jeff's the expert on namespace stuff. I know next to nothing about it.

Comment 18

[jefft]

We seem not removing staled entries from panacea.dat. And for some reason we
have double entries in it (e.g., Drafts-1, Drafts ...) Staled entries casue the
flashing of the folder view before and after login. Staled entries also cause
the wrong local *.msf name created not matching the folder name. On the server
we have "Inbox, Inbox.Drafts, Inbox.Templates, Inbox.Sent, Inbox.Trash" folder
however we created Inbox.sbd/Drafts-1.msf. The database name saved in
Drafts-1.msf has the correct folder name "Inbox/Drafts".

// <!-- <mdb:mork:z v="1.4"/> -->
< <(a=c)> // (f=iso-8859-1)
  (8A=boxFlags)(8B=hierDelim)(8C=onlineName)
  (80=ns:msg:db:row:scope:folders:all)(81=ns:msg:db:table:kind:folders)
  (82=key)(83=flags)(84=totalMsgs)(85=totalUnreadMsgs)
  (86=pendingUnreadMsgs)(87=pendingMsgs)(88=expungedBytes)(89=charset)>

....
   =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com)(1C7F
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com\\INBOX.msf)
  (1C9B=83004)(2159=99f)(2046=2e)(1C99
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com\\Inbox.sbd\\Trash.msf)
  (1C9A=Inbox/Trash)(1C9D
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com\\INBOX.sbd\\Drafts.ms\
f)(1C9F=2004)(1CA0=INBOX.Drafts)(1CA1
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com\\INBOX.sbd\\Templates\
.msf)(1CA5=INBOX.Templates)(1CA3
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com\\INBOX.sbd\\Sent.msf)
  (1CA6=INBOX.Sent)(1CA7
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com\\INBOX.sbd\\Trash.msf)
  (1CA9
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com\\Inbox.sbd\\Drafts-1.\
msf)(1CAA=Inbox/Drafts)(1CAB
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com\\Inbox.sbd\\Sent-1.ms\
f)(1CAC=Inbox/Sent)(1CAD
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com\\Inbox.sbd\\Templates\
-1.msf)(1CAE=Inbox/Templates)(1CAF
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com\\Inbox.sbd\\Drafts.ms\
f)(1CB0=F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com\\Inbox.sbd\\Sent.m\
sf)(1CB1
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com\\Inbox.sbd\\Templates\
.msf)(1CF0
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com\\INBOX.sbd\\Drafts-1.\
msf)(1D9E=INBOX/Drafts)(1CF1
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com\\INBOX.sbd\\Sent-1.ms\
f)(1D9F=INBOX/Sent)(1CF2
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com\\INBOX.sbd\\Templates\
-1.msf)(1CF8
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com\\Inbox.sbd\\Drafts-2.\
msf)(1CF9
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com\\Inbox.sbd\\Sent-2.ms\
f)(1CFA
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com\\Inbox.sbd\\Templates\
-2.msf)(1CFB
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com\\INBOX.sbd\\Drafts-2.\
msf)(1CFC
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com\\INBOX.sbd\\Sent-2.ms\
f)(1CFD
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com\\INBOX.sbd\\Templates\
-2.msf)(1DA0=INBOX/Templates)(1DA1
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com)(1DA2
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com\\INBOX.msf)
  (215A=1cb0)(1DA5
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com\\Inbox.sbd\\Trash.m\
sf)(1DA6
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com\\Inbox.sbd\\Trash-1\
.msf)(1DA7
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com\\INBOX.sbd\\Templat\
es.msf)(1DA8
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com\\INBOX.sbd\\Sent.ms\
f)(1DA9
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com\\INBOX.sbd\\Trash-1\
.msf)(1DAA
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com\\INBOX.sbd\\Trash.m\
sf)(1DAB
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com\\INBOX.sbd\\Drafts.\
msf)(1DF0=F:\\JeffTsai\\work\\jefft\\Mail\\nsmail-2\\)(1DF1
    =F:\\JeffTsai\\work\\jefft\\Mail\\nsmail-2\\Drafts.msf)(1DF2
    =F:\\JeffTsai\\work\\jefft\\Mail\\nsmail-2\\Inbox.msf)(1EEC=10e)
  (1EEB=10d)(1EEE=5670)(1DF3=F:\\JeffTsai\\work\\jefft\\Mail\\nsmail-2\\Sent.m\
sf)(1DF4=F:\\JeffTsai\\work\\jefft\\Mail\\nsmail-2\\Templates.msf)(1DF5
    =F:\\JeffTsai\\work\\jefft\\Mail\\nsmail-2\\Trash.msf)(2047
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com\\Inbox.sbd\\Drafts.\
msf)(2048
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com\\Inbox.sbd\\Sent.ms\
f)(2049
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com\\Inbox.sbd\\Templat\
es.msf)(204A
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom.com\\INBOX.sbd\\Trash-1.m\
sf)(204B=INBOX/Trash-1)(204C
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com\\INBOX.sbd\\Drafts-\
1.msf)(204D=INBOX/Drafts-1)(204E
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com\\INBOX.sbd\\Sent-1.\
msf)(204F=INBOX/Sent-1)(2050
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com\\INBOX.sbd\\Templat\
es-1.msf)(2051=INBOX/Templates-1)(2072
    =F:\\JeffTsai\\work\\jefft\\Mail\\Local Folders\\foo.msf)(2151
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com\\Inbox.sbd\\Drafts-\
2.msf)(2152
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com\\Inbox.sbd\\Sent-2.\
msf)(2153
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com\\Inbox.sbd\\Templat\
es-2.msf)(2154
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com\\Inbox.sbd\\Trash-2\
.msf)(2155
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com\\INBOX.sbd\\Drafts-\
2.msf)(2156
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com\\INBOX.sbd\\Sent-2.\
msf)(2157
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com\\INBOX.sbd\\Templat\
es-2.msf)(2158
    =F:\\JeffTsai\\work\\jefft\\ImapMail\\buggy.mcom-1.com\\INBOX.sbd\\Trash-2\
.msf)>
{...

Comment 19

[David :Bienvenu]

Jeff, panacea.dat is not used to create nsImapMailFolders - it is used to avoid
opening up DB's. So there's no code that looks through the rows in panacea.dat
and creates a folder in the view for each of them. We only look into panacea.dat
when we have an nsImapMailFolder that corresponds to an entry into panacea.dat
(which can be because we had an .msf file on disk). Does this make sense?

Comment 20

[jefft]

You are right. The folder view is saved in localstore.rdf. There are some
oddness there.

Comment 21

[David :Bienvenu]

Jeff, can you send me info for a test account so I can try this myself?

What's probably going on is some conflict between the URI's we create for the
folders, and the online names we receive from the imap server (just a guess) and
store in panacea.dat and the .msf file . I don't think local store.rdf is used
to put folders in the folder view either.

Comment 22

[David :Bienvenu]

This has nothing to do with namespaces, I believe. It has to do with non-default
hierarchy delimiters (in this case, '.'). We don't handle them correctly. The
hierarchy delimiter is specified by the server, so we can't just remove
hierarchy delimiter support from the client :-)

I have it working better in my tree, but there's no way the changes would be
accepted for rtm.

Comment 23

[Michael La Guardia]

This doesn't strike me as pull-it-off-the-wire severe.

Comment 24

[David :Bienvenu]

I don't think we should pull it off the wire, but I should point out that this
means we're broken for any imap server thtat doesn't use '/' as a hierarchy
delimiter - this broadens considerably the number of servers that we won't work on.

Comment 25

[David :Bienvenu]

Attachment: first part of fix, return correct error status

Comment 26

[David :Bienvenu]

Attachment: second part of fix, store online name with correct hierarchy delimiter, related fixes

Comment 27

[jefft]

Reassign to David. He has a fix already. Thanks, David.

Comment 28

[Phil Peterson]

PDT marking [rtm-]. Bummer that this regression wasn't caught until now, but
this seems less serious than the bug 58686

Comment 29

[David :Bienvenu]

given the scope of the things that were wrong, I'm pretty sceptical that this
ever worked well in 6.0

Comment 30

[David :Bienvenu]

to answer Steve's question, yes, Cyrus users had to muck with the personal
namespace to get various things to work in 4.x. But this bug has nothing to do
with namespaces.

Comment 31

[Karen Huang]

I know that there were related bugs: bug 31431, bug 36772 logged for Cyrus first 
level folders disappear problem before and it has been solved and verified 
worksforme on nsbeta2...

Comment 32

[David :Bienvenu]

Fix checked in.

Comment 33

[Karen Huang]

Reopening this bug based on 11-15-10-Mtrunk build verification:
I got the following folder pane view screen shot from the cyrus mail account.
Shouldn't the folder pane view be the same as above 4.7 10/31/00 18:49 attached 
screen shot?

Comment 34

[Karen Huang]

Attachment: Cyrus folder pane view for current 11-15-10-Mtrunk build

Comment 35

[David :Bienvenu]

darn, this worked on my branch build before I checked the diffs onto the trunk.
But I had to delete my branch build to build the commercial tree, so now I don't
know what the fix is anymore.

Comment 36

[Karen Huang]

By using today's build, it's weird that I got different folder pane view (as
following screen shot) from the Linux platform, I don't know why it only
displays Cyrus Inbox on the Linux platform. I couldn't verify bug 58686 on Linux
because it didn't display Trash folder at all on the Linux platfom. Changing
platform to ALL.

Comment 37

[Karen Huang]

Attachment: Screen Shot for Cyrus folder pane on the Linux platform

Comment 38

[David :Bienvenu]

this was never a linux only bug. the platform should always have been all.

Comment 39

[cls]

Bumping severity to blocker.  This bug has caused me to revert to using comm 4.x
again as I cannot see any of my subfolders in mozilla.

Comment 40

[David :Bienvenu]

blocker means that QA can't test the functionality of IMAP, not that an
individual user can't use it on a particular server. Would you keep the tree
closed for all checkins? I don't think this bug is that severe. I'm working on it.

Comment 41

[David :Bienvenu]

Attachment: proposed fix

Comment 42

[David :Bienvenu]

cls, if you can, try applying the latest patch, and let me know if it works for you.

Comment 43

[David :Bienvenu]

adding keywords. Scott, can I get an sr=? If it works for Chris, I'll check it
in. It seems to work for me for Cyrus and Iris servers.

Comment 44

[cls]

bienvenu, I applied the patch and it works for me using courier-imapd. thanks.
wrt blockers, I was under the impression that any bug w/o a sufficient
workaround that would stop the user from using the product was considered a
blocker.  I wasn't aware that it was only limited to QA.

Comment 45

[Scott MacGregor]

sr=mscott

Comment 46

[David :Bienvenu]

fix checked in, thanks for trying it out, Chris.

Comment 47

[Karen Huang]

Great! This has been fixed by David based on the verification on all the 
platforms:
WinNT 11-20-08-Mtrunk
Linux 11-20-08-Mtrunk
Mac 11-20-08-Mtrunk
Marking as verified.