82821 - Unable to finish folder discovery on account; get multiple .msf files generated for one folder

Status: VERIFIED FIXED

(MailNews Core :: Database, defect, P1)

Description

[lchiang]

Unable to finish folder discovery on account; get multiple .msf files generated 
for one folder

Mac OS 9  2001-05-25-08 commercial trunk build.
Win32 does not have this problem.

1.  Create new profile
2.  Start the product
3.  Cancel out of activation dlg
4.  Browser appears
5.  Click Buddy List tab and set up for IM screenname
6.  Exit and restart.
(note: step 5,6 are necessary due to another bug on commercial build where this 
IM dlg will cause some other problems if not set up for IM).
7.  Browser appears.  Tasks | Mail
8.  Set up for my IMAP netscape.com account
9.  Log into IMAP
10.  The status bar will eventually read "looking for folders..."
11.  After a short while, the throbber stops moving and my system is locked up.
12.  Must force reboot.
13.  Now, when I look into the Finder into my IMAPMail folder, I see many (up to 
33!) .msf files created for one folder I have.  

Note:  This does not have to do with the folder name because I've renamed the 
folder and then I see this problem with another folder name.

In my conversation with David, he writes:
LisaChiang (6:12:56 PM): interesting note.  when I crashed, the next time I 
started on the Mac during "looking for folders" I see the following:
LisaChiang (6:13:08 PM): I have a folder called archive (it's at the top level).
LisaChiang (6:13:51 PM): I see folders:  Archive, Archive-1 all the way through 
a very high number (ie. Archive-30) or more.  What do you think would cause 
these other archive folders to show up?
dbienvenu (6:13:59 PM): oh.
LisaChiang (6:14:32 PM): It's very strange.
dbienvenu (6:14:38 PM): the -1, -2, etc is an attempt to create unique database 
names
LisaChiang (6:14:43 PM): I think I now have tons of those msf files.
LisaChiang (6:14:56 PM): why would we do that when I just have one archive 
folder.
dbienvenu (6:15:07 PM): so what happens is we discover a folder foo and we find 
a database called foo.msf, but it doesn't seem like the same folder to us, for 
some reason.
dbienvenu (6:15:12 PM): so we create foo-1.msf
LisaChiang (6:15:21 PM): I think that is where I am hanging on the mac.
dbienvenu (6:15:33 PM): so the problem is that we don't think the database is 
the right db for the folder.
dbienvenu (6:15:44 PM): Seth and Cavin made some changes in this area for 
migration.
LisaChiang (6:15:48 PM): this is on a new profile, first time logging in.
dbienvenu (6:15:53 PM): right.
LisaChiang (6:16:01 PM): I will file bug then for them to debug this.
dbienvenu (6:16:08 PM): it creates all those .msf files for just one run?

Note2:  I tried another IMAP account with folders in it and it works fine.  Able 
to get through folder discovery and loading folders.

If you need my account to debug this, pls let me know.

Comment 1

[lchiang]

cc: seth and cavin.

Comment 2

[(not reading, please use [e-mail] instead)]

get this on the 0.9.1 radar.  david, I've got a mac build here at home, I can
look into it.

Comment 3

[lchiang]

Seth - if you want to debug this further, give me a page and, unless I'm asleep,
I will call you with my mail password. Thanks.

Comment 4

[David :Bienvenu]

Seth graciously offered to look at this.

Comment 5

[(not reading, please use [e-mail] instead)]

updating status.

Comment 6

[lchiang]

Reproduced on more than one system.  Will work with Seth to reproduce this on
this Mac build.

Comment 7

[(not reading, please use [e-mail] instead)]

so far, here's what it looks like.

we aren't properly handling long file names on the mac.

lisa has some folders on her imap server that exceed the minimum.

there is code to hash long file names.  my guess (so far) is that we aren't 
using it in all places.  specifically, looking at my panacea.dat (where we 
store paths to the summary files) we aren't using the hashed names.

still investigating...

Comment 8

[(not reading, please use [e-mail] instead)]

getting somewhere, I think.

nsMsgDBFolder::CreatePlatformLeafNameForDisk() needs to do some magic hashing. 
(NS_MsgHashIfNecessary()).

that gets called on folder discovery [by 
nsImapMailFolder::CreateClientSubfolderInfo()]

I'll start with that problem and see how much further it gets me.  (there might 
be more places we need the magic hashing code)

Comment 9

[(not reading, please use [e-mail] instead)]

Attachment: rough cut of the fix.

Comment 10

[(not reading, please use [e-mail] instead)]

the following patch is the start of the fix.  it fixes the problem that lisa 
found on the mac.

(note, that patch also has a change to allow me to test the fix on win32 / 
linux, which I won't check in.)

three things need to be done:

1) I need to add back the code that will make the .msf unique if it already 
exists (I removed that)
2) move the "illegal char substitution" code into NS_MsgHashIfNecessary(), and 
if the file doesn't get hashed, I need to make the substitution because...
3) NS_MsgHashIfNecessary() is really NS_MakeFilenameSafeForOS().  so I'll 
rename it.

I'll work on these three things tomorrow, and have something to land tomorrow 
or thursday.

Comment 11

[Sheela Ravindran]

changing qa contact and assigning it to myself

Comment 12

[Sheela Ravindran]

commercial buildid: 2001052508 Mac-OS-9.0.4
I tried to recreate this problem on my mac machine by doing the following: 
-On my win98 machine I created the folders below in my IMAP account
-I created the two folders which apparently was causing this problem.  1st Level 
folder name:    ! Me - To Do and created a sub folder                   
                  ->nscp6_problems_to_investigate.  
-I copied few messages to both the folders 
I followed the steps described from 1 to 10 in the initial description of the 
bug and saw different results:
1.The application did not hang and created the folder structure and downloaded 
some 1500 messages in my mail box.
2.I checked my IMAPMail folders and did not see multiple .msf files created for 
one folder
3.I noticed that the subfolder 'nscp6_problems_to_investigate' was no where to 
be found in the 3 pane. Folder ! Me - TO DO did not have the twisty to indicate 
the subfolder at all.  
4.I closed the application and relaunched it again. I still could not see the 
subfolder
5. I checked ImapMail Folder on my hard drive and I see '! Me -To Do.sbd'. When 
I open the twisty I should see 'nscp6_problems_to_investigate.msf' file but I 
don't see it.  

Could this behavior be related to this problem?

Comment 13

[(not reading, please use [e-mail] instead)]

that sounds related.  when I test my final fix on the mac, we can test that 
scenario.

here comes a cleaned up fix...

Comment 14

[(not reading, please use [e-mail] instead)]

Attachment: patch

Comment 15

[David :Bienvenu]

yikes. Looks OK, except for the K&R braces style creeping in :-)

Comment 16

[(not reading, please use [e-mail] instead)]

some comments:

1)  I tried to keep this patch clean and minimal.  (trying to reduce the risk 
so close to 0.9.1 & beta).  [That's the reason I didn't rename 
NS_MsgHashIfNecessary()]

2)  the illegal chars / length code [which was not turned on] that was in 
nsMsgDBFolder.cpp is now in NS_MsgHashIfNecessary().  If the suggested name is 
too long, we do the same old hashing.  If (and this is new) it contains illegal 
characters, I simply hash the whole string.  (hash results are always safe).  
this should prevent collision and always make it so we hash to a unique file 
name.  the worry is that "foo_bar" and "foo:bar" will map to the same summary 
file ("foo_bar.msf").  but by hashing completely, we avoid this edge case

3)  CreatePlatformLeafNameFromDisk() is now CreateFileSpecForDB(), and we 
return the file spec.  I've moved all the "unique" magic in CreateFileSpecForDB
().  that, plus the hashing code, makes us very safe.

4)  the existing callers of NS_MsgHasIfNecessary() are safe.  OS/2 now hashes 
when the name is > 8 (they have 8.3 limits).  imap also uses this, but I'm was 
broken (and using it consistently will fix it.)  news is safe, since 
the "illegal characters" are not valid characters in newsgroup or host names.

5)  I've added comments in the code to explain other changes

Comment 17

[(not reading, please use [e-mail] instead)]

Attachment: same patch, better comments.

Comment 18

[(not reading, please use [e-mail] instead)]

Attachment: new patch, changes braces style (for bienvenu)

Comment 19

[David :Bienvenu]

I tried a bunch of tests on win32, new profile, existing profile, deleting
panacea.dat, all worked fine. I did see a problem on a new profile when
selecting the inbox failed the first time, because of Sera flakiness, but I
don't think that had to do with this patch.

Comment 20

[(not reading, please use [e-mail] instead)]

thanks for testing.

an extra note:  since I'm hashing (instead of simply replacing illegal chars 
with '_'), this can cause some .msf (for imap folders only) to get ignored the 
first time someone runs with a build with this patch.

according to bienvenu, losing (and rebuilding) the .msf file one time is 
acceptable.

Comment 21

[(not reading, please use [e-mail] instead)]

tested on the mac.  looking good.

can I get an official r/sr?

Comment 22

[David :Bienvenu]

r/sr=bienvenu, whichever you need.

Comment 23

[lchiang]

With your patch builds, does my IMAP account successfully get set up with all
folders discovered and loading?  If you want, perhaps we can try this on the Mac
tomorrow too on your debug builds (if we don't have release builds in time with
your patches in it).

Comment 24

[(not reading, please use [e-mail] instead)]

I'm at home, and my mac is outside the firewall so I can't test your problem.

but, I tested original "proof of concept" patch inside the firewall yesterday 
on your account and it fixed the problem.

outside the firewall, I've tested on my mac against an imap account I have with 
long folder names.   the problems I was seeing are gone.

(I also tested some edge cases on win32 by setting the max len to 5 and adding 
common characters to the illegal char list, to test out that code)

I hope to land this patch tonight.

Comment 25

[Scott MacGregor]

sr=mscott

Comment 26

[(not reading, please use [e-mail] instead)]

updating.

Comment 27

[Daniel (Leaf) Nunes]

a=leaf, we must protect the users from weak filesystems!

Comment 28

[(not reading, please use [e-mail] instead)]

fixed.

note, another side effect of this bug was you could end up with a file 
named ".msf" in the application directory.

lisa / sheela, can you test again with tomorrow?  I don't think you should have 
to do anything special (like remove any files or create a new profile.)

it should just work out of the box.  (famous last words...)

Comment 29

[lchiang]

Creating an .msf file at a higher directory has always been there on the Mac, I
think:  http://bugzilla.mozilla.org/show_bug.cgi?id=6402

Comment 30

[(not reading, please use [e-mail] instead)]

lisa:  not "<server>.msf", literally ".msf" in the same folder as "prefs.js"

Comment 31

[Sheela Ravindran]

buildid: 2001053108 
My problem seemed to have been fixed which I mentioned in comments 2001-05-30 
15:36. 
I checked for the folders that I had created previously on existing profile and 
also on a new profile. I was able to see the subfolder and messages in that 
folder. I also did not see any .msf created in the same directory as perf.js
In verifying this bug I created some more folders with long names and with 
different characters and they all seemed to work fine. Checked this on imap 
account. Still need to check this for pop and will update the bug after I do so. 
 
Also, I will verify this bug after checking with lisa on her mac machine. 

Comment 32

[Sheela Ravindran]

I am going to verify this bug since it fixed my problem as mentioned in my
comments below.  I will log a new bug for local folders since creation of a new
folder with long name created a .msf file in application directory and as well
does not create a folder at all. 

Seth and I created lisa's imap account on my mac machine and were able to
successfully discover all the folders on the imap server. However, this problem
still exists on lisa's machine which could be a different problem. I will log a
new bug for that specific problem too.  Hmmm....Also multiple .msf files getting
created for folders has also been resovled by this fix.

Comment 33

[(not reading, please use [e-mail] instead)]

I logged #84008 to cover the problem we saw on esther's and lisa's machine 
after this fix landed.

Comment 34

[Karen Huang]

Actually, this bug is hitting part of bug 20324 problems, so that's when I
didn't verify for bug 20324 yet. I am adding this bug in bug 20324 for future
reference.....