229522 - Folder starting with . (dot) disappears after restart

Status: VERIFIED FIXED

(MailNews Core :: Database, defect)

Description

[Alexandre Gomes]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6b) Gecko/20031208
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6b) Gecko/20031208

Creating a folder like ".5" works fine until you restart mozilla.

Reproducible: Always

Steps to Reproduce:
1. Create a folder named ".Test" for example
2. put some messages on it
3. restart mozilla

Actual Results:  
after restarting the folder is no longer listed

Expected Results:  
We have two choises... or mozilla doesn't allow to create folders with dots in
the name or you enable it.

It is very easy to solve the problem. We go to the mail files and we change the
name of the folder.
Doing this is easy to see that we have 2 files per mail-folder one of this files
have no extension and this is the problem. You can create an index file with the
list of the files that store the mail-folders or you put and extension on that
files.

Comment 1

[WADA:World Anti-bad-Duping Agency]

DUPE of Bug 117840
> a dot at the end of folder name causes mozilla to lose access to mail messages

See also Bug 124287
> Meta bug: Problems with folders having names with illegal characters

Comment 2

[David :Bienvenu]

confirming.  We do actually handle folder names with "illegal" characters in
them, but we don't consider '.' as an illegal character.

Comment 3

[David :Bienvenu]

this is not a dup of Bug 117840 - I believe this is specifically related to
folder names that start with '.'. We ignore them because on Unix, file names
starting with '.' are ignored. We could make this code #ifdef XP_UNIX, which I
really don't like, or we could force folder names beginning with '.' to be
hashed like other folder names with illegal characters.

Comment 4

[WADA:World Anti-bad-Duping Agency]

David Bienvenu, read my Bug 117840 Comment #10.
I already reported what happens on Win-9x in Bug 117840 more than one year ago.
My test result indicates logic for hidden file of Unix ignores file start with
"." as you say.

Comment 5

[David :Bienvenu]

bug 117840's summary says: "a dot at the end of folder name causes mozilla to
lose access to mail messages". You mention other things that don't work in that
bug, but those are separate issues with different causes.

My fix for this bug is to force hashing of folder names that we would ignore
during folder disovery (folders starting with '.' or '#', ending with '~', or
matching any of the special file names). That should fix this problem, but it
won't do anything about folder names that end with '.'.

Comment 6

[WADA:World Anti-bad-Duping Agency]

I did care only local folder file for POP3 on MS-Win when I reported starting
"." problem in bug 117840 one year ago.
So I misunderstood that all "." related problems will be resolved if "." is
escaped or treated as an illegal file name charater. 
Sorry for my reporting to inapproriate bug.
I should have report new bug because I was aware on Unix hidden file.

Anyway, I think this bug should be listed on meta bug 124287 although "." is not
a really illegal file name character, and summary of this bug should be changed
for ease of search - indication to "starting period and other specials" is required.
David, what do you think?

Comment 7

[WADA:World Anti-bad-Duping Agency]

Added this bug to dependency tree of bug 124287 for ease of access.

Could someone change summary? (I have no privilege).

Comment 8

[WADA:World Anti-bad-Duping Agency]

I think summary should contain "starting dot(or period)".
Reporter or David, change summary, please. 

Comment 9

[Jo Hermans]

*** Bug 256478 has been marked as a duplicate of this bug. ***

Comment 10

[WADA:World Anti-bad-Duping Agency]

Since Bug 219586 and Bug 264467 resolved "/" in folder name problems (I think 
also resolved "?" problem, bug 41944), if same logic for "/" is applied to
"." in folder name, I think this bug(also bug 117840, ending dot) can be easily
fixed, although real file name becomes converted string(HEXA-string).
Only problem is that file name for mid dot('.') becomes converted string, even
though same file name as folder name currently.

I believe this type of problem(remainig "#", starting '.', ending '.') is a mine.
If user striked a mine, the mine explodes, then user get damaged, then user
loves  Outlook Express more than before.

David, what do you think?

Comment 11

[WADA:World Anti-bad-Duping Agency]

*** Bug 273753 has been marked as a duplicate of this bug. ***

Comment 12

[Jeongkyu Kim]

Hello,

It seems that thunderbird can not handle folder name correctly when it starts
with '.' or ends with '.' and '~' (see bug 273753 for details). To solve this
problem, I am proposing two patches.

The first one (simple.patch) is simply to append '.' and '~' to
ILLEGAL_FOLDER_CHARS (in nsMsgUtils.cpp) so that hashed file name is used for
the folder name with those characters. This works fine for the problematic case
but might cause some confusion with legal folder name such as 'abc~xyz'.

To be precise, I propsoed the second one (normal.patch) which checks the first
and the last character of the folder name.  This needs more lines of code but
makes meaning clear.

Comment 13

[Jeongkyu Kim]

Attachment: simple way to check illegal cases

Comment 14

[Jeongkyu Kim]

Attachment: more precise way to check illegal cases

Comment 15

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #12)
Kim, thanks for patch.
Patch for Bug 262018 (attachment 166396 [details] [diff] [review]by Mscott) has introduced
ILLEGAL_FOLDER_CHARS ";#" and I believe this patch has fixed bug 94124(# in
local  folder name problem).
So I think your simple patch is better - only apply same logic for "#" to ".".
Mysterious hashed file name problem is to be resolved by another enhancement
because the problem is common on all real illegal and psedo-illegal file name
characters in folder name.
(There are some bugs complain on hashed(hexa-string) file name when imported.)
A solution for this can be changing hashig algorythm ;
 - Keep llegal file name character and their position as many as possiple  
   in hashing.
If folder name of "abc.xyz" is converted to file name of "abc<hashed-string>xyz"
instead of "<hashed-string>", many users can understand and accept it, I think.

Comment 16

[Jeongkyu Kim]

(In reply to comment #15)
> So I think your simple patch is better - only apply same logic for "#" to ".".
> Mysterious hashed file name problem is to be resolved by another enhancement
> because the problem is common on all real illegal and psedo-illegal file name
> characters in folder name.
> (There are some bugs complain on hashed(hexa-string) file name when imported.)
> A solution for this can be changing hashig algorythm ;
>  - Keep llegal file name character and their position as many as possiple  
>    in hashing.

WADA, I also agree with you. I belive that simple patch will be better solution
with proper comment and variable name ( I mean that the characters are not
illegal ones in general context). Thank you for your kind explanation. I just
stepped into the world of mozilla development and your help makes me very exciting. 

Comment 17

[WADA:World Anti-bad-Duping Agency]

Comment on attachment 168582 [details] [diff] [review]
simple way to check illegal cases

David, this is Kim's first patch.
Please review.

Comment 18

[David :Bienvenu]

Comment on attachment 168582 [details] [diff] [review]
simple way to check illegal cases

we need to allow '.' in the middle of folder names. Looks like second approach
deals with that...

Comment 19

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #18)
> (From update of attachment 168582 [details] [diff] [review] [edit])
> we need to allow '.' in the middle of folder names.
For IMAP?
(Sorry but I don't know about IMAP and I can't imagine needs when local file.)

Comment 20

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #14)
Comments/Questions on attachment 168583 [details] [diff] [review] (more precise way) :

(1) How about introducing new defines.
I think special characters should not be hard_coded in program logic.
Kim, how about introducing new defines?
> #define ILLEGAL_FOLDER_CHARS_AS_FIRST_LETTER "." 
> #define ILLEGAL_FOLDER_CHARS_AS_LAST_LETTER  ".~"
(In addition to above, ILLEGAL_FOLDER_NAME_STRING "%25" maybe will be required
in future. See Bug 93666. This is "Die Hard".)

(2) What will happen when both starting(and/or ending) dot/tilde and middle
dot(s) exist?
Because current hashing logic (probably) converts all file name string to
hexa-string, middle dot(s) in file name will be lost if hashing is executed when
first/last dot or last tilde exists.
I believe this will cause problems since David denied your simple patch,
although I don't have understood the exact reason why mid dot is required yet. 

Comment 21

[WADA:World Anti-bad-Duping Agency]

*** Bug 275034 has been marked as a duplicate of this bug. ***

Comment 22

[WADA:World Anti-bad-Duping Agency]

*** Bug 272617 has been marked as a duplicate of this bug. ***

Comment 23

[Jeongkyu Kim]

Attachment: more precise way to check illegal cases with defines

Comment 24

[Jeongkyu Kim]

> (1) How about introducing new defines.

I posted new patch with defines.

> (2) What will happen when both starting(and/or ending) dot/tilde and middle
> dot(s) exist?
> Because current hashing logic (probably) converts all file name string to
> hexa-string, middle dot(s) in file name will be lost if hashing is executed when
> first/last dot or last tilde exists.
> I believe this will cause problems since David denied your simple patch,
> although I don't have understood the exact reason why mid dot is required yet. 
> 

David,
May I have more information on the case which needs '.' in the middle of folder
name?

Comment 25

[WADA:World Anti-bad-Duping Agency]

David, same question.
Thunderbird 1.0(release build, on Japanese MS Win-2K) created file/directry set
of "d03075c9" , "d03075c9.msf" , "d03075c9.sbd" for folder name of "A/B.C#D".
Even though you say middle dot is required, current Thunderbird does not keep
middle dot of folder name of "A/B.C#D".
Why?

Comment 26

[WADA:World Anti-bad-Duping Agency]

As Jeongkyu Kim discovered in Bug 273753 Comment #1,
/mailnews/local/src/nsLocalMailFolder.cpp has logic to ignore starting ".",
starting "#" and ending "~".

In /mailnews/local/src/nsLocalMailFolder.cpp,
192 static PRBool
193 nsShouldIgnoreFile(nsString& name)
194 {
195   PRUnichar firstChar=name.CharAt(0);
196   if (firstChar == '.' || firstChar == '#' || name.CharAt(name.Length() - 1)
== '~')
197     return PR_TRUE;

David, is this your comment #3?
Why ignored? (I know meaning of starting "." only, hidden on Unix)

I think disappear problem won't occur when starting "."/ending "~", if above
logic is disabled.
  (Note: "#" problem is already resolved by using hashed name instead of "#".)
But I cannot say on bad affect of it, since I don't know why they are ignored.
David, can this logic be disabled?

Comment 27

[David :Bienvenu]

The reason I wanted to allow "." in the middle of names is that, as has been
pointed out elsewhere, it's better not to hash when we don't have to...I believe
having a "." in the middle of the name doesn't cause problems...I don't know why
having a trailing "." causes problems, to be honest...

Comment 28

[Jeongkyu Kim]

(In reply to comment #27)
> The reason I wanted to allow "." in the middle of names is that, as has been
> pointed out elsewhere, it's better not to hash when we don't have to...I believe
> having a "." in the middle of the name doesn't cause problems...I don't know why
> having a trailing "." causes problems, to be honest...

A trailing '.' causes a problem (at least) on Windows. For example, when I
created mailbox 'test.', thunderbird tried to create 'test.', 'test..msf' and
'test..sbd'. Even though the latter two were created correctly, I had 'test' for
first one becuase Windows regarded 'test.' as 'test'. When I open thunderbird
next time, it found that there is mailbox 'test' without 'test.msf' and
'test.sbd' so it created those two. So, I ended up with mailbox 'test' with
total five files. 

Comment 29

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #27)
> it's better not to hash when we don't have to...I believe
I believe hashed name is far more better than automatically varnishing folder :-)
How about to discuss on incovenience of hashed file name at Bug 275770?

> I don't know why having a trailing "." causes problems, to be honest...

MS Windows creates file without ending "." when file name with ending "." is
requested through Command Prompt.
Try rename xxx.txt yyyy. or rename zzz.txt yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy.
on MS DOS Prompt of MS Windows.
I do not know whether this is also applicable to requet through API or not.
But when I created folder name with ending dot, file name without dot was
created on MS Windows.
  
Because last "." is the delimiter of name and extention, not part of file name
nor part of extention, in 8-3 format.
So I can accept ignoring ending "." for 8-3 format file name, but I think this
shouldn't be applied to long file name.
However, MS Windows does even on long file name.
MS loves file extention very deeply. :-)

I don't know treatment of ending "." on Linux, Mac OS 9, Mac OS X(free BSD?), OS/2.
(I hope OS/2 uses ending dot as file name when long file name...)

Comment 30

[David :Bienvenu]

>I believe hashed name is far more better than automatically varnishing folder :-)
I don't believe that if you create a folder name with a "." in the middle, the
folder will vanish. Am I confused about that? That's all I was saying, and
that's why I prefer the second patch.

Comment 31

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #30)
> that's why I prefer the second patch.

I already undersand the reason for your denial on Kim's "simple way", hashed
name will be used even when normal <name>.<ext> format, and already agree on it.
But I misunderstood that your "middle dot is required" is "middle dot is
required even when starting dot is used".
Sorry for my misunderstood.

After Kim's "more precise way", hashing name will be used when and only when ;
 (A) Illegal file name character is used in folder name
 (B) Legal file name character but special character such as "/", "#"
     is used in folder name
 (C) Startting "." or endig "." or ending "~" is used in folder name.
(A) was already implemented long time ago.
(B) was already implemented recently by Mscott(with Jshin's follow up).
Middle dots are already lost when (A) and (B).
Middle dots will never be lost if all of (A),(B),(C) is not used.
Therefore, increase of user's complaints on hashed name due to additonal (C) can
be neglected.

David, could you please review Kim's revised patch of "more precise way".

Comment 32

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #23)
> Created an attachment 169344 [details] [diff] [review]
Jeongkyu Kim, set "review flag" of your revised patch, please.
David is the best person for "Requestee:", I believe.

Comment 33

[Jeongkyu Kim]

(In reply to comment #32)
> Jeongkyu Kim, set "review flag" of your revised patch, please.
> David is the best person for "Requestee:", I believe.
> 

WADA, I did set the flag. And I just realized that I have to request review
myself. Is this correct? If then, is there any rule to request reivew? I
proposed several patches recently, and have not gotten any comment on that. 

Comment 34

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #33)
> I just realized that I have to request review myself. Is this correct?
I think is is the best way because patch can be opened for trial or getting
comments from commenters or readers of bug.
So official review by developer is to be requested by patch creater.

> is there any rule to request reivew?
Sorry but I don't know.
However, I think rule of "setting David or Mscott as Requestee: when you want"
is sufficient if patch for "Mailer", since official mail related developers of
mozilla.org seem to be they and a few persons only :-)

Comment 35

[Jeongkyu Kim]

(In reply to comment #34)
> However, I think rule of "setting David or Mscott as Requestee: when you want"
> is sufficient if patch for "Mailer", since official mail related developers of
> mozilla.org seem to be they and a few persons only :-)
> 

Got it!
Thanks a lot for your information :-)

Comment 36

[WADA:World Anti-bad-Duping Agency]

*** Bug 278371 has been marked as a duplicate of this bug. ***

Comment 37

[WADA:World Anti-bad-Duping Agency]

Comment on attachment 169344 [details] [diff] [review]
more precise way to check illegal cases with defines

Changing requestee of review to Mscott who is owner of bug 274558.

I hope this patch will be  shiped with 1.5.
If this bug fixed, remaining folder name issue becomes "%25" problem only...

Comment 38

[WADA:World Anti-bad-Duping Agency]

'#define ILLEGAL_FOLDER_NAME_ONLY_WHEN_MS_WIN  "CON AUX ..."' is now needed if
MS Win, since Bug 311339 has been discovered...

Comment 39

[Scott MacGregor]

Comment on attachment 169344 [details] [diff] [review]
more precise way to check illegal cases with defines

If we were still interested in this patch, David would be the better reviewer so passing the buck. This may be obsolete or something we aren't interested in anymore. 

Seems like an edge case to me :)

Comment 40

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #39)
> or something we aren't interested in anymore. 
Do you mean "hiding starting dot file/directry" issue has been already resolved in other place?

> Seems like an edge case to me :)
"Starting dot/Ending dot/Ending semicolon/%25/CON,PRN,..." problems are edge case for mee too.
But this kind of naming issue can not be avoided by user, and if a user met these kind of mines, and if a user became a victim, user would usually never believe the data integurity of the mailer, I think.
I believe this kind of problem is usually proof of careless design/poor implementation of software.  

Comment 41

[Hiro]

As for this problem, Mac version Thunderbird reproduces. 

Mac OS X 10.3.9
Thunderbird 1.8branch nightly build 

Comment 42

[WADA:World Anti-bad-Duping Agency]

I saw a bug which says "Including/excluding of dot-started file/directry is optional in directry listing routine call." recently.
The bug also says "excluding" is default.
"Starting dot of mail folder name" issue is possibly avoided by requesting "include dot-started file" when calling "directry listing routine".  

Comment 43

[WADA:World Anti-bad-Duping Agency]

The bug in my comment #42 was Bug 314011.
> Choose Profile Folder does not display directories starting with dot eg .mozilla

Comment 44

[WADA:World Anti-bad-Duping Agency]

I've recalled my Bug 115091 comment #24 in last year...

Ignoring of starting "."/starting "#"/ending "~" was done by following logic.  
> http://lxr.mozilla.org/seamonkey/source/mailnews/local/src/nsLocalMailFolder.cpp
> 195 nsShouldIgnoreFile(nsString& name)
> 196 {
> 197   PRUnichar firstChar=name.CharAt(0);
> 198   if (firstChar == '.' || firstChar == '#' ||
>           name.CharAt(name.Length() - 1) == '~')
> 199     return PR_TRUE;
> (Ignoring logic for msgfilterrules.dat, popstate.dat etc. follows.)

Comment 45

[Stefan [:stefanh]]

*** Bug 321605 has been marked as a duplicate of this bug. ***

Comment 46

[Hiro]

(In reply to comment #41)
> As for this problem, Mac version Thunderbird reproduces. 
> 
> Mac OS X 10.3.9
> Thunderbird 1.8branch nightly build 
> 
In Mac, when Profile is seen with Finder, the ".DS_Store" file is made by the automatic operation. 
This file is a special file for OS. 
If this file exists in the Mail directory, this should not show it by the mail folder list. 
(The special file that I know is only this.  But other special files might exist. )

Comment 47

[Hiro]

(In reply to comment #46)
> In Mac, when Profile is seen with Finder, the ".DS_Store" file is made by the
> automatic operation. 
> This file is a special file for OS. 
> If this file exists in the Mail directory, this should not show it by the mail
> folder list. 
> (The special file that I know is only this.  But other special files might
> exist. )

file bug325883

Comment 48

[Tim Wunder]

*** Bug 356163 has been marked as a duplicate of this bug. ***

Comment 49

[Tim Wunder]

Please consider changing the O/S specified on this bug to All

Comment 50

[Magnus Melin [:mkmelin]]

Yes, I see this on linux too.

Comment 51

[Magnus Melin [:mkmelin]]

*** Bug 364744 has been marked as a duplicate of this bug. ***

Comment 53

[David :Bienvenu]

fixed on trunk, thx for the patch. I'll try to get this landed on the 2.0 branch.

Comment 54

[David :Bienvenu]

marking blocking tb 2? to keep on radar.

Comment 55

[David :Bienvenu]

fixed on 2.0 branch

Comment 56

[Carsten Book [:Tomcat]]

Verified fixed for 1.8.1.2 with Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.8.1.2pre) Gecko/20070124 Thunderbird/2.0b2pre Mnenhy/0.7.4.10002 ID:2007012504

Comment 57

[WADA:World Anti-bad-Duping Agency]

Regression on Linux(Bug 379101) has been reported.
It's found that Linux creates directory/file of "ending dot" normally. "Ending dot" case should have been limited to MS Win only. (I don't know Mac OS X case)

Comment 58

[WADA:World Anti-bad-Duping Agency]

(In addition to comment #44)
I've found description about why file of "ending tilde(~)" was excluded.
> http://linux.about.com/library/gnome/blgnome6n6r.htm
> 6.6.r. Hidden Files
>  * Hidden files, whose filename begins with a period (.),
>  * Backup files, whose filename ends with a tilde (~)