Open Bug 124287 (folders-with-special-characters) Opened 18 years ago Updated 3 months ago

[Meta] Problems with folders having names with illegal(or special) characters or special name

Categories

(MailNews Core :: Backend, defect)

defect
Not set

Tracking

(Not tracked)

People

(Reporter: sheelar, Unassigned)

References

(Depends on 43 open bugs)

Details

(Keywords: intl, meta)

Attachments

(1 file, 1 obsolete file)

This bug tracks both imap and local folders having illegal characters as the
name of the folder.
Tracking the bug numbers with this problem.  As per our discussion leaving
Esther as the owner and qa contact for this bug.  
Adding two more bugs which should be tracked as well. Thanks Cavin.  
Depends on: 117385, 117840
Depends on: 125868
No longer depends on: 125868
Depends on: 157768
Depends on: 140212
Depends on: 120559
Depends on: 131013
Depends on: 142464
Attached patch patch (obsolete) — Splinter Review
I collected the characters which are problematic from the depending bugs and
made a simple patch that just checks the names for New Folder and Rename Folder
and displays an alert if one of them is contained. I think this could be done
better by escaping the problematic chars, but I don't know enough about where
that is done and if I may change the methods without destroying other
functionality. Setting helpwanted.
I just noticed I'm not allowed to set helpwanted. Could someone please do that
for me?
Attached patch patch v2Splinter Review
Changed the patch because " " and "@" are only problematic if used alone for
folder names.
Attachment #126194 - Attachment is obsolete: true
The current status is this: 

The OK Button of New Folder and Rename Folder was disabled if the name was
empty. This has to be extended for the case the name starts or ends with a space
(which also handles filenames like " ", "  " etc.). Alternatively a message
could be displayed, e.g. "Folder names may not start or end with a space
character." 

For the other chars, I'm going to modify the FILE_ILLEGAL_CHARACTERS constants
in nsCRT.h but I can only test on my Windows PC. So I need some help from Linux
and Mac users to try some filenames containing the characters "#./<>?@\".

Testcase: 

Create a folder named e.g. "abc". Copy a message into it. Rename the folder.
Check if

- the folder is displayed with correct name and path in the Folder pane
- the message shows up in the Thread pane
- the message, when selected, is displayed in the Message Pane

Then enter into the table (below) if everything works or if problems occur.
Before the next test, delete the new folder and be sure to view a different
message in the Message Pane.

char | begin | mid | end
==========================
#    |       |     |  
.    |       |     | 
/    |       |     | 
<    |       |     | 
>    |       |     | 
?    |       |     | 
@    |       |     | 
\    |       |     | 
Depends on: 232001
Wouldn't it be possible to have folder name be stored as a string and display
that string name instead?  Just a thought here...
Depends on: 239502
Depends on: 230961
Depends on: 93666
Problem caused by special characters in folder name is one of the most
puzzleded, embarrassed and strange problems for Mozilla Mail users.
Since problems live long time, not so small number of people ask about these
problems at forums such as Mozillazine.
So I think this bug should be listed in "Known Issues" section of "Release
Notes" as Bug 2654 is listed.
I believe listing-up will also reduce many DUPEs.
all this bugs seen more or less related to each others, if they arent really
dupes...

there are 2 main bugs:

-unable to use of bad folder names
-allow the creating for bad names

fix the 1º, the 2º would be little importance
fix the 2º, the 1º one still needs to be fixed or workaround

esther email is invalid (esther@formerly-netscape.com.tld) and its the owner of
many of this bugs

this are important bug as advocating people to move from OE to mozilla and
hitting this bug is very bad and might cost several users ("if it can open/use
some simple folders, its still full of bugs and so useless")

adding the radar for 1.8a
Flags: blocking1.8a?
This is more of a bug tracking purpose.  This shouldn't block the Mozilla 1.8a
release.
(In reply to comment #6)
> The current status is this: 
> 
> The OK Button of New Folder and Rename Folder was disabled if the name was
> empty. This has to be extended for the case the name starts or ends with a space
> (which also handles filenames like " ", "  " etc.). Alternatively a message
> could be displayed, e.g. "Folder names may not start or end with a space
> character." 
> 
> For the other chars, I'm going to modify the FILE_ILLEGAL_CHARACTERS constants
> in nsCRT.h but I can only test on my Windows PC. So I need some help from Linux
> and Mac users to try some filenames containing the characters "#./<>?@\".
> 
> Testcase: 
> 
> Create a folder named e.g. "abc". Copy a message into it. Rename the folder.
> Check if
> 
> - the folder is displayed with correct name and path in the 

(In reply to comment #6)
> The current status is this: 
> 
> The OK Button of New Folder and Rename Folder was disabled if the name was
> empty. This has to be extended for the case the name starts or ends with a space
> (which also handles filenames like " ", "  " etc.). Alternatively a message
> could be displayed, e.g. "Folder names may not start or end with a space
> character." 
> 
> For the other chars, I'm going to modify the FILE_ILLEGAL_CHARACTERS constants
> in nsCRT.h but I can only test on my Windows PC. So I need some help from Linux
> and Mac users to try some filenames containing the characters "#./<>?@\".
> 
> Testcase: 
> 
> Create a folder named e.g. "abc". Copy a message into it. Rename the folder.
> Check if
> 
> - the folder is displayed with correct name and path in the Folder pane
> - the message shows up in the Thread pane
> - the message, when selected, is displayed in the Message Pane
> 
> Then enter into the table (below) if everything works or if problems occur.
> Before the next test, delete the new folder and be sure to view a different
> message in the Message Pane.
> 
> char | begin | mid | end
> ==========================
> #    |       |     |  
> .    |       |     | 
> /    |       |     | 
> <    |       |     | 
> >    |       |     | 
> ?    |       |     | 
> @    |       |     | 
> \    |       |     | 
> 



Mozilla/5.0 (X11; U; Linux i686; fr-FR; rv:1.6) Gecko/20040122 Debian/1.6-1
+enigmail
+adblock

Y: working : this could stay as it is.
N: NOT working

(1): '.' on *NIX this is the meta for selecting current working directory... so
if it's the only char. we get a dialog saying it already exists... it works...

(2): '/' on *NIX this is the meta for selecting root directory or nothing (a
file could be called "/tmp/test" or "/tmp/////test") ... It generates a SEGFAULT
: killing Moz!!! it's CRITICAL and shoud be an illegal character...

(3): '<' and '>' are shells specials characters : moz creates directory but then
you can't add message to it... it's saying it's not writable... and then you
can't delete it... FIXME: it should be an illegal character...


 char | begin | mid | end
 ==========================
 #    |   N   |  N  |  N
 .    |   N   |     |        (1)
 /    |  !N   |     | 	     (2)
 <    |  !N   |  N  | 	     (3)
 >    |  !N   |  N  | 
 ?    |   N   |     |  	     as (1)
 @    |   Y   |  Y  |  Y
 \    |   Y   |  Y  |  Y
 

I do apoligize for my bad english.
Flags: blocking1.8a? → blocking1.8a-
Still no comment from Mac users...
Status: NEW → ASSIGNED
Keywords: qawanted
Hardware: PC → All
Assignee: sspitzer → mbockelkamp
Status: ASSIGNED → NEW
Status: NEW → ASSIGNED
Test on a Mac (macosX 10.3.4)
Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7) Gecko/20040616

legend:

H= folder is hidden/disapears
one Y= show up as a folder
YY= the folder and message show up
YN= folder show up, the message is shown in the summary but not in the preview
or not even shows in the summary
L = Mozilla Locks up or crash (critical)
D = rename will not apply, after reboot the name returns to the old name
D2 = 2 folders, the old name with the message, the new name empty

char | begin | mid | end	just this char
==========================
#    |   H   |  YN |	YN	H	
.    |   H   |  YY |	YY	dont allow, but not give any error, lost access to
mailbox until close mozilla
.. also fails the same way
 (the .name in unix will make a "hidden" file/folder, dont show in finder also)
/    |   L   |  D  |	D2	L
<    |   YY  |  YY |	YY	YY
>    |   YY  |  YY |	YY	YY
?    |   YN  |  YN |	YN	YN		
@    |   YY  |  YY |	YY	YY	
\    |   YY  |  YY |	YY	YY	

~	YY	YY	H	H
±	YY	YY	YN?	YN?
;	YN	YN	YN	YN

below everything was OK

´	YY	YY	YY	YY
`	YY	YY	YY	YY
|	YY	YY	YY	YY
"	YY	YY	YY	YY
$	YY	YY	YY	YY
%	YY	YY	YY	YY
!	YY	YY	YY	YY
(	YY	YY	YY	YY
)	YY	YY	YY	YY
=	YY	YY	YY	YY
*	YY	YY	YY	YY
+	YY	YY	YY	YY
-	YY	YY	YY	YY
:	YY	YY	YY	YY
&	YY	YY	YY	YY
space	YY	YY	YY	YY
(i think there is a bug here somehere, renaming will only apply after reboot, so
copy a folder with the same name will merge the old and new mailbox... i have to
confirm this when i have more time, but if someone can confirm this its welcome 8)

ƒ	YY	YY	YY	YY (mac folder simbol, not f 8)
§	YY	YY	YY	YY
,	YY	YY	YY	YY
[				YY
{				YY
€				YY
_				YY

didnt test all combinations, i'm assuming its OK
also tried just to confirm this groups
\$a
$(a)
${a}

i tried to use all simbols that mac keyboard have directly and some more used ones
i hope that i did check well everything, but its already too late, i may have
miss something 8)

i will try to do the same test in linux, specially for those that also fail in mac

also, i think that this should block aviary, this bugs make mozilla and friends
look very bad when migrating
this is a tracker, but i think that this bug should be 2/3 bugs:

1 local names (rename and friends)
2 remote names (imap)
3 importing names from other clients

all the dependent bugs are dupes of this 3 main bugs

at least 1 and 3 should block aviary and with 1 fixed, 2 is less critical unless
other client break it

bug 3 should replace bad chars with _ , % - and if needed, give numbers to the end:

folder# -> folder%
folder/ -> folder-
folder? -> folder%_1
bug #219586 might have correct some of this problems...

testing is welcome to see what are the problems that still remain

i hope that it fixed what i called bug 1 and 3 in my latest message

Depends on: 219586
Flags: blocking-aviary1.0?
renominate any serious bugs on the dependence list that may still exist... 
minusing the meta bug.
Flags: blocking-aviary1.0? → blocking-aviary1.0-
Keywords: intl
Maybe it would at least be a good idea to prevent people to create such a folder
by means of mozilla/thunderbird itself. This would be no final cure, but an
improvement for people that use mozilla/thunderbird only
Product: Browser → Seamonkey
(In reply to comment #16)
> Maybe it would at least be a good idea to prevent people to create such a folder
> by means of mozilla/thunderbird itself.
Bug 120559 is the bug of such request, although it refers to "/" problem only
and "/" problem in local folder seems to be fixed.
No longer depends on: 239502
Adding Bug 95114 and Bug 84045 (special character in IMAP path name case) to Dependency tree.
Depends on: 84045, 95114
Summary: Meta bug: Problems with folders having names with illegal characters → Meta bug: Problems with folders having names with illegal(or special) characters
Depends on: 360961
Depends on: 361255
Depends on: 366789
No longer depends on: 366789
No longer depends on: 380594
Depends on: 408506
No longer depends on: 408506
Matthias,
Are you still working on this ?
QA Contact: esther → search
Depends on: 291403
Number of critical bugs due to illegal file name character declined. And edge cases of special character case or special name case are being found.
So morphing this bug for tracking of;
 (a) Illegal file name character,
 (b) non-illegal but special file name character and special position in filename
 (c) special file names
     e.g. CON, PRON etc. on MS Win, Creation of saved search folder named Inbox at
     Local Folders before creation of standard Inbox, double hashing due to special
     illegal/character and long file name, etc. etc.
Depends on: 497035
Summary: Meta bug: Problems with folders having names with illegal(or special) characters → [Meta] Problems with folders having names with illegal(or special) characters or special name
No longer depends on: 497035
QA Contact: search → message-display
Component: MailNews: Message Display → Backend
Product: SeaMonkey → MailNews Core
QA Contact: message-display → backend
Depends on: 538942
Depends on: 547564
No longer blocks: 298033
Depends on: 298033
Depends on: 603013
Depends on: 730449
Depends on: 752768
FYI.
A reason why problem by ">", "^" "|" or around it may occur in IMAP.
> http://mxr.mozilla.org/comm-central/source/mailnews/imap/src/nsImapCore.h#71
> 71 #define kOnlineHierarchySeparatorUnknown '^'
> 72 #define kOnlineHierarchySeparatorNil '|'
> 74 #define IMAP_URL_TOKEN_SEPARATOR ">"
Assignee: mbockelkamp → nobody
Status: ASSIGNED → NEW
On July 23rd 2014 I upgraded Mozilla Thunderbird (originally installed from the tar.bz2-Installer, german localization) from 24.6.0 (20140610001341) to 31.0 (20140717165725).

Today I noticed two of my folders under "Local Folders" appearing empty (named "a*Gesichtsbuch" and "a*StudiVZ"). [Not that I couldn't bare with the loss of their content... *SCNR*]
Note the special character "*" in these folders' names.

I closed TB and backupped my mail profile. There the mail is still existent (data file as well as msf file). 
After the next application start both folders appear duplicated in the folder pane, but still empty.
Data is still available in data files in the profile, however you can't see any duplicate files there.

What puzzles me however are two empty folders with 9 characters long hexedecimal names as well as two corresponding msf files in my "Local Folders" folder in the profile, having a creation date identical to the time of the application update.

Additional Version strings:
"Funnelcake January 2011"
"mozillamessaging04 - 1.0"
"Update channel: release"

OS:
Debian Linux 6.0.10 (oldstable), Kernel 3.2.0-0.bpo.4-686-pae
Depends on: 1047788
Sir George, we generally don't do tracking on meta bugs.   I think the bug you want is bug 992879, which already has a patch and patch author already requested it for TB31. This is the bug were tracking would be appropriate.
Depends on: 992879
Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.
Depends on: 1431307
Keywords: qawanted
Blocks: 1570826
Depends on: 1568845
Depends on: 1512053
Duplicate of this bug: 1570826

I've made some notes about the various flavours of folder names in use. I think a lot of the bugs we've got can be solved by being really clear on what the naming requirements are in each case, and how the various naming schemes should be mapped to one another.

Places where folder names are used:

Folder Name

  • The 'canonical'/internal name of the folder.
  • exposed via the nsIMsgFolder name attribute
  • Usually the same as the user-facing name (but there is a prettyName attr too)
  • Used to derive URI in nsIMsgFolder getURI() implementations.
  • There is some special-case handling for certain folder names - case tweaking, localisation etc. See nsMsgDBFolder::SetPrettyName() for an example.
  • There is also abbrieviatedName, which uses account-specific rules to shorten long names for nicer display. I think only Newsfolders use it (eg "comp.sys.lang.basic" => "c.s.l.basic").

User-facing names

  • As displayed in the folder tree UI panel.
  • Should probably be able to display any printable characters: non-latin chars, '/','' etc.
  • TODO: look in the UI code to see where display name is taken from.

MailStore (ie mbox/maildir on filesystem)

  • The names of files and directories on disk.
  • Allowed characters depends on OS
  • Case-sensitivity depends on OS
  • some names illegal on some OSes (eg CON, PRN, AUX etc on windows)
  • Some names have special significance? "INBOX", "Trash" etc. Sets flags on the folder.
  • Folder discovery iterates over filesystem names, and uses those names for the UI
    => some names are tweaked/localised when mapped to UI, eg "INBOX" -> "Inbox"
    => TODO: where is this mapping performed?
  • should users be able to copy folders across OSes? This means enforcing a superset of naming rules. eg Treat Stuff and stuff as the same folder, even on case-sensitive filesystems.
  • Numeric suffixes sometimes added to deduplicate... eg "INBOX-1" (TODO: why is this needed? How does it work with folder discovery?)
  • some name extensions have special meaning (eg .sbd is used to distinguish subfolders) .

Windows filename rules: https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file

It would be nice if filesystem naming schemes were entirely handled by the appropriate mailstore, but currently there are a lot of places all over the codebase which make assumptions about the filesystem. Maybe over time this can change, particularly if new mailstores are added which aren't directly filesystem-based (eg a database-backed store).

IMAP folders

  • TB queries IMAP server to find remote folders.
  • Base IMAP has ascii names, but extensions handle full unicode (TODO: confirm)
  • Takes time to ask server. Usually, if there's a mailstore (filesystem) folder that will be picked up first.
  • Missing mailstore folders are created as needed (TODO: identify code that does this). Eg, for a new TB install.

Folder URIs

  • Full identity of a folder within TB
  • stored in VirtualFolders.dat, and elsewhere
  • what are the current encoding rules? What should they be?

=> I think the folder parts of the URI should be UTF-8, then percent-encoded. For example, a subfolder of Inbox called "はい/いいえ" ("yes/no") would end up with a URI something like:
imap://bobsmith%40example.com@imap.example.com/INBOX/%E3%81%AF%E3%81%84%2F%E3%81%84%E3%81%84%E3%81%88

VirtualFolders.dat

  • Contains URI of virtual folder to create
  • Upon loading, creates a nsDBMsgFolder with name and parent folder based upon URI
  • see nsMsgAccountManager::LoadVirtualFolders() and nsMsgAccountManager::SaveVirtualFolders().

Here's a particularly extra special one - on macos there is a difference between composed and decomposed utf8 chars, meaning the folder name is not what you think it is if using it for a key. Manifests in Bug 1219084 and maybe elsewhere.

Depends on: 1219084
Depends on: 1581219
Depends on: 1588391
Alias: folders-with-special-characters
Depends on: 1529246
Depends on: 1608318
You need to log in before you can comment on or make changes to this bug.