124287 - (folders-with-special-characters) [Meta] Problems with folders having names with illegal(or special) characters or special name

Status: NEW

(MailNews Core :: Backend, defect)

Description

[Sheela Ravindran]

This bug tracks both imap and local folders having illegal characters as the
name of the folder.

Comment 1

[Sheela Ravindran]

Tracking the bug numbers with this problem.  As per our discussion leaving
Esther as the owner and qa contact for this bug.  

Comment 2

[Sheela Ravindran]

Adding two more bugs which should be tracked as well. Thanks Cavin.  

Comment 3

[Matthias Bockelkamp]

Attachment: patch

I collected the characters which are problematic from the depending bugs and
made a simple patch that just checks the names for New Folder and Rename Folder
and displays an alert if one of them is contained. I think this could be done
better by escaping the problematic chars, but I don't know enough about where
that is done and if I may change the methods without destroying other
functionality. Setting helpwanted.

Comment 4

[Matthias Bockelkamp]

I just noticed I'm not allowed to set helpwanted. Could someone please do that
for me?

Comment 5

[Matthias Bockelkamp]

Attachment: patch v2

Changed the patch because " " and "@" are only problematic if used alone for
folder names.

Comment 6

[Matthias Bockelkamp]

The current status is this: 

The OK Button of New Folder and Rename Folder was disabled if the name was
empty. This has to be extended for the case the name starts or ends with a space
(which also handles filenames like " ", "  " etc.). Alternatively a message
could be displayed, e.g. "Folder names may not start or end with a space
character." 

For the other chars, I'm going to modify the FILE_ILLEGAL_CHARACTERS constants
in nsCRT.h but I can only test on my Windows PC. So I need some help from Linux
and Mac users to try some filenames containing the characters "#./<>?@\".

Testcase: 

Create a folder named e.g. "abc". Copy a message into it. Rename the folder.
Check if

- the folder is displayed with correct name and path in the Folder pane
- the message shows up in the Thread pane
- the message, when selected, is displayed in the Message Pane

Then enter into the table (below) if everything works or if problems occur.
Before the next test, delete the new folder and be sure to view a different
message in the Message Pane.

char | begin | mid | end
==========================
#    |       |     |  
.    |       |     | 
/    |       |     | 
<    |       |     | 
>    |       |     | 
?    |       |     | 
@    |       |     | 
\    |       |     | 

Comment 7

[Zook Valem]

Wouldn't it be possible to have folder name be stored as a string and display
that string name instead?  Just a thought here...

Comment 8

[WADA:World Anti-bad-Duping Agency]

Problem caused by special characters in folder name is one of the most
puzzleded, embarrassed and strange problems for Mozilla Mail users.
Since problems live long time, not so small number of people ask about these
problems at forums such as Mozillazine.
So I think this bug should be listed in "Known Issues" section of "Release
Notes" as Bug 2654 is listed.
I believe listing-up will also reduce many DUPEs.

Comment 9

[Daniel Mota Leite]

all this bugs seen more or less related to each others, if they arent really
dupes...

there are 2 main bugs:

-unable to use of bad folder names
-allow the creating for bad names

fix the 1º, the 2º would be little importance
fix the 2º, the 1º one still needs to be fixed or workaround

esther email is invalid ([e-mail]) and its the owner of
many of this bugs

this are important bug as advocating people to move from OE to mozilla and
hitting this bug is very bad and might cost several users ("if it can open/use
some simple folders, its still full of bugs and so useless")

adding the radar for 1.8a

Comment 10

[Zook Valem]

This is more of a bug tracking purpose.  This shouldn't block the Mozilla 1.8a
release.

Comment 11

[NextGen$]

(In reply to comment #6)
> The current status is this: 
> 
> The OK Button of New Folder and Rename Folder was disabled if the name was
> empty. This has to be extended for the case the name starts or ends with a space
> (which also handles filenames like " ", "  " etc.). Alternatively a message
> could be displayed, e.g. "Folder names may not start or end with a space
> character." 
> 
> For the other chars, I'm going to modify the FILE_ILLEGAL_CHARACTERS constants
> in nsCRT.h but I can only test on my Windows PC. So I need some help from Linux
> and Mac users to try some filenames containing the characters "#./<>?@\".
> 
> Testcase: 
> 
> Create a folder named e.g. "abc". Copy a message into it. Rename the folder.
> Check if
> 
> - the folder is displayed with correct name and path in the 

(In reply to comment #6)
> The current status is this: 
> 
> The OK Button of New Folder and Rename Folder was disabled if the name was
> empty. This has to be extended for the case the name starts or ends with a space
> (which also handles filenames like " ", "  " etc.). Alternatively a message
> could be displayed, e.g. "Folder names may not start or end with a space
> character." 
> 
> For the other chars, I'm going to modify the FILE_ILLEGAL_CHARACTERS constants
> in nsCRT.h but I can only test on my Windows PC. So I need some help from Linux
> and Mac users to try some filenames containing the characters "#./<>?@\".
> 
> Testcase: 
> 
> Create a folder named e.g. "abc". Copy a message into it. Rename the folder.
> Check if
> 
> - the folder is displayed with correct name and path in the Folder pane
> - the message shows up in the Thread pane
> - the message, when selected, is displayed in the Message Pane
> 
> Then enter into the table (below) if everything works or if problems occur.
> Before the next test, delete the new folder and be sure to view a different
> message in the Message Pane.
> 
> char | begin | mid | end
> ==========================
> #    |       |     |  
> .    |       |     | 
> /    |       |     | 
> <    |       |     | 
> >    |       |     | 
> ?    |       |     | 
> @    |       |     | 
> \    |       |     | 
> 



Mozilla/5.0 (X11; U; Linux i686; fr-FR; rv:1.6) Gecko/20040122 Debian/1.6-1
+enigmail
+adblock

Y: working : this could stay as it is.
N: NOT working

(1): '.' on *NIX this is the meta for selecting current working directory... so
if it's the only char. we get a dialog saying it already exists... it works...

(2): '/' on *NIX this is the meta for selecting root directory or nothing (a
file could be called "/tmp/test" or "/tmp/////test") ... It generates a SEGFAULT
: killing Moz!!! it's CRITICAL and shoud be an illegal character...

(3): '<' and '>' are shells specials characters : moz creates directory but then
you can't add message to it... it's saying it's not writable... and then you
can't delete it... FIXME: it should be an illegal character...


 char | begin | mid | end
 ==========================
 #    |   N   |  N  |  N
 .    |   N   |     |        (1)
 /    |  !N   |     | 	     (2)
 <    |  !N   |  N  | 	     (3)
 >    |  !N   |  N  | 
 ?    |   N   |     |  	     as (1)
 @    |   Y   |  Y  |  Y
 \    |   Y   |  Y  |  Y
 

I do apoligize for my bad english.

Comment 12

[Matthias Bockelkamp]

Still no comment from Mac users...

Comment 13

[Daniel Mota Leite]

Test on a Mac (macosX 10.3.4)
Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7) Gecko/20040616

legend:

H= folder is hidden/disapears
one Y= show up as a folder
YY= the folder and message show up
YN= folder show up, the message is shown in the summary but not in the preview
or not even shows in the summary
L = Mozilla Locks up or crash (critical)
D = rename will not apply, after reboot the name returns to the old name
D2 = 2 folders, the old name with the message, the new name empty

char | begin | mid | end	just this char
==========================
#    |   H   |  YN |	YN	H	
.    |   H   |  YY |	YY	dont allow, but not give any error, lost access to
mailbox until close mozilla
.. also fails the same way
 (the .name in unix will make a "hidden" file/folder, dont show in finder also)
/    |   L   |  D  |	D2	L
<    |   YY  |  YY |	YY	YY
>    |   YY  |  YY |	YY	YY
?    |   YN  |  YN |	YN	YN		
@    |   YY  |  YY |	YY	YY	
\    |   YY  |  YY |	YY	YY	

~	YY	YY	H	H
±	YY	YY	YN?	YN?
;	YN	YN	YN	YN

below everything was OK

´	YY	YY	YY	YY
`	YY	YY	YY	YY
|	YY	YY	YY	YY
"	YY	YY	YY	YY
$	YY	YY	YY	YY
%	YY	YY	YY	YY
!	YY	YY	YY	YY
(	YY	YY	YY	YY
)	YY	YY	YY	YY
=	YY	YY	YY	YY
*	YY	YY	YY	YY
+	YY	YY	YY	YY
-	YY	YY	YY	YY
:	YY	YY	YY	YY
&	YY	YY	YY	YY
space	YY	YY	YY	YY
(i think there is a bug here somehere, renaming will only apply after reboot, so
copy a folder with the same name will merge the old and new mailbox... i have to
confirm this when i have more time, but if someone can confirm this its welcome 8)

ƒ	YY	YY	YY	YY (mac folder simbol, not f 8)
§	YY	YY	YY	YY
,	YY	YY	YY	YY
[				YY
{				YY
€				YY
_				YY

didnt test all combinations, i'm assuming its OK
also tried just to confirm this groups
\$a
$(a)
${a}

i tried to use all simbols that mac keyboard have directly and some more used ones
i hope that i did check well everything, but its already too late, i may have
miss something 8)

i will try to do the same test in linux, specially for those that also fail in mac

also, i think that this should block aviary, this bugs make mozilla and friends
look very bad when migrating
this is a tracker, but i think that this bug should be 2/3 bugs:

1 local names (rename and friends)
2 remote names (imap)
3 importing names from other clients

all the dependent bugs are dupes of this 3 main bugs

at least 1 and 3 should block aviary and with 1 fixed, 2 is less critical unless
other client break it

bug 3 should replace bad chars with _ , % - and if needed, give numbers to the end:

folder# -> folder%
folder/ -> folder-
folder? -> folder%_1

Comment 14

[Daniel Mota Leite]

bug #219586 might have correct some of this problems...

testing is welcome to see what are the problems that still remain

i hope that it fixed what i called bug 1 and 3 in my latest message

Comment 15

[chris hofmann]

renominate any serious bugs on the dependence list that may still exist... 
minusing the meta bug.

Comment 16

[Alexander Sack]

Maybe it would at least be a good idea to prevent people to create such a folder
by means of mozilla/thunderbird itself. This would be no final cure, but an
improvement for people that use mozilla/thunderbird only

Comment 17

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #16)
> Maybe it would at least be a good idea to prevent people to create such a folder
> by means of mozilla/thunderbird itself.
Bug 120559 is the bug of such request, although it refers to "/" problem only
and "/" problem in local folder seems to be fixed.

Comment 18

[WADA:World Anti-bad-Duping Agency]

Adding Bug 95114 and Bug 84045 (special character in IMAP path name case) to Dependency tree.

Comment 19

[Serge Gautherie (:sgautherie)]

Matthias,
Are you still working on this ?

Comment 20

[WADA:World Anti-bad-Duping Agency]

Number of critical bugs due to illegal file name character declined. And edge cases of special character case or special name case are being found.
So morphing this bug for tracking of;
 (a) Illegal file name character,
 (b) non-illegal but special file name character and special position in filename
 (c) special file names
     e.g. CON, PRON etc. on MS Win, Creation of saved search folder named Inbox at
     Local Folders before creation of standard Inbox, double hashing due to special
     illegal/character and long file name, etc. etc.

Comment 21

[WADA:World Anti-bad-Duping Agency]

FYI.
A reason why problem by ">", "^" "|" or around it may occur in IMAP.
> http://mxr.mozilla.org/comm-central/source/mailnews/imap/src/nsImapCore.h#71
> 71 #define kOnlineHierarchySeparatorUnknown '^'
> 72 #define kOnlineHierarchySeparatorNil '|'
> 74 #define IMAP_URL_TOKEN_SEPARATOR ">"

Comment 22

[Sir George]

On July 23rd 2014 I upgraded Mozilla Thunderbird (originally installed from the tar.bz2-Installer, german localization) from 24.6.0 (20140610001341) to 31.0 (20140717165725).

Today I noticed two of my folders under "Local Folders" appearing empty (named "a*Gesichtsbuch" and "a*StudiVZ"). [Not that I couldn't bare with the loss of their content... *SCNR*]
Note the special character "*" in these folders' names.

I closed TB and backupped my mail profile. There the mail is still existent (data file as well as msf file). 
After the next application start both folders appear duplicated in the folder pane, but still empty.
Data is still available in data files in the profile, however you can't see any duplicate files there.

What puzzles me however are two empty folders with 9 characters long hexedecimal names as well as two corresponding msf files in my "Local Folders" folder in the profile, having a creation date identical to the time of the application update.

Additional Version strings:
"Funnelcake January 2011"
"mozillamessaging04 - 1.0"
"Update channel: release"

OS:
Debian Linux 6.0.10 (oldstable), Kernel 3.2.0-0.bpo.4-686-pae

Comment 23

[Wayne Mery (:wsmwk)]

Sir George, we generally don't do tracking on meta bugs.   I think the bug you want is bug 992879, which already has a patch and patch author already requested it for TB31. This is the bug were tracking would be appropriate.

Comment 24

[Ludovic Hirlimann [:Usul]]

Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

Comment 26

[Ben Campbell]

I've made some notes about the various flavours of folder names in use. I think a lot of the bugs we've got can be solved by being really clear on what the naming requirements are in each case, and how the various naming schemes should be mapped to one another.

Places where folder names are used:

Folder Name

• The 'canonical'/internal name of the folder.
• exposed via the nsIMsgFolder name attribute
• Usually the same as the user-facing name (but there is a prettyName attr too)
• Used to derive URI in nsIMsgFolder getURI() implementations.
• There is some special-case handling for certain folder names - case tweaking, localisation etc. See nsMsgDBFolder::SetPrettyName() for an example.
• There is also abbrieviatedName, which uses account-specific rules to shorten long names for nicer display. I think only Newsfolders use it (eg "comp.sys.lang.basic" => "c.s.l.basic").

User-facing names

• As displayed in the folder tree UI panel.
• Should probably be able to display any printable characters: non-latin chars, '/','' etc.
• TODO: look in the UI code to see where display name is taken from.

MailStore (ie mbox/maildir on filesystem)

• The names of files and directories on disk.
• Allowed characters depends on OS
• Case-sensitivity depends on OS
• some names illegal on some OSes (eg CON, PRN, AUX etc on windows)
• Some names have special significance? "INBOX", "Trash" etc. Sets flags on the folder.
• Folder discovery iterates over filesystem names, and uses those names for the UI
  => some names are tweaked/localised when mapped to UI, eg "INBOX" -> "Inbox"
  => TODO: where is this mapping performed?
• should users be able to copy folders across OSes? This means enforcing a superset of naming rules. eg Treat Stuff and stuff as the same folder, even on case-sensitive filesystems.
• Numeric suffixes sometimes added to deduplicate... eg "INBOX-1" (TODO: why is this needed? How does it work with folder discovery?)
• some name extensions have special meaning (eg .sbd is used to distinguish subfolders) .

Windows filename rules: https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file

It would be nice if filesystem naming schemes were entirely handled by the appropriate mailstore, but currently there are a lot of places all over the codebase which make assumptions about the filesystem. Maybe over time this can change, particularly if new mailstores are added which aren't directly filesystem-based (eg a database-backed store).

IMAP folders

• TB queries IMAP server to find remote folders.
• Base IMAP has ascii names, but extensions handle full unicode (TODO: confirm)
• Takes time to ask server. Usually, if there's a mailstore (filesystem) folder that will be picked up first.
• Missing mailstore folders are created as needed (TODO: identify code that does this). Eg, for a new TB install.

Folder URIs

• Full identity of a folder within TB
• stored in VirtualFolders.dat, and elsewhere
• what are the current encoding rules? What should they be?

=> I think the folder parts of the URI should be UTF-8, then percent-encoded. For example, a subfolder of Inbox called "はい/いいえ" ("yes/no") would end up with a URI something like:
imap://[e-mail]/INBOX/%E3%81%AF%E3%81%84%2F%E3%81%84%E3%81%84%E3%81%88

VirtualFolders.dat

• Contains URI of virtual folder to create
• Upon loading, creates a nsDBMsgFolder with name and parent folder based upon URI
• see nsMsgAccountManager::LoadVirtualFolders() and nsMsgAccountManager::SaveVirtualFolders().

Comment 27

[alta88]

Here's a particularly extra special one - on macos there is a difference between composed and decomposed utf8 chars, meaning the folder name is not what you think it is if using it for a key. Manifests in Bug 1219084 and maybe elsewhere.

Comment 28

[Thorolf E.R. Weißhuhn]

// A header search of "" is illegal for IMAP and will cause us to
//  explode.  You don't want that and I don't want that.  So let's check
//  if the bloody term is a subject search on a blank string, and if it
//  is, let's secretly not add the term.  Everyone wins!

Add a check "is IMAP" and if it is not, search as written by user.
And document that IMAP is different here because of <reason>, else it will be seen as inconsistent behaviour and reported as bug.

Comment 29

[Thorolf E.R. Weißhuhn]

^^^ wrong bug, sorry, please ignore

Comment 30

[Ben Campbell]

Some more win32 file naming quirks to deal with:
https://learn.microsoft.com/en-us/troubleshoot/windows-client/shell-experience/file-folder-name-whitespace-characters

(edit: I don't think these apply to the API level - they're just rules applied when the user enters filenames in the windows shell/file explorer.)

Comment 31

[Richard Leger]

Also just experienced recently in TB Beta, after moving folders, some of the sub-folders disappeared from TB in my IMAP account without error or noticed, while there were still present in webmail. This prevented access to the folders and emails it contained.

After some lengthy investigation, it turned out the issue was TB not able to create the .msf file for few of the sub-folders because the 255 full file path limit was exceeded on Windows, so Windows itself refused to create such files with too long path.

At least TB shall raise/log the error in such situation, not just silently hide the folder(s).

Part of the issue is that the prefix path to the mailbox folder is already very long something like:
C:\Users\richard\AppData\Roaming\Thunderbird\Profiles\cnantXXX.default\ImapMail\mail.my-domain.com
...so if you then have a many sub-folders the full file path can easily reach/exceed the 255 characters limit of Windows.

Should it be time for TB to start using \\?\ prefix to specify an extended-length path?

For example, \\?\C:\<very long path> rather than just C:\<very long path> which is causing issues on Windows.