Open Bug 29926 Opened 20 years ago Updated 2 months ago

IMAP Folders' names with "/" problems for different platforms

Categories

(MailNews Core :: Backend, defect, P3)

defect

Tracking

(Not tracked)

People

(Reporter: huang, Unassigned)

References

(Blocks 1 open bug)

Details

Attachments

(3 files)

Used 03-01-08/09-M15 commercial build:

We should have a way to solve folders' names with "/" problems for different 
platforms:

From bug#17681:
A folder name cannot end with a hierarchy separator.  
In IMAP, a request to CREATE a name which ends with a hierarchy separator is a 
request to create a directory. 

(My comment is: So if not end with a hierarchy separator, users should be 
allowed to create folder name ex: "o/p")

From bug#20324 John's comments & my testing results for "/":
* John's comments: 
My server always puts a "=" character after every "/" in order to avoid this 
potential problem.

*My test results for "/" & "o/p" on WinNT & Linux: 
"/": "The current command did not succeed, the mail server responded:  Invalid 
mailbox name" 
"o/p": will create "o" for the first level folder & create "p" for the 2nd level 
subfolder since there was no way to aviod this problem in seamonkey yet!!

*Additional information from Alec is: 
Unix "illegal" character is "/" 

So we should have a way to handle "/" problem for different platforms.
Change QA contact to me.
QA Contact: lchiang → huang
Blocks: 20324
My 20324 comment was to avoid the NT special names "aux" "con", etc.

We need to get QA a test server which uses a different hierarchy separator.  We 
should try separators of ".", "\\", and ":", since those seem to be in use.  We 
should file a separate bug for the task of writing such a test server.
1) I have tried "aux" on WinNT -> it seemed no problem for creating this folder 
on WinNT, not try "con" yet...I will try that!!

2) Yes. As I know, we do have the Cyrus test server/account for testing "." 
hierarchy separator. I was thinking that I am going to try on Cyrus server and 
log another bug....
And do you know which servers' use "\\" & ":" hierarchy separators?
While Cyrus uses "." as the hierarchy separator, it does not permit "/" in folder 
names.  I have heard of an NT server with "\\" and a mac server with ":" but 
don't remember specifics.
Hmm...need to do more investigation!!
I think David's looking at these IMAP folder name bugs. cc'ing jefft too.
Assignee: phil → bienvenu
accept
Status: NEW → ASSIGNED
Target Milestone: M16
Karen, not, that the platform (and product) of the server is relevant, not the
client's.
Just put some comments here in order to tracking/investigate this problem:

1) From NMS 4.1 IMAP server (Enable support dual-use folder):
After I setup the "Enable support dual use folder", I created folder
name end with slash (ex: ABC/), I found that this folder not been
subscribed initially....shouldn't this "ABC" folder be subscribed &
included in the .mailboxlist file? 

2) From UW IMAP server (Disable support dual-use folder):
After I setup the "Disable support dual use folder", I created folder
name same as above: end with slash (ex: ABC/) , I found that this folder
had been subscribed initially......

3) From above both servers (No matter Enable or Disable dual-use folder
or not):
If I created "A/B" folder, it will created "A" folder with subfolder
"B", and I found that "A" folder not been subscribed initially and won't
allow to subscribed ( "B" has been subscribed)....
Do we know what IMAP protocol the client is sending?



If the client is doing a "tag SUBSCRIBE ABC/" then that is incorrect protocol 

(even if it appears to work with c-client).  The name of the folder is "ABC" 

without the slash, even though it is a non-selectable folder.  It is a special 

case of the CREATE command that appending a "/" to the name of the folder 

requests it be created as something that can contain subfolders.  This special 

case of accepting a "/" after the folder name only applies to the CREATE command.  

It does not apply to, say, the SUBSCRIBE command.

Thanks god, I thought that I cannot contact with you anymore, why I got an error 
message from your email address?

Anyway, thanks for replying on this bug.
I will try to catch those logs again.....
After returning from my vacation, I've been busy with iPlanet's move to Santa 
Clara.  The mail bounces were fallout from the move; I did actually get the 
messages.

I'm not very knowlegable about the U. Washington server, so I can't answer too 
many of your questions.  I do suspect the SUBSCRIBE command incorrectly has a 
trailing "/", someone should look at a protocol telemetry log.

Attached file attached IMAP log file
Yes. You are right!! I saw "ABC/" incorrect IMAP protocol from above log for the 
client side: 
--------------------------------------------------------------------------------
23 create "ABC/"-3617371[3885210]:208.12.40.110:S-INBOX:CreateNewLineFromSocket: 
23 OK CREATE completed-3617371[3885210]: 208.12.40.110:S-INBOX:SendData: 
24 
subscribe"ABC/"-3617371[3885210]:208.12.40.110:S-INBOX:CreateNewLineFromSocke:
24 OK SUBSCRIBE completed-3617371[3885210]: 208.12.40.110:S-INBOX:SendData: 
25 list "" "ABC/"-3617371[3885210]: 
208.12.40.110:S-INBOX:CreateNewLineFromSocket: * LIST (\NoSelect) "/" 
ABC/-3617371[3885210]: 208.12.40.110:S-INBOX:CreateNewLineFromSocket: 25 OK LIST 
completed
--------------------------------------------------------------------------------
Mail Review recommends beta2 stopper.
Keywords: nsbeta2
Mass moving M16 to M17 - look for nsbeta2 before anything else.
Target Milestone: M16 → M17
Putting on [nsbeta2+] radar for beta2 fix. 
Whiteboard: [nsbeta2+]
Copying Jeff's comments from bug 20879:

-------------------------------------------------------------------------------
notice that folder only mailbox end with "/". It's all up to the server to set
\Noselect or \Noinferiors flag as it likes. Client doesn't get the control for
it. So for servers that allows both mailbox there is no difference between

1 create "foobar/"

vs

2 create "shooshoo"

they can create child of "shooshoo" by issuing

3 create "shooshoo/foofoo"

which will not be allowed in a uw server.
--------------------------------------------------------------------------------
If I understand this bug correctly, 4.0 and 4.5 shipped this way. Why is this a 
beta2 stopper? Clearing status.
Whiteboard: [nsbeta2+]
Putting on [nsbeta2-] radar assuming bienvenu is correct.
Whiteboard: [nsbeta2-]
Nominate this bug for nsbeta3.
Keywords: nsbeta3
Mail triage is marking nsbeta3-
Whiteboard: [nsbeta2-] → [nsbeta2-][nsbeta3-]
Target Milestone: M17 → Future
adding mail3 keyword
Keywords: mail3
changing priorities
Priority: P3 → P2
marking nsbeta1+ and moving to mozilla0.8
Keywords: nsbeta2, nsbeta3nsbeta1
Whiteboard: [nsbeta2-][nsbeta3-] → [nsbeta1+]
Target Milestone: Future → mozilla0.8
again, 4.0, 4.5, and 6.0 shipped with this problem - I'm not sure that it
affects very many people.
any endorsement to not fix a bug is worth looking at! So I'm removing the + and 
renominating for triage again.
Whiteboard: [nsbeta1+]
Target Milestone: mozilla0.8 → ---
marking nsbeta1-
Keywords: nsbeta1nsbeta1-
Update the current status: 
By using the latest good build 08-15-06-trunk build
Now if I create the folder name with the slash. (e.g. karen1/karen2)
On the UI, our client will create additional unknown subfolder "karen1^karen2".
After exit and relaunch again, it will disappear. 
I willl attach a scrren shot as following.
It seems fine for creating the "karen1/karen2" folder from the following log:

257[4620eb0]: nsmail-1:S-INBOX:SendData: 12 create "karen1/karen2"
257[4620eb0]: nsmail-1:S-INBOX:CreateNewLineFromSocket: 12 OK Completed
257[4620eb0]: nsmail-1:S-INBOX:SendData: 13 subscribe "karen1/karen2"
257[4620eb0]: nsmail-1:S-INBOX:CreateNewLineFromSocket: 13 OK Completed
257[4620eb0]: nsmail-1:S-INBOX:SendData: 14 list "" "karen1/karen2"
257[4620eb0]: nsmail-1:S-INBOX:CreateNewLineFromSocket: * LIST (\HasNoChildren) 
"/" karen1/karen2
257[4620eb0]: nsmail-1:S-INBOX:CreateNewLineFromSocket: 14 OK Completed

-----------------------------------------------------------------------

But, our client shouldn't create an unknown "karen1^karen2" subfolder...
Following is: when I select the "karen1^karen2" folder, I got alerts from the 
log...

131[526da10]: nsmail-1:A:SendData: 3 LANGUAGE en-us
131[526da10]: nsmail-1:A:CreateNewLineFromSocket: 3 NO Unrecognized language
131[526da10]: nsmail-1:A:SendData: 4 select "karen1^karen2"
131[526da10]: nsmail-1:A:CreateNewLineFromSocket: 4 NO Mailbox does not exist
131[526da10]: nsmail-1:A:SendData: 5 getacl "karen1^karen2"
131[526da10]: nsmail-1:A:CreateNewLineFromSocket: 5 NO Mailbox does not exist
-------------------------------------------------------------------------------
I have just gotten bitten by this bug. I stupidly created an IMAP subfolder
called "a/b", which seemed to work fine. But I immediately noticed the folder
was only called a, and that it was greyed out.

I then noticed it had a subfolder called b! So now I can't erase either a or b,
and I can't put any messages in either a or b! This is using 1.3.1 on Linux. 
Product: MailNews → Core
*** Bug 301714 has been marked as a duplicate of this bug. ***
Duplicate of this bug: 378140
Hi, I tried to create a folder with name "a/b" using Outlook and I can't. Outlook shows a warning message alerting to not use "/" on folder names and don't let me create the folder.
Strange which Outlook version are you using? I'm running Office 2003 at work and I'm able to rename folders which can have a '/' inside. Outlook shows them correctly but in Thunderbird they are invisible. 
FWIW, here's the gmail imap case:

I tried on gmail imap to create / and // named subfolders and goes ok, only that it actually names them _test and __test. Well, gmail may be a specific case here (labels vs folders..) and I don't have another imap to test with. 

But I would ask (as a very undocumented imap one), could this be a server/imap side also case? Cause I'm intrigued by the fact that gmail (which uses labels "translated" to folders actually) obviously avoids this / naming "mistake" by transforming it into _ (underscore). Which makes me think that they may be aware of this kind of bug and if so, maybe not only a TB one. 

If I make a bit of sense in this idea, maybe it's about the way TB handles these kind of error from the server/imap side rather than inside error.
This looks like a completely hopeless bug to me. After reading all the comments I don't really even understand what people are talking about in here. The reason I'm here is because my bug 301714 was marked as a duplicate to this, most likely because comment #10 happens to describe the same thing. But that doesn't seem to have anything to do with the rest of the discussion in here.

You're also talking about "/" as if there's something special about it. There isn't really. Except that it's the most commonly used hierarchy separator and it also happens to be directory separator in UNIX.

Then you're also talking about some special Windows filenames which have absolutely nothing to do with IMAP as well. If there's a problem with creating such files on Thunderbird/Windows, it's the problem with the local filesystem handling code, not with IMAP code.

As for how TB should be handling hierarchy separators when creating mailbox names:

 - If mailbox is being created as "Folders only", add the separator to mailbox name. 
 - If separator is manually added to the end of mailbox name, treat it the same way as when creating "Folders only", but don't add the second separator.
 - If separator is used in the middle of a mailbox name, such as "a/b/c", you can assume that mailbox "a/b/c" was created which can contain messages. You can't assume anything about "a" and "a/b". They may or may not have been created. Only LIST shows what is really the case.

As for other special characters and whatever in the mailbox names, as long as it's valid UTF-7 the client has no business whatsoever in guessing server's actions. Some servers may disallow some characters or mailbox names, others may not. Don't guess. If it fails, just return the error message.

I'd vote to reopen my bug 301714 since it's clearly a TB bug (and should be easy to fix) and close this bug as INVALID.
Oh, and for the GMail case in the previous comment: If GMail changes the mailbox name when creating it, I'd argue that it's a GMail bug and it should instead fail the command with something like "NO '/' characters not supported at beginning of mailbox names". Clients can't be expected to guess that server happens to rename some mailbox names when creating them.

(If CREATE /foo is used with Dovecot, it either fails with "Invalid mailbox name" or it actually creates "foo" to root directory, depending both on configuration and permissions. There's no way TB would be able to guess which one of these happens.)
(In reply to comment #39)
...
> The
> reason I'm here is because my bug 301714 was marked as a duplicate to this,
> most likely because comment #10 happens to describe the same thing. But that
> doesn't seem to have anything to do with the rest of the discussion in here.
...
> 
> I'd vote to reopen my bug 301714 since it's clearly a TB bug (and should be
> easy to fix) and close this bug as INVALID.
> 

I think you are right about your bug 301714 being too easily duped for this one!  Seams to me about a different thing.

As about this one invalid, dunno ..
(In reply to comment #40)
> Oh, and for the GMail case in the previous comment: If GMail changes the
> mailbox name when creating it, I'd argue that it's a GMail bug and it should
> instead fail the command with something like "NO '/' characters not supported
> at beginning of mailbox names". Clients can't be expected to guess that server
> happens to rename some mailbox names when creating them.
> 
I agree on that, gmail changes my / for a _ without ask or alert. But that could be another topic. I just find that interesting from the point of view of gmail avoiding this / which is the subject here. This gives me another idea: is anyone able to ask gmail stuff what is that all about? or maybe I should just go for a help/forum there for that q?

OK, looks like there really are some issues with '/' character and Thunderbird. These can be easily tested with Dovecot v1.1, Maildir and http://dovecot.org/patches/1.1/listescape-plugin.c using '^' as hierarchy separator.

Then you'll see the following behavior:

1. Create "bar" mailbox under "foo" mailbox.

Expected results: In mailbox list "bar" shows up under "foo" as a subtree, like:
+ foo
   bar

Actual results: In mailbox list "bar" shows up as "foo/bar", not as a subtree:
foo
foo/bar

2. Open up subscribe dialog.

Expected results: In mailbox list "bar" shows up under "foo" as a subtree.
Actual results: In mailbox list "foo^bar" shows up.

3. Create "foo2/bar2" mailbox.

Expected results: In mailbox list "foo/bar2" shows up.
Actual results: In mailbox list "bar" shows up under "foo" as a subtree.

4. Select "foo2/bar2" mailbox.

Expected results: Mailbox is opened using SELECT "foo2/bar2" command.
Actual results: Mailbox is tried to be opened using SELECT "foo2^bar" command, which fails.

I tested this using Debian's Icedove v2.0.0.12. Before testing I deleted all Thunderbird's state by deleting ~/.mozilla-thunderbird directory. The same happens both with and without "show only subscribed folders".

When opening up TB it issues commands correctly:

3 namespace

* NAMESPACE (("" "^")) NIL NIL

3 OK Namespace completed.


4 list "" "%"

* LIST (\HasNoChildren) "^" "Trash"

* LIST (\HasChildren) "^" "foo"

* LIST (\HasNoChildren) "^" "foo2/bar2"

* LIST (\HasNoChildren) "^" "INBOX"

4 OK List completed.


5 list "" "%^%"

* LIST (\HasNoChildren) "^" "foo^bar"

5 OK List completed.


6 list "" "INBOX"

* LIST (\HasNoChildren) "^" "INBOX"

6 OK List completed.


Since it believes "bar2" is a folder under "foo2", expanding "bar2" to show its subtree sends a bit pointless commands:

15 list "" "test/%"

* LIST (\HasNoChildren) "^" "test/box"

15 OK List completed.


16 list "" "test/%/%"

16 OK List completed.

Sorry, miswrote the mailbox names here:

> 3. Create "foo2/bar2" mailbox.
> 
> Expected results: In mailbox list "foo/bar2" shows up.
> Actual results: In mailbox list "bar" shows up under "foo" as a subtree.

I meant to say foo2 and bar2 everywhere, so:

Expected results: In mailbox list "foo2/bar2" shows up.
Actual results: In mailbox list "bar2" shows up under "foo2" as a subtree.
No longer blocks: 20324
QA Contact: huang → backend
Product: Core → MailNews Core
David, are you working on this or assign should be cleared?
Duplicate of this bug: 504588
(In reply to comment #43)
> OK, looks like there really are some issues with '/' character and
> Thunderbird. These can be easily tested with Dovecot v1.1, Maildir and
> http://dovecot.org/patches/1.1/listescape-plugin.c using '^' as hierarchy
> separator.

I'd bet that the behavior Timo noted in comment #43 are due to how Thunderbird uses ^ in a few places.
See http://mxr.mozilla.org/comm-central/source/mailnews/imap/src/nsImapCore.h#99
And where it gets used: http://mxr.mozilla.org/comm-central/ident?i=kOnlineHierarchySeparatorUnknown
Hi Everybody! I may discoverd an issue like this one here.

I'm using Dovecot 2.x with listescape and '$' as a seperator.
Somehow creating folders with foldername 'tests/stuff' does not work. When i try do create such a folder using outlook for example it works fine.
Assignee: mozilla → nobody
Status: ASSIGNED → NEW
I have this problem, in that my users are sharing a mailbox via IMAP, and if someone creates a new folder with a slash in it (e.g. "Main St E/W"), somewhere between TB and our (3rd party) server, things cack up and ALL of the subfolders on the account get deleted, then this deletion populates across everyone's TB installations. 

My current workaround is to periodically remind staff not to use slashes in folder names, but I would rather a more robust workaround that prevents users from creating these invalid filenames in the first place, such as:

- A configuration setting called "illegal_folder_characters" (or "Gadsby") where I can insert a value like "./:$" and it will prevent users from creating folders with periods, slashes, colons, and dollar signs in the name; or
- An extension that accomplishes this.

I would even be satisfied if, initially, this only works for the "new folder" dialog, or if it just replaced any of these characters with an underscore.

This wouldn't help in the generalized case but it would at least allow some of us with specific restrictions to be able to enforce them, and may be a step in the direction of solving this bug.

- RG>
Priority: P2 → P3
You need to log in before you can comment on or make changes to this bug.