148460 - Need new canonical format for IMAP folders, should include information on the namespace of the folder

Status: NEW

(MailNews Core :: Backend, enhancement)

Description

[Michael Klose]

While working for fixes for bugs #27002 (about making special folders work on
cyrus and courier IMAP servers) and #105385 (server directory for these servers)
and seeing the mess between the two different concepts of namespaces and server
directories, it has become clear in my opinion and that of a few others that to
properly fix this, the internal representation - the canonical format in which
folder names are stored and are used to build the tree and files on the disk,
needs to be reworked to somehow include information on the namespace of the folder.

This bug is for discussion on how this can be done.

Comment 1

[Michael Klose]

I will start with my idea:

I have been thinking of this for some time now and have asked people to send me
NAMESPACE responses from a lot of different IMAP servers they use. From this and
earlier ideas which I had in bug #27002, this is my current plan on who I would
intend to implement this:

The IMAP NAMESPACE command returns namespaces for private, shared and public
folders.

We should make use of this and have a format like:

/<namespacetype>/<namespace>/<foldername>

where namespacetype is either "personal", "shared" or "public".

This would give a nice representation in the tree as it would group all
personal, all shared and all public folders.

For IMAP servers that do not publicise a namespace, I would use a neutral string
like "folders".

The only problem I see is for personal namespaces that are "". For this, the
resulting canonical namespace would be "/personal//Sent", which is probably not
good. The empty namespace somehow needs to be escaped and then not shown in the
tree. Perhaps we can do this by using another namespace type string, for example
"personal%" or so.

I would also leave the prefs for the 3 different types of namespaces, give them
a new role though. Their new role would be to filterthe namespace you are
interested in for each of the hierarchies. This would turn their functionality
into what today the "server directory" does today.

Now to comment on another idea.

Henry Jia commented in bug #27002:

"4. About the canonical format of the directories, I think we can use the format
"<namespace>//<directory>/<directory>...". This can give us flexibility for
namespace handling in the future without any information losing. P.S. the
delimiter between directories and namespace is '//' instead of '/'."

I now agree that the namespace should be part of the canonical format of the
directories as you see in my proposal. I do not like the use of // as a
delimiter. // is normally only used to destinguish between the URL and the
actual protocol. Also, many operations seperate the strings into substrings
using / as the delimiter. This would cause an empty string. This could cause
problems building the folder tree.


 

Comment 2

[Henry Jia]

Adding myself to track this issue.

Comment 3

[Jeremy Howard]

Is there really a need to track whether a namespace is personal, shared, or
public? It doesn't really make much difference, does it?

I'll try and find some time soon to mock up the UI I was thinking of for
handling namespaces better. In general I certainly agree that the current
situation is messy and needs to be cleaned up, and that Server Directory and
Namespace are to concepts that need to be consolidated (indeed, the IMAP RFC has
no concept of "Server Directory").

Comment 4

[Michael Klose]

Adding the type of namespace would give a really nice tree in my opinion.  And if we only have one namespace under a namespace type, like I said, I would set the "server directory" for this namespace automatically.   This way, all personal folders regardless of server typ would appear maybe under "Personal folders" (coded as /personal), all shared folders (if exists) under "Shared folders" (coded as /shared) and so on. The tree will look the same no matter what type of server you are using.  Namespace type necessary? No, not really. Also if we have the namespace we can actually find out what type of namespace it is pretty simply. The reason I want to have it is just to construct a nicer tree and together with the "server directory" (set automatically) for each namespace, then the tree will look the same no matter what IMAP server you are using.  I know there is no server director defined in any RFC. But is it a bad thing? The reason I still want it is so that in 99% of all cases we can have /personal/Sent as a folder no matter what type of server we are using. /personal/Sent would be expanded to "/personal/INBOX/Sent" because of the (99% automatically) set Server directory (for this type of namespace), and this would be "expanded" to "INBOX.Sent".  I may hack up some code for this next week to illustrate what I mean.   

Comment 5

[Michael Klose]

Change the name in my posts of "server directories" to "namespace selection"  
and you get more what I mean. 
 
They are used to specify which namespace(s) we are interested in. If there is 
only one, then the namespace is removed from the canonical folder string for 
this type of namespace. To do this absolutely requires having the type of 
namespace in the canonical diectory name so we can do a reconstruction. 
 
For this, the GUI could be left as it is (although we could and should remove 
the server directory).  
 
Call this "namespace selection", keep them in server "namespace" and then we 
are back to where the RFC is and we have removed the concept sort of of a 
server directory totally. 
 
What I aim at is the following:  
 
1. automatic configuration but manual possible 
2. canonical directory names the same on just about every type of server (not 
possible e.g. when we have multiple personal namespaces. Currently not really 
supported at all, but this concept will support them).  
 
2 is the reason canonical names are used at all. 
 
I know I am developing this idea from post to post. I now think this is an 
idea we could work with. 
 
I like thinking things out before I start coding. 
 
Thoughts, suggestions, is this all total b&ll s&%t? 
 
I'd like to start coding this, but I won't do so until I've heard some 
feedback. This is actually something I could implement pretty quickly. I know 
where to make the changes to support this. 
 

Comment 6

[Henry Jia]

I think we can use the canonical format like this "/namespace/folder" for an
IMAP folder. And thus "imap://username@host/namespace/folder" for an IMAP URI.

Now, I agree with Michael Klose to not to use "//" between namespace and folder.
It's mainly for a protocol. We can just use "/".

I think we can omit the "namespacetype". Because this can be easily gotten if we
need it.

Marking as new and change to 'Normal'.

Comment 7

[philip zhao]

NAMESPACE command gives us what the server suports. The only useness of this
command is to let us have a record for the server's namespacetype and it's prefs.
We can use them later then.
Compared with  server directory, namespace gives us more info about the server.
While server directory is only mannully created and only helps us manage the
directory tree more easily.(For example,to show the tree in a different format
like.) 

Comment 8

[philip zhao]

Henry, if the IMAP URI were in the format
"imap://username@host/namespace/folder" as you mentioned comment#6, what will we
do with the problem in bug #71383 (where 
namespace is "Shared Folders/User/")?

Comment 9

[Henry Jia]

'/' is the cononical seperator, server side's '/' seperator should be changed to
'^' when it is translated to cononical format.

Comment 10

[philip zhao]

Mozilla keep canonical formated IMAP folder name instead of real folder name
now. When we set IMAP server directory in advanced server setting, the canonical
formated folder name will lost some infomation. For example, the exact heirarchy
level of the folder in server and the folder's namespace prefix. 

Mozilla use canonical formated folder name to display folder pane and recreate
the tree structure at client side. 

This means that when we set IMAP server directory, the structure of folders will
be different with the structure in server. This brings us many trouble.

When we create a new folder, mozilla again only keep the not-real folder name
(which may have lost whole path infomations of the folder) into the folder DB. 

And because mozilla has used canonical formated folder name, we can't get the
realname of folder after mozilla has lsubed the folders. This cause many errers
on namespace.

Thus, we need two kinds of folder name in mozilla now. 
1, canonical folder name which can be used to create and display folder pane.
2, real canonical folder name which can be used to do create, rename, and other
intercomunicate with server actions.

Comment 11

[philip zhao]

Now we know that mozilla will convert the realName of a folder into its
canonical formate, then only keeps the 'allocatedPathName' of folder.

This means that if a folder 'INBOX.a.b' on server side, when it came to
mozilla's side, mozilla will convert and keep only 'a/b'.

If I want to keep the realName('INBOX.a.b') of the folder in mozilla and use it
later when needed. Where should I change the code? Who can give me some
suggestions, Thx!:-)

Comment 12

[philip zhao]

We sure need to keep namespace prefix in url in order to know which namespace a
folder are in!

Comment 13

[Wayne Mery (:wsmwk)]

There has been recent discussion of URI handling, but I don't remember where.  Did that include URI format?

Comment 14

[Jorg K (CEST = GMT+2)]

(In reply to Wayne Mery (:wsmwk) from comment #13)
> There has been recent discussion of URI handling, ...
You're speaking riddles. Almost everything in TB is addressed via URLs/URIs:
Messages in folders, attachments in messages, protocol access, even via SMTP or POP3. So we're talking about URLs *all the time*.

Looks like this bug from 2002 is about changing the format of IMAP URLs. Without reading any of it, that looks like a WONTFIX to me after 15 years on the books. We surely won't rework IMAP URLs now, maybe in TB:NG.

Comment 15

[Wayne Mery (:wsmwk)]

(In reply to Jorg K (GMT+1) from comment #14)
> (In reply to Wayne Mery (:wsmwk) from comment #13)
> > There has been recent discussion of URI handling, ...
> You're speaking riddles. Almost everything in TB is addressed via URLs/URIs:
> Messages in folders, attachments in messages, protocol access, even via SMTP
> or POP3. So we're talking about URLs *all the time*.

Certainly. Sorry not able to be more specific because I don't remember the context of what I was reading.  SWAG maybe it was bug 922464. But I have conflated that with IMAP urls.

 
> Looks like this bug from 2002 is about changing the format of IMAP URLs.
> Without reading any of it, that looks like a WONTFIX to me after 15 years on
> the books. We surely won't rework IMAP URLs now, maybe in TB:NG.

Perhaps gene or magnus will have some perspective about the usefulness of this

Comment 16

[Magnus Melin [:mkmelin]]

I think it would be useful in general, but it's not a pressing matter so would need someone to step up to do the work. Doesn't sound like very high gain for what would probably be a lot of work.

Comment 17

[gene smith]

Not sure I really understand what the original complaint here is.

In my recent work with namespaces, I think the main user complaint is "why doesn't my shared folder appear". This has recently been fixed for at least one user.

There are two remaining small bug that I have noticed while working with namespaces. 

Suppose you have multiple namespaces in a given category, e.g., two "personal namespaces" called INBOX and a-bear and INBOX is the "default" namespace (lists first in the imap namespace command response). New folders appear under INBOX when created in tb at the account level without a prefix. If you look in the advanced server setting under personal namespaces you will see:

"a-bear.","INBOX."

This kind of implies that a-bear is default and not INBOX. The namespaces that show here should be in the same order as the imap namespace command returns, i.e.,

"INBOX.","a-bear."

The same order reversal occurs for user and shared namespaces if there is more than one of a type.

The other very small bug is that the order shown on the advanced server page for "Public (shared)" and "Other Users" is in the opposite order from the actual imap namespace command response. Also, the naming for "Public (shared)" doesn't quite match how it is described in the namespace rfc2342. So, I think the page should show like this to match the order and terminology of the rfc:

Personal Namespace:
Other Users' Namespace:          now: Public (shared):
Shared (Public) Namespace:       now: Other Users:

Of course, if anyone wants these namespace changes it should be written up as a new bug.