557718 - Make folders of the personal namespace appear as siblings of the INBOX

Status: REOPENED

(MailNews Core :: Networking: IMAP, enhancement)

Description

[Ben Bucksch (:BenB)]

I have many accounts where the personal folders appear as children of the Inbox. I don't like that, they should be siblings. That's what the IMAP NAMESPACE extension solves. We support that. But it doesn't seem to work, at least it doesn't have the desired effect.

Reproduction:
1. Create fastmail.fm account. Create corresponding account in Thunderbird.
2. Open the account, look at folder pane
3. Go to Account Settings | Server Settings | Advanced...
4. Check "Personal Namespace" setting.
5. Change "IMAP server directory" to "INBOX."
6. Restart Thunderbird, open account again, look at folder pane again.

Expected result:
In step 2: "Trash" is on same level as "Inbox"
In step 6: ditto
In step 4: "Personal Namespace" = "INBOX."

Actual result:
In step 2: "Trash" is a child of "Inbox"
In step 6: "Trash" is on same level as "Inbox"
In step 4: "Personal Namespace" = "INBOX."

Server:
Supports NAMESPACE extension and reports:
* NAMESPACE (("INBOX." ".")) (("user." ".")) (("" "."))
(Complete chat below)

Diagnosis:
So, the server properly tells us about the user's personal namespace, and out UI says the same, we apparently just don't use it.

----

$ netcat mail.messagingengine.com 143
* OK IMAP4 ready
1 capability
* CAPABILITY IMAP4 IMAP4rev1 LITERAL+ COMPRESS=DEFLATE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE CATENATE CONDSTORE SCAN XLIST DIGEST=SHA1 IDLE URLAUTH SASL-IR AUTH=PLAIN STARTTLS
1 OK completed
1 login [e-mail] password
1 OK User logged in SESSIONID=<...>
1 namespace
* NAMESPACE (("INBOX." ".")) (("user." ".")) (("" "."))
1 OK Completed

Comment 1

[David :Bienvenu]

adjusting summary.

We could either do this in the front end, i.e., the folder pane, or we could change the URI's for the folders to strip out the INBOX. part. This would break existing profiles in fun ways, but would give us everything else for free.

The online server directory does the same thing, but I'd rather not use it, since I'd rather get rid of it, and just use the namespace as is...

Comment 2

[Ben Bucksch (:BenB)]

This is a dup of bug 80858, but not marking as such (or maybe other way around), because this bug here is more focused and has less comments.

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #1)
> The online server directory does the same thing, but I'd rather not use it,
> since I'd rather get rid of it, and just use the namespace as is...

I also prefer "[ ? ] Show namespace as root folder of children" like option(required mainly for diagnosis) to current Server Directory, because;
(1) I never want to analyze problem reported by user who intentionally sets "Server Directory: [Gmail]". (Bug 507200)
(2) If such option, it can be extended to other than personl namespace.
And, if possible, I expect feature like;
(a) folder of namespace(usually Inbox) is shown as root or not at UI depends on above option setting only,
(b) internal setting in pref.js etc. is always full real IMAP mbox name(with decoded version of modified utf-7 if required),
for ease of diagnosis, and to avoid problem like bug 480393, and useless Bug 511331(I opened based on Bug 507200 and Bug 508838).

Comment 4

[WADA:World Anti-bad-Duping Agency]

Does NAMESPACE extension state "IMAP client MUST hide namespace any time, any where, when he shows IMAP folder to client user"? AFAIR, it's not, and there really is manual way to hide it - IMAP server directory:, and it works even if it's not as designed or as intended. So, I think enhancement of "auto-hide if namespace is specified". Is it wrong?

Comment 5

[David :Bienvenu]

(In reply to comment #4)
> Does NAMESPACE extension state "IMAP client MUST hide namespace any time
No, not at all. That's just the opinion of the bug filer :-) But I agree it's something we should do by default for the personal namespace of Inbox.

Comment 6

[WADA:World Anti-bad-Duping Agency]

I've opened bug 567705 for "remove UI of IMAP server directory:" part only. xref for ease of searching and tracking.

Comment 7

[WADA:World Anti-bad-Duping Agency]

Developers already know about/are aware of issues you stated by this bug in bug 80858. I don't think increasing number of bugs at B.M.O by opening dup bug for already known problem for your satisfaction. So, closing as DUP of bug 80858.

To bug opener:
If you believe your this bug should superceed already known bug 80858, please re-open this bug with duping already known bug 80858 to your this bug.
Please never increase dup bugs at B.M.O, even though you know that your complaint is same as existent bug report at B.M.O.

Comment 9

[WADA:World Anti-bad-Duping Agency]

Correction. I wanted to say;

I don't think "increasing number of bugs at B.M.O by opening dup bug for
already known problem for his satisfaction" is not correct action at B.M.O. It merely increases workload of QA team/triage team, and it merely forces contributers in QA area/Triage area to waste their valuable time.

Comment 10

[Ben Bucksch (:BenB)]

WADA, unlike bug 80858, this bug was clearly focused and had a good chance of being fixed.
I leave it up to bienvenu which bug to use, but I think this one should be fixed soon (i.e. ship in 3.2).

Comment 11

[Ben Bucksch (:BenB)]

bienvenu in comment 1 wrote:
> We could either do this in the front end, i.e., the folder pane, or we could
> change the URI's for the folders to strip out the INBOX. part. This would break
> existing profiles in fun ways, but would give us everything else for free.

bienvenu, which option do you prefer?
And can you point me to the relevant code, and how you would change it?
I might give it a whack myself. It's irritating for new users to have "Sent" as subfolder of Inbox.

Comment 12

[Rob Mueller]

I'm one of the developers at Fastmail. As an FYI, the server we use is cyrus, which is a pretty common open source one. It defaults to "all user folders are sub-folders of INBOX" mode. It uses the "user." namespace for shared folder from other users, and the "" namespace for global shared mailboxes.

It has an "altnamespace" option that does all the promotion internally, and re-orders the namespaces. We've got that running on port 142 and 992 (ssl) for users if they want. In that mode, the personal namespace becomes "", the shared user folders go in "user." and the global shared mailboxes (which we don't use in our setup) into "Other Folders.".

. namespace
* NAMESPACE (("" ".")) (("user." ".")) (("Other Folders." "."))

While this would be a nicer default for our setup, unfortunately we can't just enable that on 143/993, because it'll confuse all existing IMAP clients that users have already setup. Massive support headache.

Internally our webui does the promotion itself, so in the web interface, all the sub-folders of INBOX look like siblings. Users like this.

From an implementation points of view, ideally I think the best way of doing this is to have a display hierarchy, and for each node, have the actual server + folder name as a completely separate thing. Then how you map server folder names to display hierarchy/names is decoupled. Unfortunately our webui doesn't do this, so it's a little messy, but if I were to do it again, I'd go this route. It means we can't support shared top level folders at all at the moment, and special case the "user." namespace as well.

One other evil annoyance to be aware of. In IMAP, "inbox" is special. It must exist and is case insensitive. eg. INBOX = Inbox = inbox. But sub-folders of inbox aren't, so INBOX.Inbox != INBOX.INBOX.

In Fastmail, users can create sub-folders of their Inbox. On the server, that creates folders named INBOX.Inbox.Name. Note that's INBOX.Inbox.Name, not INBOX.INBOX.Name. That's because our normalisation goes:

INBOX           -> Inbox
INBOX.Inbox.Abc -> Inbox.Abc
INBOX.Xyz       -> Xyz
INBOX.Xyz.Abc   -> Xyz.Abc
user.other      -> user.other

So that all ends up displaying nicely for the user and mapping to sane folder names on the server.

Comment 13

[Andrew Sutherland [:asuth] (he/him)]

For further fastmail-related context, fastmail hosts a number of non-HTTPS autoconfig.* domains that tell Thunderbird to use port 992 and thereby the "altnamespace" option and I've just now landed the messagingengine.com entry in the ISPDB on bug 683783 so MX lookups will also pick up that configuration.  (The autoconfig will not re-check self-hosted domains after performing an MX lookup, just the ISPDB, so the ISPDB effectively must host all entries we want MX lookups to work for.)