476260 - support for IMAP XLIST- not necessary to configure common folder names

Status: VERIFIED FIXED

(MailNews Core :: Networking: IMAP, enhancement, P2)

Description

[Frantisek Boranek]

User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2008121717 Mandriva/1.9.0.5-0.1mdv2009.0 (2009.0) Firefox/3.0.5
Build Identifier: verze 2.0.0.19 (20090114) 

There are no well-know name for common folders such as Drafts, Trash, Spam, ... on IMAP server. User has to manually configure which folder is used for what. If user doesn't do it IMAP client create folder with his favorite names on server. It causes duplicated/multiplicated folders on server e.g. Deleted Items / Deleted Messages / Trash. The problem is even worse when you use localized version of IMAP client.

Google and Apple developed special IMAP command XLIST to address this issue. XLIST command is supported on Gmail and iPhone.

IMAP XLIST command return list of folder and their type (\Inbox, \Drafts, \Trash, \Sent, \Spam).
Therefore, IMAP client gets information about folder type from server automatically. User do not need to configure it.

Common folder names can be also localized on server side. User can see the same name on any clients.

Support of this extension would improve user experience in using Thunderbird with Gmail.

Reproducible: Always




See: 
[frido@frido ~]$ telnet-ssl -z ssl imap.gmail.com 993
Trying 64.233.183.111...
Connected to gmail-imap.l.google.com.
Escape character is '^]'.
* OK Gimap ready for requests from 81.201.58.35 f4if1005730nfh.69
1 capability
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA XLIST CHILDREN XYZZY
1 OK Thats all she wrote! d2if946932nfc.47
2 login name ******
2 OK [e-mail] authenticated (Success)
3 xlist "" "%"
* XLIST (\HasNoChildren \Inbox) "/" "Doru&AQ0-en&AOE- po&AWE-ta"
* XLIST (\HasChildren) "/" "Trash"
* XLIST (\Noselect \HasChildren) "/" "[Gmail]"
3 OK Success
4 xlist "" "%/%"
* XLIST (\HasChildren) "/" "Trash/ff"
* XLIST (\HasNoChildren \Drafts) "/" "[Gmail]/Koncepty"
* XLIST (\HasNoChildren \Trash) "/" "[Gmail]/Ko&AWE-"
* XLIST (\HasNoChildren \Sent) "/" "[Gmail]/Odeslan&AOE- po&AWE-ta"
* XLIST (\HasNoChildren \Starred) "/" "[Gmail]/Ozna&AQ0-en&AOk-"
* XLIST (\HasNoChildren \Spam) "/" "[Gmail]/Spam"
* XLIST (\HasNoChildren \AllMail) "/" "[Gmail]/V&AWE-echny zpr&AOE-vy"
4 OK Success
5 logout
* BYE LOGOUT Requested
5 OK 73 good day (Success)
Connection closed by foreign host.

Comment 1

[Frantisek Boranek]

Folder 'Trash' created Thunderbird after then I deleted folder 'ff'.

Comment 2

[WADA:World Anti-bad-Duping Agency]

Following is Brian Kennelly's comments relates to XLIST of Gmail(found by Google search for "IMAP XLIST").
> http://markmail.org/message/fxasmdizupidc4bk#query:IMAP%20XLIST+page:1+mid:akdd6lrgnen7eyb5+state:results
> http://markmail.org/message/fxasmdizupidc4bk#query:IMAP%20XLIST+page:1+mid:wql6f73ylsj635cf+state:results
He kindly posted Bug 467860 Comment #25 for us, providing idea of solution by XLIST command.
However, I couldn't find RFC for XLIST extension.
Frantisek Boranek(bug openr), do you know RFC number? If RFC for XLIST exists, is it already "Standards Track" status?

Comment 3

[Frantisek Boranek]

Unfortunately, I couldn't find specification too. I have experience with IPhone and its works well with gmail. I try find specification on google and apple, noting write about this good idea.

Comment 4

[WADA:World Anti-bad-Duping Agency]

FYI. See Brian Kennelly's Bug 467860 Comment #31 for his idea of new "picker mode" with XSLT extension of Gmail.

Comment 5

[Nikolay Shopik]

rfc3501 7.2.1. clearly says - Capability names MUST either begin with "X" or be standard or standards-track IMAP4rev1 extensions, revisions, or amendments registered with IANA.
So XLIST isn't any RFC yet.

Comment 6

[WADA:World Anti-bad-Duping Agency]

Thanks for pointing RFC 3501 : http://www.faqs.org/rfcs/rfc3501.html
RFC 7.2.1 said also;
> A server MUST NOT offer unregistered or non-standard capability names,
> unless such names are prefixed with an "X".
So XLIST is RFC 3501 compliant capability name.

I could find next two documents only by search for "XLIST" at Gmail Help. Brian Kennelly's comments I pointed in Comment #2 look to be copy of these Gmail Help Discussions articles.
> Help Discussions > POP and IMAP > Writing IMAP client for Gmail Options
> http://groups.google.com/group/Gmail-Help-POP-and-IMAP-en/browse_thread/thread/92a57e77fa2867b0
> Help Discussions > POP and IMAP > Language independent IMAP folder names
> http://groups.google.com/group/Gmail-Help-POP-and-IMAP-en/browse_thread/thread/a154105c54f020fb?pli=1

Basic back end support of XLIST(issue XLIST and parse its response) doesn't seem to be difficult, even if there is no official document/spec of Gmail's XLIST. But it seems that design of new "picker mode" and related UI will be tough work. I feel implementation will probably be much tough work, if implementation of full "picker mode" is requested. I think initial implementation is better to be feature like "Gmail IMAP account definition assistance". It can be implemented as add-on/extension. (I hope Google will provide extension at Gmail Lab... :-) )

Comment 7

[Nikolay Shopik]

Bug 467860 discus very same problem but just for drafts, shouldn't we dupe to it? Marking it as NEW for now since its discus XLIST feature. I'm personally searching for such extension like for years and still wondering understand why its not went into RFC 3501 as standard part of IMAP like INBOX before. Hope XLIST make into standards track. Probably many who used exchange+outlook would die to have this feature in TB, currently only Apple iPhone supported AFAIK.

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #7)
> shouldn't we dupe to it?
Many issues are found by Bug 467860. No separate request for XLIST until this bug is simply a result that no one who knows Bug 467860 opened separate bug for specific request of XLIST support ( I'm a "no one" :-) )

Comment 9

[Wayne Mery (:wsmwk)]

there is also Bug 467735

Comment 10

[WADA:World Anti-bad-Duping Agency]

(Off-Topic)
(In reply to comment #9)
> there is also Bug 467735
Wayne Mery, please read Bug 467735 Comment #7 and comments of Bug 467860 pointed by Bug 467735 Comment #7. It's simply one of many issues discovered in Bug 467860, for which no one(who knows Bug 467860) opened separate bug for the problem yet ( favorably speaking, problem analysis in Bug 467735 is still in progress :-) ).

Comment 11

[Magnus Melin [:mkmelin]]

Sounds very useful...

Comment 12

[Nikolay Shopik]

I've been spoke to Dovecot developer this extension also will be added in some future versions of Dovecot.

Comment 13

[David :Bienvenu]

yes, we should definitely support this. One tricky part is that we don't want to override the user if the user has overridden the default settings. But first I need to write the code to detect the xlist capability, issue xlist and store those flags.

We may want a hidden pref to turn off XLIST, since it's always a little risky to not have an escape hatch for new server features.

Comment 14

[David :Bienvenu]

From what I can tell, Kerio also supports this.

I've moved this to blocking, since it's really the only way we're going to reliably recognize IMAP special folders for GMail across languages, since GMail switches the imap folder names based on what language preference you set. It also allows us to find the All Mail folder, which can be handy.

I think we want to force the use of XLIST with Gmail accounts, which most likely means turning off subscription by default for GMail accounts.

Comment 15

[Brian Kennelly]

I am not certain why using XLIST requires turning off subscription.  At worst, it would mean subscribing to the special folders.

It will still be necessary to fall back on reasonable defaults, because, with the Advanced IMAP Controls lab in Gmail, the users can hide any of the special folders from IMAP.

Comment 16

[David :Bienvenu]

(In reply to comment #15)
> I am not certain why using XLIST requires turning off subscription.  At worst,
> it would mean subscribing to the special folders.

I mean turning off the mode where we only show you subscribed folders. You can subscribe/unsubscribe folders at will, but we'll show you all folders. I'd rather just use XLIST instead of LIST, when its supported, and if we turn off the use of subscription, we'll use LIST/XLIST instead of LSUB. 

Technically, we could use LSUB and XLIST together, but it complicates the code more than I want right now, and if we're doing XLIST anyway, the performance advantages of LSUB are lost, and we're left with the very geeky desire to have folders that exist but aren't displayed.  I don't think a lot of gmail users need that, and if they do, they can turn off subscription.

> 
> It will still be necessary to fall back on reasonable defaults, because, with
> the Advanced IMAP Controls lab in Gmail, the users can hide any of the special
> folders from IMAP.

All I plan to do is if Gmail tells us about special folders, we'll use those. If the user has hidden them from IMAP, IMAP won't tell us about them.

Comment 17

[David :Bienvenu]

Attachment: wip

this makes us issue an XLIST and parse XLIST responses, and set folder flags. I still have to deal with setting the special folder uri prefs, and decide what to do with existing profiles (e.g., potentially different Trash folder, different Archives folder, etc).

Comment 18

[David :Bienvenu]

Attachment: proposed fix

this adds support for xlist. If the server has xlist capabilities, we issue it. We try to set up special folders to use the xlist specified folder, if the current pref folder doesn't exist. And we set trash to be the xlist trash, and clear the trash flag on the old trash folder.

This also adds an attribute that we can check to see if an imap server is a gmail server, which we'll need for other stuff.

Comment 19

[[e-mail]]

Comment on attachment 375744 [details] [diff] [review]
proposed fix

>-    identity->GetStationeryFolder(folderUri);
>-    if (!folderUri.IsEmpty() && NS_SUCCEEDED(rdf->GetResource(folderUri, getter_AddRefs(res))))
>-    {
>-      folder = do_QueryInterface(res, &rv);
>-      if (NS_SUCCEEDED(rv))
>-        rv = folder->SetFlag(nsMsgFolderFlags::Templates);
>-    }
What happened to this code?

>+#define kImapTrash   0x10 /* Navigator flag */
>+#define kJustExpunged 0x20 /* This update is a post expunge url update. */
>+#define kPersonalMailbox 0x40 /* this mailbox is in the personal namespace */
>+#define kPublicMailbox 0x80 /* this mailbox is in the public namespace */
>+#define kOtherUsersMailbox 0x100 /* this mailbox is in the other users' namespace */
>+#define kNameSpace 0x200 /* this mailbox IS a namespace */
>+#define kNewlyCreatedFolder 0x400 /* this folder was just created */
>+#define kImapDrafts 0x800 /* XLIST says this is the drafts folder */
>+#define kImapSpam 0x1000 /* XLIST says this is the spam folder */
>+#define kImapSent 0x2000 /* XLIST says this is the sent folder */
>+#define kImapInbox 0x4000 /* XLIST says this is the INBOX folder */
>+#define kImapAllMail 0x8000 /* XLIST says this is AllMail (GMail) */
>+#define kImapXListTrash 0x10000 /* XLIST says this is the trash */
So, what's the difference between kImapTrash and kImapXListTrash?
Nit: you use a slightly different order for each list of common folders ;-)

Comment 20

[David :Bienvenu]

(In reply to comment #19)
> (From update of attachment 375744 [details] [diff] [review])
> >-    identity->GetStationeryFolder(folderUri);
> >-    if (!folderUri.IsEmpty() && NS_SUCCEEDED(rdf->GetResource(folderUri, getter_AddRefs(res))))
> >-    {
> >-      folder = do_QueryInterface(res, &rv);
> >-      if (NS_SUCCEEDED(rv))
> >-        rv = folder->SetFlag(nsMsgFolderFlags::Templates);
> >-    }
> What happened to this code?

Very good catch - I had replaced it with the code that checked for an XLIST folder first, but then removed that code because XLIST doesn't specify a templates folder. I'll put it back. 



> So, what's the difference between kImapTrash and kImapXListTrash?

kImapTrash is the flag the imap protocol code sets when it matches the trash name; The XList flavor is when XList says a folder is the trash. I use the distinction to favor the xlist specified folder.

> Nit: you use a slightly different order for each list of common folders ;-)

Comment 21

[David :Bienvenu]

p2, since I need this for gmail-specific archive. Adding blocked bug.

Comment 22

[David :Bienvenu]

Attachment: address Neil's comment about removed code.

Comment 23

[Frantisek Boranek]

I am developer of Kerio Mailserver and I added support of XLIST to our product now. I am not sure if this code will recognize XLIST capability of our mailserver. See:
+              PRBool clearFlag;
+              if (isGMailServer)
+              {
+               ...

Comment 24

[David :Bienvenu]

that code is to remove the trash flag on gmail accounts where we've already created a trash folder but we'd rather use the trash folder specified by the server. So it only affects existing accounts - it's not central to the xlist support.

Comment 25

[[e-mail]]

(In reply to comment #20)
> Very good catch - I had replaced it with the code that checked for an XLIST
> folder first, but then removed that code because XLIST doesn't specify a
> templates folder. I'll put it back.
Maybe XLIST should specify a templates folder too ;-)

> > So, what's the difference between kImapTrash and kImapXListTrash?
> kImapTrash is the flag the imap protocol code sets when it matches the trash
> name; The XList flavor is when XList says a folder is the trash. I use the
> distinction to favor the xlist specified folder.
Out of interest, which do we see first, or does it vary? Why doesn't the problem occur with, say, the Drafts folder?

Comment 26

[David :Bienvenu]

(In reply to comment #25)
> Maybe XLIST should specify a templates folder too ;-)

I need to talk to the gmail folks about this - they're open to using a more standard approach to this whole issue.

> 
> Out of interest, which do we see first, or does it vary? Why doesn't the
> problem occur with, say, the Drafts folder?

We see our trash folder first, then GMail's. But we do the cleanup when discovery is done.

The trash folder is special because we create it on first contact with an imap server. Drafts is much less of a problem because we default to a local drafts folder (iirc), and we don't create it until the user composes a message.  There are certainly gmail imap accounts out there that have a top-level drafts folder that we created, but I'm less worried about cleaning those up. And I think the existing auto-config files for gmail accounts (e.g., in the 2.0 new account wizard) specify all the other folders correctly. So it's a somewhat arbitrary decision on my part, but basically I felt more strongly about fixing the trash folder...and I have to clear the flag in order for the user to delete the duplicate trash folder.

Comment 27

[[e-mail]]

So, to clarify, the detection is:
1. Create a trash folder according to preferences
2. Look for other special folders according to preferences
3. If not found, look for them according to XList
4. If not found, create them lazily using preferences
5. If GMail, use the XList trash anyway, since everything else breaks

Comment 28

[David :Bienvenu]

(In reply to comment #27)
> So, to clarify, the detection is:
> 1. Create a trash folder according to preferences
> 2. Look for other special folders according to preferences
> 3. If not found, look for them according to XList
> 4. If not found, create them lazily using preferences
> 5. If GMail, use the XList trash anyway, since everything else breaks

Yes, with the caveat that we shouldn't be creating the Trash folder on gmail anymore, but even if we previously created one, we use the XLIST one.

Comment 29

[[e-mail]]

(In reply to comment #28)
> we shouldn't be creating the Trash folder on gmail anymore
Sorry, I'm not following... earlier you said we create our Trash folder first.

Comment 30

[David :Bienvenu]

(In reply to comment #29)
> (In reply to comment #28)
> > we shouldn't be creating the Trash folder on gmail anymore
> Sorry, I'm not following... earlier you said we create our Trash folder first.

I meant that we create it before you ever delete a message, i.e., up front instead of lazily. After folder discovery is done, we check if there's a trash folder, and if not, we create one. Without XLIST support, if the user is running GMail in a non-english language, we won't recognize the GMail trash, and we'll create our own.

Comment 31

[[e-mail]]

(In reply to comment #30)
> After folder discovery is done,
This is going to be via XList if possible?
> we check if there's a trash folder, and if not, we create one.
So, won't we already have discovered the XList trash folder by now?

Comment 32

[David :Bienvenu]

(In reply to comment #31)
> (In reply to comment #30)
> > After folder discovery is done,
> This is going to be via XList if possible?
Yes.
> > we check if there's a trash folder, and if not, we create one.
> So, won't we already have discovered the XList trash folder by now?

Yes, we will, which is why, with XLIST support, we won't create an extra trash.

Comment 33

[[e-mail]]

So the extra Gmail check is just for existing Gmail IMAP users, because the Gmail trash is a special case and we really need to use it if it's there?

Comment 34

[David :Bienvenu]

(In reply to comment #33)
> So the extra Gmail check is just for existing Gmail IMAP users, because the
> Gmail trash is a special case and we really need to use it if it's there?

We really should use it, yes. And then the user can delete our trash once it no longer has the trash flag set.

Comment 35

[David :Bienvenu]

pinging for review; this is blocking an other b3 blocker...

Comment 36

[[e-mail]]

Comment on attachment 375817 [details] [diff] [review]
address Neil's comment about removed code.

OK, sr=me given that you comment that you're using a separate flag for XList trash so that you can fix existing GMail users.

Comment 37

[David :Bienvenu]

I'll add something like this:

/**
 * We use a separate xlist trash flag so we can prefer the GMail trash
 * over an existing Trash folder we may have created.
 */

Comment 38

[Mark Banner (:standard8)]

Comment on attachment 375817 [details] [diff] [review]
address Neil's comment about removed code.

>+    // GMail gives us a localized name for the inbox but doesn't let 
>+    // us select that localized name.
>+      if (boxFlags & kImapInbox)
>+        imapFolder->SetOnlineName(NS_LITERAL_CSTRING("INBOX"));
>+      else if (onlineName.IsEmpty() || !onlineName.Equals(dupFolderPath))
>         imapFolder->SetOnlineName(dupFolderPath);

nit: incorrect comment indentation.

Looks good and works nicely. r=Standard8

Comment 39

[David :Bienvenu]

fix checked in, with nits addressed.

Comment 40

[Nikolay Shopik]

This works almost flawlessly on gmail. I've changed language to Russian from English recreate account and every folder except trash take correct icon. I've changed back to english w/o creating account and it get correct icons for all folders except trash.
Is this expected behavior for trash?

Comment 41

[[e-mail]]

I have a nasty feeling that the GMail trash correction feature only works on accounts created using older versions of Thunderbird, not on language changes.

Comment 42

[Nikolay Shopik]

(In reply to comment #41)
> I have a nasty feeling that the GMail trash correction feature only works on
> accounts created using older versions of Thunderbird, not on language changes.

How older, before patch? And what reason behind this?

Comment 43

[Nikolay Shopik]

I was wrong Trash folder is working but I had to restart TB to see proper icon.

Comment 45

[Ben Bucksch (:BenB)]

RFC 6154 has been published, see bug 558659 (we morphed that bug to it).