support for IMAP XLIST- not necessary to configure common folder names

VERIFIED FIXED in Thunderbird 3.0b3

Status

enhancement
P2
normal
VERIFIED FIXED
11 years ago
6 years ago

People

(Reporter: fboranek, Assigned: Bienvenu)

Tracking

(Blocks 1 bug, {intl})

unspecified
Thunderbird 3.0b3
Dependency tree / graph
Bug Flags:
blocking-thunderbird3 +
wanted-thunderbird3 +

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 2 obsolete attachments)

Reporter

Description

11 years ago
User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2008121717 Mandriva/1.9.0.5-0.1mdv2009.0 (2009.0) Firefox/3.0.5
Build Identifier: verze 2.0.0.19 (20090114) 

There are no well-know name for common folders such as Drafts, Trash, Spam, ... on IMAP server. User has to manually configure which folder is used for what. If user doesn't do it IMAP client create folder with his favorite names on server. It causes duplicated/multiplicated folders on server e.g. Deleted Items / Deleted Messages / Trash. The problem is even worse when you use localized version of IMAP client.

Google and Apple developed special IMAP command XLIST to address this issue. XLIST command is supported on Gmail and iPhone.

IMAP XLIST command return list of folder and their type (\Inbox, \Drafts, \Trash, \Sent, \Spam).
Therefore, IMAP client gets information about folder type from server automatically. User do not need to configure it.

Common folder names can be also localized on server side. User can see the same name on any clients.

Support of this extension would improve user experience in using Thunderbird with Gmail.

Reproducible: Always




See: 
[frido@frido ~]$ telnet-ssl -z ssl imap.gmail.com 993
Trying 64.233.183.111...
Connected to gmail-imap.l.google.com.
Escape character is '^]'.
* OK Gimap ready for requests from 81.201.58.35 f4if1005730nfh.69
1 capability
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA XLIST CHILDREN XYZZY
1 OK Thats all she wrote! d2if946932nfc.47
2 login name ******
2 OK name@gmail.com authenticated (Success)
3 xlist "" "%"
* XLIST (\HasNoChildren \Inbox) "/" "Doru&AQ0-en&AOE- po&AWE-ta"
* XLIST (\HasChildren) "/" "Trash"
* XLIST (\Noselect \HasChildren) "/" "[Gmail]"
3 OK Success
4 xlist "" "%/%"
* XLIST (\HasChildren) "/" "Trash/ff"
* XLIST (\HasNoChildren \Drafts) "/" "[Gmail]/Koncepty"
* XLIST (\HasNoChildren \Trash) "/" "[Gmail]/Ko&AWE-"
* XLIST (\HasNoChildren \Sent) "/" "[Gmail]/Odeslan&AOE- po&AWE-ta"
* XLIST (\HasNoChildren \Starred) "/" "[Gmail]/Ozna&AQ0-en&AOk-"
* XLIST (\HasNoChildren \Spam) "/" "[Gmail]/Spam"
* XLIST (\HasNoChildren \AllMail) "/" "[Gmail]/V&AWE-echny zpr&AOE-vy"
4 OK Success
5 logout
* BYE LOGOUT Requested
5 OK 73 good day (Success)
Connection closed by foreign host.
Reporter

Comment 1

11 years ago
Folder 'Trash' created Thunderbird after then I deleted folder 'ff'.
Following is Brian Kennelly's comments relates to XLIST of Gmail(found by Google search for "IMAP XLIST").
> http://markmail.org/message/fxasmdizupidc4bk#query:IMAP%20XLIST+page:1+mid:akdd6lrgnen7eyb5+state:results
> http://markmail.org/message/fxasmdizupidc4bk#query:IMAP%20XLIST+page:1+mid:wql6f73ylsj635cf+state:results
He kindly posted Bug 467860 Comment #25 for us, providing idea of solution by XLIST command.
However, I couldn't find RFC for XLIST extension.
Frantisek Boranek(bug openr), do you know RFC number? If RFC for XLIST exists, is it already "Standards Track" status?
Reporter

Comment 3

10 years ago
Unfortunately, I couldn't find specification too. I have experience with IPhone and its works well with gmail. I try find specification on google and apple, noting write about this good idea.
FYI. See Brian Kennelly's Bug 467860 Comment #31 for his idea of new "picker mode" with XSLT extension of Gmail.

Comment 5

10 years ago
rfc3501 7.2.1. clearly says - Capability names MUST either begin with "X" or be standard or standards-track IMAP4rev1 extensions, revisions, or amendments registered with IANA.
So XLIST isn't any RFC yet.
Thanks for pointing RFC 3501 : http://www.faqs.org/rfcs/rfc3501.html
RFC 7.2.1 said also;
> A server MUST NOT offer unregistered or non-standard capability names,
> unless such names are prefixed with an "X".
So XLIST is RFC 3501 compliant capability name.

I could find next two documents only by search for "XLIST" at Gmail Help. Brian Kennelly's comments I pointed in Comment #2 look to be copy of these Gmail Help Discussions articles.
> Help Discussions > POP and IMAP > Writing IMAP client for Gmail Options
> http://groups.google.com/group/Gmail-Help-POP-and-IMAP-en/browse_thread/thread/92a57e77fa2867b0
> Help Discussions > POP and IMAP > Language independent IMAP folder names
> http://groups.google.com/group/Gmail-Help-POP-and-IMAP-en/browse_thread/thread/a154105c54f020fb?pli=1

Basic back end support of XLIST(issue XLIST and parse its response) doesn't seem to be difficult, even if there is no official document/spec of Gmail's XLIST. But it seems that design of new "picker mode" and related UI will be tough work. I feel implementation will probably be much tough work, if implementation of full "picker mode" is requested. I think initial implementation is better to be feature like "Gmail IMAP account definition assistance". It can be implemented as add-on/extension. (I hope Google will provide extension at Gmail Lab... :-) )

Comment 7

10 years ago
Bug 467860 discus very same problem but just for drafts, shouldn't we dupe to it? Marking it as NEW for now since its discus XLIST feature. I'm personally searching for such extension like for years and still wondering understand why its not went into RFC 3501 as standard part of IMAP like INBOX before. Hope XLIST make into standards track. Probably many who used exchange+outlook would die to have this feature in TB, currently only Apple iPhone supported AFAIK.
Status: UNCONFIRMED → NEW
Ever confirmed: true
(In reply to comment #7)
> shouldn't we dupe to it?
Many issues are found by Bug 467860. No separate request for XLIST until this bug is simply a result that no one who knows Bug 467860 opened separate bug for specific request of XLIST support ( I'm a "no one" :-) )
there is also Bug 467735
(Off-Topic)
(In reply to comment #9)
> there is also Bug 467735
Wayne Mery, please read Bug 467735 Comment #7 and comments of Bug 467860 pointed by Bug 467735 Comment #7. It's simply one of many issues discovered in Bug 467860, for which no one(who knows Bug 467860) opened separate bug for the problem yet ( favorably speaking, problem analysis in Bug 467735 is still in progress :-) ).

Updated

10 years ago
Component: General → Networking: IMAP
Product: Thunderbird → MailNews Core
QA Contact: general → networking.imap
Sounds very useful...
Flags: wanted-thunderbird3+

Comment 12

10 years ago
I've been spoke to Dovecot developer this extension also will be added in some future versions of Dovecot.
Assignee

Comment 13

10 years ago
yes, we should definitely support this. One tricky part is that we don't want to override the user if the user has overridden the default settings. But first I need to write the code to detect the xlist capability, issue xlist and store those flags.

We may want a hidden pref to turn off XLIST, since it's always a little risky to not have an escape hatch for new server features.
Assignee: nobody → bienvenu
Status: NEW → ASSIGNED
Target Milestone: --- → Thunderbird 3.0b3
Assignee

Comment 14

10 years ago
From what I can tell, Kerio also supports this.

I've moved this to blocking, since it's really the only way we're going to reliably recognize IMAP special folders for GMail across languages, since GMail switches the imap folder names based on what language preference you set. It also allows us to find the All Mail folder, which can be handy.

I think we want to force the use of XLIST with Gmail accounts, which most likely means turning off subscription by default for GMail accounts.
Flags: blocking-thunderbird3+

Comment 15

10 years ago
I am not certain why using XLIST requires turning off subscription.  At worst, it would mean subscribing to the special folders.

It will still be necessary to fall back on reasonable defaults, because, with the Advanced IMAP Controls lab in Gmail, the users can hide any of the special folders from IMAP.
Assignee

Comment 16

10 years ago
(In reply to comment #15)
> I am not certain why using XLIST requires turning off subscription.  At worst,
> it would mean subscribing to the special folders.

I mean turning off the mode where we only show you subscribed folders. You can subscribe/unsubscribe folders at will, but we'll show you all folders. I'd rather just use XLIST instead of LIST, when its supported, and if we turn off the use of subscription, we'll use LIST/XLIST instead of LSUB. 

Technically, we could use LSUB and XLIST together, but it complicates the code more than I want right now, and if we're doing XLIST anyway, the performance advantages of LSUB are lost, and we're left with the very geeky desire to have folders that exist but aren't displayed.  I don't think a lot of gmail users need that, and if they do, they can turn off subscription.

> 
> It will still be necessary to fall back on reasonable defaults, because, with
> the Advanced IMAP Controls lab in Gmail, the users can hide any of the special
> folders from IMAP.

All I plan to do is if Gmail tells us about special folders, we'll use those. If the user has hidden them from IMAP, IMAP won't tell us about them.
Assignee

Comment 17

10 years ago
Posted patch wip (obsolete) — Splinter Review
this makes us issue an XLIST and parse XLIST responses, and set folder flags. I still have to deal with setting the special folder uri prefs, and decide what to do with existing profiles (e.g., potentially different Trash folder, different Archives folder, etc).
Assignee

Comment 18

10 years ago
Posted patch proposed fix (obsolete) — Splinter Review
this adds support for xlist. If the server has xlist capabilities, we issue it. We try to set up special folders to use the xlist specified folder, if the current pref folder doesn't exist. And we set trash to be the xlist trash, and clear the trash flag on the old trash folder.

This also adds an attribute that we can check to see if an imap server is a gmail server, which we'll need for other stuff.
Attachment #374757 - Attachment is obsolete: true
Attachment #375744 - Flags: superreview?(neil)
Attachment #375744 - Flags: review?(bugzilla)
Comment on attachment 375744 [details] [diff] [review]
proposed fix

>-    identity->GetStationeryFolder(folderUri);
>-    if (!folderUri.IsEmpty() && NS_SUCCEEDED(rdf->GetResource(folderUri, getter_AddRefs(res))))
>-    {
>-      folder = do_QueryInterface(res, &rv);
>-      if (NS_SUCCEEDED(rv))
>-        rv = folder->SetFlag(nsMsgFolderFlags::Templates);
>-    }
What happened to this code?

>+#define kImapTrash   0x10 /* Navigator flag */
>+#define kJustExpunged 0x20 /* This update is a post expunge url update. */
>+#define kPersonalMailbox 0x40 /* this mailbox is in the personal namespace */
>+#define kPublicMailbox 0x80 /* this mailbox is in the public namespace */
>+#define kOtherUsersMailbox 0x100 /* this mailbox is in the other users' namespace */
>+#define kNameSpace 0x200 /* this mailbox IS a namespace */
>+#define kNewlyCreatedFolder 0x400 /* this folder was just created */
>+#define kImapDrafts 0x800 /* XLIST says this is the drafts folder */
>+#define kImapSpam 0x1000 /* XLIST says this is the spam folder */
>+#define kImapSent 0x2000 /* XLIST says this is the sent folder */
>+#define kImapInbox 0x4000 /* XLIST says this is the INBOX folder */
>+#define kImapAllMail 0x8000 /* XLIST says this is AllMail (GMail) */
>+#define kImapXListTrash 0x10000 /* XLIST says this is the trash */
So, what's the difference between kImapTrash and kImapXListTrash?
Nit: you use a slightly different order for each list of common folders ;-)
Assignee

Comment 20

10 years ago
(In reply to comment #19)
> (From update of attachment 375744 [details] [diff] [review])
> >-    identity->GetStationeryFolder(folderUri);
> >-    if (!folderUri.IsEmpty() && NS_SUCCEEDED(rdf->GetResource(folderUri, getter_AddRefs(res))))
> >-    {
> >-      folder = do_QueryInterface(res, &rv);
> >-      if (NS_SUCCEEDED(rv))
> >-        rv = folder->SetFlag(nsMsgFolderFlags::Templates);
> >-    }
> What happened to this code?

Very good catch - I had replaced it with the code that checked for an XLIST folder first, but then removed that code because XLIST doesn't specify a templates folder. I'll put it back. 



> So, what's the difference between kImapTrash and kImapXListTrash?

kImapTrash is the flag the imap protocol code sets when it matches the trash name; The XList flavor is when XList says a folder is the trash. I use the distinction to favor the xlist specified folder.

> Nit: you use a slightly different order for each list of common folders ;-)
Assignee

Comment 21

10 years ago
p2, since I need this for gmail-specific archive. Adding blocked bug.
Blocks: 475852
Priority: -- → P2
Assignee

Comment 22

10 years ago
Attachment #375744 - Attachment is obsolete: true
Attachment #375817 - Flags: superreview?(neil)
Attachment #375817 - Flags: review?(bugzilla)
Attachment #375744 - Flags: superreview?(neil)
Attachment #375744 - Flags: review?(bugzilla)
Reporter

Comment 23

10 years ago
I am developer of Kerio Mailserver and I added support of XLIST to our product now. I am not sure if this code will recognize XLIST capability of our mailserver. See:
+              PRBool clearFlag;
+              if (isGMailServer)
+              {
+               ...
Assignee

Comment 24

10 years ago
that code is to remove the trash flag on gmail accounts where we've already created a trash folder but we'd rather use the trash folder specified by the server. So it only affects existing accounts - it's not central to the xlist support.
(In reply to comment #20)
> Very good catch - I had replaced it with the code that checked for an XLIST
> folder first, but then removed that code because XLIST doesn't specify a
> templates folder. I'll put it back.
Maybe XLIST should specify a templates folder too ;-)

> > So, what's the difference between kImapTrash and kImapXListTrash?
> kImapTrash is the flag the imap protocol code sets when it matches the trash
> name; The XList flavor is when XList says a folder is the trash. I use the
> distinction to favor the xlist specified folder.
Out of interest, which do we see first, or does it vary? Why doesn't the problem occur with, say, the Drafts folder?
Assignee

Comment 26

10 years ago
(In reply to comment #25)
> Maybe XLIST should specify a templates folder too ;-)

I need to talk to the gmail folks about this - they're open to using a more standard approach to this whole issue.

> 
> Out of interest, which do we see first, or does it vary? Why doesn't the
> problem occur with, say, the Drafts folder?

We see our trash folder first, then GMail's. But we do the cleanup when discovery is done.

The trash folder is special because we create it on first contact with an imap server. Drafts is much less of a problem because we default to a local drafts folder (iirc), and we don't create it until the user composes a message.  There are certainly gmail imap accounts out there that have a top-level drafts folder that we created, but I'm less worried about cleaning those up. And I think the existing auto-config files for gmail accounts (e.g., in the 2.0 new account wizard) specify all the other folders correctly. So it's a somewhat arbitrary decision on my part, but basically I felt more strongly about fixing the trash folder...and I have to clear the flag in order for the user to delete the duplicate trash folder.
So, to clarify, the detection is:
1. Create a trash folder according to preferences
2. Look for other special folders according to preferences
3. If not found, look for them according to XList
4. If not found, create them lazily using preferences
5. If GMail, use the XList trash anyway, since everything else breaks
Assignee

Comment 28

10 years ago
(In reply to comment #27)
> So, to clarify, the detection is:
> 1. Create a trash folder according to preferences
> 2. Look for other special folders according to preferences
> 3. If not found, look for them according to XList
> 4. If not found, create them lazily using preferences
> 5. If GMail, use the XList trash anyway, since everything else breaks

Yes, with the caveat that we shouldn't be creating the Trash folder on gmail anymore, but even if we previously created one, we use the XLIST one.
(In reply to comment #28)
> we shouldn't be creating the Trash folder on gmail anymore
Sorry, I'm not following... earlier you said we create our Trash folder first.
Assignee

Comment 30

10 years ago
(In reply to comment #29)
> (In reply to comment #28)
> > we shouldn't be creating the Trash folder on gmail anymore
> Sorry, I'm not following... earlier you said we create our Trash folder first.

I meant that we create it before you ever delete a message, i.e., up front instead of lazily. After folder discovery is done, we check if there's a trash folder, and if not, we create one. Without XLIST support, if the user is running GMail in a non-english language, we won't recognize the GMail trash, and we'll create our own.
(In reply to comment #30)
> After folder discovery is done,
This is going to be via XList if possible?
> we check if there's a trash folder, and if not, we create one.
So, won't we already have discovered the XList trash folder by now?
Assignee

Comment 32

10 years ago
(In reply to comment #31)
> (In reply to comment #30)
> > After folder discovery is done,
> This is going to be via XList if possible?
Yes.
> > we check if there's a trash folder, and if not, we create one.
> So, won't we already have discovered the XList trash folder by now?

Yes, we will, which is why, with XLIST support, we won't create an extra trash.
So the extra Gmail check is just for existing Gmail IMAP users, because the Gmail trash is a special case and we really need to use it if it's there?
Assignee

Comment 34

10 years ago
(In reply to comment #33)
> So the extra Gmail check is just for existing Gmail IMAP users, because the
> Gmail trash is a special case and we really need to use it if it's there?

We really should use it, yes. And then the user can delete our trash once it no longer has the trash flag set.
Assignee

Updated

10 years ago
Whiteboard: [needs review standard8, sr neil]
Assignee

Comment 35

10 years ago
pinging for review; this is blocking an other b3 blocker...

Updated

10 years ago
Attachment #375817 - Flags: superreview?(neil) → superreview+
Comment on attachment 375817 [details] [diff] [review]
address Neil's comment about removed code.

OK, sr=me given that you comment that you're using a separate flag for XList trash so that you can fix existing GMail users.
Assignee

Updated

10 years ago
Whiteboard: [needs review standard8, sr neil] → [needs review standard8]
Assignee

Comment 37

10 years ago
I'll add something like this:

/**
 * We use a separate xlist trash flag so we can prefer the GMail trash
 * over an existing Trash folder we may have created.
 */
Attachment #375817 - Flags: review?(bugzilla) → review+
Comment on attachment 375817 [details] [diff] [review]
address Neil's comment about removed code.

>+    // GMail gives us a localized name for the inbox but doesn't let 
>+    // us select that localized name.
>+      if (boxFlags & kImapInbox)
>+        imapFolder->SetOnlineName(NS_LITERAL_CSTRING("INBOX"));
>+      else if (onlineName.IsEmpty() || !onlineName.Equals(dupFolderPath))
>         imapFolder->SetOnlineName(dupFolderPath);

nit: incorrect comment indentation.

Looks good and works nicely. r=Standard8
Whiteboard: [needs review standard8]
Assignee

Comment 39

10 years ago
fix checked in, with nits addressed.
Status: ASSIGNED → RESOLVED
Last Resolved: 10 years ago
Resolution: --- → FIXED

Comment 40

10 years ago
This works almost flawlessly on gmail. I've changed language to Russian from English recreate account and every folder except trash take correct icon. I've changed back to english w/o creating account and it get correct icons for all folders except trash.
Is this expected behavior for trash?
Status: RESOLVED → VERIFIED
I have a nasty feeling that the GMail trash correction feature only works on accounts created using older versions of Thunderbird, not on language changes.

Comment 42

10 years ago
(In reply to comment #41)
> I have a nasty feeling that the GMail trash correction feature only works on
> accounts created using older versions of Thunderbird, not on language changes.

How older, before patch? And what reason behind this?

Comment 43

10 years ago
I was wrong Trash folder is working but I had to restart TB to see proper icon.

Updated

10 years ago
Keywords: intl

Updated

10 years ago
Depends on: 530805

Updated

8 years ago
Duplicate of this bug: 666412
RFC 6154 has been published, see bug 558659 (we morphed that bug to it).
You need to log in before you can comment on or make changes to this bug.