871200 - Support filters based on X-GM-LABELS

Status: UNCONFIRMED

(MailNews Core :: Filters, enhancement)

Description

[David Rees]

Now that TB 17 supports X-GM-LABELS thanks to bug #721316, it would be great if they could be used in Filters/Searches. Even if a "Customize"-type filter would be a great start

Comment 1

[rsx11m]

So, if you go into the Message Filters dialog and click "New" there, you can't select "Customize" and then add X-GM-LABELS as a new header manually? Or, does that filter not work as expected?

Comment 2

[rsx11m]

I see, those are held in the .msf files but apparently are not available in the regular message headers, thus the question is if/how the filter would see them.

Comment 3

[Kent James (:rkent)]

(In reply to rsx11m from comment #2)
> I see, those are held in the .msf files but apparently are not available in
> the regular message headers, thus the question is if/how the filter would
> see them.

This is done by adding the header to a preference, I think it is mailnews.customHeaders IIRC at one point I was going to add them automatically to that preference, not sure if that ever got done or not.

The other alternative is custom searches and custom actions that are done by an extension. So there are plenty of ways to do this if you are motivated.

Since with custom filters and searches it is pretty easy for an extension to extend traditional search and filter actions, the bar is pretty high to justify including something new in core, as it tends to clutter up an already long list of possible searches and actions.

Comment 4

[rsx11m]

Yes, mailnews.customHeaders was my first guess as well, and that's what you modify with the Message Filters > New > Custom dialog. That works fine for any regular RFC 822 headers, but I'm not sure if that would catch the X-GM headers, or are those coming with the regular IMAP message? (No-shows in Message Source.)

Comment 5

[David Rees]

Yes, I tried creating a Custom X-GM-LABELS in Filter/Search and it didn't find them.

I'll add I tried making it a server search also,  but it still didn't work even though GMAIL accepts X-GM-LABELS as an attribute to the IMAP SEARCH command as described in https://developers.google.com/google-apps/gmail/imap_extensions:
a012 SEARCH X-GM-LABELS foo
* SEARCH 1 2
a012 OK SEARCH (Success)

I tried logging the IMAP commands to see what TB was actually sending to Gmail using the steps described in http://email.about.com/od/mozillathunderbirdtips/qt/et_mail_log.htm, but they don't seem to work for server searches.


On a related note, Gmail IMAP also support a X-GM-RAW attribute to SEARCH which supports full Gmail search syntax. I can think of some great use cases for creating search folders that leverage X-GM-RAW (running the search on the server of course). A user could have TB search folders built using the full GMail search syntax.

Comment 6

[David Rees]

(In reply to rsx11m from comment #4)
> headers, or are those coming with the regular IMAP message? (No-shows in
> Message Source.)

The X-GM-LABELS are not part of the message source, but bug 721316 adds them as dbmsg hdrs in the database. Not sure how that related to TB local filtering, and of course TB server filtering must work differently.

Comment 7

[WADA:World Anti-bad-Duping Agency]

X-GM-LABELS is fetched by following command upon message header fetch of newly detected mail.
   UID fetch 5:10 (UID X-GM-MSGID X-GM-THRID X-GM-LABELS RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (...)] 
   * 1 FETCH (X-GM-THRID 1491263731375521726 X-GM-MSGID 1491263731375521726 
      X-GM-LABELS ("[Gmail]" DraftsX)
      UID 5 RFC822.SIZE 599
      FLAGS (\Seen mboxa)
      BODY[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)]
      {231} 
As easily known by this command, X-GM-LABELS != message header. X-GM-LABELS is rather a friend of FLAGS.
This value is saved as StringProperty named X-GM-LABELS in msgDBHdr. So value should be obtained by msgHdr.getStringProperty("X-GM-LABELS").

However, "message filter on a new mail for X-GM-LABELS" can be used only for following.
   At Mbox named AAA, "If X-GM-LABEL contains PPP" == "If this mail's copy is already held in Mox named PPP".
   At Mbox named AAA, "If X-GM-LABEL contians AAA" is nonsense, because Gmail IMAP doesn't return Gmail Label corresponds to this Mbox.
   At any Mbox, "If X-GM-LABEL contians \\AllMail etc." is nonsense, because Gmail IMAP never returns Gmail Label such as \\AllMail,
  [Gmail]/All Mail, \\Trash, [Gmail/Trash, \\Spam, [Gmail]/Spam.
And, because X-GM-LABEL is currently obtained upon message header fetch only, after the fact filter can do nothing on next:
   At Mbox named AAA, "If X-GM-LABEL contains BBB" where BBB is Gmail Label which is added to original of the mail after Gmail Label AAA is added.
This is true even at [Gmail]/All Mail. Because X-GM-LABELS is fetched upon message header fetch only, "X-GM-LABELS added after message header fetch at [Gmail]/All Mail" won't be fetched, unless "Repair Folder of [Gmail]/All Mail" is executed.

What is use case of this bug even though X-GM-LABELS is currently fetched upon message header fetch only?

To do mail data management with utilizing X-GM-MSGID/X-GM-THRID and X-GM-LABELS, following is mandatory.
   Create/maintain mail database based on  X-GM-MSGID.
        X-GM-MSGID = 1403445501350191849  
          THRID = 1403445501350191849
          LABELS] = { // archive is done twice, with Year, with Month
            ArchivesX/2012,
            ArchivesX/2012/2012-05
          }
         Original is held in = [Gmail]/All Mail, UID=2467 (one of [Gmail]/All Mail, [Gmail]/Trash, [Gmail]/Spam)
         This mail is held as =
            Mbox = ArchivesX/2012                  UID=7
            Mbox = ArchivesX/2012/2012-05]  UID=3

To keep data like above correct state or sync'ed state, periodical  X-GM-LABELS check is needed, as "new mail check" is needed to know new mail.
     uid fetch 1:* Flags X-GM-LABELS (CHANGEDSINCE known_modseq)
And, if this kind of database will be constructed, there is no need of searching or filterling.
    "Where copy of this X-GM-MSGID = 1403445501350191849 is held" is known at a glance, because this database is for it.

What is use case of "filters based on X-GM-LABELS"?

> a012 SEARCH X-GM-LABELS foo

What is purpose of this search?
X-GM-LABELS=foo exists? If so, All Gmail Label is shown in Folder pane of Thunderbird...
Mails which has X-GM-LABELS=foo? If so, it's always shown in Thread pane for Mbox named foo if View=All...

Comment 8

[WADA:World Anti-bad-Duping Agency]

FYI
If "If X-GM-LABAELS contains" is supported by View", "Gmail IMAP access to [Gmail]/All Mail, [Gmail]/Trash, [Gmail]/Spam only" can be easily achieved by simply utilizing View.
   Inbox = Virtual Folder named Inox. Search Target=[Gmail]/All Mail, Condition = X-GM-LABELS contains \\Inbox
   [Gmail]/Starred = Virtual Folder named [Gmail]/Starred. Search Target=[Gmail]/All Mail, Condition = X-GM-LABELS contains \\Starred
   Top level Drafts = Virtual Folder named Drafts. Search Target=[Gmail]/All Mail, Condition = X-GM-LABELS contains [Imap]/Drafts
Further, following is possible.
   [Gmail]/All Mail is hidden at Folder Pane.
   Virtual Folder named "All Mail", Search Target=hidden [Gmail]/All Mail, Condition = All Mail
   Virtual Folder named "Sent Mail", Search Target=hidden [Gmail]/All Mail, Condition = X-GM-LABELS contains \\Sent
   => Same display as Gmail Web UI without confusing [Gmail] and its subfolders.
See bug 697409 comment #11, please.
I believe that feature requested by this bug is one of most important features for better Gmail IMAP access support.

Comment 9

[WADA:World Anti-bad-Duping Agency]

Attachment: JavaScript code sample : get all messages which has GM Label of \\Inbox in a msgFolder

Logic is pretty simple.
1. get all msgDBHdr in a msgFolder
2. get X-GM-xxx string property of each message
3. Because X-GM-LABELS is string like \\Inbox Inbox/ABC/DEF "My Folder/XYZ"  "[Imap]/Tash" Archives/2015/2015-01,
    split to associtive array for future prosessing
4. Select message which has Gmail Label of \\Inbox

Comment 10

[Kent James (:rkent)]

Re comment 8 "I believe that feature requested by this bug is one of most important features for better Gmail IMAP access support."

I'm not a big GMail user so I don't understand where this fits into the importance of things. Could I have more comments on why this is important?

If "X-GM-LABELS are not part of the message source, but bug 721316 adds them as dbmsg hdrs in the database" is true, then can't you search for them already?

Comment 11

[WADA:World Anti-bad-Duping Agency]

(In reply to Kent James (:rkent) from comment #10)
> Re comment 8 "I believe that feature requested by this bug is one of most important features for better Gmail IMAP access support."

Feature I called in previous comment is "backend code or action which can find specific GM-LABEL of mails". This is similar to current predefined AgeInDays. AgeInDays is not for Date header. It's for date related msgDHdr properties.
If action/function for "X-GM-LABELS contiains" is usable in View/Search Folder, "Virtual Folder which shows all mails of \Inbox in X-GM-LABELS" can pretty easily be obtained. I this is possible, there is no need to access Inbox via IMAP. "IMAP access for [Gmail]/All Mail only" is sufficient.
I don't think UI for it is needed. Rather, it should be hidden, because it's useless for searching/filtering by users. It's useful for Tb's feature developers or addon developers.

As known by my JavaScript sample, "finding all mails which has \Inbox in X-GM-LABELS" itself is pretty simple job. Function like my JavaScript sample can be held in MsgUtils etc.
Problem is : How to utilize such function at where. My want is AgeInDays like predefined "X-GM-LABELS contains" condition in View.
So I posted to this bug.

> but bug 721316 adds them as dbmsg hdrs in the database" is true, (snip)

It's not true. As known by my script code, X-GM-xxx response data is currently saved in StringProperties of msgDBHdr.
Needless to say, msDBHdr.X_GM_MSGID which is similar to msgDBHdr.messageKey, msDBHdr.X_GM_LABELS array, are better for people who tries to write code.

Comment 12

[WADA:World Anti-bad-Duping Agency]

FYI.
If "X-GM-LABELS contains xxx" will be supported, and Gmail will resolve problem of bug 1128947, and bug 1123617 will be fixed, following is possible.
1. Server access for [Gmail]/All Mail, [Gmail]/Trash, [Gmail]/Spam only.
        Upon folder open        : uid fetch 1:* Flags X-GM-MSGID X-GM-TRDID X-GM-LABELS 
        Upon new mail check : uid fetch 1:* Flags X-GM-MSGID X-GM-TRDID X-GM-LABELS (CHANGEDSINCE modseq)
    "Local copy of mail in Offline-Store" is for these 3 folders only => No duplicaate copy in Offline-store.
2. Because all Mbox except above 3 is merely Gmail Label in Gmail,
    and because  "mapping between X-GM-LABEL and MboxName in Gmail IMAP" is known,
    "MboxName in Gmail IMAP" can be pretty easily known by X-GM-LABEL.
3. If "Offline-Store file for [Gmail]/All Mail, [Gmail]/Trash, [Gmail]/Spam" is similar one to Inbox of POP3 account/Inbox of Local Folders,
    problem like "Offline-store file content was lost!" won't occur.
    If user want "sync'ed staus of Mbox with server", it can be done merely by using X-GM-MSGID and X-GM-LABELS.
    If user want "message filter like POP3", user can create local "Inbox" by filter of "if X-GM-LABELS contains \Inbox, copy to Inbox" :-)

=> user can freely use Gmail IMAP as if POP3 which user pretty deeply loves :-)