697409 - gmail offline support should only default to archiving the 'All Mail' folder, so data is not stored locally twice.

Status: RESOLVED DUPLICATE of bug 721316

(MailNews Core :: Networking: IMAP, defect)

Description

[Anand Kumria]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:8.0) Gecko/20100101 Firefox/8.0
Build ID: 20111019081014

Steps to reproduce:


Setup thunderbird to use my gmail account


Actual results:


Everything, EVERYTHING, was downloaded.


Expected results:


Instead, since Thunderbird is downloading the special folder 'All Mail', it should not need to store a copy of other folders (doubling disk space used for offline messages)

Comment 1

[rsx11m]

That's primarily a Gmail issue, their labels are mapped to folders to be in compliance with the IMAP standard (which doesn't know that concept). You can simply disable synchronization for the "All Mail" folder by right-clicking on it and going into the Synchronization tab of the "Properties" dialog. That should do, or you can unsubscribe from "All Mail" entirely (which may conflict with the archiving default, I don't know about that part).

Comment 2

[Anand Kumria]

It is also a usability thing. 

I'm well aware, but thanks for the info., of how to disable synchronisation of a particular folder. 

But Thunderbird *already* knows I am using GMail.

Why can't it do the right thing to start with?

IMO the right thing would be to actually sync all the '[GMail]'-type folders and *then* for other labels keep an index into the appropriate '[GMail]' folder (whether that be 'All Mail', 'Trash', etc.)

Comment 3

[rsx11m]

I think there is already a special handling for the "All Mail" folder given that it's the default folder for archiving, and also some special handling for not allowing subfolders as they are not supported by Gmail, etc. Just changing the order in which [Gmail] folders are synchronized wouldn't resolve the issue of duplicate synchronization, though. Coming up with some heuristics to only synchronize those messages which aren't presented in other folders (hence synchronized there already) would sound like quite a bit effort for just the purpose of avoiding duplicate synchronization (or it would have to be coupled with the process of archiving explicitly, but again a special case just to be implemented for Gmail accounts).

A "lite" solution might be to disable synchronization for "All Mail" by default. There is the argument of message duplication you mentioned in favor of such a step, but there are also arguments against it (e.g., messages may be in "All Mail" but nowhere else, and if you access a message in "All Mail" it would nevertheless be downloaded from the server even if it was synchronized in another folder representing a different label). Thus, it's not clear to me that this is the best thing to do either.

Comment 4

[Wayne Mery (:wsmwk)]

vincent, sid0, any thoughts?

This is a perf issue.
And we see these issues in support forums

Comment 5

[Anand Kumria]

(In reply to rsx11m from comment #3)
> Just changing the order in which [Gmail] folders are synchronized wouldn't
> resolve the issue of duplicate synchronization, though. Coming up with some
> heuristics to only synchronize those messages which aren't presented in
> other folders (hence synchronized there already) would sound like quite a
> bit effort for just the purpose of avoiding duplicate synchronization (or it
> would have to be coupled with the process of archiving explicitly, but again
> a special case just to be implemented for Gmail accounts).

Huh?

> A "lite" solution might be to disable synchronization for "All Mail" by
> default. 

The right solution would be to *default* archiving of the 'All Mail' folder, since it is - by definition - all the email.

Every other folder should be marked by *default* to not being archivied.

> There is the argument of message duplication you mentioned in favor
> of such a step, but there are also arguments against it (e.g., messages may
> be in "All Mail" but nowhere else, and if you access a message in "All Mail"
> it would nevertheless be downloaded from the server even if it was
> synchronized in another folder representing a different label). Thus, it's
> not clear to me that this is the best thing to do either.

Frankly I do not understand what you've written:
 - "e.g. message be in "All Mail" but nowhere else"

Fine.

 - "and if you access a message in "All Mail" it would nevertheless be downloaded"

Which is what you want, if it isn't already downloaded

 - "downloaded from the server even if it was synchronised in another folder representing a different label"

Huh?

Basically:
 - default 'All Mail' to being archived
 - default all other to not.

Advanced users, who are the uncommon case, can tweak an adjust things if the need to.

That would address the pressing disk-space issue; tweaking the GMail provider to handle indexes from one folder type to another can be addressed in another bug.

I've updated the subject to reflect the reduced scope of this particular report.

I'd appreciate it if someone could confirm this as a problem (as per comment 4).

Comment 6

[rsx11m]

(In reply to Anand Kumria from comment #5)
> The right solution would be to *default* archiving of the 'All Mail' folder,
> since it is - by definition - all the email.
> 
> Every other folder should be marked by *default* to not being archivied.
> 
> Frankly I do not understand what you've written:
>  - "e.g. message be in "All Mail" but nowhere else"

The problem with this approach is that the synchronized messages would *only* be accessible when actually looking into the "All Mail" folder, but not when trying to open it from any other folder (e.g., "Inbox"). It's considered a different folder from Thunderbird's perspective (and rightfully so as Gmail has an unusual concept of using folders not directly covered by the standard) and thus the message would be requested from the server every time when looking at it from a folder other than "All Mail".

Wayne has added some experts to the CC list in his comment #4, thus let's wait for their ideas how to identify such multi-label messages (the Message-ID may be sufficient already) and thus to avoid duplicate synchronization of identical messages. Selectively disabling or enabling synchronization for folders based on the server type would be the second step after that, and may be rendered unnecessary if the general solution is comprehensive enough.

Comment 7

[WADA:World Anti-bad-Duping Agency]

FYI.
If Gmail IMAP Extension of X-GM-MSGID, X-GM-LABELS is fully utilized, there is no need to access Gmail IMAP folder other than [Gmail]/All Mail, [Gmail]/Trash, [Gmail]/Spam, although access of [Gmail]/Drafts may be needed for draft handling unless "delete old draft" will be changed to "move to [Gmail]/Trash".
If X-GM-THRID is also fully utilized, Tb can become free from Threading of mails in Gmail IMAP folder. Required work in threading by Tb is "theading in a thread" only if Gmail IMAP folder.
See bug 721316 for Gmail IMAP Extension.

Comment 8

[David :Bienvenu]

I think this is important to do, and not too difficult.

Comment 9

[WADA:World Anti-bad-Duping Agency]

(In reply to David :Bienvenu from comment #8)
> and not too difficult.

X-GM-MSGID and X-GM-THRID is great and simple and easy to use, so they are very useful and usable for us, but X-GM-LABELS looks chaos. Is distingushing of standard IMAP flag/keyword, Tag by Tb(also IMAP keyword/flag), Tag by other IMAP clients including other Tb, Gmail's System Label(returned to XLIST as folder name), and ordinal user defined Gmail Label, easy?

Comment 10

[WADA:World Anti-bad-Duping Agency]

To do mail data management with utilizing X-GM-MSGID/X-GM-THRID and X-GM-LABELS, following is mandatory.
   Create/maintain mail database based on  X-GM-MSGID.
        X-GM-MSGID = 1403445501350191849  
          THRID = 1403445501350191849
          LABELS] = { // example of "archive is done twice, with Year, with Month"
            ArchivesX/2012,
            ArchivesX/2012/2012-05
          }
         Original is held in = [Gmail]/All Mail, UID=2467 (one of [Gmail]/All Mail, [Gmail]/Trash, [Gmail]/Spam)
         This mail is held as =
            Mbox = ArchivesX/2012                  UID=7
            Mbox = ArchivesX/2012/2012-05]  UID=3
   Creation of such table is pretty simple job because above is merely a summary table of following response, 
       X-GM-THRID 1491263731375521726 X-GM-MSGID 1491263731375521726  X-GM-LABELS ("[Gmail]" DraftsX)
       UID 5 RFC822.SIZE 599
       ("This mail is held as" in above table is not essentially needed)
   and if JavaScript object, "save as JSON" is possible.

To keep table like above correct-state or sync'ed-state, periodical  X-GM-LABELS check is needed, as "new mail check" is needed to know new mail.
     uid fetch 1:* Flags X-GM-LABELS (CHANGEDSINCE known_modseq)

After it, following is possible.
(1) IMAP access to [Gmail]/All Mail, [Gmail]/Trash, [Gmail]/Spam) only.
(2) For other than  [Gmail]/All Mail, [Gmail]/Trash, [Gmail]/Spam,
      Faked MsgDatabase for a Gmail Label  (read only, similar to Virtual Folder and/or View)
         =     msgDBHdr array == subset of msgDBHdr array of [Gmail]/All Mail (collection of mail whch has the Gmail Label)
             + msgStore == [Gmail]/All Mail
(3) "Modification at Faked MsgDatabase for a Gmail Labell" is done on [Gmail]/All Mail
(4) Copy.mail between two Gmail Labels is pretty simple.
       "copy to  Mbox" == "add corresponding Gmail Label", so, At [Gmail]/All Mail, uid copy nn Inbox/ABC/DEF
      If move, At [Gmail]/All Mail, uid move nn [Gmail]/All Mail/Dummy + delete [Gmail]/All Mail/Dummy + uid copy nn Inbox/ABC/DEF.
      Any action is merely "add a Gmail Label" and/or  "remove a Gmail Label".
      Mail in [Gmail]/All Mail is reoved only by "copy or move to [Gmail]/Trash, [Gmail]/Spam ".
(5) Copy/Move.mail among [Gmail]/All Mail, [Gmail]/Trash, [Gmail]/Spam is also pretty simple.
      At the Mbox, SELECT SpecialA, uid copy xx [Gmail]/SpecialB where SpecialA and SpecialB is two in "All Mail, Trash, Spam".
      When copy/move to [Gmail]/Trash or [Gmail]/Spam, any Gmail Label is hidden by Gmail(same as remove from perspective of imap client) 
      and EXPUNGED(VANISHED) is notified at the SELECTed Mbox.
      When copy/move to [Gmail]/All Mail, EXPUNGED(VANISHED) is notified at the SELECTED Mbox,
      and "new mail is notified at [Gmail]/All Mail via IDLE [] or by new mail check, and all Gmail Label is re-shoewn(restored) by Gmail.
That's all.
Trick is : "uid copy/move nn GmalLabel" is merely "add/remove Gmail Label" if othe than [Gmail]/All Mail, [Gmail]/Trash, [Gmail]/Spam.
This can be called : Efficient Gmail Label maintenance by Thundebird with convenient mail viewing than Gmail's Web UI without excess server access.

Comment 11

[WADA:World Anti-bad-Duping Agency]

"Faked MsgDatabase for a Gmail Label" is almost same as current Virtual Folder = Search Folder.
Difference is;
  - Search Target Folder :  [Gmail]/All Mail only
  - Search condition : All mail which has X-GM-LABELS contains specific Gmail Label == MboxName in Gmail IMAP
  - Required msgDBHdr array : no need of re-construction of MboxName.msf. Subset of msgDBHdr Array of [Gmail]/All Mail is always sufficient.
View of "X-GM-LABELS contains MboxName" + "dead copy or pointer to [Gmail]/All Mail .msf" + "overlay of some fields for name" may be sufficient.

Because "excellent View(gDBView)" and "Virtual Folder based on the View" are already available, and because Gmail is simple(Big and flat and externally simple mail database structure with Gmail Label), I think "Gmail/Gmail IMAP support utilizing X-GM-MSGID/X-GM-THRID and X-GM-LABELS" is not so complicated. Simple structure like table of comment #10 becomes visible, so I feel imeplementing feature based on such "simple table structure" is not so complicated job.

Comment 12

[WADA:World Anti-bad-Duping Agency]

Attachment: Sample JavaScript code to extract all message data which has \\Inbox etc. from a msgFolder

1. get all msgDBHdr in a msgFolder
2. get X-GM-xxx string property of each message
3. Because X-GM-LABELS is string like \\Inbox Inbox/ABC/DEF "My Folder/XYZ"  "[Imap]/Tash" Archives/2015/2015-01,
    split to associtive array for future prosessing
4. Select message which has Gmail Label such as \\Inbox

Comment 13

[WADA:World Anti-bad-Duping Agency]

Attachment: Gmail Label and Mbox name in Gmail IMAP

Comment 14

[WADA:World Anti-bad-Duping Agency]

If condition of "X-GM-LABELS contains xxx" is supported by Virtual Folder/View, following is possible.
   -  Faked Gmail IMAP account/folder with Virtual Folder utilized : All folders is Virtual folder.
          Inbox        : "X-GM-LABELS of mail in \AllMail folder" contains Gmail Label named \Inbox
          Sent          : "X-GM-LABELS of mail in \AllMail folder" contains Gmail Label named \Sent
          ABC/DEF   :  "X-GM-LABELS of mail in \AllMail folder" contains Gmail Label named ABC/DEF
          All Mail      :   All mails(MSGIDs) in [Gmail]/All Mail
          Trash         :   All mails(MSGIDs) in [Gmail]/Trash
          Spam         :   All mails(MSGIDs) in [Gmail]/Spam
      No confusing [Gmail] folder.
      Localised Mbox name at Gmail Web UI is known by XLIST response.
      If needed, "use GMLABEL as Label instead of IMAP Mbox, as done at Gmail Web UI" is also possible.
  - At actual Gmail IMAP accont in Tb.
       Access \AllMail, \Trash, \Spam only.
       Due to current implementton, access to \Inbox folder is needed in order to detect new mails in \AllMail, \Trash, \Spam 
       Auto-sync of  \AllMail, \Trash, \Spam only.       
:     If new mail check at \AllMail, \Trash, \Spam  is following, state of all GMLABEL is known.
          uid fetch 1:* Flags X-GM-MSGID X-GM-THRID X-GM-LABELS BODY.PEEK[HEADER] (CHANGEDSINCE known_modseq).
       If auto-sync is used, following can be "new mail check" at \AllMail, \Trash, \Spam.
          uid fetch 1:* Flags X-GM-MSGID X-GM-THRID X-GM-LABELS BODY.PEEK[] (CHANGEDSINCE known_modseq)
          Thiere is no difference from downlod in POP3. This is a way to use IMAP as if POP3  :-) .
          Rather, better than POP3, because there is no need to keep popstate.dat like file.
          If this "actual Gmail IMAP accont in Tb" is hidden at folder pane, this is similar to defferedtoccont in Global Inbox for POP3.

If new server type=pop3likegmailimap is introduced for above faked Gmail IMAP account, it's perhaps good for users who are eager to use IMAP as if POP3.
      No need of "Keep Messages in Server"       <-> "Keep Messages in Server" in POP3
      Offlime-use=Off of [Gmail]/All Mail             <-> "Fetch header only" in POP3

Comment 15

[Ludovic Hirlimann [:Usul]]

Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

Comment 16

[Wayne Mery (:wsmwk)]

With perhaps minor exceptions, which I believe we have bug reports for, this should have been resolved by bug 721316 ?

Comment 17

[Mark Banner (:standard8)]

Probably? I don't recall enough about the specifics for that.