493455 - subscription manager completely broken with gmail -imap (after XLIST support is implemeted)

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[u294409]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; pl; rv:1.9.0.10) Gecko/2009042315 Firefox/3.0.10
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; pl; rv:1.9.1b5pre) Gecko/20090514 Lightning/1.0pre Shredder/3.0b3pre

I'm using GMail via IMAP.

After an update [using nightly build] I noticed that folders I don't, are subscribed.

I tried unsubscribing them, but it's impossible.

I need to do this, because Thunderbird downloads every message and tries to index all the messages. In folder GMail/All I've got few thousands of messages.

Reproducible: Always

Actual Results:  
Can't unsubscribe.

Expected Results:  
Possible to unsubscribe.

Comment 1

[Ludovic Hirlimann [:Usul]]

(In reply to comment #0)
> User-Agent:       Mozilla/5.0 (X11; U; Linux i686; pl; rv:1.9.0.10)
> Gecko/2009042315 Firefox/3.0.10
> Build Identifier: Mozilla/5.0 (X11; U; Linux i686; pl; rv:1.9.1b5pre)
> Gecko/20090514 Lightning/1.0pre Shredder/3.0b3pre

Is the issue present when Lightning is not running (start thunderbird in -safemode)
 
> I'm using GMail via IMAP.
> 
> After an update [using nightly build] I noticed that folders I don't, are
> subscribed.
> 
> I tried unsubscribing them, but it's impossible.


When you do that are there any error messages in Tools->Error console ? 

> 
> I need to do this, because Thunderbird downloads every message and tries to
> index all the messages. In folder GMail/All I've got few thousands of messages.

Is gloda enabled on your account ? see (https://wiki.mozilla.org/Thunderbird:Using_Gloda)

Comment 2

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> After an update [using nightly build]

(Q1) Updated from what? From Tb 2? Updated from recent trunk build before 5/14 to 5/14 build?
What is difference between environment in which problem occurs and environment in which problem doesn't occur? Tb's build only?
What action did you take just before your problem started to occur?

> I'm using GMail via IMAP.
> I noticed that folders I don't, are subscribed.
> I need to do this, (snip)

All mail folders is always returned to LSUB by Gmail IMAP. i.e. Gmail IMAP curently doesn't support unsubscribe. 
And, If "Show only subscribed folder" is checked, Tb holds subscription list locally, then "deleted folder by other client" is displayed at folder pane, until the folder is unchecked at "Subscribe" panel.

To hide Gmail's folder/label(similar to unsubscribe), you need to use "Advanced IMAP Controls" of Gmail's "Labs".

> I need to do this, because Thunderbird downloads every message and tries to index all the messages.

"Mail download of all mail folder" can happen when;
 (a) mail.check_all_imap_folders_for_new = yes
 (b) auto-sync is enabled.
     Account Setting/Syncronization&Storage
       Message Synchronizing
        [X] Keep messages for this account on this computer
     and all folder are checked in "Advanced" of it.
     (if option is checked, all mail folders are checked as default.) 
"Mail download of a folder" can happen when;
 (c) Offline-use = enabled (by new auto-sync)

(Q2) Do you enable such option? (auto-sync is enabled as default by latest-trunk)

> I'm using GMail via IMAP.
> I noticed that folders I don't, are subscribed.

(Q3) Do you use googlemail.com instead of gmail.com? Do you select other than "Display Language:English(US)" at Gmail's setting at Web Interface?

> and tries to index all the messages.

As Ludovic says, it's by new feature of "Gloda"(auto indexing). It's probably defaulted to "Enabled" by recent build.
(Q4) Do you enable the option?
> Tools/Options/Advanced/General
>   Advanced Configuration
>     [?] Enable Global Search and Indexer

Comment 3

[WADA:World Anti-bad-Duping Agency]

>(bug summary)
> subscription manager completely broken with gmail -imap

If bug summary correctly represents your problem, DUP of Bug 413254(INVALID).
See Dependency Tree for meta Bug 402793(with "Show Resolved").

Comment 4

[u294409]

> Is the issue present when Lightning is not running 
> (start thunderbird in -safemode)

Haven't tried yet, I took a look on options below.

> When you do that are there any error messages 
> in Tools->Error console ? 

Same as above.

> Is gloda enabled on your account ? 
> see (https://wiki.mozilla.org/Thunderbird:Using_Gloda)

I was disabled until I disabled it today, on my own.

>  (Q1) Updated from what? From Tb 2? Updated 
> from recent trunk build before 5/14
> to 5/14 build?

From a nightly build, don't remember which one.

> What is difference between environment 
> in which problem occurs and environment
> in which problem doesn't occur? Tb's build only?

Yes, only Tb nightly build makes the difference.

> What action did you take just before 
> your problem started to occur?

After update from nightly to more recent nightly, using Tb's integrated updater, problem started to occur.

> (a) mail.check_all_imap_folders_for_new = yes

Was true. Not enabled my me.

> (b) auto-sync is enabled.
>     Account Setting/Syncronization&Storage
>       Message Synchronizing
>        [X] Keep messages for this account on this computer
>     and all folder are checked in "Advanced" of it.
>     (if option is checked, all mail folders are checked as default.) 

Was enabled too. Not by me.

> (Q2) Do you enable such option? (auto-sync 
> is enabled as default by latest-trunk)

I don't need such option and find it rather as crucial mistake.

> (Q3) Do you use googlemail.com instead of gmail.com? 
> Do you select other than "Display Language:English(US)"
>  at Gmail's setting at Web Interface?

I'm using mail.google.com with Polish (pl_PL/PL).

> As Ludovic says, it's by new feature of "Gloda"
> (auto indexing). It's probably defaulted 
> to "Enabled" by recent build.
> (Q4) Do you enable the option?

No, I need search only in folder I'm currently having opened, not more.

> If bug summary correctly represents your problem, 
> DUP of Bug 413254(INVALID).
> See Dependency Tree for meta Bug 402793(with "Show Resolved").

The "cannot unsubscribe problem" occurs for me since I filed this bug, never before. Folders unchecked in subscription manager weren't displayed in Tb's folder pane.

Comment 5

[Brian Kennelly]

(In reply to comment #2)
> All mail folders is always returned to LSUB by Gmail IMAP. i.e. Gmail IMAP
> curently doesn't support unsubscribe. 

That is demonstrably false.  Gmail supports unsubscribe, and "[Gmail]/All Mail" can be unsubscribed.

Comment 6

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #5)
> Gmail supports unsubscribe, (snip)

There was no text like "subscribe is not supported" in next page any more. 
> https://mail.google.com/support/bin/answer.py?answer=78761&topic=12762
It seems already enhanced.

2009/4/14 build : Issued unsubscribe & LSUB  => folder disappered from folder pane.
2009/5/16 build : Issued unsubscribe & XLIST => folder stayed in folder pane.

Confirming.

Comment 7

[WADA:World Anti-bad-Duping Agency]

Attachment: IMAP log for Gmail IMAP. XLIST after unsubscribe instead of LSUB.

Comment 8

[David :Bienvenu]

Yes, now we're using XLIST instead of LSUB for GMail - XList ignores subscription. I can see why you'd want to not subscribe to All Mail though we'd really like to fix that by knowing that we probably already have the messages in All Mail in some other offline store. Maybe we could figure out some way of using gloda to look up messages by message-id, and if we already have that message in an offline store, instead of downloading it from the gmail server, we just copy the offline msg into the All Mail offline store (or vice versa, depending on which folder we stored the message offline for first). The async nature of gloda might make this tricky, plus I guess gloda is really designed to be used from js, and not c++. Or a related idea - we could add the idea of delegated offline store copy of a message, to avoid duplicate local copies of a message, but that makes things complicated when a message is deleted.

An other awkward thing is that we want to use the All Mail folder as our Archive folder, and if you don't subscribe to it, we can't do that.

Combining XLIST and LSUB use is possible but it's a bit of work. I think we would issue the LSUB first and build up the folder list. Then we would issue an XLIST but only pay attention to folders that we had already seen with LSUB. This would mean double the amount of folder discovery network traffic, and an extra round trip to the server on every startup. Perhaps we could turn off imap subscription by default for gmail imap accounts and those small number of users who want imap subscription for gmail could turn it on.

Comment 10

[u294409]

I disabled gloda, mail.check_all_imap_folders_for_new = yes, "Keep messages for this account on this computer" but I noticed that after that when I'm opening INBOX, I see no messages inside.

Something went wrong. I used to see mail even when others had problems though. Now I've got the problem with GMail's IMAP/Tb ^^ .

Comment 11

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #8)
David, no problem if non-"[Gmail]/xxx" folder which is not returned to XLIST?
I tried mail.server.serverN.is_gmail=false(Gloda was disabled in my test), but behaviour looked same as one when is_gmail=true. Subscription related behaviour depends on "XLIST extension is supported or not" only?

Comment 12

[Magnus Melin [:mkmelin]]

Hm, subscription don't seem to work anywhere anymore? Is that the same problem? 

Subscriptions are shown ok in the subscription dialog again today, but both subscribed and unsubscribed folders are shown in the folder pane.

Comment 13

[Magnus Melin [:mkmelin]]

I meant to say, i see it also with normal imap servers.

Comment 14

[Bob Riley]

I am not really a pc tech person, but by "normal imap servers," do you mean imap servers other than gmail's?  I have heard that in some way gmail's IMAP system is different from others (say, fastmail.fm's system), but I frankly have no idea what those difference might be and what that means to a mail program.

Comment 15

[Magnus Melin [:mkmelin]]

Yes.

Comment 16

[Magnus Melin [:mkmelin]]

Ah, correction. Other servers are mainly ok, it's just that subscribing to a folder with subfolders doesn't work.

Comment 17

[Magnus Melin [:mkmelin]]

er, *un*subscribing

Comment 21

[junk]

Just to add, I can't unsubscribe to my gmail spam folder after updating to tb 3b3. I could and was unsubscribed in tb 3b2.

Comment 22

[David :Bienvenu]

yeah, so there are enough issues with this that I'm going to have to combine XLIST and LSUB for gMail servers.

Comment 23

[Mark Banner (:standard8)]

The Thunderbird drivers wish to release Thunderbird 3 as soon as possible. As a
result, we feel that this bug shouldn't stand in the way of all the other good
work getting into the hands of users sooner rather than later. Therefore we are
retargeting it for 3.1. See http://ccgi.standard8.plus.com/blog/archives/242
for more details. The 3.1 release is expected to be a quick release soon after
Thunderbird 3.

Comment 24

[David :Bienvenu]

My recollection is that we talked about this in the drivers meeting and I said I needed to fix this...

Comment 25

[David :Bienvenu]

I thought a little more about how to do this - my current thinking is to detect the case of a gmail server using subscription, and in that case, I'll do a special xlist, and build up the list of folders with special flags, and not tell the mailnews backend about them. Then, when we do our normal lsub, I'll check the results of the lsub against the list of folders I've saved, and for the matches, add the special flag to the lsub response I send over to the mailnews backend. I may use a string hash table for the list of folders with special flags just to keep things speedy.

Comment 26

[David :Bienvenu]

Attachment: proposed fix

This should fix it. I'll run with it a bit just to be sure...

Comment 27

[David :Bienvenu]

Comment on attachment 396926 [details] [diff] [review]
proposed fix

this should fix a whole raft of issues with servers that support XLIST like gMail...

Comment 28

[ahu]

have the binary Mac version, don't know how to patch that. I just believe you and hope that I will not have this problem again in the final version. :)

Comment 29

[Mark Banner (:standard8)]

Comment on attachment 396926 [details] [diff] [review]
proposed fix

+          PRInt32 hashValue = 0;
+          if (m_specialXListMailboxes.Count() > 0)
+          {
+            PRUint32 hasHashValue = 0;
+            nsCString strHashKey(adoptedBoxSpec->mAllocatedPathName);
+            m_specialXListMailboxes.Get(strHashKey, &hashValue);
+            adoptedBoxSpec->mBoxFlags |= hashValue;
+          }

hasHashValue isn't used, but hasValue doesn't need to be used outside the loop either.

+  if (hasXLIST)
+  {
+    if (usingSubscription)
+    {
+      m_hierarchyNameState = kXListing;
+      // we use "%.%" since gmail special folders are unlikely
+      // to be more than 2 levels deep.
+      List("%.%", PR_TRUE, PR_TRUE);
+    }
+  }

Nit: could combine if statements into one.
 
+  //mapping between special xlist mailboxes and the corresponding flag

nit: space after //

+  nsDataHashtable<nsCStringHashKey,PRInt32> m_specialXListMailboxes;

nit: space after comma.

This seems to work fine for gmail subscription.

r/sr=Standard8

Comment 30

[David :Bienvenu]

fix checked in, with nits addressed. I think this means we can close several bugs, though I don't know all of them off the top of my head.

Comment 32

[WADA:World Anti-bad-Duping Agency]

See Dependency Tree of Bug 506095 for bug reports of "IMAP subfolder is not shown".

Comment 33

[WADA:World Anti-bad-Duping Agency]

Checked with next build, with Gmail IMAP.
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.4pre) Gecko/20090831 Shredder/3.0b4pre

Unsubscribe worked well. But, once unsubscribed, it was impossible to subscribe again, because "+" mark is not displayed for folder who has childs and child folders are not displayed at subscription list. Unchecking of "Show only subscribed folder" showed child folders at folder pane, but it was impossible to get child folders in subscription list again.

Re-opening.
David, are we better to open separate bug for above?

Comment 34

[David :Bienvenu]

yeah, I think opening a separate bug for the above would be good. I have to try to reproduce it first, though. Re-resolving fixed.

Comment 35

[David :Bienvenu]

OK, I saw that happen the first time, but expand/collapsing the parent folder will eventually make the child folder have the + mark. I highly doubt that has to do with gmail at all, and I'd be surprised if it wasn't an existing bug.

Comment 36

[WADA:World Anti-bad-Duping Agency]

I've opened Bug 514106 for problem of Comment #33 to 35.

Comment 37

[WADA:World Anti-bad-Duping Agency]

Adding XLIST in bug summary for ease of search.

Comment 38

[David :Bienvenu]

WADA, see bug 369964 which was reported long before XLIST support was added...it looks like the same bug to me.

Comment 40

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #38)
This bug is same as bug 369964? Or comment #33(Bug 514106) is same as bug 369964?
If latter, I agree with you. By fix of this bug, LSUB was issued to Gmail IMAP again, then I could see phenomenon/IMAP flow of Bug 369964.

Comment 41

[David :Bienvenu]

Attachment: xlist gmail correctly.

the checked in patch broke xlist support for gmail, because gmail uses '/' as a hierarchy delimiter. This patch makes us use the correct hierarchy delimiter for the xlist commands, and makes us list both the first and second level, in case some server has top level special folders.

Comment 42

[David :Bienvenu]

I'm going to re-open this to keep the bug on the radar...

Comment 43

[Mark Banner (:standard8)]

Comment on attachment 398938 [details] [diff] [review]
xlist gmail correctly.

Presumably you've kept the first List call because that gives us the sub directory separator? or is there some other reason?

r/sr=Standard8 assuming there's a good reason or you remove it.

Comment 44

[David :Bienvenu]

I want both the first and second levels, and yes, listing the first level does give me the hierarchy delimiter - fix checked in.