1759883 - Imap servers using oauth2 don't show subscription changes in folder pane without a restart

Status: RESOLVED FIXED

(Thunderbird :: Folder and Message Lists, defect)

Description

[andysem]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0

Steps to reproduce:

I have an IMAP mail box with a number of folders. One of the folders, on the same level as Inbox and other root level folders, is a backup of another mail box which is maintained by the server. From Thunderbird perspective, this is just a folder.

The problem is that I cannot unsubscribe from this folder.

1. Right-click on the email account, select "Subscribe...".
2. In the opened window, deselect the folder and all its nested folders. Press Ok.

Actual results:

The deselected folder stays present in the UI, and new messages in it continue to be downloaded. In the "Subscribe..." window, the folder, and all its subfolders, remain selected.

Expected results:

Unsubscribing should work. The folder should disappear from the UI, new messages should no longer be downloaded and previously downloaded messages should be deleted.

This is with Thunderbird 91.5.0 x86_64 on Kubuntu 21.10.

Comment 1

[Wayne Mery (:wsmwk)]

Server issue?

Comment 2

[andysem]

I don't know. The server is mail.ru.

Shouldn't it work locally?

Comment 3

[gene smith]

Not sure what you mean by "work locally"?

I just tried it on my personal account on a folder I don't use much and it worked OK. The folder and its sub-folder went away. Then when I re-subscribed they came back.
But the signaling from different servers can vary so you might also need to try two other things to make them disappear after unsubscribing:

• Collapse and then expand the account at the server name using the > widget, or
• Restart tb to ensure a folder re-discovery

Comment 4

[gene smith]

In the "Subscribe..." window, the folder, and all its subfolders, remain selected.

I missed that statement. So you are saying they still show subscribed in the dialog even after you un-checked the boxes there? If that's true, it does sound like the server is reporting the folders as subscribed even after you unsubscribed them. Hard to tell without the usual IMAP:5 log.

Comment 5

[andysem]

(In reply to gene smith from comment #3)

Not sure what you mean by "work locally"?

I mean, TB should just not request new messages from these folders.

(In reply to gene smith from comment #4)

In the "Subscribe..." window, the folder, and all its subfolders, remain selected.

I missed that statement. So you are saying they still show subscribed in the dialog even after you un-checked the boxes there?

Yes. I also tried restarting and collapsing/expanding the account - doesn't help.

Comment 6

[andysem]

To illustrate the "work locally", I have Aqua Mail client on my Android phone, which is also configured to use the same email account. In that client, the folder in question is not subscribed, which means:

• It is visible (with its subfolders) in the full list of folders in the account settings, available for subscribing.
• It is not enabled for synchronization in those settings.
• Therefore it is not visible in the main UI (folder tree), and new messages in it are not downloaded from the server.

I'm not sure if Aqua Mail subscribes to folders on the server or works in some other way. My point is that, if unsubscribing on the server doesn't work for some reason, then it must work in the client locally. I assume, this can be implemented simply by not requesting messages from the server. If this is not implemented in TB currently then please consider this issue a feature request.

Comment 7

[gene smith]

Well, if the server is not respecting the clients request to unsubscribe and still reports the mailbox as subscribed, tb will show it. There is no purely local subscription effect as you correctly point out.

With a properly behaving server, if client A and B are subscribed to folder X and B unsubscribes from folder X, client A will not see folder X except in the subscribe dialog. So client B (or A) can override the other clients subscription setting for folder X.

Do you have more than one client accessing and subscribing or unsubscribing from the subject folder? Could the aqua mail somehow be overriding TB's setting at the server?

Another remote possibility is under advanced server settings you might have "show only subscribed folders" unchecked.

Again, the only way to be sure what is really happening is with an IMAP:5 log. I.e., is tb really unsubscribing and is the server reporting the folder as unsubscribed.

Comment 8

[andysem]

(In reply to gene smith from comment #7)

Well, if the server is not respecting the clients request to unsubscribe and still reports the mailbox as subscribed, tb will show it. There is no purely local subscription effect as you correctly point out.

With a properly behaving server, if client A and B are subscribed to folder X and B unsubscribes from folder X, client A will not see folder X except in the subscribe dialog. So client B (or A) can override the other clients subscription setting for folder X.

Well, this is certainly not the expected behavior. Subscribing should be client-specific. I may want to see this folder in one client (on one machine) but not the other.

Do you have more than one client accessing and subscribing or unsubscribing from the subject folder? Could the aqua mail somehow be overriding TB's setting at the server?

No. Aqua Mail is not subscribed to the folder in question and never was. My actions in TB wrt. unsubscribing to the folder have no effect on Aqua Mail.

At this time I have no other mail clients connecting to this account.

Another remote possibility is under advanced server settings you might have "show only subscribed folders" unchecked.

"Show only subscribed folders" is checked for this email account.

Again, the only way to be sure what is really happening is with an IMAP:5 log. I.e., is tb really unsubscribing and is the server reporting the folder as unsubscribed.

How can I collect the logs?

And I'd like to reiterate. It sounds like server-side subscriptions are both unreliable and don't work well with multiple clients. I would really prefer to have a local subscription mechanism.

Comment 9

[andysem]

Attachment: tb_imap.log

Here is a log excerpt where I'm unchecking the folders in the "Subscribe..." menu.

Comment 10

[gene smith]

Thanks for the log. It does show that the server accepts all the unsubscribe commands for your backup folder and the about 25 sub-folders. When I do the same thing I see the same thing but then I see in the log "discoverallboxes" URL occur which triggers in TB a list (subscribed) "" "*" return (special-use) imap command which returns all the subscribed folders. Since the ones I unsubscribed are not in the list, TB erases them in the UI.

So maybe you cut off the log before the list occurred? Or maybe it never happened for some unknown reason. So if you could provide the log after the last

95 unsubscribe "[e-mail]/x265"
95 OK UNSUBSCRIBE done

it might help.

The next imap commands I see are:

list (subscribed) "" "*" return (special-use)

This is in response to the URL command discoverallboxes which occurs after the unsubscribe imap commands finish.

If you don't see any "discoverallboxes" or "list" commands in the log, it would help if you could record a log with TB starting up. There should always be a "discoverallboxes" at that time so any now unsubscribed folders should not appear in the "list" response or on the UI.
Note: Depending on server capabilities, you might see an imap lsub command instead of list in response to the "discoverallboxes" URL.

Comment 11

[andysem]

Attachment: thunderbird.log.gz

Here is a more complete log, from TB start to finish. I don't see discoverallboxes after unsubscribing.

PS: In case you wonder, yes, the log ends mid-word. This is not me cutting it early.

Comment 12

[gene smith]

Ok, thanks again for the log.

I assume that before you recorded the log the last thing you did was attempt to unsubscribe the backup folder and its sub-folders. Then you restarted tb? If that's what you did, it appears that the original unsubscribe was not respected by the server.

Or maybe before you recorded the log you checked that the backup folder and sub-folders were subscribed (according to the subscribe dialog) and then you ran tb while recording the log, unsubscribed the folders and we see there is no attempt by TB to re-discover the folders after what looks like successful unsubscribes.

After the unsubscribes, I see tb checking the status on all the folders, including the ones you unsubscribed, but, you're right, there is no discoverallboxes. Do you have tb set to check for new mail on all folders?

Anyhow, something maybe looks wrong on the TB side. How folder discovery occurs is very dependent on the server capabilities so I need to look closer at your server's imap CAPABILITY response and see if I can duplicate the problem with another server.

By any chance do you know the internal server type used by mail.ru? Maybe they developed their own or are using something like dovecot, courier or cyrus?

Comment 13

[gene smith]

I can partially duplicate the problem with gmail. It supports XLIST like mail.ru. However, when I unsubscribe (or subscribe) I don't see a discoverallboxes occur so the UI does not update to reflect the change. Collapse/expand also doesn't cause an update. However, on TB restart the UI change occurs (due to initial discoverallboxes URL) and the folder disappears (or appears for subscribe) as it should.

So there seems to be a bug in TB that it doesn't do discoverallboxes on a subscription change with XLIST in effect.
However, it appears that mail.ru is also not respecting the unsubscribe either.

Comment 14

[gene smith]

Re: comment 8:

Well, this is certainly not the expected behavior. Subscribing should be client-specific. I may want to see this folder in one client (on one machine) but not the other.

See https://superuser.com/questions/432106/is-imap-unsubscribe-meant-to-work-across-mail-clients

TB doesn't currently have a mechanism to hide or reveal a folder (mailbox) independent of imap subscribe.

Comment 15

[andysem]

(In reply to gene smith from comment #12)

Ok, thanks again for the log.

I assume that before you recorded the log the last thing you did was attempt to unsubscribe the backup folder and its sub-folders. Then you restarted tb? If that's what you did, it appears that the original unsubscribe was not respected by the server.

I'm not sure I did exactly that. I mean, the only thing I did wrt. subscriptions is unsubscribing (because the folders always stay subscribed), but I'm not sure I unsubscribed immediately before restarting.

In any case, I've just reproduced this the way you describe:

• Unsubscribe from folders.
• Close TB and start with logs enabled.
• The unsubscribed folder is present in the UI and log.

So, if the server supports subscribing, it appears to be not persistent.

Do you have tb set to check for new mail on all folders?

I think so. I don't remember an option like this in settings, and new messages do appear in all folders. The folder is enabled "for offline use" in the Synchronization tab of its properties (which is the default).

By any chance do you know the internal server type used by mail.ru? Maybe they developed their own or are using something like dovecot, courier or cyrus?

Unfortunately, I don't know. It's a public service, much like Gmail, and I haven't found any documentation on its internals.

Comment 16

[gene smith]

Ok, I know what the problem is and I've seen it before. It really has nothing to do with XLIST or LIST or anything like those. On both servers you are using OAUTH2 authentication. There is an issue (bug?) in the TB code the prevents the discoverallboxes URL from occurring since TB can't get a password with OAUTH2 in effect: bug 1714572 comment 36

My proposed fix was just to remove the check for a password in the PerformExpand() function. However, I'm not an expert on OAUTH2 and no one responded yes or no to my proposal. The reporter of the original bug lost interest and the bug was closed due to lack of response.

I asked: Do you have tb set to check for new mail on all folders?

I think so. I don't remember an option like this in settings, and new messages do appear in all folders. The folder is enabled "for offline use" in the Synchronization tab of its properties (which is the default).

I think it's this hidden pref: mail.server.default.check_all_folders_for_new

Anyhow, not really relevant to the bug. I was just wonder why all the folders were being checked for status.

Comment 17

[gene smith]

If you can login to mail.ru with a plain password instead of OAUTH2, you might see different behavior. (I do see PLAIN authentication in the capabilities.) However, don't know if that will cause the unsubscribe to be respected but you should then see the "discoverallboxes" in the log after the unsubscribe.

Comment 18

[andysem]

(In reply to gene smith from comment #16)

I asked: Do you have tb set to check for new mail on all folders?

I think so. I don't remember an option like this in settings, and new messages do appear in all folders. The folder is enabled "for offline use" in the Synchronization tab of its properties (which is the default).

I think it's this hidden pref: mail.server.default.check_all_folders_for_new

This property is set to true.

Comment 19

[andysem]

Switching to "Normal password" auth method does bring discoverallboxes after all unsubscribe responses. However, after that XLIST, LIST and LSUB commands return all folders, including the unsubscribed ones.

Comment 20

[Alfred Peters [:infofrommozilla]]

Dupe of Bug 560200!?
Even the issue with mail.ru is already known: Bug 560200 comment #8

Comment 21

[gene smith]

You would think after at least 12 years that server would have been updated to conform to RFC. Anyhow, yes, does look like a dupe of bug 560200. Thanks for digging that up.

However this bug still points out the problem in bug 1714572 where the discovery is skipped when OAUTH2 is in effect and a TB restart is needed to see that the unsubscribe actually worked.

Reporter Andysem, if you care about it, you could submit an enhancement bug requesting a feature to hide folders independent of the subscribe mechanism. I think the reporter of bug 560200 also requested something like that.

Comment 22

[andysem]

I'm reopening this bug as a request to implement client-side folder subscriptions. Unfortunately, I can't modify the original bug description, but the idea is described in comment #6. If someone is able to update bug description, please feel free to do so.

Comment 23

[gene smith]

Attachment: 1759883-fix-oauth2-imap-folder-subscriptions.patch

There are several bugs that mention this issue that are not really duplicates. The imap password is empty as returned by GetPassword() when the mailbox uses oauth2. This causes changes in the subscription dialog to not be immediately reflected in the folder tree pane. It also causes folders when newly added or removed to not be reflected until a restart occurs when oauth2 is in effect.
Note: This doesn't implement a new "client-side folder subscription" as requested in comment 22.

A similar fix was applied in bug 1745205 also caused by empty oauth2 password returned by GetPassword().

A try build for this seems to succeed: https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=f196253a86c5d3fb42f82742673a6fe6ecef41bd
I've since fixed the "clang-format" complaint.

Comment 24

[Rachel Martin]

Comment on attachment 9275785 [details] [diff] [review]
1759883-fix-oauth2-imap-folder-subscriptions.patch

Seems reasonable. Would you like us to test this? On Yahoo?

Comment 25

[gene smith]

(In reply to Rachel Martin from comment #24)

Comment on attachment 9275785 [details] [diff] [review]
1759883-fix-oauth2-imap-folder-subscriptions.patch

Seems reasonable. Would you like us to test this? On Yahoo?

Thanks, another check would be good. I tested it on AOL (yahoo based). It still requires the number of cached conns to be set down to 3 or you still get some flaky results even with this patch. Gmail should be fixed even with default 5 conns.

Comment 26

[gene smith]

(In reply to andysem from comment #22)

I'm reopening this bug as a request to implement client-side folder subscriptions. Unfortunately, I can't modify the original bug description, but the idea is described in comment #6. If someone is able to update bug description, please feel free to do so.

This feature request is already documented in bug 560200.

Comment 27

[Rachel Martin]

(In reply to gene smith from comment #25)

Thanks, another check would be good.

In our brief testing on Yahoo with 3 connections, without the patch changes to the subscribed folders didn't appear to take effect, with the patch they did. So this appears to be a definite improvement.

Comment 28

[gene smith]

Thanks Ben!

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/86aca28d9c81
Subscribe/unsubscribe to oauth2 imap folders requires restart. r=benc