1624699 - Thunderbird refuses to subscribe to more than a certain number of folders.

Status: RESOLVED INVALID

(MailNews Core :: Networking: IMAP, defect)

Description

[Garret Wilson]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0

Steps to reproduce:

I'm using Thunderbird 68.6.0 on Windows 10. I'm accessing an IMAP server from a well established email provider, which I have used for years. I have also used Thunderbird for well over a decade.

I am attempting to use Thunderbird to backup my IMAP account locally. I installed Thunderbird from scratch on a separate machine. I connected with no problems. For some reason Thunderbird arbitrarily was subscribed to some folders but not others; I don't know why it was subscribed to some folders and not others on a fresh install. (This is a separate issue; while puzzling, I don't want this ticket to get sidetracked.)

Nevertheless I right-clicked on the email account, selected "Subscribe...", refreshed the folders, selected all folders (hundreds of them), and clicked the "Subscribe" button, and then "OK". After a while Thunderbird started downloading messages.

I let Thunderbird run overnight, but this morning perhaps half the folders have not been downloaded. I looked at the subscribed folders, and only about the first half of them are selected. I tried to select the second half, but Thunderbird apparently ignores the updated selection. That is, I will select some new folders after the "cutoff line" where folders are no longer subscribed, and I'll select "OK", but when I go back into the subscribed folders, the new folders I selected are no longer selected.

It doesn't matter if I click the new folders manually, or if I select several folders and then hit the "Subscribe" button. It doesn't matter if I select a folder immediately after the "cutoff line", or if I go to the end and select the last few folders. Whatever I do, Thunderbird does not add new subscribe folders after a set number of selected folders. (I counted and it's way over 256; then I stopped counting.)

Note that there is no pattern related to the number of nested folders. Literally in my nested folder tree in the folder subscription dialog, all the folders are subscribed down to a certain line, after which only a few folders are subscribed—ones that were subscribed already. After this invisible "line" I cannot add a new folder to be subscribed. It simply doesn't "take" or "stick" when I close the dialog.

I have no bandwidth limitations. The profile is on an almost empty volume with literally dozens of gigabytes available.

Does Thunderbird have a limit on the number of folders it can subscribe to?

Actual results:

Thunderbird did not subscribe to additional folders. In fact it unchecked the boxes I had checked in the folder subscription dialog.

Expected results:

Thunderbird should have subscribed to the new folders, left the checkboxes checked in the folder subscription dialog, and downloaded the messages in the folders

Comment 1

[Garret Wilson]

Now all of a sudden 24 hours later Thunderbird is letting me add more folders to the list of subscribed folders. And oddly after starting it up again, it starts downloading one of the large folders that happened to be subscribed after that invisible "dividing line", such as "Sent". (Remember that I had mentioned that a few folders at the bottom had been subscribed already, for whatever reason. But not just "Sent"—a lot of arbitrary folders, too.)

But now that it seems to be subscribing to additional folders and downloading them, in the profile folder for that account a lot of the names are ending in -1 and -1.msf. So for example I'll have all these three files:

• foo.msf
• foo-1
• foo-1.msf

These all have file dates of today, a few minutes apart, in that order. What is the meaning of this -1 extension?

Comment 2

[Firefox Bug Husbandry Bot]

Because this bug's Severity has not been changed from the default since it was filed, and it's Priority is -- (Backlog,) indicating it has has not been previously triaged, the bug's Severity is being updated to -- (default, untriaged.)

Comment 3

[Firefox Bug Husbandry Bot]

Because this bug's Severity has not been changed from the default since it was filed, and it's Priority is -- (Backlog,) indicating it has has not been previously triaged, the bug's Severity is being updated to -- (default, untriaged.)

Comment 4

[Firefox Bug Husbandry Bot]

Because this bug's Severity has not been changed from the default since it was filed, and it's Priority is -- (Backlog,) indicating it has has not been previously triaged, the bug's Severity is being updated to -- (default, untriaged.)

Comment 5

[Firefox Bug Husbandry Bot]

The severity of these bugs was changed, mistakenly, from normal to S3.

Because these bugs have a priority of --, indicating that they have not been previously triaged, these bugs should be changed to Severity of --.

Comment 6

[Wayne Mery (:wsmwk)]

In my experience, the -N files and folders appear when thunderbird is unsuccessful in contacting the server.
Alfred, do you agree?

Comment 7

[gene smith]

Not Alfred, but I've seen the -N folders appear when you unsubscribed and then re-subscribe to the folder. Tb has to re-download headers (foo.msf) and mbox files (foo) after a resubscribe so it keeps the previous named files as backup and and creates new ones with a incremental suffix.

Reporter, I know it's been a while, but have you resolved the issue?

I don't know of any hard limit on number of folders or the number that can be subscribed to at all.

Tb must see folders as reported in the imap "list" server response that have the \subscribed state set. So probably the server is ignoring the request by tb to subscribe to certain folders. If they are not reported by the server as subscribed, they won't show up in the folder list and messages will not be fetched into those folders.

The only way to know for sure what is happening is by looking at the IMAP:5 log that you can produce and attach above. See https://wiki.mozilla.org/MailNews:Logging. The subscribe list should be returned by the server on each tb startup and is called "folder discovery".

Might be helpful to also know the imap provider and exactly how many folders you have which I assume is more than 256.

Comment 8

[Alfred Peters [:infofrommozilla]]

(In reply to gene smith from comment #7)

Not Alfred, but I've seen the -N folders appear when you unsubscribed and then re-subscribe to the folder. Tb has to re-download headers (foo.msf) and mbox files (foo) after a resubscribe so it keeps the previous named files as backup and and creates new ones with a incremental suffix.

We should discuss this in the appropriate bug. -> Bug 1287223

Comment 9

[Alfred Peters [:infofrommozilla]]

I can confirm, that TB has a problem with many folders. I had seen this already in Bug 1554188 but can confirm it now outside of a virtual folder.

First: I have no problem to subscribe to over 1000 folders. I can subscribe and unsubscribe at will.

But if let TB synchronize those folders I get an Error: "Unable to open the summary file for <folder>. Perhaps ther was an error on disk..."
I don't think, it is really a disk issue.

(In reply to gene smith from comment #7)

Reporter, I know it's been a while, but have you resolved the issue?

I see it in TB trunk.

I don't know of any hard limit on number of folders or the number that can be subscribed to at all.

Agreed.

Might be helpful to also know the imap provider and exactly how many folders you have which I assume is more than 256.

This varies from time to time. So there is really no fix limit.

Comment 10

[Alfred Peters [:infofrommozilla]]

Attachment: TB_Err_Subfolder.png

Tb must see folders as reported in the imap "list" server response that have the \subscribed state set. So probably the server is ignoring the request by tb to subscribe to certain folders. If they are not reported by the server as subscribed, they won't show up in the folder list and messages will not be fetched into those folders.

The folder structure contains 10 * 10 * 10 folders with one e-mail each. Here the problems started after a little more than 410 folders.

Comment 11

[Garret Wilson]

(In reply to gene smith from comment #7)

Reporter, I know it's been a while, but have you resolved the issue?

I think I was the reporter on this ticket.

As I mentioned in Comment 1, after I opened Thunderbird again the next day, it magically started downloaded the remaining folders. So for this particular set of folders, I've at least worked around the issue.

To find out if it is "resolved" I would have to delete the entire mailbox and start downloading everything from scratch. In addition, I have ensured that all the folders are subscribed on the IMAP server side, now, so to reproduce this I would also have to unsubscribe them on the server and then try again to subscribe them all on Thunderbird. I just don't have time to do this right now.

Note that there are two separate "issues", with the first one being the worse problem:

• Thunderbird not subscribing all selected folders during a session.
• These odd, extraneous, and undesired folder -1 suffixes.

I don't know if these problems are related.

I don't know of any hard limit on number of folders or the number that can be subscribed to at all.

It seemed to be limited to the session, because the next day (after a Thunderbird restart and indeed a system reboot) I was able to subscribe to the folders.

Might be helpful to also know the imap provider and exactly how many folders you have which I assume is more than 256.

I don't know the number of folders, but as mentioned in the description it was way over 256.

Right now I've successfully subscribed to all the folders and I just don't have time to unsubscribe them and do tests. So I hope you can find the problem, but at least now I know how to work around them. But I really would like to know about the ugly -1 files in Bug 1287223.

I suggest you look at the code that talks to the UI to get the selected folders and then goes out to do the actual subscriptions. There may be a particular data structure or a bug limiting the number of subscriptions for a particular session.

Comment 12

[gene smith]

(In reply to Alfred Peters from comment #9)

I can confirm, that TB has a problem with many folders. I had seen this already in Bug 1554188 but can confirm it now outside of a virtual folder.

FirI [First] have no problem to subscribe to over 1000 folders. I can subscribe and unsubscribe at will.

Reporter Garret,
Based on Alfred's input above, for now I will assume there is no problem in tb that prevents subscribing to an unlimited number of folders. If you or someone reading this sees the problem again then the IMAP:5 log, as requested in comment 7 above, will be helpful in debugging the problem and we can do that then.

The problem with the file suffixes and other problems caused by a large number of folders will be handled in the other bug reports since those issues don't prevent subscriptions. So for lack of a better category I'm closing this as INVALID.

Note: A workaround if you are unable to subscribe to folders is to just allow tb to discover and show all folder whether subscribed or not. This is set in the Server setting | Advanced... by un-selecting "Show only subscribed folders".

Sorry for the big delay in getting back to you on this.

Comment 13

[Garret Wilson]

(In reply to gene smith from comment #12)

Note: A workaround if you are unable to subscribe to folders is to just allow tb to discover and show all folder whether subscribed or not. This is set in the Server setting | Advanced... by un-selecting "Show only subscribed folders".

Gene, I am aware of that option, but I think there was some drawback to it. (I don't remember; it's been a while since I tried his.)

Will using this option will that cause Thunderbird to download all messages in all folders automatically, even those that aren't subscribed to? The whole purpose of this scenario for me was to download all messages as a poor-persons backup mechanism. I need a way to tell Thunderbird to download all the message in all the folders, not just show them.

Comment 14

[gene smith]

"Show only sub'd folders" unchecked will cause all folders on the server to appear. The drawback to this is that on some imap servers like UW IMAP (Mark Crispin's original imap reference implementation) you could end up seeing all the directories you own or have access to on the server system, not just IMAP folders. So the "subscription" concept (along with namespaces) was invented to preclude this. But this is not a problem with most modern servers, AFAIK. You may still end up seeing some folder you don't care about, especially on servers like gmail and outlook.

If you can see the folder listed in tb they should be downloaded automatically. This assume you have autosync enabled which is by default. You may have to go though the list and select (click on) any folders you want downloaded first but eventually all folders will be sync'd. But just to hurry the process up, I would recommend quickly selecting each of the folders to let tb know you want the content loaded. You don't have to sit and wait on each one to load. Note: I haven't actually tried/tested this with >256 folder account.
You can also set in folder properties that the folder is checked along with Inbox for new mail based on the timer or clicking "get mail" button. However, this would require going through all your folders and setting each individually. This would be faster if you didn't have to go through every folder and set the property individually.

Comment 15

[gene smith]

Assume you are seeing all the folders of interest in tb (subscribed to or show even if not subscribed) probably a better and quicker way to complete the sync process is to go offline. You will be prompted to download the folders for the selected account before tb goes offline. This will scan through all your folders and do an imap select on each so tb gets a picture of what may still need to be fetched (downloaded). This may be faster than waiting for autosync to "lite-select" (do imap status) on each folder and download them in the background. When going offline you will get definite feedback on when the sync is done when you see tb go to offline mode.