Closed Bug 1843637 Opened 1 year ago Closed 1 year ago

v115 fails to subscribe Imap subfolders at folder hierarchy-depth > 2; v <=102 works OK

Categories

(Thunderbird :: Folder and Message Lists, defect, P1)

Product:
Thunderbird
Component:
Folder and Message Lists
Version:
Thunderbird 115
Platform:
x86_64
All
Type:
defect

Tracking

(thunderbird_esr102 unaffected, thunderbird_esr115+ fixed, thunderbird116 fixed)

Status:
RESOLVED FIXED
Milestone:
117 Branch
Tracking Flags:
Tracking Status
thunderbird_esr102 --- unaffected
thunderbird_esr115 + fixed
thunderbird116 --- fixed

People

(Reporter: pgnd, Assigned: gds, NeedInfo)

References

(Blocks 1 open bug,
URL
)

Details

(Keywords: regression, Whiteboard: [Supernova3p])

Attachments

(2 files, 2 obsolete files)

before-after.png
5.74 KB, image/png
Details
Bug 1843637 - Fix to trigger imap folder discovery on expand. r=darktrojan
48 bytes, text/x-phabricator-request
wsmwk
: approval-comm-beta+
wsmwk
: approval-comm-esr115+
Details | Review

Steps to reproduce:

i run tb115/linux, upstream

Application Basics

Name: Thunderbird
Version: 115.0
Build ID: 20230711125218
Distribution ID:

Update Channel: release
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0 Thunderbird/115.0
OS: Linux 6.3.12-200.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Jul  6 04:05:18 UTC 2023
OS Theme: Breeze / Adwaita

Multiprocess Windows: 0/0
Fission Windows: 0/0
          Enabled by user
Remote Processes: 5
Enterprise Policies: Inactive
Google Location Service Key: Missing
Google Safebrowsing Key: Missing
Mozilla Location Service Key: Missing
Safe Mode: false
Memory Size (RAM): 62.6 GB
Disk Space Available: 168 GB

my IMAP server is a local Dovecot instance

i've a 'deep' MAIL folder hiearchy

  • myAccount
    -- INBOX
    -- Sent
    -- Junk
    -- Trash
    -- Drafts
    -- folderA
    --- subA1
    ---- subA11
    ---- subA12
    --- subA2
    ---- subA11
    ----- subA111
    ------ subA1111
    ----- subA112
    ------ subA1121
    ...

i do NOT save in TB for Offline -- I keep all @ IMAP

for any TB version <=v102, on account creation/setup, all the folders are seen, and the full (sub)folder hierarchy is dsiplayed & accessible in the client

if I

stop TB
cd /home/pgnd/.thunderbird/xA1B2C3y.Dovecot
rm -rf ImapMail
start TB

the hierarchy is correctly rebuilt, and, as above, all's good.

on upgrade to TB v115, whether I

create a new account

or

stop + delete ImapMail + start

TB only finds/displays to limited depth,

  • myAccount
    -- INBOX
    -- Sent
    -- Junk
    -- Trash
    -- Drafts
    -- folderA
    --- subA1
    --- subA2
    ...

NONE of

  • myAccount
    ...
    --- subA1
    ---- subA11
    ---- subA12
    ...
    ---- subA11
    ----- subA111
    ------ subA1111
    ----- subA112
    ------ subA1121

or their contents are seen/displayed.

i'm unable to get v115 to walk to greater depth

drop back to v102, all's good again.

this is reproducible here, atm, on multiple TB clients, with 2 different Dovecot server instances, and one CyrusMail server instance (@ FastMail)

as a result, 1000's of messages per account are unavailable/hidden from TB v115 client view

checked subscriptions @ depth with

Outlook/Windows
Clawsmail/Linux
Evolution/Linux

All are good -- as TB <=v102.

So far, this is unique to TB 115.

OS: Unspecified → Linux
Hardware: Unspecified → x86_64
Flags: needinfo?(gds)

I hereby confirm the bug.

This as well is a show stopper and makes TB 115 impossible to use in companies and for some private persons as well. We have to stay with 102 until this is fixed, or use another MUA if 102 is abandoned before it is fixed.

I can add some additional information, though (Environment: TB 115 64-bit, Windows, IMAP / no local folders):

  • The OP says that he had synchronization turned off. In our case, synchronization is turned on (all messages in all folders), but the behavior is the same, at least with folders that have not been synchronized yet before the upgrade. This means that 115 show all folder correctly (regardless of the nesting level) that had been synchronized before the upgrade. Deeply nested folders that are created elsewhere after a client has been upgraded are inaccessible by that client. If 115 is newly installed (i.e., there are no synchronized messages or folders yet), all folders that are deeply nested are inaccessible after the installation.
  • In 115, when we try to subscribe to folders, in the subscription window the folder list is complete as far as I can tell. This means that 115 in general can correctly list and access all folders and subfolders, but the UI (folder hierarchy at the left side) is flawed.

This has drastic consequences. As the OP already has stated,

  • for environments where synchronization had been turned before the upgrade, 115 makes all messages inaccessible that are in deeply nested folders;
  • for environments with synchronization, 115 after a new installation does not provide access to any messages that are in deeply nested folders;
  • for environments with synchronization, 115 after an upgrade makes all messages inaccessible that are in deeply nested folders, except those messages that had been synchronized already before the upgrade.

Due to limited time, I can't do further investigations right now, but chances are that all of those message are inaccessible to the search as well.

To everybody who may think that this is harmless because it only happens with IMAP: Please note that no reasonable company uses local folders / POP3 on client machines today (no centralized backup as with IMAP, no public or shared folders as with IMAP, not possible to manage the same account / messages at different PCs as with IMAP, and so on).

As explained above, this bug, depending on circumstances, may make the most part of messages in companies inaccessible. I guess we don't need to explain that it therefore should be fixed with highest priority.

(Example of a typical environment at one of our clients: 1e6 messages, several thousands of public folders, folder nesting level up to 8, at some workplaces no synchronization, at other workplaces full synchronization; I personally know quite a few private persons who also use only IMAP just because they want to manage each of their account on every of their gadgets the same way (PC, smart phone, tablet, each with different clients); these private persons are affected as well).

Clarification: it should read "for environments where synchronization had been turned off before the upgrade, 115 makes all messages inaccessible that are in deeply nested folders;" (never try to file bug reports when you're in time pressure).

Update:

Further investigation has shown that TB 115 sometimes actively deletes folders from the local synchronization copy. We have conducted the following test:

  • Install TB 102 and TB 115 and let them use different profiles.
  • Using 102, create a deeply nested folder and move some messages into it.
  • Let 102 run a complete synchronization (File -> Offline ...) after having made sure that the new folder will be included in the synchronization (should be the case anyway when synchronization of all messages is enabled in the account setting, but additionally checking won't hurt).
  • Using the file explorer, go to 102's profile, navigate to "ImapMail" and work through the folders until you have found the newly created folder.
  • Using the file explorer, verify that this folder and its messages actually have been synchronized.
  • Close TB 102.
  • Delete the folder "ImapMail" from 115's profile.
  • Move the folder "ImapMail" from 102's profile into 115's profile.
  • Using the file explorer, go to 115's profile, navigate to "ImapMail" and work through the folders until you have found the newly created folder mentioned above.
  • Start TB 115.
  • In our case, we then could observe that TB 115 deleted that newly created folder and its contents from the file system.
  • For clarification: That folder still existed on the IMAP server, but got deleted by TB 115 from the local file system along with the messages it contained, making it inaccessible from within TB.

Further remarks:

This issue (115 actively deleting previously synchronized folders and messages) may be related to what parts of the folder structure are opened at the left side in the moment TB starts. We repeated the test described above with all accounts / folders being collapsed when TB 115 started and then didn't observe that fatal behavior.

Due to time pressure, we can't investigate the behavior in detail right now.

Finally, we are unsure whether the "active file deletion" that TB 115 shows is related to the actual subject of this bug report (TB 115 disabling access to deeply nested folders). However, it's not completely illogical to assume a relationship, so we think that this post may eventually help the developers figure out what's going on.

Best regards,

Binarus

As a quick test with what I having running (self-built daily 116) I created a new profile and created a dovecot imap account having deeply nested folders. I set it so it doesn't use offline store since I don't want to download all my many large test messages. I don't see a problem seeing or accessing existing messages in a very deep folder (deeper) having access URL this (from folder properties):
imap://gene@wally.dbnet.lan/4/rubish/baskette/truck/deep/deeper
I had put messages into "deeper" before creating the new profile.

I haven't tried this with official 115 (beta?) yet and I'm not sure if what I did matches what has been reported above.

Flags: needinfo?(gds)

I tried it with cyrus that uses / delimiters and dovecot using . delimiters. Both seem to work for me on new account creation with 116 in new profile. Will try official 115.

I think Binarus is saying the missing deeply nested problem occurs when another client creates the deeply nested folder and it doesn't show up in 115.

I did this:

Created new profile with dovecot and cyrus account using official 115.0 having existing deeply nested folders. All folders appear as expected in both accounts and messages are accessible, even at the deepest level.

With daily running with another profile, created even deeper folders in cyrus and dovecot accounts and put messages in them.

Looking with 115 the new folders don't show up instantly which is expected. I thought a new "folder discovery" would occur if I collapse and then expand the dovecot and cyrus account at the root (account name line). However, that didn't seem to work and the new folders didn't appear.

Then I restarted 115 and the new folders (deeply nested) appeared so a new folder discovery occurred then, as it should.

So the bug I see here is why collapse/expand of the account doesn't trigger a new folder discovery like I thought it would.
Not sure if Binarus and the reporter did a restart on 115 and if that made the missing nested folders appear.

From comment 0:

on upgrade to TB v115, whether I
create a new account
or
stop + delete ImapMail + start
TB only finds/displays to limited depth,

This works for me on linux with dovecot and cyrus accounts.

Not sure if Binarus and the reporter did a restart on 115 and if that made the missing nested folders appear.

in v115, i toggled parent folder tree selector & subfolder (as deep as they're displayed ....). No effect / trigger. same with TB restart.

in v102, toggling & restarts both trigger the (re)discovery & full display,

in v115, reproducibly not ...

I think you are saying restart of 115 didn't make the new folders appear.
And I think you are also saying that collapse/expand at the account level doesn't show the new folders also.

But both of these work to discover new folders with <=102. Right?

With 115, if you right click on account and check the subscription, are the missing folders subscribed?

(In reply to gene smith from comment #10)

I think you are saying restart of 115 didn't make the new folders appear.

yes

And I think you are also saying that collapse/expand at the account level doesn't show the new folders also.

yes

But both of these work to discover new folders with <=102. Right?

yes

With 115, if you right click on account and check the subscription, are the missing folders subscribed?

yes

all listed, to full depth, and all subscribed

Maybe an IMAP:4 log would be helpful to see if you are getting the initial folder discovery (a lot of "LIST" imap stuff at startup).
See https://wiki.mozilla.org/MailNews:Logging for detail on how to record the IMAP:4 log.
(It recommends IMAP:5 but IMAP:4 is smaller and OK.)

Don't know why reporter is not seeing the initial "discoverallboxes" URL to list the new folders. Need the log to find out. (Edit: Actually it is occurring.)
However, the account collapse/expand is definitely not working with 115 (and with my daily 116). Look like the expand should be detected here:
https://searchfox.org/comm-central/rev/708124bf27a7b743c2cf307d9625c1a521d8859b/mailnews/base/content/msgSynchronize.js#156 but the function onSynchronizeClick() never gets called and its file doesn't seem to be accessible with the JS debugger.

FWIW, a workaround to see the new folders is to toggle a subscription on a single folder off and then back on. That will trigger the discoverallboxes URL.

Edit: Actually discoverallboxes IS getting triggered. However "listFolders" URL is NOT getting triggered in DiscoveryDone() due to "unverifiedFolders" count being zero.

(In reply to gene smith from comment #12)

Maybe an IMAP:4 log would be helpful to see if you are getting the initial folder discovery (a lot of "LIST" imap stuff at startup).
See https://wiki.mozilla.org/MailNews:Logging for detail on how to record the IMAP:4 log.
(It recommends IMAP:5 but IMAP:4 is smaller and OK.)

on exec

 export MOZ_LOG=imap:4
 export MOZ_LOG_FILE=/tmp/mozlog.txt
 thunderbird -P Dovecot

log (way too much PII, so munged ...),

cat mozlog.txt.moz_log | grep folderA
	[Parent 28679: IMAP]: I/IMAP 7f6e3ef34f00:mx.example.net:A:CreateNewLineFromSocket: * LIST (\HasChildren \UnMarked) "/" folderA
	[Parent 28679: IMAP]: I/IMAP 7f6e3ef34f00:mx.example.net:A:CreateNewLineFromSocket: * LIST (\HasChildren \UnMarked) "/" folderA/subA1
	[Parent 28679: IMAP]: I/IMAP 7f6e3ef34f00:mx.example.net:A:CreateNewLineFromSocket: * LIST (\HasChildren \UnMarked) "/" folderA/subA2
	[Parent 28679: IMAP]: I/IMAP 7f6e3ef34f00:mx.example.net:A:CreateNewLineFromSocket: * LIST (\HasChildren \UnMarked) "/" folderA/subA3
	[Parent 28679: Main Thread]: D/IMAP proposed url = folderA folder for connection INBOX has To Wait = false can run = false
	[Parent 28679: Main Thread]: I/IMAP exceeded connection cache limit:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA
	[Parent 28679: Main Thread]: I/IMAP queuing url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA
	[Parent 28679: Main Thread]: I/IMAP considering playing queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA
	[Parent 28679: Main Thread]: I/IMAP creating protocol instance to play queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA
	[Parent 28679: Main Thread]: D/IMAP proposed url = folderA folder for connection INBOX has To Wait = false can run = false
	[Parent 28679: Main Thread]: I/IMAP exceeded connection cache limit:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA
	[Parent 28679: Main Thread]: I/IMAP failed creating protocol instance to play queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA
	[Parent 28679: Main Thread]: I/IMAP considering playing queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA
	[Parent 28679: Main Thread]: I/IMAP creating protocol instance to play queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA
	[Parent 28679: Main Thread]: D/IMAP proposed url = folderA folder for connection INBOX has To Wait = false can run = false
	[Parent 28679: Main Thread]: I/IMAP playing queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA
	[Parent 28679: IMAP]: I/IMAP 7f6e3ef34f00:mx.example.net:S-INBOX:ProcessCurrentURL:imap://me%40example%2Ecom@mx.example.net:52993/folderstatus%3E/folderA:  = currentUrl
	[Parent 28679: IMAP]: I/IMAP 7f6e3ef34f00:mx.example.net:S-INBOX:SendData: 123 STATUS "folderA" (UIDNEXT MESSAGES UNSEEN RECENT)
	[Parent 28679: IMAP]: I/IMAP 7f6e3ef34f00:mx.example.net:S-INBOX:CreateNewLineFromSocket: * STATUS folderA (MESSAGES 0 RECENT 0 UIDNEXT 110500 UNSEEN 0)
	[Parent 28679: Main Thread]: D/IMAP proposed url = folderA/subA1 folder for connection INBOX has To Wait = false can run = false
	[Parent 28679: Main Thread]: I/IMAP exceeded connection cache limit:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA1
	[Parent 28679: Main Thread]: I/IMAP queuing url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA1
	[Parent 28679: Main Thread]: I/IMAP considering playing queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA1
	[Parent 28679: Main Thread]: I/IMAP creating protocol instance to play queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA1
	[Parent 28679: Main Thread]: D/IMAP proposed url = folderA/subA1 folder for connection INBOX has To Wait = false can run = false
	[Parent 28679: Main Thread]: I/IMAP exceeded connection cache limit:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA1
	[Parent 28679: Main Thread]: I/IMAP failed creating protocol instance to play queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA1
	[Parent 28679: Main Thread]: I/IMAP considering playing queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA1
	[Parent 28679: Main Thread]: I/IMAP creating protocol instance to play queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA1
	[Parent 28679: Main Thread]: D/IMAP proposed url = folderA/subA1 folder for connection INBOX has To Wait = false can run = false
	[Parent 28679: Main Thread]: I/IMAP playing queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA1
	[Parent 28679: IMAP]: I/IMAP 7f6e3ef34f00:mx.example.net:S-INBOX:ProcessCurrentURL:imap://me%40example%2Ecom@mx.example.net:52993/folderstatus%3E/folderA/subA1:  = currentUrl
	[Parent 28679: IMAP]: I/IMAP 7f6e3ef34f00:mx.example.net:S-INBOX:SendData: 124 STATUS "folderA/subA1" (UIDNEXT MESSAGES UNSEEN RECENT)
	[Parent 28679: IMAP]: I/IMAP 7f6e3ef34f00:mx.example.net:S-INBOX:CreateNewLineFromSocket: * STATUS folderA/subA1 (MESSAGES 110 RECENT 0 UIDNEXT 111 UNSEEN 6)
	[Parent 28679: Main Thread]: D/IMAP proposed url = folderA/subA2 folder for connection INBOX has To Wait = false can run = false
	[Parent 28679: Main Thread]: I/IMAP exceeded connection cache limit:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA2
	[Parent 28679: Main Thread]: I/IMAP queuing url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA2
	[Parent 28679: Main Thread]: I/IMAP considering playing queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA2
	[Parent 28679: Main Thread]: I/IMAP creating protocol instance to play queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA2
	[Parent 28679: Main Thread]: D/IMAP proposed url = folderA/subA2 folder for connection INBOX has To Wait = false can run = false
	[Parent 28679: Main Thread]: I/IMAP exceeded connection cache limit:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA2
	[Parent 28679: Main Thread]: I/IMAP failed creating protocol instance to play queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA2
	[Parent 28679: Main Thread]: I/IMAP considering playing queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA2
	[Parent 28679: Main Thread]: I/IMAP creating protocol instance to play queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA2
	[Parent 28679: Main Thread]: D/IMAP proposed url = folderA/subA2 folder for connection INBOX has To Wait = false can run = false
	[Parent 28679: Main Thread]: I/IMAP playing queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA2
	[Parent 28679: IMAP]: I/IMAP 7f6e3ef34f00:mx.example.net:S-INBOX:ProcessCurrentURL:imap://me%40example%2Ecom@mx.example.net:52993/folderstatus%3E/folderA/subA2:  = currentUrl
	[Parent 28679: IMAP]: I/IMAP 7f6e3ef34f00:mx.example.net:S-INBOX:SendData: 125 STATUS "folderA/subA2" (UIDNEXT MESSAGES UNSEEN RECENT)
	[Parent 28679: IMAP]: I/IMAP 7f6e3ef34f00:mx.example.net:S-INBOX:CreateNewLineFromSocket: * STATUS folderA/subA2 (MESSAGES 0 RECENT 0 UIDNEXT 116742 UNSEEN 0)
	[Parent 28679: Main Thread]: D/IMAP proposed url = folderA/subA3 folder for connection INBOX has To Wait = false can run = false
	[Parent 28679: Main Thread]: I/IMAP exceeded connection cache limit:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA3
	[Parent 28679: Main Thread]: I/IMAP queuing url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA3
	[Parent 28679: Main Thread]: I/IMAP considering playing queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA3
	[Parent 28679: Main Thread]: I/IMAP creating protocol instance to play queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA3
	[Parent 28679: Main Thread]: D/IMAP proposed url = folderA/subA3 folder for connection INBOX has To Wait = false can run = false
	[Parent 28679: Main Thread]: I/IMAP exceeded connection cache limit:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA3
	[Parent 28679: Main Thread]: I/IMAP failed creating protocol instance to play queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA3
	[Parent 28679: Main Thread]: I/IMAP considering playing queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA3
	[Parent 28679: Main Thread]: I/IMAP creating protocol instance to play queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA3
	[Parent 28679: Main Thread]: D/IMAP proposed url = folderA/subA3 folder for connection INBOX has To Wait = false can run = false
	[Parent 28679: Main Thread]: I/IMAP playing queued url:imap://me@example.com@mx.example.net:52993/folderstatus>/folderA/subA3
	[Parent 28679: IMAP]: I/IMAP 7f6e3ef34f00:mx.example.net:S-INBOX:ProcessCurrentURL:imap://me%40example%2Ecom@mx.example.net:52993/folderstatus%3E/folderA/subA3:  = currentUrl
	[Parent 28679: IMAP]: I/IMAP 7f6e3ef34f00:mx.example.net:S-INBOX:SendData: 126 STATUS "folderA/subA3" (UIDNEXT MESSAGES UNSEEN RECENT)
	[Parent 28679: IMAP]: I/IMAP 7f6e3ef34f00:mx.example.net:S-INBOX:CreateNewLineFromSocket: * STATUS folderA/subA3 (MESSAGES 1 RECENT 0 UIDNEXT 121080 UNSEEN 1)

if you need other specific log detail, pls specify

FWIW, a workaround to see the new folders is to toggle a subscription on a single folder off and then back on. That will trigger the discoverallboxes URL.

here,
toggling the subssciption on any/all of (1) account top, (2) 'shallowest' account subfolder under (1), or (3) 'deepest' subfolder under (2) seems to have no effect.

nothing's triggered/refreshed

fwiw,

in

Account -> Server Settings -> Advanced ...

settings

IMAP server directory:
[ ] Show only subscribed folders
[X] Server supports folders that contain sub-folders and messages
Maximum number of server connections to cache	1
These preferences specify the namespaces on your IMAP server
 Personal namespace: "virtual/",""
 Public (shared):
 Other Users: "shared/"
[X] Allow server to override these namespaces

Ok, I'm also only seeing 2 levels in the list responses when I set my "advanced" config like yours. I assume you had it set this way for 102. If so, I suspect that setting the number of "cached" connection back to 5 (the default) and checking "show only sub'd folders" will fix it. But I'll try to determine why that breaks it.

Note: I wrote all this below here before seeing your comment 15.
My request to discover folders and the first few list responses look like this for dovecot:

wally.dbnet.lan:A:ProcessCurrentURL:imap://gene@wally.dbnet.lan:993/discoverallboxes:  = currentUrl
wally.dbnet.lan:A:SendData: 20 list (subscribed) "" "*" return (special-use)
wally.dbnet.lan:A:CreateNewLineFromSocket: * LIST (\Subscribed) "." 4
wally.dbnet.lan:A:CreateNewLineFromSocket: * LIST (\Subscribed) "." 4.rubish
wally.dbnet.lan:A:CreateNewLineFromSocket: * LIST (\Subscribed) "." 4.rubish.baskette
wally.dbnet.lan:A:CreateNewLineFromSocket: * LIST (\Subscribed) "." 4.rubish.baskette.truck
wally.dbnet.lan:A:CreateNewLineFromSocket: * LIST (\Subscribed) "." 4.rubish.baskette.truck.deep
wally.dbnet.lan:A:CreateNewLineFromSocket: * LIST (\Subscribed) "." 4.rubish.baskette.truck.deep.deeper
wally.dbnet.lan:A:CreateNewLineFromSocket: * LIST (\Subscribed) "." 4.rubish.baskette.truck.deep.deeper.deepest
wally.dbnet.lan:A:CreateNewLineFromSocket: * LIST (\Subscribed) "." 4.rubish.baskette.bicycle
wally.dbnet.lan:A:CreateNewLineFromSocket: * LIST (\Subscribed) "." "4.rubish.baskette.car(automobile)"
<snip>

Notice that it's showing folders at all the levels. Yours server is not returning all the levels it seems.
I need to see the see the imap request that's asking for the list like this:

wally.dbnet.lan:A:SendData: 20 list (subscribed) "" "*" return (special-use)

toggling the subssciption on any/all of (1) account top, (2) 'shallowest' account subfolder under (1), or (3) 'deepest' subfolder under (2) seems to have no effect.

Well, if the server's not LISTing the deeper folders as it isn't on startup then toggling subscriptions won't help either.

I also need to see the response to imap CAPABILITY for folderA. This may affect how the discoverallboxes does the imap LIST command. (Actually, the response should be the same for all folders on the imap server.)
FYI, here's my dovecot capability response from my imap log:

2023-07-17 20:59:29.059838 UTC - [Parent 496706: IMAP]: I/IMAP 7fffb3fd7a00:wally.dbnet.lan:NA:CreateNewLineFromSocket: 22 OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE SPECIAL-USE LITERAL+ NO

Gene, thank you very much for working on the problem!

Some hints:

  • In our case, the server is Cyrus imapd (currently 3.2.6) / Linux.
  • If the server is the culprit (which of course is not impossible), why did 102 not show the misbehavior? We literally copied the profile from a flawlessly working 102 installation and used it from the beginning on with 115 without altering anything.
  • Yesterday, we took the time and spent the whole work day with tests and bug reports for 115. While doing so, we have restarted it countless times, but it didn't find those folders.
  • Collapsing and expanding the folder tree structure (beginning at the account level, then at every possible level) countless times did not help either.
  • In our tests, we mainly concentrated on a certain folder that previously had been synchronized in 102 and thus was still synchronized in 115 (because we have copied the profile as described above). Deleting this folder and its associated .msf file from the ImapMail folder in the profile did not help either (we did this in the hope that it would make TB 115 re-scan the folders at the respective level).
  • We have the server settings as described in comment #15, with the following differences: Personal Namespace = "", Public (shared) = "public/", Other Users = "other/"; Maximum number of server connections to cache = 5. For the time being, we only use the personal and the public namespace, not the other users namespace.

I'll now test whether "Show only subscribed folders" makes a difference (as suggested in comment #16) and report the result.

In my previous post, I forgot to state again the following:

  • In the subscribe window, TB 115 lists all folders in all nesting levels correctly. I have stated this in one of my previous posts, but perhaps not prominently enough. I believe that this means that the server it not the culprit. Perhaps the problem can be analyzed further by comparing the code path that is used to create the folder list in the subscribe window to the code path that is used to create the folder list in the main window.
  • In contrast, in the account settings, in Synchronization and Storage, in the Advanced ... window, the same folders as in the folder list in the main window are missing.

One more time, all folders are shown in the subscribe window, but many folders are missing in the list of the individual folders to be synchronized and in the folder list in the main window.

And while preparing the test as promised in my previous post, I just have caught TB 115 deleting several dozens of synchronized folders from the file system while starting, although those folders had been synchronized completely, although they definitely are on the IMAP server and although synchronization of all folders and messages is still active for the account in question.

Gene, thanks a lot again!

I confirm that ticking "Show only subscribed folders" in the server configuration of the respective account and a subsequent restart made TB 115 re-scan all folders. Now all folders at all nesting levels appear in the folder list at the left side of the main window.

While this workaround is interesting, it unfortunately is not a viable solution. In our clients' environments, new folders are frequently created and others are deleted, and all folders mostly should be visible at each workplace. I guess we cannot make people check their list of subscribed folders every 15 minutes just to see whether somebody else has created a new folder they need :-) Notably given the fact that there are thousands of deeply nested folders ...

This workaround (IMHO) also shows that the server software is not the culprit. If it was, the workaround wouldn't change the situation, would it?

As an addition, I confirm that unsubscribing and re-subscribing the respective folders did not help in the previous tests as long as "Show only subscribed folders" was inactive.

@pgnd Could you please try Gene's suggestion, tick that checkbox, restart TB 115, and report the result? Thank you very much in advance.

(In reply to gene smith from comment #16)

I assume you had it set this way for 102.

yes

If so, I suspect that setting the number of "cached" connection back to 5 (the default) and checking "show only sub'd folders" will fix it.

setting

[X] Show only subscribed folders
Maximum number of server connections to cache	1

and restarting TB (just toggling folders in tree is NOT sufficient) does the trick

While this workaround is interesting, it unfortunately is not a viable solution. In our clients' environments, new folders are frequently created and others are deleted, and all folders mostly should be visible at each workplace.

agreed

I also need to see the response to imap CAPABILITY for folderA. This may affect how the discoverallboxes does the imap LIST command. (Actually, the response should be the same for all folders on the imap server.)
FYI, here's my dovecot capability response from my imap log:

mine, from an auth'd login

a OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE SPECIAL-USE LITERAL+ NOTIFY SPECIAL-USE QUOTA ACL RIGHTS=texk] Logged in

Thanks for all the good info. I thought I was getting a handle on this yesterday but still a unclear to me why the change to 115 seems to change the behavior. I need to take a fresh look again today.

Are you both just using 1 imap connection? If so, curious as to why you don't allow more connection or the default of 5. Not sure, but maybe that is affecting the type of imap LIST that occurs. Or it may be the "show unsubscribed folders" causing it, as Binarus said.

Also, I suppose you are wanting to see all folders, not just subscribed ones, so the folders in "public" namespace are immediately visible to everyone? (Folders in personal space are auto-subscribed if created in TB, but maybe not when created by other clients.)

Are you both just using 1 imap connection?

I am

it's a 'legacy' setting -- i.e., been that way ( =1 )here for a looooong time ...

according to my notes, in response to

https://electrictoolbox.com/thunderbird-exceeded-maximum-connections-imap-server/

my dovecot settings include

protocol imap {
    mail_plugins = $mail_plugins imap_acl imap_quota mail_log notify quota imap_sieve
    mail_max_userip_connections = 20
    imap_client_workarounds = tb-extra-mailbox-sep
    ...
}

service imap {
    vsz_limit = 256M
    process_limit = 256
    ...
}

in any case, it doesn't appear to be relevant. setting =1, or =5, makes no difference that I'm aware. just toggling the "show unsubscribed folders" makes a diff.

Also, I suppose you are wanting to see all folders, not just subscribed ones, so the folders in "public" namespace are immediately visible to everyone?

yes, that's been the way it's worked here.
i've 3-classes of folder namespaces -- private (user only), public (all) and shared (acl controlled)

still a unclear to me why the change to 115 seems to change the behavior

Note that https://hg.mozilla.org/comm-central/rev/17ad6ea84651 changed the intialisation of the the notification system. This intialisation visits all folders to count unread messages:
https://searchfox.org/comm-central/rev/2b828b09c4566a8e17c55c5b77eb6ef77aaf7528/mailnews/base/src/MailNotificationService.jsm#77-114
Maybe visiting all the folders early had some positive side effects on later IMAP processing (avoiding a "cold start"). We've moved the intialisation to before the point where messages are fetched, see here (ignore the hunk in MailNotificationService.jsm). This is of course only a wild guess which could be miles away from the source of the problem.

The error pointed out in the electrictoolbox link above is really a "cache all" for any network error. It probably doesn't mean your max connections were exceeded at the server but just a possible reason. So you should be able to set the max cached connections to 20 and be OK in that regard.
My dovecot protocol imap section looks like this:

protocol imap {
  # Space separated list of plugins to load (default is global mail_plugins).
  # gds note: listescape plugin breaks "select" unless "list" done first on shared folder
  mail_plugins = $mail_plugins imap_quota imap_acl listescape
# mail_plugins = $mail_plugins quota imap_quota imap_acl

  # Maximum number of IMAP connections allowed for a user from each IP address.
  mail_max_userip_connections = 10
# mail_max_userip_connections = 2
}

So using 3 to 5 imap connections in TB I would recommend. Also, having more connections available will speed up the initial folder discovery since it will occur in it's own thread which may be significant if you have a lot of folders. Anyhow, just a suggestion.

thx. seems like the tweaks may help performance, but are unrelated to the problem here. i'll changes values some more, and see if there are any surprises

setting back to

[ ] Show only subscribed folders

remains the outstanding issue

We've moved the intialisation to before the point where messages are fetched, see here (ignore the hunk in MailNotificationService.jsm). This is of course only a wild guess which could be miles away from the source of the problem.

Seems like a good guess since, AFAIK, I haven't changed anything regarding folder discovery after 102. Re: bug 163964.

Well, bug 163964 had regressions, firstly bug 1832395. We also suspect that initialising the notification system later or too late might be the root cause of bug 1824889 and heaps of duplicates.

just to confirm ...

this

Maximum number of server connections to cache	1

is per account, per server -- right?

if i have, e.g., 10 imap accounts, all to the same server ... with that set to, say, =5,
what should

mail_max_userip_connections = ??

be?

= 10account * 5connections = 50 ?

= 10account * 5connections = 50 ?

That sounds right, if the accounts are on the same IP address. (But I'm not a dovecot expert, I only use it test TB stuff.)

I looked again comparing my 116 to 103.13 with just the "show only subscribed folders" unchecked so it is supposed to shows all folders. I'm seeing the same folder discovery results with both (actually results of LIST commands). In both cases I'm only seeing two folder levels in the IMAP log and on the screen.
I created a new profile using 116 for just my dovecot account and I only see 2 levels in the folder pane. Looking at imap log, the LIST responses never go to more than 2 levels. I then shutdown 116 and run 102.13 on the same profile and see the same results in the folder pane and the same LIST commands and LIST responses in the log.

So it appears this isn't really a regression but a bug that has been around probably before 102 and it only occurs when you create a new profile for a server having more than 2 folder levels and you want to see unsubscribed folders.

From comment 18:

I just have caught TB 115 deleting several dozens of synchronized folders from the file system while starting, although those folders had been synchronized completely, although they definitely are on the IMAP server and although synchronization of all folders and messages is still active for the account in question.

This may be just because subfolders below level 2 are not being found so TB thinks they are gone and removes their files locally (but not on server). So probably just a side-effect of the discovery/LIST bug described in comment 30 -- but I could be wrong.

Anyhow, I need to look closer at the internals of "folder discovery" code which I don't think I have ever touched. (The changes I made in bug 163964 treated the discovery stuff as a "black box".)

this isn't really a regression but a bug that has been around probably before 102 and it only occurs when you create a new profile for a server having more than 2 folder levels and you want to see unsubscribed folders.

when i drop back to v102, i do not have any issues with deep-folder discovery/expansion in any case -- new profile, more than 2 folder leverl, or see unsubscribed folders ... or not

this only occurs, here, with v115.

(In reply to pgnd from comment #32)

when i drop back to v102, i do not have any issues with deep-folder discovery/expansion in any case -- new profile, more than 2 folder leverl, or see unsubscribed folders ... or not
this only occurs, here, with v115

You are correct. It was a user error on my part (again). I was running 102 with the profile I had created with 116 by having --allow-downgrade option set on the command line to run 102. Without that option, it won't let me run TB on the 116 profile and requires a new profile to run 102, which then works fine.
Also, applied the BB patch to 116 linked to in comment 24 and it didn't seem to make a difference with this bug.

(In reply to gene smith from comment #31)

From comment 18:

I just have caught TB 115 deleting several dozens of synchronized folders from the file system while starting, although those folders had been synchronized completely, although they definitely are on the IMAP server and although synchronization of all folders and messages is still active for the account in question.

This may be just because subfolders below level 2 are not being found so TB thinks they are gone and removes their files locally (but not on server). So probably just a side-effect of the discovery/LIST bug described in comment 30 -- but I could be wrong.

I totally agree. This reasoning is logical. TB 115 not finding those folders for sure is the reason for deleting them from the file system. I have only described that behavior to show that TB 115 is quite consequent in not finding those folders. Amongst others, this observation proves that the missing folders in the folder pane are not only just a problem of the UI in the folder pane, but that TB indeed thinks that those folders are gone.

(In reply to gene smith from comment #22)

Are you both just using 1 imap connection?

To cite myself :-) (comment #17):
We have left this setting at its default value (Maximum number of connections to cache = 5). This is true for all accounts. For example, at my primary workplace I have configured 8 accounts, each of of them using the same IMAP server. This probably means a maximum of 40 connections from my workplace to the server.

If so, curious as to why you don't allow more connection or the default of 5. Not sure, but maybe that is affecting the type of imap LIST that occurs. Or it may be the "show unsubscribed folders" causing it, as Binarus said.

It seems that the number of connections to cache does not help in solving the problem. I don't know whether it would become worse when we would set it to 1, because we have not tested that (and currently don't plan to conduct such tests). But as we had it left at its default value, it shouldn't cause the problem.

In contrast, in our environments, it's pretty clear that "Show only unsubscribed folders" causes the problem when not activated and solves the problem when activated (but then causes other non-technical problems so that this setting is not a long-term solution).

I guess that the technical reason is easy to understand, but the solution may be not that easy:

  • If "Show only subscribed folders" is active, TB does not need to search for folders on the IMAP server. Instead, it can use the data from the subscription database, which contains a list of folders that TB just needs to work through. Is this correct or is it too naive (no clue about IMAP ...)?
  • If "Show only subscribed folders" is not active, TB needs to search folders on the server, i.e. to recursively query the folder structure from the server. That's the thing which goes wrong.

Also, I suppose you are wanting to see all folders, not just subscribed ones, so the folders in "public" namespace are immediately visible to everyone? (Folders in personal space are auto-subscribed if created in TB, but maybe not when created by other clients.)

Exactly. Given that IMAP and public folders are a great collaboration tool, folders in the public namespace are frequently created at one workplace and should be visible at other workplaces as fast as possible.

This is also true for folders in the personal space. Please consider the following examples:

  • In a company, we often have a situation where every PC can be used by everybody (but everybody needs to login first with his own credentials to get his own work environment, including TB with his individual configuration), or where one person uses several PCs. Now imagine that a person works in the office and creates some folders in his personal IMAP account. Then he walks down into the production hall, logs into another PC and wants to continue working in his personal IMAP account. He then should see the folders he had created in the office before.
  • It is nearly the same situation when employees work via VPN. They log in to the IMAP server in the company via VPN from the comfort of their home (sometimes even when they are sick) and work on their emails, using their own gadgets, creating folders in their personal account. Next day they return to the company and of course should see those folders at their workplace.
  • In companies, when an employee is out of office, other people often must monitor his personal account for important messages that need immediate action. With IMAP, this can easily be implemented using shared folders (in contrast to public and personal folders). When connected to the shared folder (i.e. to the personal folder of a colleague), TB should correctly show the nested folder structure in that "foreign" account up to any nesting level.

That's only a few examples, but I hope they make clear why the problem that is discussed here not only affects public folders. It affects everybody who stores emails on IMAP servers and accesses them from different MUAs or different devices. This applies also to private persons (many people have their email messages stored in the IMAP server of their email provider and want to access them via IMAP from any place in the world, using a variety of different devices (e.g., a tablet when on holiday, a PC when at home), possibly creating folders at each of those devices).

As a side question (I'm eager to learn): Why is the status of this bug still "unconfirmed"? We have two people who have described very thoroughly what happens, sacrificing at least a work day, and we have one TB IMAP developer who could reproduce the problem. Just my two cents ...

As a side question (I'm eager to learn): Why is the status of this bug still "unconfirmed"? We have two people who have described very thoroughly what happens, sacrificing at least a work day, and we have one TB IMAP developer who could reproduce the problem. Just my two cents ...

I just hadn't yet clicked the box :).

Still trying to determine what the problem is. Thanks for the good info!

Status: UNCONFIRMED → NEW
Ever confirmed: true

I'm seeing maybe two issues that are causing problems when subscriptions are ignored in advanced imap server settings:

With versions <= 110 (pre-SN changes) when you create a new account, the none of the nested folders initially appear expanded. Only the top level shows with the "twisty" indicating there is something below. With newer versions, the top two levels are shown (but nothing lower) and the "twisty" is not shown on any 2nd level folder even if there is actually something below. In both cases, on first run after account creation, the "discoverallbox" URL runs and the list responses show only the top two levels.
I don't know why (or where) the UI is auto-expanding the folders right after account creation.

With <= 110, when you click the "twisty" and expand the folder, a call is made to nsImapMailFolder::PerformExpand() which when subscriptions are ignored causes a "discoverChildren" URL to occur which finds recursively all the subfolders beneath the folder. This never occurs for newer versions when you click the "twisty". It appears there is no longer a trigger in the UI code to cause PerformExpand() to get called as I pointed out in comment 13 above (which applies to expanding at the server and folder level).

On restart with <= 110, "discoverallboxes" occurs which actually only finds the top two levels. Then when in "OnStopRunningURL" all the lower level folders (children of the 2nd level) are hit with a "listFolders" URL which, similar to the "discoverChildren", finds everything below. But this never occurs for new versions since the "unverifiedCount" is always zero as checked here: https://searchfox.org/comm-central/rev/67e5345e0506b7ed56694ce1b3d334735f077b36/mailnews/imap/src/nsImapIncomingServer.cpp#1544

So a possible fix for this in UI code might be:

  • Don't auto-expand folders after first display after account creation. At subsequent start-ups leave the folder expansion as it was in last session (this may already be the case).
  • Fix the code so that expanding the folders, e.g., with "twisty", triggers PerformExpand(), both at the server and folder levels like it did with <= 110.

Requesting info from Geoff and/or Magnus

Flags: needinfo?(mkmelin+mozilla)
Flags: needinfo?(geoff)
Attached patch partial-fix-expand.diff (obsolete) — — Splinter Review

All this does is allow you to expand the server or a folder and trigger discoverallboxes or discoverChildren respectively.
However the discoverChildren only finds the next folder(s) down so you have to click again to see each level expand, but you finally can see all levels. And discoverallboxes doesn't discover everything either since "verifiedOnline" count remains 0, I think. Edit: Actually that's working correctly but there is still a display problem shown in comment 40 below.
So only a very partial "fix".

Assignee: nobody → gds
Attached image before-after.png —

This show a problem with the UI display in the folder pane. On the left is before I collapse/expand on folder "rubish". Then on the right is after collapse/expand on "rubish" and it shows the sub-folders under "baskette". The problem is that on the left there is no "twisty" widget shown on "baskette" which has sub-folders.

The same problem would occurs on the right since "truck" also contains sub-folders and the only way to show them is to collapse/expand "baskette".

Other than this, the attached diff is working OK and fixes this bug, but requires the user to collapse/expand one level up instead of just expanding at the correct level if folders "baskette" and "truck" had the twisty when initially displayed. Obviously, this is not acceptable.

Gene, thank you very much for the detailed explanation in comment #38 and the partial fix in comment #39! You're absolutely great.

That should give the developers an idea how to solve the problem, and I really hope that somebody will act upon it. I can guarantee that a great part of users will be really upset once TB 115 is distributed via the normal update mechanism and then they can't access their folders and messages any more. Again, nobody in any serious company uses local folders, and even most private persons I know are on IMAP nowadays, at least partially, and most of the users want to see every folder, not only the subscribed ones, and have configured TB accordingly.

Hence, this bug needs to be fixed immediately to prevent substantial damage to TB's reputation, to prevent loss of market share and to prevent damage to users (e.g., expensive admin time or inability to access data).

I also would advise the TB team to kick off an internal discussion about how this bug could remain undetected. The problem would have shown itself even in the most basic IMAP tests that should have been a matter of course before publishing the new version. In that sense, probably the quality assurance processes should be reviewed.

(In reply to gene smith from comment #40)

Created attachment 9344772 [details]
before-after.png

This show a problem with the UI display in the folder pane. On the left is before I collapse/expand on folder "rubish". Then on the right is after collapse/expand on "rubish" and it shows the sub-folders under "baskette". The problem is that on the left there is no "twisty" widget shown on "baskette" which has sub-folders.

The same problem would occurs on the right since "truck" also contains sub-folders and the only way to show them is to collapse/expand "baskette".

Other than this, the attached diff is working OK and fixes this bug, but requires the user to collapse/expand one level up instead of just expanding at the correct level if folders "baskette" and "truck" had the twisty when initially displayed. Obviously, this is not acceptable.

Gene, thank you much for your update.

However, the behavior at our side differs from your description.

To stay with your example, we do not see the subfolders of "baskette" when we collapse / expand "rubish". Likewise, we would not see the subfolders of "truck" when we would collapse / expand "baskette".

We have tried that sort of things countless times, and if it would have worked, we would have considered the problem as a minor glitch and wouldn't have participated in this bug report.

But unfortunately, in our case, there is absolutely no chance to get the vanished subfolders back. With several subfolders at nesting level 5 or 6 (don't remember), we have collapsed and expanded its parent folders at every other nesting level, but to no avail. We even tried countless restarts with various combinations of parent folders collapsed or expanded, and varied the order in which we collapsed and expanded the parent folders. Nothing helped.

In other words, we have absolutely no chance to access the vanished folders. Collapsing / expanding parent folders at any level and in any order does not help. That's the actual problem here. The only chance to see all folders is to use TB 102.

To make sure we're talking about the same situation:

  • We have two installations of TB on two different PCs. Call them PC1 and PC2.
  • On both PCs, we have "Show only subscribed folders" off (since we want to see all folders).
  • It does not matter whether message synchronization is on or off.
  • On PC2, create a deeply nested folder, e.g., "road" as subfolder of "bicycle" in your example.
  • Note that there is no chance to see the newly created folder "road" on PC1, even if you collapse / expand its parent folders at any nesting level a zillion times. At least, that's the situation here.

For that test, it is essential that you use two different TB installations. The problem shows when you try to see new folders that have been created elsewhere (e.g., at a different PC, or at least with a different TB profile).

Can you confirm that you used two different TB installations (in the sense above) when testing (i.e., did you create the folders in one installation and try to view them in the other installation)?

As an addition:

The trick with collapsing / expanding the parent folder to make subfolders visible works in TB 102. TB 102 also sometimes does not show new folders that had been created elsewhere immediately. However, this often could be cured by collapsing / expanding parent folders, and in cases where that didn't work, a restart helped.

But in TB 115, neither restarting nor collapsing / expanding parent folders helps; there is no chance to access those folders.

Blocks: sn-folderpane
Severity: -- → S2
status-thunderbird116: --- → affected
status-thunderbird_esr102: --- → unaffected
status-thunderbird_esr115: --- → affected
tracking-thunderbird_esr115: --- → +
Component: Networking: IMAP → Folder and Message Lists
Flags: needinfo?(alessandro)
Keywords: regression
OS: Linux → All
Priority: -- → P1
Product: MailNews Core → Thunderbird
Whiteboard: [Supernova3p]

this bug needs to be fixed immediately to prevent substantial damage to TB's reputation, to prevent loss of market share and to prevent damage to users (e.g., expensive admin time or inability to access data).

less of this, diluting the actual bug detail, will help devs focus on the bug

Binarius,
I realized I didn't make my example in comment 40 clear. This is with my proposed patch applied to my 116 daily. So, you are correct, the steps I describe won't work at all to reveal the sub-folders with official 115 since the twisty widget does not trigger anything in the imap code.
Also, in the example in comment 40, I didn't indicate that this was right after I created a new profile for the existing email account having lots of deeply nested folders and I had already done a collapse/expand at folder "4" to reveal folder "baskette". (Almost need a video to show exactly what I'm talking about.)

For that test, it is essential that you use two different TB installations. ...

Yes, I haven't tested my patch in that situation. Will do.

Yes, I haven't tested my patch in that situation. Will do.

I didn't test with 2 PCs but by running two instances of TB 116 (with my patch and configured with ignore subscriptions in "advanced" imap settings) both with the same deeply nested account. I'm able to create deeply nested folder and see them on the other instance with new folders set to subscribed or un-subscribed.

Also, FWIW, after folders are fully expanded one time after initial account creation in the TB instance, the folder collapse/expand can be done on the immediate folder, not the one above. So the issue shown in comment 40 probably only applies right after initial account creation. So you probably won't see this if you just run 115 with my patch from an existing profile/account.

I can make a "try" build on release 115.0 with my proposed patch applied if you would like to test it for yourself. Let me know.

Blocks: 1844599

Comment on attachment 9344758 [details] [diff] [review]
partial-fix-expand.diff

Went ahead and submitted patch via moz-phab.

Attachment #9344758 - Attachment is obsolete: true

From comment 46 I wrote:

So the issue shown in comment 40 probably only applies right after initial account creation. So you probably won't see this if you just run 115 with my patch from an existing profile/account.

Well wrong again. I created a 3 level tree from the top level of unsubscribed folders in tb profile A
Then on another tb instance on profile B that tree initially auto-expands to 2 levels and I have to collapse/expand the top level to see the 3rd level.
But at least the the folder expansion of my patch triggers the discovery actions now.

This fixes the collapse/expand issue by using the LIST-EXTENDED capability supported by most imap servers to do the folder discovery when subscriptions are ignored, just like when subscriptions are respected (the default) which already uses it.

Seems to work well but won't help with servers that don't support LIST-EXTENDED. Need to find out tomorrow what supports it and what doesn't and test it some more. I know dovecot does support it.

The other patch is still needed.

Gene,

thank you very much for your tests and all your work. It seems that we had a misunderstanding. I supposed that your trick with collapsing and expanding the parent folders worked in TB 115 without any patch, which wasn't the case here. Thanks for the clarification.

So I'll test TB 116 with your patch and report back the result; please allow me a day to do that. I also believe that I've got your patch in a version for TB 115, and of course will try that, too.

Since I am not acquainted with TB's release management: Is there a chance that your patch makes it into one of the next official releases of TB 115?

Best regards, and thanks again,

Binarus

Binarus,

So I'll test TB 116 with your patch and report back the result

First just test with this one that I submitted formally to lead developers for review: https://bugzilla.mozilla.org/attachment.cgi?id=9344913

The patch I attached yesterday, https://bugzilla.mozilla.org/attachment.cgi?id=9344999, is not quite ready for formal submission but you can apply that and test it as well if you want to.

Since I am not acquainted with TB's release management: Is there a chance that your patch makes it into one of the next official releases of TB 115?

A simple patch with big impact like https://bugzilla.mozilla.org/attachment.cgi?id=9344913 should be a prime candidate for uplifting to 115.x. But I haven't heard back from the lead developers about this at all. I suppose they have a lot on their plates after the major release with a LOT of changes in the UI code.

-gene

Comment on attachment 9344999 [details] [diff] [review]
fix-ignore-subs-with-list-extended.diff

My latest moz-phab diff makes this change to c++ discovery code unnecessary.

Attachment #9344999 - Attachment is obsolete: true

Here's the note I added to my latest moz-phab diff I just submitted:

This now completely fixes the bug. Before new folders are "auto-expanded" I now trigger a performExpand() which, 
when subscriptions are ignored, triggers a "discoverchildren" URL. This actually is an improvement over 102 in that,
when subscriptions are ignored, the folder tree completely auto-expands just like it does when subscriptions are
respected when a new account with nested folders is added to a profile.
No changes are required to any C++ code with this diff.

The patch is here: https://d2mfgivbiy2fiw.cloudfront.net/file/data/hqfceocfdc776gwrcdkf/PHID-FILE-67zw4cueujwhsnpagc45/D184145.diff

Thank yo so much Gene for tackling this!

Flags: needinfo?(alessandro)
Duplicate of this bug: 1844748

The patch is here: https://d2mfgivbiy2fiw.cloudfront.net/file/data/hqfceocfdc776gwrcdkf/PHID-FILE-67zw4cueujwhsnpagc45/D184145.diff

Do you have a test build available ?
In my case, a linux/x86_64 build?

I just started a "try" build here:
https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=c09f00c4cecafbcf46789b66ef2b8612cf819b6d
The patch is on top of the current daily. I'm building for all archs but I'll provide you a directly link for the linux64 tar when it finishes.

https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/e1jS0esiRLyqA66gYAWVXQ/runs/0/artifacts/public/build/target.tar.bz2
Linux64 build already finished. A lot faster when changes are all JS.
Edit: you can just extract this in place and run the internal file thunderbird/thunderbird.

Gene, a big thanks for caring about that very important problem, and please excuse the delay; I've had a lot of unforeseen other tasks over the weekend.

I still couldn't test 116 with your patch, but another person has provided me a special build of 115 with the newest version of your patch included. I confirm that your patch solves the problem completely for us.

I have conducted several tests quite thoroughly, e.g., creating a deeply nested folder on the IMAP server at PC 1 and trying to see that folder with the patched TB 115 at PC 2 (of course having "Show only subscribed folders" turned off there). This now works even better than in TB 102:

If a new folder that has been created elsewhere doesn't appear anyway in the patched TB 115, it is sufficient to collapse and expand that folder's grandparent to make that folder visible reliably; it is not necessary to restart TB for this to happen.

There's a subtle difference between 102 and the patched 115: When collapsing and expanding that new folder's grandparent, 102 just prepends the twisty widget before that new folder's parent, while the patched 115 additionally expands that new folder's parent.

This even works when that new folder already has siblings. That is, whenever you collapse and expand a folder, the patched TB 115 automatically expands all of that folder's children that contain new subfolders. That's very nice and a progress on top of 102.

I believe that the situation now cannot be improved further. Maybe it would be nice if TB could detect new folders without any user interaction (i.e., without collapsing and expanding folders), but this IMHO is not possible because on one hand IMAP does not provide push notifications for folders (please correct me if I am wrong), but on the other hand continuous active recursive scans ("polling") for new folders would put substantial load on the network and the IMAP server and would make TB very slow (probably).

Again, thank you very much for your fantastic support and your strong commitment, and for communicating with users in an exemplary manner!

Best regards,

Binarus

(In reply to gene smith from comment #59)

https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/e1jS0esiRLyqA66gYAWVXQ/runs/0/artifacts/public/build/target.tar.bz2
Linux64 build already finished. A lot faster when changes are all JS.
Edit: you can just extract this in place and run the internal file thunderbird/thunderbird.

thanks o/

disabled

	[ ] show only subscribed folders

then, stopped TB

rm -rf ~/.thunderbird/x.../ImapMail

launched your 'try' build,

	Name 	Thunderbird
	Version 	117.0a1
	Build ID 	20230724162327

on start, all folders' discovery is triggered and appears that all (sub)folders are discovered, completely displayed in the UI tree, and mail in them is accessible

look good; haven't found any further/similar issues yet.

i do notice that on (clean) start as above, all (sub)folders are expanded in the tree. i'll check to see if closed/open/prior tree state is persisttent ...

since it's v117, 1/2 my usual extensions are disabled ... so will hope to see this fix in v115.x

i'll check to see if closed/open/prior tree state is persistent ...

I was concerned about that too. But it only auto-expands all the way the first time and, once new folders discovered, you can collapse at any level and on restart you see the same level. At least that's what I saw when I tested it.

Maybe it would be nice if TB could detect new folders without any user interaction (i.e., without collapsing and expanding folders), ...

Probably not easy :).

I was concerned about that too. But it only auto-expands all the way the first time and, once new folders discovered, you can collapse at any level and on restart you see the same level. At least that's what I saw when I tested it.

I confirm that it is the same here. It works as desired and expected.

Maybe it would be nice if TB could detect new folders without any user interaction (i.e., without collapsing and expanding folders), ...

You might want to have a look into my previous post (comment #60) where I (try to) explain why this probably is not possible (please note the long paragraph quite at the bottom).

Apart from that, there are only two situations where you actually need to to something to make new folders visible; both situations (in my experience) are encountered often during tests, but rarely in practice. Suppose you have a folder A that contains a subfolder B, and that a new folder C is created as a subfolder of B elsewhere. The two situations I am talking about are then:

  1. If B didn't contain subfolders yet (meaning that C hasn't siblings), B doesn't have the twisty widget prepended. In this case, if A is already expanded, you need to collapse and expand it to make C visible. If A is collapsed, no additional action is required to see C, because you would need to expand A anyway to see any of its contents. Upon expanding A, TB even auto-expands B because it detects that there is a new subfolder there.

  2. If B does contain subfolders already, C is a sibling of them. In this case, B already has the twisty widget prepended. If B is already expanded, you need to collapse and expand it to make C visible. If B is collapsed, no additional action is required to see C, because you would need to expand B anyway to see any of its contents.

In practice, I have observed that most users (including me) collapse folders one they're done with them, especially if there are many folders, because they would lose the plot otherwise. That probably means that for most users neither of the two situations described above arises often.

Off-topic:
In this context, it would be very nice if we could recursively collapse all folders in an account at once. That would enable us to get on track again quickly if there are too many folders open, and we could be sure that we (when navigating to a subfolder again) see every new folder that has been recently created at any nesting level of the path we navigate. However, that "recursive collapse" feature definitely is not subject of this bug report and already has been requested, but it seems that there are no plans to implement it (see bug 539140, 14 years ago). Possibly there's an extension (addon) that implements it, but I don't know.

I was concerned about that too. But it only auto-expands all the way the first time and, once new folders discovered, you can collapse at any level and on restart you see the same level. At least that's what I saw when I tested it.

yup, same here.

Duplicate of this bug: 1845107

The patch has been approved. Thanks Geoff/darktrojan!
Wayne is wanting to fast-track this to 115.

Keywords: checkin-needed-tb

Thanks Gene. Is comment 58 try build still sufficient for testing purposes?

Flags: needinfo?(mkmelin+mozilla)
Flags: needinfo?(geoff)
Flags: needinfo?(gds)

Binarus, are you able to test the try build?

Flags: needinfo?(info)

Pushed by thunderbird@calypsoblue.org:
https://hg.mozilla.org/comm-central/rev/bbcab53d356e
Fix to trigger imap folder discovery on expand. r=darktrojan

Status: NEW → RESOLVED
Closed: 1 year ago
Keywords: checkin-needed-tb
Resolution: --- → FIXED
Target Milestone: --- → 117 Branch

Comment on attachment 9344913 [details]
Bug 1843637 - Fix to trigger imap folder discovery on expand. r=darktrojan

[Triage Comment]
Approved for beta

more checking to be done before approving for esr115

Attachment #9344913 - Flags: approval-comm-beta+

(In reply to Wayne Mery (:wsmwk) from comment #68)

Thanks Gene. Is comment 58 try build still sufficient for testing purposes?

I did a quick test to make sure it still worked with the latest changes required by Geoff before committing it. I just now tried to kick off a new try build but it kept saying something is wrong with my "initialization parameters". Maybe because the patch I was pushing is now at tip of C-C, don't know. Anyhow, I think Binarus has a way to patch 115.0 JS code without a try build so if he want to test with the latest code, he can.
The OP reporter pgnd (on linux) can test with tonight's daily build which my patch should be in (I hope) if he wants to.

Flags: needinfo?(gds)

The OP reporter pgnd (on linux) can test with tonight's daily build which my patch should be in (I hope) if he wants to.
...
Thunderbird 116.0b7: https://hg.mozilla.org/releases/comm-beta/rev/9d411d73687a

https://hg.mozilla.org/releases/comm-beta/archive/9d411d73687a5c000f400163f80c1402a306b023.zip

looks good here

Comment on attachment 9344913 [details]
Bug 1843637 - Fix to trigger imap folder discovery on expand. r=darktrojan

[Triage Comment]
APproved for esr115

Attachment #9344913 - Flags: approval-comm-esr115+
See Also: → 1864776
See Also: → 1798436
See Also: → 1902236
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: