2087454 - Thunderbird 159.0: IMAP folders disappear after restart despite successful LIST responses, because of IMAP COMPRESS=DEFLATE

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Ron]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:159.0) Gecko/20100101 Firefox/159.0

Steps to reproduce

1. Use Thunderbird 159.0 (aarch64) on macOS.
2. Configure an IMAP account with freenet.de (mx.freenet.de, port 143, STARTTLS).
3. Complete the account setup. All subscribed IMAP folders are initially discovered and displayed correctly.
4. Quit Thunderbird completely.
5. Restart Thunderbird and expand the freenet.de account in the folder pane.
6. Open the folder subscription dialog. The missing folders are still listed as subscribed, but the dialog remains stuck at "Please wait...".

The issue is reproducible after completely removing and recreating the IMAP account.

Other IMAP accounts in the same Thunderbird profile are unaffected.

Actual results

After restarting Thunderbird, almost all IMAP subfolders disappear from the folder pane. Usually only the Inbox remains visible.

The local IMAP storage directory contains INBOX.msf but no longer contains the INBOX.sbd subfolder structure.

IMAP protocol logging confirms that Thunderbird receives the complete folder list from the server:

* NAMESPACE (("INBOX." ".")) NIL NIL
* LIST (\Subscribed \UnMarked) "." INBOX.Archives
* LIST (\Subscribed \UnMarked) "." INBOX.Bandcamp
* LIST (\Subscribed \UnMarked) "." INBOX.paypal
* LIST (\Subscribed \UnMarked \Sent) "." INBOX.sent
101 OK List completed

The server also exposes folder names containing ^, such as INBOX.INBOX^Junk, which Thunderbird displays as INBOX^^Junk.

All folders remain accessible through freenet webmail and Thunderbird for Android.

Rebuilding .msf files, resetting folderCache.json, using Troubleshoot Mode, and recreating the account did not resolve the issue.

Potentially related: Bug 1604736, Bug 2005701 and Bug 2078247. The underlying cause has not been confirmed.

Expected results

All subscribed IMAP folders should remain visible and accessible after restarting Thunderbird.

Thunderbird should correctly restore the IMAP folder hierarchy, including folders within the INBOX. namespace, without removing the local folder structure.

Comment 1

[Ron]

I found another potentially related report: Bug 2087735.

The reporter describes a very similar issue with Thunderbird 159 on Windows, using a different IMAP provider (Network Solutions).

In both cases:

• IMAP subfolders disappear after restarting Thunderbird.
• The folders remain available on the server.
• The local ImapMail directory loses its previously existing subfolder structure.
• Reinstalling or recreating the account does not provide a lasting solution.

My issue occurs on macOS with freenet.de, so the behavior may not be limited to a specific operating system or IMAP provider.

It is not yet clear whether both reports share the same underlying cause, but the similarities may be useful for investigating a possible Thunderbird 159 regression.

See Bug 2087735.

Comment 2

[Wayne Mery (:wsmwk)]

Perhaps also related - several items at https://support.mozilla.org/en-US/questions/1615185 and https://www.reddit.com/r/Thunderbird/comments/1xkd385/account_folders_disappeared_and_expandcollapse/

Comment 3

[Olav Seyfarth]

I tried to reproduce this with the official Thunderbird 159.0 (Linux) against a local IMAP server that mimics your setup: NAMESPACE (("INBOX." ".")), the same folders including INBOX.INBOX^Junk, both with plain LIST/LSUB and with LIST-EXTENDED (list (subscribed)). After three restarts, all folders were still there, in the folder pane and in INBOX.sbd. So bug 2005701 doesn't seem to cause this by itself. Could you attach a full IMAP log of a startup where the folders are missing (MOZ_LOG=IMAP:5), and say whether 158 worked? It would also help to know whether renaming INBOX^Junk in webmail (to a name without ^) changes anything, as ^ is Thunderbird's internal placeholder for an unknown hierarchy delimiter (bug 1604736).

Comment 4

[Ron]

Attachment: thunderbird-imap-freenet-bugzilla.log

Comment 5

[Ron]

Attachment: thunderbird-imap-freenet-anonymized.log

Comment 6

[Ron]

I was able to reproduce the issue again with Thunderbird 159.0 on macOS and captured another IMAP log during the affected startup.

Reproduction:

1. Started Thunderbird with IMAP logging enabled.
2. During the first startup, the folders were displayed normally.
3. Closed Thunderbird and started it again with logging enabled.
4. This time, all folders of the Freenet IMAP account were missing immediately after startup, including the Inbox.
5. Closed Thunderbird without attempting to repair or refresh the folders.

Findings from the affected startup:

• Authentication against mx.freenet.de succeeded.
• The server returned NAMESPACE (("INBOX." ".")) NIL NIL.
• Thunderbird requested the subscribed folders using LIST (subscribed).
• The server returned 14 subscribed folders, including the Sent, Drafts and Trash folders.
• A separate LIST response confirmed that INBOX exists and has children.
• All three LIST commands completed successfully with OK List completed.

Despite these successful responses, Thunderbird did not display any folders for this account.

I also noticed the following sequence:

LIST "" "INBOX.INBOX/Trash"
CREATE "INBOX.Trash"
NO [ALREADYEXISTS] Mailbox already exists

The server had already reported INBOX.Trash with the \Trash special-use flag. I am not sure whether this unexpected mailbox creation attempt is related to the issue.

Another observation is that two folders are returned with the \Junk special-use flag, while the existing INBOX.Junk folder is listed without that flag.

Comparison with the earlier IMAP log:

The earlier log also shows successful folder discovery, but Thunderbird continues processing individual folders with additional STATUS and SELECT operations.

In contrast, the affected startup log ends shortly after folder discovery and the unsuccessful attempt to create the already existing Trash folder.

This suggests that Thunderbird receives the folder information from the server but may fail to process or display it correctly during startup.

I have attached both anonymized IMAP logs for comparison:

• thunderbird-imap-freenet-bugzilla.log — earlier, more extensive log
• thunderbird-imap-freenet-anonymized.log — startup where all folders were missing

Please let me know if any additional diagnostics would be helpful.

Comment 7

[Wayne Mery (:wsmwk)]

If the same issue, https://support.mozilla.org/en-US/questions/1615185#answer-1859554 reports workaround as ...

Right click on the account name in the left pane that has its folders missing.
Subscribe.
Click OK.
The missing folders will reappear and FB will then resync.

Comment 8

[Ron]

Yeah but with the next restart all folders are gone again. I went through it more than 5 times while the issue is reported. As other ones already reported, this issue exists only on Thunderbird at the desktop. The android version, webmail and other mail clients didn't have this trouble. All folders are visible there.

Comment 9

[Olav Seyfarth]

Thanks for the logs, they show the cause. It isn't the folder handling but the IMAP connection getting out of step right after COMPRESS DEFLATE.

In the failing log, the first line after compression is enabled starts with five stray bytes, 00 00 00 FF FF, which the log shows as spaces and ``:

70 OK Begin compression.
71 ID ("name" "Thunderbird" "version" "159.0")
\0\0\0\xFF\xFF* ID ("name" "Dovecot")

That is an empty deflate block (a sync flush), which Dovecot sends right after OK Begin compression. The code explains what follows:

1. nsMsgLineStreamBuffer::ReadNextLine() reads everything the socket has available into its buffer (and turns NUL bytes into spaces, hence the spaces in the log). If the flush block arrives together with the OK line, it is already in that buffer.
2. nsImapProtocol::BeginCompressing() only wraps m_inputStream. Bytes already in m_inputStreamBuffer never go through the inflater, so they are glued in front of the next line as plain text.
3. In nsImapServerResponseParser::ParseIMAPServerResponse(), the line's first token starts with 0xFF, not *. So it is counted as the tagged response to command 71, and response_done() compares it with the tag 71. That fails, so response_fatal() and resp_cond_bye() run: SetConnected(false) and kNonAuthenticated, which is the A → NA in the log.
4. The real 71 OK is still pending, so every following command gets the response to the previous one. After 72 list (subscribed) Thunderbird reads 71 OK ID completed, the LIST lines end up as the answer to 73 list "" "INBOX", and the folder list is lost.

Both logs show the same thing: whenever the * ID line comes with these bytes, the connection gets out of step Whenever it comes clean, it works. In the earlier log, the discovery connection happened to get a clean line. Whether the flush block arrives in the same read as the OK line depends on timing, which is why it comes and goes. Whether something changed on freenet's side or in Thunderbird, I can't tell from here; I found no change to this code in recent months.

Workaround to confirm: in the Config Editor, set mail.server.serverN.use_compress_deflate to false for the freenet account (N from the account's server key), restart, and repair the folders once via Subscribe. If the folders then survive restarts, this is the cause.

The create "INBOX.Trash" / ALREADYEXISTS is a separate issue, bug 2087812, and doesn't cause the missing folders.

Comment 10

[Robert Flatt]

mail.server.serverN.use_compress_deflate to false

Worked for me on two Windows machines, more detail Bug 2087735

Comment 12

[Robert Flatt]

And in my case the IMAP provider is Network Solutions.

Comment 13

[Magnus Melin [:mkmelin]]

Looks like a few things go wrong here.

Comment 15

[Olav Seyfarth]

Attachment: Bug 2087454 - Inflate data the line buffer already holds when IMAP COMPRESS starts. r?mkmelin

After "OK Begin compression", Dovecot immediately flushes an empty deflate block (00 00 00 FF FF). If it arrives in the same read as the OK line, nsMsgLineStreamBuffer already holds it when BeginCompressing() wraps m_inputStream, so these bytes bypass the inflater and are parsed as the start of the next response. That response is then taken for an unknown tagged one, the connection drops to the non-authenticated state, and every later response is matched to the wrong command, so folders seem to vanish.

BeginCompressing() now takes the unread bytes out of the line buffer and hands them to nsMsgCompressIStream, which inflates them before reading from the socket. To keep these bytes intact, the line buffer no longer replaces NULs in its buffer but only in the lines it returns, and finds line ends with memchr.

Co-Authored-By: Claude Opus 5.5 <[e-mail]>
Signed-off-by: Olav Seyfarth <[e-mail]>

Comment 16

[Olav Seyfarth]

I have attached a patch: D338669. In BeginCompressing(), the bytes the line buffer has already read are now passed through the inflater. With Dovecot this is the empty sync-flush block 00 00 00 FF FF that arrives together with the OK. Before, these bytes ended up in front of the next response; the "20 20 20 FF FF" in bug 2087926 is exactly this block after the NUL-to-space replacement. The line buffer now keeps NULs until it returns a line. The patch includes gtests; the xpcshell tests for imap, local and news pass. Two Claude Code "/code-review high" runs over the patch found nothing relevant to the fix.

Unrelated, pre-existing issues the review noticed, which I won't pursue: nsMsgCompressIStream::StreamStatus() dereferences m_iStream after Close(); DoInflation() treats Z_STREAM_END like Z_OK and can drop input after a final block; m_zstream.next_in/avail_in are not initialized before inflateInit2(); the null checks after new/make_unique are dead code; HandleIdleResponses() checks only the line buffer, not data already buffered inside nsMsgCompressIStream.

Comment 17

[Magnus Melin [:mkmelin]]

Bug 2075517 caused the regression, but only together with an older compression bug.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/e6c1de85f3f1
Inflate data the line buffer already holds when IMAP COMPRESS starts. r=mkmelin

Comment 19

[Magnus Melin [:mkmelin]]

Bug 2088009 will fix the regression that exposed this issue. It seems it was working by luck back on 158.

Comment 20

[Olav Seyfarth]

Comment on attachment 9672340 [details]
Bug 2087454 - Inflate data the line buffer already holds when IMAP COMPRESS starts. r?mkmelin

[Approval Request Comment]
Regression caused by (bug #): 2075517 (together with an older COMPRESS bug, see comment 17)
User impact if declined: With servers offering COMPRESS=DEFLATE (e.g. Dovecot at freenet, Network Solutions), IMAP responses get out of sync right after compression starts. Folder lists come out wrong, and local folder contents can be lost (bug 2087735).
Testing completed (on c-c, etc.): Landed on c-c on Nov 2 (e6c1de85f3f1). New gtests (TestMsgLineStreamBuffer, TestMsgCompressIStream); xpcshell imap/local/news pass. Two reporters confirmed that use_compress_deflate=false avoids the problem, which matches the cause.
Risk to taking this patch (and alternatives if risky): Low. The change only affects the start of COMPRESS: bytes the line buffer has already read now go through the inflater. Alternative: set mail.server.default.use_compress_deflate=false on the branches.
String changes made/needed: none

Uplift Approval Request

• Please state case for uplift consideration and ensure bug severity is set: Regression from bug 2075517 together with an older COMPRESS bug (comment 17). 159 and 160 are affected; duplicates bug 2087735 and bug 2087926.
• User impact if declined: With servers offering COMPRESS=DEFLATE (e.g. Dovecot at freenet, Network Solutions), IMAP responses get out of sync right after compression starts. Folder lists come out wrong, and local folder contents can be lost (bug 2087735).
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Daily?: No
• Has the fix been verified in Beta?: N/A
• Needs manual test from QA?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Only the start of COMPRESS changes: bytes the line buffer has already read now go through the inflater. New gtests; xpcshell imap/local/news pass. Alternative: mail.server.default.use_compress_deflate=false on the branches.
• Does the fix cause any migrations to be skipped?: N/A
• String changes made/needed: none

Comment 21

[Corey Bryant]

Comment on attachment 9672340 [details]
Bug 2087454 - Inflate data the line buffer already holds when IMAP COMPRESS starts. r?mkmelin

[Triage Comment]
Approved for beta

This seems fairly urgent, however there is a work-around. Can we let it ride 160 beta to release? 160.0 releases Nov 10.

Comment 22

[gene smith]

(In reply to Corey Bryant from comment #21)

This seems fairly urgent, however there is a work-around. Can we let it ride 160 beta to release? 160.0 releases Nov 10.

The workaround requires user to go into config editor and set mail.server.default.use_compress_deflate to false.
I tested this with local dovecot which I set to support compress=deflate capability (which isn't on by default). Also tested with cyrus which does default compress=deflate capability. Tested with 159.0 release and 161 self-built (maybe a week ago) daily and never see the problem on either account after many restarts. Not sure why I don't see the problem but I think Olav implied above that this can depend on how the bytes line up or on timing. And based on reports, seems to only occur for a certain dovecot setup (maybe depends on degree of compression).

FWIW, I also tested several restarts with gmail and yahoo (both support compress=deflate) and don't see the problem with them either.

Looking at the attachments above, I don't see the capability COMPRESS=DEFLATE in the imap logs. All I see is COM and COMP in the responses. Maybe it got edited out?

Comment 23

[Wayne Mery (:wsmwk)]

So far I find 11 likely reports between SUMO and reddit, plus additional users in some of them. So the number is not a crazy high, but it is a lot. And take into consideration that the majority of reports only started 3 days ago means that many users must still be updating. (update rate 100 was only set on the afternoon of 2026-10-29)

Comment 24

[Wayne Mery (:wsmwk)]

Anecdotal data from google...

A. Dovecot: 76.9% global market share (with an installed base of at least 2.9 million to over 5 million reachable/observable IMAP servers worldwide).

B. ("it is estimated that roughly 15% to 30% of active Dovecot installations use some form of native or IMAP deflate compression, while under 5% specifically rely on compress=deflate for mail storage in modern environments.") "In-Transit IMAP Line Compression (imap_zlib). Estimated Usage: 20% – 40%. Many administrators enable the imap_zlib plugin to support the IMAP COMPRESS=DEFLATE extension.

Comment 25

[Olav Seyfarth]

(In reply to Gene Smith from comment #22)

I don't see the capability COMPRESS=DEFLATE in the imap logs. All I see is COM and COMP in the responses. Maybe it got edited out?

The IMAP log cuts each line at 400 characters (%.400s in nsImapProtocol::Log()), so the CAPABILITY line ends in COM. The server did offer it: Thunderbird sends COMPRESS DEFLATE (line 19 of attachment 9672084 [details]) and gets OK Begin compression back.

The failure needs the server's first deflate bytes (here an empty sync-flush block, 00 00 00 FF FF) to arrive in the same socket read as the OK Begin compression line. That depends on the server and on timing, so a local test server may never trigger it.

Comment 26

[Corey Bryant]

We'll be picking this fix up in 160.0b4 which will be available as a candidate later today and officially release tomorrow. Beta feedback from any affected users would be useful.

Comment 27

[Evandro Baginski [:ebaginski]]

Thunderbird 160.0b4
https://hg.mozilla.org/releases/comm-beta/rev/00f28ed9b07a

Comment 28

[Ron]

You guys are great. Thanks a mill to take care of it!

Comment 30

[Corey Bryant]

Comment on attachment 9672340 [details]
Bug 2087454 - Inflate data the line buffer already holds when IMAP COMPRESS starts. r?mkmelin

This will be in 160.0 next Tues