1737514 - cannot receive email from IMAP server with UTF8 enabled

Status: RESOLVED INVALID

(MailNews Core :: Networking: IMAP, defect)

Description

[chaoskmg]

Attachment: IMAP log after reproducing the problem

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0

Steps to reproduce:

Our email system uses the commercial Coremail XT3 server (http://www.coremail.cn/).
Install Thunderbird 91, and add a new account as IMAP+SMTP to the server.

Actual results:

No email is fetched from the server.
The status bar in the left bottom corner shows "Checking capabilities" until timeout.
Thunderbird >= 91.0 seem to do nothing after sending the CAPABILITY and ID command.

Expected results:

We should be able to see the list of emails in Thunderbird. This is the actual behavior for Thunderbird <= 78.14.0.

Comment 1

[chaoskmg]

The problem seem to be related with a change in Thunderbird 91

Sending backend, SMTP, and IMAP protocols now operate only in UTF-8 mode

However, I've no idea why Thunderbird suddenly stopped doing anything during the process of receiving emails.

Comment 2

[chaoskmg]

After some digging, I think Thunderbird is not parsing the response of ENABLE UTF8=ACCEPT according to the RFC5161 specification.
The IMAP server should return untagged responses to that command.
However, from the code in nsImapProtocol::EnableUTF8Accept, which calls nsImapProtocol::ParseIMAPandCheckForNewMail and then nsImapServerResponseParser::ParseIMAPServerResponse, the client is expecting one tagged response from the server.
Since tagged responses is never sent by the server in this circumstance, Thunderbird would wait for a non-existent response, and trigger a timeout finally.

For earlier versions of Thunderbird, since UTF8 is not enabled by default even when it's supported, no problem will be encounted in this case.

Comment 3

[Magnus Melin [:mkmelin]]

As a workaround you can set the mail.server.default.allow_utf8_accept to false

Comment 4

[gene smith]

From your imap log:

SendData: 39 ENABLE UTF8=ACCEPT
CreateNewLineFromSocket: * ENABLED UTF8=ACCEPT
CreateNewLineFromSocket: * OK ENABLE completed

According to the "Enable" rfc, https://datatracker.ietf.org/doc/html/rfc5161, the "OK" response above must be tagged. The RFC provides this example on page 3:

   In the following example, the client enables CONDSTORE:

      C: a1 ENABLE CONDSTORE
      S: * ENABLED CONDSTORE
      S: a1 OK Conditional Store enabled

So the 3rd line from your server appears to be wrong. It should be like this, tagged:

CreateNewLineFromSocket:  39 OK ENABLE completed

Also, it appears that your server does not list the UTF8=ACCEPT capability. Instead it lists several other UTF8=* capabilities that are now obsolete and have never been supported by TB. So if your server actually doesn't support the UTF8=ACCEPT capability, the attempt to enable it should respond like this (leaving off the untagged ENABLED line):

SendData: 39 ENABLE UTF8=ACCEPT
CreateNewLineFromSocket: 39 OK ENABLE completed

Edit/Addendum:
Here an except from an email exchange with Arnt Gulbrandsen, author of ENABLE rfc referenced above. I had misunderstood how ENABLE works and he corrected me:

On Sunday 2 August 2020 22:00:59 CEST, Gene Smith wrote:
> So I think you saying that if I see the ENABLE capability I should just go ahead
 and send ENABLE UTF8=ACCEPT and not worry about checking the capability UTF8=ACCEPT.
 If it fails with NO (or BAD) response then I know the server doesn't support it. If response is
 OK then the server has enabled UTF8=ACCEPT and I'm good to go.

Close, but not quite. You'll get NO or BAD if ENABLE fails. If ENABLE works but 
has no effect, you'll get OK.

What you should do is look for the ENABLE capability, and if you see that,
you send ENABLE UTF8=ACCEPT, and then you look at the ENABLED response.
If the ENABLED response contains UTF8=ACCEPT you're good to go. See the
last three lines of RFC 5161 page 3:

https://tools.ietf.org/html/rfc5161#page-3

Comment 5

[Magnus Melin [:mkmelin]]

So something we should correct?

Comment 6

[gene smith]

So something we should correct?

No.
I looked again at the attached log and their server is failing to respond correctly to ENABLE.

They response like this:

215ebea5000:mail.fudan.edu.cn:A:CreateNewLineFromSocket: * OK ENABLE completed

should be:

215ebea5000:mail.fudan.edu.cn:A:CreateNewLineFromSocket: 39 OK ENABLE completed

The bad response results in our timeout since we are waiting for the "tagged" response and never see it. This is how all commands work with imap.
Like you suggested, disabling UTF8-ACCEPT until they get a server fix is the proper solution.
I suggest closing this as INVALID.

Comment 7

[Magnus Melin [:mkmelin]]

Thanks!