1673158 - Send message cannot be saved "Send" folder - "The current command did not succeed ... [CLIENTBUG] syntax: unexpected char: '('."

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[juraj.masiar]

Attachment: Screenshot 2020-10-24 09.48.17.png

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0

Steps to reproduce:

My Thunderbird updated to 83.0b1 (32-bit) few days ago and now every time I send email, I get this error (see attachment).
I didn't changed anything.

Comment 1

[Wayne Mery (:wsmwk)]

FWIW https://mzl.la/3ok4amK

Comment 2

[Arthur K. (he/him)]

(In reply to juraj.masiar from comment #0)

Created attachment 9183606 [details]
Screenshot 2020-10-24 09.48.17.png

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0

Steps to reproduce:

My Thunderbird updated to 83.0b1 (32-bit) few days ago and now every time I send email, I get this error (see attachment).
I didn't changed anything.

Any chance you can do a CTRL-SHIFT-J in TB to open Error Console and see if there's anything there? With Error Console open, click the trash can icon (top-left) to clear the screen. Then, try to send this email that is causing the issue. It should cause the error to appear and then go back to Error Console and see if there's anything useful you might be able top post here. Do strip out any identifying info.

Comment 3

[juraj.masiar]

Hello,
Nope, the browser console stays empty, even when I click Retry and the error reappears again, it's still empty.
Also I've noticed it happens only to my work account hosted at "zoho.com", my personal Gmail can send emails without issues.

Also I have still my old Thunderbird 78.4.0 (32-bit) installed where it still works fine. So I'm pretty sure this is a regression in 83.0b1 because it started after the update.

Comment 4

[Arthur K. (he/him)]

(In reply to juraj.masiar from comment #3)

Hello,
Nope, the browser console stays empty, even when I click Retry and the error reappears again, it's still empty.
Also I've noticed it happens only to my work account hosted at "zoho.com", my personal Gmail can send emails without issues.

Also I have still my old Thunderbird 78.4.0 (32-bit) installed where it still works fine. So I'm pretty sure this is a regression in 83.0b1 because it started after the update.

Thanks for checking for me. Were you on 82.0b3 before going to 83.0b1? You could always roll back to 82.0b3 and launch it using the --allow-downgrade option if you're sure it's 83.0b1 that's the issue.

Any chance you might also want to try and use Mozregression (https://mozilla.github.io/mozregression/) to figure out which bug might be causing it? There's a How-To video on that site that explains what it does and how to use it.

Comment 5

[gene smith]

Sounds like an error reported by yahoo when trying to imap APPEND the sent messages to the Sent mailbox. I've seen yahoo complain like this when the message is OK and appends fine to other servers. Of course I'm just guessing. Is this using Yahoo or a yahoo derivative like AOL?

If you just send a real simple test message to yourself do you see this?

Comment 6

[gene smith]

I see it is zoho and not yahoo (should have read closer). Probably need an IMAP:5 log to understand what is going on: https://wiki.mozilla.org/MailNews:Logging
My next guess is the server they are using is dovecot.

Comment 7

[juraj.masiar]

Sure thing, I will check it out in the morning and send you the logs.

Comment 8

[gene smith]

I think the problem is that zoho server (now I suspect Courier, but a log will tell) supports the newly tb supported UTF8=ACCEPT capability. This features also uses a slightly different syntax for imap APPEND. We support the new and correct append syntax as described here and in another linked-to bug: bug 1571672 comment 57. The first changed char is a "(" as the error message shows.

Reporter juraj, you can attach the log above. But if you prefer to send it by other means, that's OK too. Be sure to record the log while sending a message.

Another thing you can try is change with the Config Editor the pref value mail.server.default.allow_utf8_accept to false and restart tb. This will cause tb to ignore the UTF=ACCEPT capability and use the original APPEND command. This probably wont cause any problems since it worked OK with 78.

Note: AFAIK only gmail and Courier servers support UTF8=ACCEPT. I am able to send OK (no append problems) with gmail and an up-to-date Courier server with allow_utf8_accept true. So my guess is that zoho is using a server that does not now properly implement UTF8=ACCEPT capability, at least the change required for imap APPEND. I missed that too until it was pointed out to me.

Comment 9

[juraj.masiar]

@gene I've send you the log to your email (you have registered here).
I would attached it here but don't know if it contains some sensitive info (it's huge - 38MB).

Comment 10

[gene smith]

Ok, thanks! Not that big once it's zipped.

I see what the problem is. The zoho server is responding to "enable utf8=accept" that is has enabled it with the untagged response * ENABLED UTF8=ACCEPT:

I/IMAP 3E86D800:imap.zoho.com:A:SendData: 5 ENABLE UTF8=ACCEPT
D/IMAP ReadNextLine [rv=0x0 stream=2DAA2C40 nb=23 needmore=0]
I/IMAP 3E86D800:imap.zoho.com:A:CreateNewLineFromSocket: * ENABLED UTF8=ACCEPT
D/IMAP ReadNextLine [rv=0x0 stream=2DAA2C40 nb=14 needmore=0]
I/IMAP 3E86D800:imap.zoho.com:A:CreateNewLineFromSocket: 5 OK Success

Tb then uses the format for the APPEND command that is required for UTF8=ACCEPT. But, apparently the zoho server doesn't really support it so it rejects the APPEND command. The zoho server also doesn't list UTF8=ACCEPT in its capabilities response.

The proper response for when UTF8=ACCEPT is attempted to be enable but is not supported is like this response from the iCloud imap server:

I/IMAP 0x7fc0e6db1800:imap.mail.me.com:A:SendData: 5 ENABLE UTF8=ACCEPT
D/IMAP ReadNextLine [rv=0x0 stream=0x7fc0eda92980 nb=23 needmore=0]
I/IMAP 0x7fc0e6db1800:imap.mail.me.com:A:CreateNewLineFromSocket: 5 OK ENABLE completed

In this case the ENABLE command produces no untagged responses indicating UTF8=ACCEPT is not supported.

I am unable to tell which actual imap server zoho is using. Maybe they have created their own. So they should be notified that they should not response with "enable utf8=accept" if they don't actually support it.

Then again, I guess I'm somewhat to blame for this too since I took the advice of ENABLE rfc author as described here: bug 1571672 comment 59.

juraj, as I mentioned before you should be able to set false this pref and work-around the problem: mail.server.default.allow_utf8_accept

Comment 11

[gene smith]

Attachment: Re: ENABLE UTF8=ACCEPT.eml

For the record, here's the relevant email from the author of the ENABLE rfc. The link to the rfc is near the end of the file.

Comment 12

[Arthur K. (he/him)]

Do we have someone to contact at Zoho? I would guess Juraj won't be the only one hitting this. Juraj, what's the full server name you're using for Zoho?

Comment 13

[Arthur K. (he/him)]

Nevermind, I might know who to contact.

Comment 14

[gene smith]

I setup a trial account with zoho ([e-mail]) and can duplicate the problem here. I changed the code to enable a bogus extension that I called UUU8=ACCEPT and it ignored it when I tried to enable it as it should (responded only with OK). So apparently zoho is actually accepting and enabling the UTF8=ACCEPT, or at least thinks it is. I then attempted to create a folder with a multi-byte UTF8 character and it accepts it. But it lists it back as MUTF7 and the name appears with <?>'s in the folder list. So the zoho server seems to have big problem with UTF8=ACCEPT. Doing the exact same things with Courier and gmail work as expected.

Here's the log entry on the append fail:

I/IMAP 0x7f499329c000:imappro.zoho.com:S-INBOX:SendData: 14 append "INBOX" (\Seen nonjunk) "23-Mar-2020 16:12:05 -0400" UTF8 (~{4090}
D/IMAP ReadNextLine [rv=0x0 stream=0x7f499a6072a0 nb=49 needmore=0]
I/IMAP 0x7f499329c000:imappro.zoho.com:S-INBOX:CreateNewLineFromSocket: 14 BAD [CLIENTBUG] syntax: unexpected char: '('

I'll probably have to go back to keying off the actual UTF8=ACCEPT capability string to fix this if we can't get an acknowledgment from zoho. But technically this shouldn't be required. If the extension is not supported or the support is incomplete they should ignore my request to enable it like they do for my bogus UUU8=ACCEPT.

Comment 15

[gene smith]

Attachment: utf8Accept-cap-fix.diff

Here's a probable fix for this bug. I'll hold off a bit on submitting a formal patch to see if we hear anything from the zoho people.
All this does is check the capability for UTF8=ACCEPT or UTF8=ONLY instead of only ENABLE before attempting to enable UTF8=ACCEPT.
I still think the bug is on their end but maybe they can convince me I'm wrong.

Comment 16

[Arthur K. (he/him)]

(In reply to gene smith from comment #15)

Here's a probable fix for this bug. I'll hold off a bit on submitting a formal patch to see if we hear anything from the zoho people.
All this does is check the capability for UTF8=ACCEPT or UTF8=ONLY instead of only ENABLE before attempting to enable UTF8=ACCEPT.
I still think the bug is on their end but maybe they can convince me I'm wrong.

Thanks. I haven't hear back yet but it is the weekend so maybe no one is monitoring the emails.

Comment 17

[Zoho Mail Support]

Hi Gene/Arthur,

Thank you for reporting this to us.

Zoho's IMAP server is developed in-house and we do not support UTF8=ACCEPT capability. We will modify the server response to the ENABLE command, to let the client know that UTF8=ACCEPT is not supported when the client attempts to enable it. It will be done in our next production update.

For the time being, we will stop advertising ENABLE extension in our capabilities. Once we modify the server response, we will re-advertise it. We would like to inform that ENABLE will be supported only for CONDSTORE capability.

Comment 18

[gene smith]

(In reply to Zoho Mail Support from comment #17)

Thanks for the quick reply. I can confirm using my zoho trial account that ENABLE capability is not now advertised. Currently TB's CONDSTORE support is switched off by default so probably not a problem that, for now, it can't be enabled for zoho.

Reporter juraj, you should now be able to reset mail.server.default.allow_utf8_accept back to default of true if you want to.

Comment 19

[Arthur K. (he/him)]

(In reply to gene smith from comment #18)

(In reply to Zoho Mail Support from comment #17)

Thanks for the quick reply. I can confirm using my zoho trial account that ENABLE capability is not now advertised. Currently TB's CONDSTORE support is switched off by default so probably not a problem that, for now, it can't be enabled for zoho.

Reporter juraj, you should now be able to reset mail.server.default.allow_utf8_accept back to default of true if you want to.

Thanks for testing!

Comment 20

[juraj.masiar]

Thank you all for the fix. It works now again.
Feel free to close this now.

Comment 21

[Arthur K. (he/him)]

(In reply to juraj.masiar from comment #20)

Thank you all for the fix. It works now again.
Feel free to close this now.

Thanks for confirming. Gene you wanna close as WFM?

Comment 22

[Magnus Melin [:mkmelin]]

Maybe the patch is still worth taking? Gene, what do you think?

Comment 23

[gene smith]

Yes, I think it is probably good to do this for servers that don't quite implement ENABLE correctly as was the case with zoho.
I'll add this to my TODOs.

Comment 24

[Wayne Mery (:wsmwk)]

(In reply to gene smith from comment #23)

Yes, I think it is probably good to do this for servers that don't quite implement ENABLE correctly as was the case with zoho.
I'll add this to my TODOs.

Comment 25

[gene smith]

Ok, sorry for the delay on this. moz-phab diff on the way...

Comment 26

[gene smith]

Attachment: Bug 1673158 - Ensure there is an imap capability UTF8=ACCEPT before enabling it. r=mkmelin

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/2cbd022daf3c
Ensure there is an imap capability UTF8=ACCEPT before enabling it. r=mkmelin

Comment 28

[Wayne Mery (:wsmwk)]

old bug - let's have this ride the train to 112