2056491 - SMTP username in account authentication not encoded in UTF-8

Status: RESOLVED FIXED

(MailNews Core :: Networking: SMTP, defect)

Description

[Francois Lesueur]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0

Steps to reproduce:

I configured a login name with non-ascii characters on my server.

Actual results:

Authentication fails with Thunderbird.

Expected results:

Authentication should have succeeded.
Authentication works with Evolution.

According to RFC 4616 regarding the PLAIN Simple Authentication and Security Layer (SASL) Mechanism, the AUTH string containing the login and the password must be encoded with UTF-8. From what I can see on the server logs, the login from Thunderbird is encoded with latin-1. The password can be encoded with UTF-8 (that was bug #1491755) but only the password, the login is still encoded with latin-1.

Comment 1

[Wayne Mery (:wsmwk)]

(assuming this also affects 153)

Comment 2

[Ótzï fón GlüϷgor]

I can't even authenticate, so I can't even configure the server settings.
(see bug 1986189)

Comment 3

[Francesco (no longer active, use PM for contact)]

And it fails with pref mail.smtp_login_pop3_user_pass_auth_is_latin1 set to false?

Comment 4

[Francesco (no longer active, use PM for contact)]

For background: This pref was introduced in bug 1500772 (also see bug 1503625) since some M$ servers required latin1 at the time and utf-8 failed.

Comment 5

[Francois Lesueur]

(In reply to Ótzï fón GlüϷgor from comment #2)

I can't even authenticate, so I can't even configure the server settings.
(see https://bugzilla.mozilla.org/show_bug.cgi?id=1986189)

You can try by removing the special characters at the first step (email address, which will be wrong but let's accept it). Then, without special characters in the address, you can specify the servers, logins and passwords.

Comment 6

[Francois Lesueur]

(In reply to Francesco from comment #3)

And it fails with pref mail.smtp_login_pop3_user_pass_auth_is_latin1 set to false?

Yes, I tried both and it fails too, for SMTP. Yet it worked on the IMAP side (I think), but that was possibly a "counter-bug" in the server (I only studied precisely the smtp side)

From what I saw, mail.smtp_login_pop3_user_pass_auth_is_latin1 handles the password's encoding but not the login's encoding, no ?

Comment 7

[Francesco (no longer active, use PM for contact)]

The pref is for SMTP and POP3 only.

https://searchfox.org/comm-central/search?q=smtp_login_pop3_user_pass_auth_is&path=&case=true&regexp=false

See this code:

https://searchfox.org/comm-central/rev/72b8ba0761b3881d926be53f77fbf75d5a9316d5/mailnews/compose/src/SmtpClient.sys.mjs#1087

Comment 8

[Francois Lesueur]

Yes, I did not look up the IMAP spec for this auth, nor if my server (dovecot) is behaving as expected : it just works with Thunderbird and Evolution for IMAP.

Regarding SMTP :

• https://searchfox.org/comm-central/source/mailnews/compose/src/SmtpClient.sys.mjs#1057 -> there is no handling of mail.smtp_login_pop3_user_pass_auth_is_latin1, and in my case it seems to send latin-1
• https://searchfox.org/comm-central/source/mailnews/base/src/MailAuthenticator.sys.mjs#273 -> Comment here seems to indicate only password is UTF-8 encoded

Comment 9

[Francesco (no longer active, use PM for contact)]

Aha, so this is about the non-ASCII in the user name, not the password. You're correct conversion to UTF-8 is missing here:
https://searchfox.org/comm-central/rev/72b8ba0761b3881d926be53f77fbf75d5a9316d5/mailnews/compose/src/SmtpClient.sys.mjs#1064
So the username would be sent in UTF-16, which looks a lot like latin1 for the lower 256 characters.

Compare to the UTF-8 conversion for the password here:
https://searchfox.org/comm-central/rev/72b8ba0761b3881d926be53f77fbf75d5a9316d5/mailnews/compose/src/SmtpClient.sys.mjs#1096

Looks like this has been wrong for a long time.

Comment 10

[Wayne Mery (:wsmwk)]

Apparently we need more tests, manual and automatic, around these types of issues. Another job for QA.

Comment 11

[Wayne Mery (:wsmwk)]

a couple adjacent items in https://mzl.la/3R7fG8t (I haven't attempted to narrow it down)

Comment 12

[Francesco (no longer active, use PM for contact)]

If this is a regression, it was regressed by the SMTP rewrite in JS. It's not an internationalisation issue, just not converting from JS/UTF-16 to UTF-8. You can trigger it using any non-ASCII character, like € or £. Sure, it surfaces when people start using "local" non-ASCII characters in usernames (or passwords).

Comment 13

[Ótzï fón GlüϷgor]

Here's a poll on Mozilla Connect regarding this topic
https://connect.mozilla.org/t5/ideas/completely-smtputf8-imap4rev2-support-rfc-6531-rfc-9051/idi-p/123992?emcs_t=S2h8ZW1haWx8ZGlnZXN0X25vdGlmaWNhdGlvbnxNUlcxRE5aVVlUMkExWHwtMXxPVEhFUlN8aEs

Comment 14

[Magnus Melin [:mkmelin]]

Attachment: Bug 2056491 - SMTP username in account authentication not encoded in UTF-8. r=#thunderbird-back-end-reviewers

Two bugs really: not encoding properly, and not storing the username pref value the right way.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/75977b75cb66
SMTP username in account authentication not encoded in UTF-8. r=jtracey

Comment 16

[Magnus Melin [:mkmelin]]

Comment on attachment 9628530 [details]
Bug 2056491 - SMTP username in account authentication not encoded in UTF-8. r=#thunderbird-back-end-reviewers

Uplift Approval Request

• Please state case for uplift consideration and ensure bug severity is set: Not to mess up migrations - https://bugzilla.mozilla.org/show_bug.cgi?id=2069089#c22
• User impact if declined:
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Daily?: Yes
• Has the fix been verified in Beta?: Yes
• Needs manual test from QA?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Has been in the tree since 156, no regressions reported.
• Does the fix cause any migrations to be skipped?: N/A
• String changes made/needed: none

Comment 17

[Helge H.]

How can you backport this if you're skipping bug 2044250 which also did a migration (apart from the fact that the patch won't apply)?

Comment 18

[Magnus Melin [:mkmelin]]

Right, bug 2044250 would be the odd one out...
I think we could skip that (https://hg-edge.mozilla.org/comm-central/diff/835e07554d036cdead02d7a26168f29626393441/mail/modules/MailMigrator.sys.mjs) migration for esr, and then add it back later before next esr.

Comment 19

[Corey Bryant]

It's a good point. How would you ensure the migration for bug 2044250 added back before the next esr? Bug 2044250 isn't really a candidate for uplift.