848365 - Thunderbird uses wrong encoding (ISO 8859-1 instead of UTF-8) for SMTP AUTH PLAIN

Status: RESOLVED INCOMPLETE

(MailNews Core :: Networking: SMTP, defect)

Description

[S]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.97 Safari/537.22

Steps to reproduce:

I used Thunderbird on MS Windows for SMTP AUTH login using PLAIN with a username containing the German umlaut "ü". When I 'm replacing the umlaut with "ü" the login is possible.




Actual results:

The login failed.


Expected results:

The login should be possible.

According to RFC 4616 the message is encoded using UTF-8. I think Thunderbird is encoding the login string with base64 based on ISO-8859-1 encoded string instead of a UTF-8 encoded string.

Comment 1

[rsx11m]

Are you indeed using Thunderbird 3.1 as the version stated when you opened this bug? (moving it to MailNews Core makes it the respective Gecko version.)

Comment 2

[rsx11m]

Hmm, the built-in log omits the username during authentication. Do you have a log somewhere showing the encoding of the username?

Comment 3

[S]

I tested it with the current Thunderbird version. In the reject.log of my local Exim the umlaut "ü" in the wrong login is logged as "\374". Exim is logging non-ascii characters in octal form.

Comment 4

[rsx11m]

That's 0xFC in hex and thus matches "ü" in the ISO 8859-1 table.
Thus, it's definitely not UTF-8 what is sent...

Comment 5

[Magnus Melin [:mkmelin]]

I suppose that should be here:
http://mxr.mozilla.org/comm-central/source/mailnews/compose/src/nsSmtpProtocol.cpp&rev=9a11f1cc1da9?rev=9a11f1cc1da9#1312

Something there not liking utf-8?

Comment 6

[Magnus Melin [:mkmelin]]

Base64 encoding assumes ASCII, so you have to convert the sting to be the an ASCII representation of the original

https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/btoa#Unicode_strings

https://searchfox.org/comm-central/rev/5862032a956262bfdda5a7903d24e1cb0fc93ae7/mailnews/compose/src/nsSmtpProtocol.cpp#1522

Comment 7

[Chris Newman]

SMTP AUTH PLAIN is defined by https://tools.ietf.org/html/rfc4616 and https://tools.ietf.org/html/rfc4954

The specification requires both the username and password to be UTF-8 encoded prior to applying the base64 encoding.

Now we're starting to see EAI deploying https://tools.ietf.org/html/rfc6531, UTF-8 usernames are likely to become more common.

Comment 8

[Wayne Mery (:wsmwk)]

(In reply to Chris Newman from comment #7)
> ...
> Now we're starting to see EAI deploying https://tools.ietf.org/html/rfc6531,
> UTF-8 usernames are likely to become more common.

bug 1259632

Comment 9

[Wayne Mery (:wsmwk)]

dup to bug 1563891 ?

Comment 10

[gene smith]

I'm still looking at the imap problems with utf8 and I want to get into the smtp issues too but just haven't had a chance yet. So don't know if this is a dup.

Comment 11

[gene smith]

Re: comment 0. Maybe this is already know but umlaut-u is U+00FC. This is encoded in UTF8 as bytes 0xc3 0xbc. When decoded with as 8859-1 they are "ü". Decoding and displaying UTF8 as 8859-1 seems to be a common problem in TB. However, I think it is a separate issue from bug 1563891 so I would say not a dup since it involves user names and not email addresses.

Comment 12

[Wayne Mery (:wsmwk)]

Version 91 has all new smtp backend code. If you can still reproduce this issue, please file a new bug report https://bugzilla.mozilla.org/enter_bug.cgi?product=MailNews%20Core&component=Networking%3A%20SMTP