Closed Bug 1500772 Opened 6 years ago Closed 6 years ago

Authentication failed (SMTP, POP3) when some special characters are in the mail password.

Categories

(Thunderbird :: Account Manager, defect)

Product:
Thunderbird
Component:
Account Manager
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(thunderbird_esr6064+ fixed, thunderbird64 fixed, thunderbird65 fixed)

Status:
RESOLVED FIXED
Milestone:
Thunderbird 65.0
Tracking Flags:
Tracking Status
thunderbird_esr60 64+ fixed
thunderbird64 --- fixed
thunderbird65 --- fixed

People

(Reporter: mac.johnson, Assigned: jorgk-bmo)

References

Details

Attachments

(5 files, 1 obsolete file)

1500772-latin1.patch (v2)
6.29 KB, patch
infofrommozilla
: review+
jorgk-bmo
: approval-comm-beta+
jorgk-bmo
: approval-comm-esr60+
Details | Diff | Splinter Review
2018-11-05_095151_thunderbird_SMTP_ASCII.log
15.60 KB, text/plain
Details
2018-11-05_100110_thunderbird_SMTP_POUND.log
21.63 KB, text/plain
Details
2018-11-05_101033_thunderbird_SMTP_UTF8.log
14.65 KB, text/plain
Details
2018-11-05_143826_thunderbird_60.3_POUND.log
8.42 MB, text/plain
Details 
User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0

Steps to reproduce:

- Change your mail password so that it contains only ASCII characters and/or digits and a 'ä' or a '§' character if your mail provider does allow this.
- Try to send or poll your mails.


Actual results:

When using SMTP or POP3 Thunderbird will show an error message: Authentication failed. It seems that the username or the password are wrong. But when using the login credentials via the web mail frontend (in this case webmail.all-inkl.com) the login works. Therefore it's probably Thunderbird having a problem with the mentioned characters 'ä' and '§'. If you remove these again and the password contains only ASCII characters and digits the connections to the SMTP and POP3 mail server are working. Maybe there are more special characters that cause this behaviour.

The issue appeared right after the upgrade from version 52.9.1 to 60.2.1 on the 16th of october on Ubuntu 18.04.1 LTS.


Expected results:

I think UTF-8 characters in passwords shouldn't throw authentication exceptions.
one of these https://mzl.la/2P9RksF ?
Flags: needinfo?(acelists)
That's very strange. Non-ASCII characters in passwords never worked in TB 52.x. This was fixed in TB 60.0 in bug 312593 with some more fixes in TB 60.2.1, like bug 1491755 and bug 1493542.

It should be working now. Alfred? I've never tried since my provider only allows ASCII passwords.
Flags: needinfo?(acelists) → needinfo?(infofrommozilla)
(In reply to Jorg K (GMT+2) from comment #2)
> That's very strange. Non-ASCII characters in passwords never worked in TB
> 52.x. This was fixed in TB 60.0 in bug 312593 with some more fixes in TB
> 60.2.1, like bug 1491755 and bug 1493542.
> 
> It should be working now. Alfred? I've never tried since my provider only
> allows ASCII passwords.

As you mentioned in Bug 1493542 comment 3, UTF-8 is required. We changed the
encoding of passwords to UTF-8.

But do all providers expect passwords UTF-8 encoded? Or do they still expect
them in Windows-1252?

What could we do? Should we undo it? Or make it configurable?
Flags: needinfo?(infofrommozilla)
First we need to find out what's going on. Reporter, your e-mail provider is all-inkl.com(?) and what are your authentication methods for POP and SMTP? I have an account with United Internet (GMX/web.de) and I have just changed a password there to contain ä§ and it worked flawlessly for IMAP and SMTP, didn't try POP.

BTW, in bug 1493542 we only changed windows-1252 to UTF-8 for two auth method here, the rest were corrections to UTF-8:
https://hg.mozilla.org/comm-central/rev/e55253fbb27b#l1.79 - LOGIN password
https://hg.mozilla.org/comm-central/rev/e55253fbb27b#l1.90 - PASS password
I thought the more common method is PLAIN login.

The changes in bug 312593 were surely more drastic:
https://hg.mozilla.org/comm-central/rev/23725f858c42#l33.90
(In reply to Jorg K (GMT+2) from comment #4)
> First we need to find out what's going on. Reporter, your e-mail provider is
> all-inkl.com(?) and what are your authentication methods for POP and SMTP? I
> have an account with United Internet (GMX/web.de) and I have just changed a
> password there to contain ä§ and it worked flawlessly for IMAP and SMTP,
> didn't try POP.

Yes, my mail provider is https://all-inkl.com/ and maybe there was a change on their side regarding the mail server. But it's really remarkable that the problem appeared right after updating to 60.2.1 because before this update the password worked without any problems. 
The authentication method for POP3 and SMTP is btw "Password, normal" and the security protocol set is "SSL/TLS".
(In reply to mac-79 from comment #5)

>  But it's really remarkable that the problem appeared right after updating to 60.2.1
> because before this update the password worked without any problems.

Not really. As Jorg K already wrote, TB 59 (and older) still had sent 8-bit characters
in the local character set. But that did not meet the requirements of the standards.

> The authentication method for POP3 and SMTP is btw "Password, normal" and

Could you please test it with the setting:  Authentication Method: "Encrypted Password"

TIA
(In reply to mac-79 from comment #5)
> Yes, my mail provider is https://all-inkl.com/

(In reply to Alfred Peters from comment #6)
> Could you please test it with the setting:  Authentication Method:
> "Encrypted Password"

That will not work. if I'm right, they only support AUTH PLAIN and AUTH LOGIN.
Alfred, we had another report were a pound sign £ stopped working, bug 1503625.
For what it's worth, changed my password to contain 䧣 and it's still working on GMX.
What I can find in five minutes is this:
https://tools.ietf.org/html/rfc5721 talks about UTF-8 for non-ASCII usernames/passwords, and https://stackoverflow.com/questions/46058485/smtp-password-with-special-characters-utf8-java-mail-encoding-issue mentions SMTPUTF8 on the server.

So maybe non-ASCII worked "by accident" on (some) non-SMTPUTF8 servers and sending them UTF-8 now stopped working.
Gene, do you know anything about SMTPUTF8?
Flags: needinfo?(gds)
But there would also have to be a server property for POP3, no? See also bug 1435903 which is about 8BITMIME.
I tried to get é and £ into my Microsoft account password and it was rejected. Also see:
https://docs.microsoft.com/en-us/azure/active-directory/authentication/concept-sspr-policy
All the special characters are pure ASCII, it explicitly says: Characters not allowed: Unicode characters.
https://kb.wisc.edu/office365/page.php?id=44994

So I have yet to see a provider which allows windows-1252 passwords but not UTF-8, as I said, GMX works fine with UTF-8.
(In reply to Jorg K (GMT+2) from comment #12)
> Gene, do you know anything about SMTPUTF8?

No, don't know much about char encoding.
Flags: needinfo?(gds)
Sorry, misunderstanding. SMTPUTF8 seems to be a server capability, like the various IMAP capabilities. Do we check SMTP server capabilities anywhere?
Found it: https://tools.ietf.org/html/rfc6531#section-3.2, SMTPUTF8 relates to mailbox names, not passwords. Passwords, at least for POP, are in https://tools.ietf.org/html/rfc5721.
    279 14:59:24.825135849 34328  587    SMTP     74    C: EHLO [192.168.1.6]
    280 14:59:24.954785578 587    34328  SMTP     238   S: 250-www180.your-server.de Hello [192.168.1.6] [75.138.172.187] | 250-SIZE 104857600 | 250-8BITMIME | 250-ETRN | 250-PIPELINING | 250-AUTH LOGIN PLAIN | 250-CHUNKING | 250-STARTTLS | 250 HELP

This shows the smtp ehlo response for the jorgk test account. Looks like various capabilities but I haven't really learned smtp. But apparently n/a based on comment 17.
(In reply to Jorg K (GMT+2) from comment #11)

> So maybe non-ASCII worked "by accident" on (some) non-SMTPUTF8 servers and
> sending them UTF-8 now stopped working.

Maybe not "by accident", but because clients like the old TB have always
sent passwords as 8-Bit SBCS.

I have also tested a bit: Freenet.de and Google require a password in US-ASCII.
While 'Web.de' allows umlauts. And there the TB trunk works fine.

I am not aware of any 'capability' that says anything about the character set
of passwords. No matter what protocol.

Only at the 'Digest-MD5' SASL mechanism (RFC 2831) the server declares if he
does support UTF-8. But 'Digest-MD5' is obsolete. That's why we probably don't
use that anymore. Only in some comments there are still traces of it.
As I said, no problem with 䧣 on GMX which is run by "United Internet" as is web.de (comment #4). Also see bug 1503625 comment #7. There must be more systems around supporting unicode passwords, since people have been asking form them since 2005 (bug 312593).

I just don't understand what happens to Google's offer in China, Japan, Korea, Thailand, Arab countries, Israel, Greece, etc. Are those users forced to use an ASCII password?
Oops, more obviously: Cyrillic, Russia, Ukraine, etc.
So what can we do? As you can see in the log (attachment 9021579 [details]), we
try the different mechanisms until we get to the simplest USER/PASS
from RFC 1939. There, nothing is said about character sets. And it
doesn't fall under SASL either. So we could switch back to the ASCII
password for this mechanism.
Servers supporting UTF-8 should already have accepted the credentials
at the SASL mechanisms. But even that is not 100% sure.
OK, according to the log, first PLAIN = 0x1000 is tried, followed by USER/PASS = 0x400.
We could revert the USER/PASS code here
https://hg.mozilla.org/comm-central/rev/e55253fbb27b#l1.90

What do you suggest we do for SMTP? There's nothing we can revert, is there?
https://hg.mozilla.org/comm-central/diff/23725f858c42/mailnews/compose/src/nsSmtpProtocol.cpp (bug 312593)
https://hg.mozilla.org/comm-central/rev/afd14a54871a (bug 1491755)

Well, in nsSmtpProtocol::AuthLoginStep2() you could split LOGIN from PLAIN and let it use an ASCII/Latin password again.

All ask the reporter from bug 1503625 for an SMTP log: Bruno, we're considering returning the POP3 auth method USER/PASS back to ASCII/Latin/windows-1252 so your £ might work again. We're not sure what to do with SMTP. Would you mind providing an SMTP log for us.
Flags: needinfo?(bruno)
Attached patch 1500772-latin1.patch (obsolete) — — Splinter Review 
Something like this?
Attachment #9022394 - Flags: feedback?(infofrommozilla)
Comment on attachment 9022394 [details] [diff] [review]
1500772-latin1.patch

Review of attachment 9022394 [details] [diff] [review]:
-----------------------------------------------------------------

POP3: ++
{JFTR: <https://dxr.mozilla.org/comm-central/source/comm/mailnews/imap/src/nsImapProtocol.cpp#5980> 😉}

SMTP: That's an ugly hack. But let's try it. +

Would it be an option to try both: UTF-8 and ASCII password (at last if it contains 8bit)?
Attachment #9022394 - Flags: feedback?(infofrommozilla) → feedback+
How about this with a pref?

We should really see whether Bruno's server drops down to SMTP LOGIN auth based on the log we don't have yet.

Bruno, would you be prepared to try a special TB 60.3 version with this patch applied to see whether it helps your problem?

(In reply to Alfred Peters from comment #25)
> Would it be an option to try both: UTF-8 and ASCII password (at last if it
> contains 8bit)?
Well, not without considerable changes to the logic, right? You'd have to run the auth method twice and that's not a small low-risk tweak like this one. Maybe a pref will help. Most users won't authenticate with LOGIN or USER/PASS auth so they won't even notice.
Assignee: nobody → jorgk
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Attachment #9022440 - Flags: review?(infofrommozilla)
Oh, I forgot to say: Instead we could do an option to always use Latin1 on all auth methods and hence clearly infringe RFC 5721 on demand.
Comment on attachment 9022440 [details] [diff] [review]
1500772-latin1.patch (v2)

Review of attachment 9022440 [details] [diff] [review]:
-----------------------------------------------------------------

That's OK.
Attachment #9022440 - Flags: review?(infofrommozilla) → review+
(In reply to Jorg K (GMT+2) from comment #27)
> Oh, I forgot to say: Instead we could do an option to always use Latin1 on
> all auth methods and hence clearly infringe RFC 5721 on demand.

That would be the luxury solution with *.
But then we would need a pref per account.
(In reply to Alfred Peters from comment #28)
> That's OK.
Thanks. Before I land this, I'd really like some user feedback, at least via an SMTP log or better, a user willing to try a specially compiled version. The reporter here seems to be on Windows, so he can easily try.
Flags: needinfo?(mac-79)
(In reply to Jorg K (GMT+2) from comment #27)
> Oh, I forgot to say: Instead we could do an option to always use Latin1 on
> all auth methods and hence clearly infringe RFC 5721 on demand.
Reading more RFCs, according to https://tools.ietf.org/html/rfc4616, PLAIN SASL uses UTF-8, so no point offering anything else there.
Flags: needinfo?(bruno)

    
    

    
  
The _ASCII _POUND and _UTF8 suffix indicates which characters were included in the password used to produce the log file.
Yes, I'm willing to test a special 60.3 with patches; I'll probably need guidance on how to compile/package/install it.
Given how Outlook 14 used to send windows-1252 encoded messages with an "iso-8859-1" header [0] and given how Exchange used to corrupt text/plain UTF8 encoded messages when they had an attachment, it would not surprise me to discover that office.com is maybe not respecting RFCs and maybe demanding windows-1252 encoded password while it is supposed to accept utf8 encoded passwords -_-'

[0] https://twrh.noblogs.org/files/2012/12/ss_20121213-110724.jpg

    
    

    
  
Thanks for the logs, the show that for POP3 both PLAIN and USER/PASS are supported, for SMTP LOGIN is supported (and PLAIN is not). PLAIN needs to use UTF-8 as per RFC 4616, but I found nothing in about the encoding for USER/PASS and LOGIN.

So we're dropping them back to Latin1, or call it windows-1252, by default, but with a pref that can raise them again to UTF-8. You are on Linux, so I will prepare a Linux 64 build which you just need to download, unpack and run. Further instructions to follow.
Flags: needinfo?(mac-79)

    
    

    
  
https://queue.taskcluster.net/v1/task/Oofl973rRYOiSSiPD5T5KQ/runs/0/artifacts/public/build/target.tar.bz2
Unpack this and run it. It's TB 60.3.1, basically 60.3.0 with this tweak and one insignificant other one.

    
    

    
  


  
I cut some parts to go under the 10 MB limit

    
    

    
  
I'm not sure why you attached that. POP said:
2018-11-05 13:38:50.288796 UTC - [5760:Main Thread]: D/POP3 [this=0x7f6a102dca60] PASS password
2018-11-05 13:38:50.288838 UTC - [5760:Main Thread]: I/POP3 [this=0x7f6a102dca60] Logging suppressed for this command (it probably contained authentication information)
2018-11-05 13:38:50.821113 UTC - [5760:Main Thread]: I/POP3 [this=0x7f6a102dca60] Entering NET_ProcessPop3 34
2018-11-05 13:38:50.821124 UTC - [5760:Main Thread]: I/POP3 [this=0x7f6a102dca60] Entering state: 3
2018-11-05 13:38:50.821138 UTC - [5760:Main Thread]: I/POP3 [this=0x7f6a102dca60] RECV: +OK User successfully logged on.
2018-11-05 13:38:50.821146 UTC - [5760:Main Thread]: I/POP3 [this=0x7f6a102dca60] Entering state: 31
2018-11-05 13:38:50.821149 UTC - [5760:Main Thread]: D/POP3 [this=0x7f6a102dca60] NextAuthStep()
2018-11-05 13:38:50.821152 UTC - [5760:Main Thread]: D/POP3 [this=0x7f6a102dca60] login succeeded

And SMTP said:
2018-11-05 13:39:51.864307 UTC - [5760:Main Thread]: D/SMTP LOGIN auth
2018-11-05 13:39:51.864318 UTC - [5760:Main Thread]: I/SMTP Logging suppressed for this command (it probably contained authentication information)
2018-11-05 13:39:51.916307 UTC - [5760:Main Thread]: I/SMTP SMTP entering state: 0
2018-11-05 13:39:51.916319 UTC - [5760:Main Thread]: I/SMTP SMTP Response: 334 UGFzc3dvcmQ6
2018-11-05 13:39:51.916326 UTC - [5760:Main Thread]: I/SMTP SMTP entering state: 18
2018-11-05 13:39:51.916328 UTC - [5760:Main Thread]: D/SMTP SMTP Login response, code 334
2018-11-05 13:39:51.916330 UTC - [5760:Main Thread]: I/SMTP SMTP entering state: 17
2018-11-05 13:39:51.916334 UTC - [5760:Main Thread]: D/SMTP SMTP AuthLoginStep2
2018-11-05 13:39:51.916336 UTC - [5760:Main Thread]: D/SMTP LOGIN auth, step 2
2018-11-05 13:39:51.916342 UTC - [5760:Main Thread]: I/SMTP Logging suppressed for this command (it probably contained authentication information)
2018-11-05 13:39:52.735045 UTC - [5760:Main Thread]: I/SMTP SMTP entering state: 0
2018-11-05 13:39:52.735065 UTC - [5760:Main Thread]: I/SMTP SMTP Response: 235 2.7.0 Authentication successful

So this appears to be working, right?

    
    

    
  
Pushed by mozilla@jorgk.com:
https://hg.mozilla.org/comm-central/rev/7fcefde4be7f
Revert POP3 USER/PASS and SMTP LOGIN auth back to ASCII/Latin1. r=AlfredPeters
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED

    
  
Target Milestone: --- → Thunderbird 65.0

    
  

        Attachment #9022440 -
        Flags: approval-comm-esr60+

        Attachment #9022440 -
        Flags: approval-comm-beta+

    
  

        Attachment #9022394 -
        Attachment is obsolete: true

    
    

    
  
I attached the log to show that the patch was working and because maybe it contained some interesting stuff.
Thanks :)

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: