218025 - International (Polish) characters cause pop3 login failure

Status: RESOLVED FIXED

(MailNews Core :: Networking: POP, defect)

Description

[Joe Infla]

Build Identifier: Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.4) Gecko/20030624

It seems that some international keyboard characters don't work in passwords,
according to bug 217841 comment 3:

"The problem with login to pop3 server is when exist in password an unnormal
sign. In my password there was a letter "ł" from polish alphabet. I changed
password and it's all OK, but Outlook goes with old passwd (with polish letters)
too. I don't know this is the bug or no, but it's true;)"

Comment 1

[Frank Schönheit]

confirming. For me, since moving to Mozilla 1.5, accounts whose password
contains an umlaut (ä,ö,ü) do not work anymore - at least not always.

Especially, when first opening the mailnews window, all respective
authentications fail.

I darkly rememeber that something like this hit me in 1.5 Beta, too, but it
wasn't that obvious (IIRC, it worked sometimes).

Comment 2

[Daniel Veditz [:dveditz]]

->Mailnews

Regression impacting Int'l folks, good candidate for any 1.5.1 that might come out.

Comment 3

[Daniel Veditz [:dveditz]]

changing assignee to match component

Comment 4

[Asa Dotzler [:asa]]

Seth, any ideas here? Should this block 1.6alpha (which is otherwise pretty much
ready to go)?  It's been open for a coupel of months with no dupes and without a
patch nearby, I'm inclined to push this to beta. 

Comment 5

[David :Bienvenu]

I wouldn't hold 1.6 for this.

a protocol log from the user would be helpful, just to see what authentication
mechanism is getting used.

http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#pop

Comment 6

[Frank Schönheit]

Attachment: pop3 log of unsuccessful authentication

This log was produced when opening the mail window the first time during the
Mozilla session. There are two accounts which are updated in this situation,
the passwords for both contain non-ascii character(s). The passwords are
retrieved from the password manager.

I debugged a little bit, and it seems that the password is properly stored in
the password manager. I don't remember the exact code location, but somewhere
on the way to the nsPop3Protocol it was converted to UTF8 using
NS_ConvertUCS2toUTF8 (IIRC), and never converted back. Something like this ...

(BTW: I would also not really consider this a blocker for 1.6a. Manually
retrieving mails afterwards works (not always, but well ...), and 1.6a IMO is
an "alpha" ....)

Comment 7

[Christian Eyrich]

Hm, I just changed my passwords for my GMX and Web.de accounts to contain
umlauts. But i still can login without problems.

For PLAIN mechanism (not used by this two accounts) I found  "Non-US-ASCII
characters are permitted as long as they are represented in UTF-8 [UTF-8]." and
"If non-US-ASCII characters are used in a username, they MUST use UTF-8. 
Passwords MAY contain arbitrary binary data excluding NUL, CR and LF
characters." for LOGIN mechanism.

As far I saw in my tests we always use ISO-8859-1 encoding when sending
credentials over the wire (resp. in the source used to get the base64 coded string).

Comment 8

[Christian Eyrich]

Huh, I was wrong. Login doesn't work here too. Now that I saved the passwords in
the manager, previously I didn't (didn't think that makes a difference).
The cause is, that for saving the password in the manager, it's converted to
UTF-8 - but not back when sending.

But why does it work after that? I don't know at the moment. It seems it gets
converted back to ISO-8859-1 after the password failed the first time.
Quite confusing.

Comment 9

[Christian Eyrich]

So, I now tested 1.5a, 1.5b, 1.5rc1, 1.5rc2 and 1.5. The password has always
(means at least since 1.5a) been saved as UTF-8 in the password manager (.s file).
MailNews of 1.5a to 1.5rc1 gets mails because password is sent in Windows-1252
(resp. single-byte umlaut). MailNews of 1.5rc2 and 1.5final fails because
password is sent as saved in UTF-8 (resp. multi-byte umlaut).

I doubt this change has been caused by a change in the Pop3Protocol handler.
IIRC we didn't change anything in the after 1.5 branch.

Comment 10

[David :Bienvenu]

I probably screwed this up - does the password prompting code convert the
password from utf8 to another charset when it pulls it out of wallet?

Comment 11

[David :Bienvenu]

I changed the password code to first try to get the password out of the password
manager, before calling the password prompting code, which also gets the
password out of the password manager if it exists, and doesn't prompt the user.
Looking at the code, I think both methods should be equivalent. What might not
be equivalent is what we do with both strings after we get them from wallet.
Christian, can you try a patch for me? I'll attach it in a minute.

Comment 12

[David :Bienvenu]

Attachment: proposed fix

this uses the same method to convert the unicode string to a cstring, so in
theory, it should work like before.

Comment 13

[Christian Eyrich]

Hm, this change (bug 219976) was also made in the 1.5 branch? If not, it
couldn't cause this.

And I applied your patch but that changed nothing. I also backed out your patch
from bug 219976 - the problem persists.

Comment 14

[David :Bienvenu]

ah, ok, thx. Never mind, then, I guess I didn't screw this up. I doubt that
password manager has changed - so this regressed between rc1 and rc2 on the 1.5
branch? that can't have been many checkins. I'll try to figure out how to query
for that.

Comment 15

[David :Bienvenu]

ah, there was another patch checked onto the branch that could be related. I'll
attach another patch (cumulative) and a suggestion for how to back out that
patch as well...

Comment 16

[David :Bienvenu]

Attachment: fix another possible case of this

Christian, could you try this patch instead? There was another change that
might be involved. I'll attach the url that shows the diffs for that one too.

Comment 17

[David :Bienvenu]

http://bonsai.mozilla.org/cvsview2.cgi?diff_mode=context&whitespace_mode=show&subdir=mozilla/mailnews/local/src&command=DIFF_FRAMESET&file=nsPop3Protocol.cpp&rev1=1.196&rev2=1.196.4.1&root=/cvsroot

http://bonsai.mozilla.org/cvsview2.cgi?diff_mode=context&whitespace_mode=show&subdir=mozilla/mailnews/local/src&command=DIFF_FRAMESET&file=nsPop3Protocol.h&rev1=1.53&rev2=1.53.8.1&root=/cvsroot

these two diffs were checked onto the 1.5 branch by scott for me to attempt to
fix Bug 219162. 

Comment 18

[Christian Eyrich]

Yes, I figured out it's this 
    server->GetIsAuthenticated(&isAuthenticated);
in nsPop3Protocol::GetPassword() from the first diff.

Your patch to GetIsAuthenticated() repaires this.

Comment 19

[David :Bienvenu]

Asa, if the 1.6a ship hasn't sailed, it wouldn't hurt to check this into 1.6a -
Christian, thx for the detective work!

Comment 20

[Asa Dotzler [:asa]]

Today's the last day to get stuff landed for 1.6a. This'll need reviews and
landing before tomorrow morning's 8am builds if it's gonna make it. 

Comment 21

[David :Bienvenu]

Christian, could you put an official r= on that last patch? thx. I'll check it
in after I get approval.

Comment 22

[Frank Schönheit]

wow, that was *fast*! :) The patch works like a charm ...
Davin, thanks for the fix, and  Christian, thanks for the investigations!

Comment 23

[David :Bienvenu]

fix checked in, a=sspitzer over aim.

Comment 24

[Scott MacGregor]

*** Bug 224377 has been marked as a duplicate of this bug. ***

Comment 25

[Jungshik Shin]

I'm afraid this fix is kinda bogus. POP3(see section 3 of
http://www.faqs.org/rfcs/rfc2449.html) doesn't allow non-ASCII password. 

The patch is just downconverting UCS2 to ISO-8859-1. It wouldn't work for 
non-Latin1 characters.

Until POP3 is extended to support password in UTF-8, it's better to 'educate' users 
NOT to use non-ASCII password. 

Comment 26

[David :Bienvenu]

sure, that's fine, but we're just fixing a regression - some passwords used to
work that didn't work after the regression and that's not really acceptable.