200436 - implement SPA (secure password authentication) using SSPI (aka NTLM for SMTP/IMAP/POP3/NNTP) - fixed for SMTP and POP3

Status: RESOLVED FIXED

(MailNews Core :: Networking, enhancement)

Description

[Martin Kutschker]

User-Agent:       Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.3) Gecko/20030312
Build Identifier: 

Now that bug #159015 is fixed, it's time to extend the NTLM authentication
support to the "mail" protocols: SMTP, IMAP, POP and NTTP.

An overwiew of the mechanism may be found here:

Microsoft Knowledge Base Article - 175440
XFOR: Protocol Authentication on Exchange Server
http://support.microsoft.com/default.aspx?scid=KB;en-us;q175440

NTLM (Windows NT Challenge/Response)

The Windows NT Challenge/Response (NTLM) authentication method is much more
secure because it uses a randomization algorithm and an encrypted password to
authenticate users.

The NTLM protocol is described by the following process:

1. The client sends an AUTH NTLM command to the server on
TCP/IP Port 110.

2. The server responds with a +(space) response to indicate
the port will accept commands.

3. A Negotiate message is sent to the server with some random
information.

4. The server responds by creating a unique challenge with
the random information.

5. The client encrypts the challenge with its password and sends
the response to the server.

6. The server then looks up the user's password and also encrypts
the challenge sent to the client. If the encrypted data matches
the client response, then a Successful Acknowledge is sent to
the client. Otherwise an Access Denied is returned again to
the client.

More info at http://www.kuro5hin.org/story/2002/4/28/1436/66154 and a Unix 
implementation (doesn't use SSPI of course) at
http://www.stafford.uklinux.net/libesmtp/index.html

AFAIR Microsoft uses these scheme also to authenticate against LDAP servers.

Reproducible: Always

Steps to Reproduce:

Comment 1

[Eric]

For a description of NTLM in general, and HTTP, SMTP, POP3 and IMAP in 
particular see:

http://davenport.sourceforge.net/ntlm.html

Comment 2

[tmh]

This is actually quite simple...  have a look at how fetchmail does it.  It's
simpler than HTTP because no encryption component is used - SPA is basically
straight NTLMv1 authentication which is already implemented in lots of places (I
managed to knock together a PLAIN->NTLM proxy in a couple of hours, which I
needed at the time because my company blocked plaintext IMAP).

Comment 3

[Jo Hermans]

Note that bug 223318 claims that you'll need "AUTH MSN" authentication for MSN
SMTP servers. "AUTH MSN" should be a synonym for "AUTH NTLM" apparently.

Comment 4

[Darin Fisher]

adding POP3 to the list of protocols which can use NTLM.  see
http://davenport.sourceforge.net/ntlm.html for details on how NTLM should be
used with the mail protocols.  i'm working on a patch which should make this bug
much easier to solve (it involves factoring the NTLM calculations out of the
current HTTP specific code).

Comment 5

[Scott MacGregor]

*** Bug 225022 has been marked as a duplicate of this bug. ***

Comment 6

[Christian Eyrich]

taking

Comment 7

[Christian Eyrich]

Attachment: proposed patch

Patch that adds support for NTLM authentication to SMTP and POP3 using Darins
NTLM utilities.

I had no opportunity to test this with an Exchange server so don't know if it
works with this "reference" implementation. But tests with another server
supporting NTLM (v1) succeeded.
If anyone has access to an Exchange server with NTLM on SMTP or POP3, feel free
to test this.

Comment 8

[Olivier Cahagne]

*** Bug 119694 has been marked as a duplicate of this bug. ***

Comment 9

[David :Bienvenu]

basically, looks good. This will be good to have

+      if (inBuf)
+        rv = m_authModule->GetNextToken(inBuf, inBufLen, &outBuf, &outBufLen);
+      else
+        rv = NS_ERROR_FAILURE;
rv = (inBuf) ? ... : NS_ERROR_FAILURE;

same here:
+            if (inBuf)
+                rv = m_authModule->GetNextToken(inBuf, inBufLen, &outBuf,
&outBufLen);
         else
-        if (TestCapFlag(POP3_HAS_AUTH_APOP))
+                rv = NS_ERROR_FAILURE;


this appears a couple places. It doesn't seem like a useful comment to me...
+            // TlRMTVNTUAABAAAABwIIAAAAAAAAgAAAAAAAACAAAAA=


+            m_authModule = do_CreateInstance(NS_AUTH_MODULE_CONTRACTID_PREFIX
"ntlm");
+            // if this fails, then it means that we cannot do NTLM auth.
+            if (!m_authModule)
+                return NS_ERROR_UNEXPECTED;

I'd prefer that you pass in the &rv to doCreateInstance and just use its error
value, instead of overwriting it with NS_ERROR_UNEXPECTED. (this occurs in a
couple places)

There's some cloned code here. I think it can go in nsMsgProtocol, since both
nsSmtpProtocol and nsPop3Protocol inherit from them, and there might be other
protocols who want this capability.

Comment 10

[Christian Eyrich]

Ok, I did the changes.

But testing inBuf after PL_Base64Decode() was wrong. The return value has to be
tested, so I have now
+  if (PL_Base64Decode(m_commandResponse.get(), m_commandResponse.Length(),
(char *)inBuf))
+    rv = m_authModule->GetNextToken(inBuf, inBufLen, &outBuf, &outBufLen);
+  else
+    rv = NS_ERROR_FAILURE;

Having this as
+  rv = (PL_Base64Decode(m_commandResponse.get(), m_commandResponse.Length(),
(char *)inBuf)) ? m_authModule->GetNextToken(inBuf, inBufLen, &outBuf,
&outBufLen) : NS_ERROR_FAILURE;
is very hard to read I think.

And cloned code, there is some in SMTP and POP. And the code can easily be used
for IMAP as well. My problem here are the different output ways.
  cmd = base64Str;
versus
  PR_snprintf(buffer, ...);

Or do you think that's the right opportunity to unify them?

Comment 11

[David :Bienvenu]

+  rv = (PL_Base64Decode(m_commandResponse.get(), m_commandResponse.Length(),
(char *)inBuf)) ? m_authModule->GetNextToken(inBuf, inBufLen, &outBuf,
&outBufLen) : NS_ERROR_FAILURE;

I agree, that is hard to read. But,

rv = (PL_Base64Decode(m_commandResponse.get(), m_commandResponse.Length(),
(char *)inBuf))
    ? m_authModule->GetNextToken(inBuf, inBufLen, &outBuf,
&outBufLen)
    : NS_ERROR_FAILURE;

is not hard to read, IMO.

Re the cloned code, I'm wondering if this needs to be in nsMsgUtils.cpp/h, like
MSGCramMD5. The problem is that IMAP doesn't inherit from nsMsgProtocol. Maybe
we just need to make IMAP inherit from nsMsgProtocol and move some stuff like
crammd5 to there from nsMsgUtils.

So, yeah, let's use nsMsgProtocol. The helper routine can return the base64
encoded string and the caller can use it and then free it.

Comment 12

[Christian Eyrich]

Attachment: patch v2 reference

Re Hard to read ?: instead if. Formatting it that way slightly improves
readability, yes. But I think you overdo in your effort to prevend normal ifs.
While the ?: style is nice to produce a short and nice one liner it's not good
in such cases. But ok, I nevertheless use it here.

I now transfered the cloned code to nsMsgProtocol. What I'm not sure about is
if
char **response or nsCAutoString &response would be better as argument.
Here is the first possibility.

Comment 13

[Christian Eyrich]

Attachment: patch v2 pointer

Hmpf, I mixed them up. The former was the reference variant, this one is with
pointers, sorry.

Comment 14

[Christian :Biesinger (don't email me, ping me on IRC)]

> nsCAutoString &response

you should never do this, if you want something like it use |nsACString& response|

Comment 15

[Christian Eyrich]

David, would you please have a look on the v2 patches say what you think?

Comment 16

[David :Bienvenu]

looks good, with these comments:

these method names don't follow our method-naming conventions : do_NTLM_Step1
and do_NTLM_Step2 - you should remove the '_' 's.

+    if (NS_SUCCEEDED(rv) && outBuf)
+    {
+        *response = PL_Base64Encode((char *)outBuf, outBufLen, nsnull);
+        nsMemory::Free(outBuf);
+    }
+
+    return rv;

in theory, PL_Base64Encode can fail for out of memory, so you should probably add:

rv = (*response) ? NS_OK : NS_ERROR_OUT_OF_MEMORY after the call to
PL_Base64Encode(). Same thing applies to the Step2 method.

Comment 17

[Christian Eyrich]

> these method names don't follow our method-naming conventions : do_NTLM_Step1

Ah yes. But wouldn't it be DoNtlmStep1 then?

> in theory, PL_Base64Encode can fail for out of memory, so you should probably
> add:

You know that the POP and SMTP handler don't look at the rv? So in practice
nothing would change - except we add tests for rv after returning.


And you prefer response to be char **? Even if I take Biesi's remark into account?

Comment 18

[David :Bienvenu]

> Ah yes. But wouldn't it be DoNtlmStep1 then?

either way, DoNTLMStep1 or DoNtlmStep1 - I guess we tend to prefer the latter,
mostly because people are lazy about typing caps :-)

>You know that the POP and SMTP handler don't look at the rv? So in practice
>nothing would change - except we add tests for rv after returning.

These are core routines now - other users could care, and POP/SMTP probably
should check the return value.

>And you prefer response to be char **? Even if I take Biesi's remark into
> account?
No, good point. nsACString& response is probably the more accepted style now.
But, if you leave the response as char **, then you should use nsXPIDLCStrings
at the call sites, and getter_Copies. It produces cleaner looking code, and less
prone to memory leaks.

Comment 19

[Christian Eyrich]

Attachment: patch v3

v2 + changes from comment #16. It uses nsACString &response.

Comment 20

[David :Bienvenu]

+        char *base64Str = PL_Base64Encode((char *)outBuf, outBufLen, nsnull);
+        rv = (*base64Str) ? NS_OK : NS_ERROR_OUT_OF_MEMORY;
+        response = base64Str;
+        PR_Free(base64Str);
+        nsMemory::Free(outBuf);

can't this just be response = PL_Base64Encode(...)
rv = response.IsEmpty() ? NS_ERROR_OUT_OF_MEMORY : NS_OK;
nsMemoryFree(outBuf)

and save the extra allocation and free of base64Str? Similarly for the other
Step method.

Comment 21

[David :Bienvenu]

Oy, I'm an idiot; you can't do that because that will leak...I don't know if
there's a getter_Copies type construct for nsACString...

Comment 22

[David :Bienvenu]

Adopt is the method I was thinking of, but I don't know if it's a method on
nsACString

Comment 23

[David :Bienvenu]

Comment on attachment 138438 [details] [diff] [review]
patch v3

I'll try to improve the string allocation handling before I check this in.

Comment 24

[Christian Eyrich]

Ok, a nicer string handling would be great here.

BTW,
  rv = (*base64Str) ? NS_OK ...
is wrong anyway. It must be
  rv = (base64Str) ? NS_OK ...
My fault.

Comment 25

[David :Bienvenu]

Attachment: improve string allocation handling

change nsACString to nsCString so we can use Adopt. Christian, does this look
OK to you? I'll get mscott to sr, if so.

 Also contains a one line change to clear the running protocol in the case of
the server being down.

@@ -728,6 +728,8 @@
   }
   // need this to close the stream on the inbox.
   m_nsIPop3Sink->AbortMailDelivery();
+  m_pop3Server->SetRunningProtocol(nsnull);
+

Comment 26

[Scott MacGregor]

Comment on attachment 138492 [details] [diff] [review]
improve string allocation handling

looks great. Question, is the NTLM module part of the default build or do you
need a build flag to pick it up? Just asking in case I need to modify the build
config for thunderbird to pick up the NTLM module.

Comment 27

[Christian :Biesinger (don't email me, ping me on IRC)]

it's in the default build (assuming you enable crypto)

Comment 28

[David :Bienvenu]

fix checked in; thx, Christian!

Comment 29

[Christian Eyrich]

Hu, I was slow.

I get a segmentation fault if PL_Base64Encode() returns null (return (char *)0;).

And the patch misses to adjust POP3_HAS_AUTH_ANY_SEC to also hold
POP3_HAS_AUTH_NTLM. Together with a change to take patch of bug 67963 into
account, hunk 2 for nsPop3Protocol.h should be

@@ -104,12 +105,13 @@ enum Pop3CapabilityEnum {
     POP3_HAS_AUTH_PLAIN         = 0x00001000,
     POP3_HAS_AUTH_CRAM_MD5      = 0x00002000,
     POP3_HAS_AUTH_APOP          = 0x00004000,
-    POP3_HAS_RESP_CODES         = 0x00008000,
-    POP3_HAS_AUTH_RESP_CODE     = 0x00010000
+    POP3_HAS_AUTH_NTLM          = 0x00008000,
+    POP3_HAS_RESP_CODES         = 0x00010000,
+    POP3_HAS_AUTH_RESP_CODE     = 0x00020000
 };
 
 #define POP3_HAS_AUTH_ANY         0x00001C00
-#define POP3_HAS_AUTH_ANY_SEC     0x00006000
+#define POP3_HAS_AUTH_ANY_SEC     0x0000E000

Comment 30

[David :Bienvenu]

OK, thx, I'll fix those.

Comment 31

[David :Bienvenu]

Attachment: fix pop3 protocol state flags per Ch.ey

Comment 32

[David :Bienvenu]

Attachment: fix error creating base64 encoding string

Comment 33

[Christian Eyrich]

Comment on attachment 138497 [details] [diff] [review]
fix error creating base64 encoding string

In case you may not have noticed - you attached the wrong patch.

Comment 34

[David :Bienvenu]

Attachment: fix error creating base64 encoding string

Comment 35

[Christian Eyrich]

Comment on attachment 138498 [details] [diff] [review]
fix error creating base64 encoding string

Ok, not as nice as your original one, but works.

Comment 36

[Scott MacGregor]

hey Christian, if I wanted to take this patch and put it on a branch that was
created around 12/18, were all the pieces for NTLM already in place before the
18th besides this patch? Or were there additional necko changes for this stuff
that were not in by then?

Comment 37

[Christian Eyrich]

It only depends on patch of bug 224653 and that was checked in 11/17. So I don't
know why it shouldn't work.

Comment 38

[MatthiasElmo]

I have been plagued with this very problem.  Took me a long time to stumble on
this bug.  I am not a programmer but if anyone needs someone to help telt let me
know.  I am forced to use Outlook at work or Outlook Express using IMAP and
"SPA".   I would really like to get rid of the dead weight.  Let me know if I
can test IMAP fixes for you. 

Comment 39

[MatthiasElmo]

Woops.... somehow "help test" became "help telt"  Sorry for any confusion that
may have caused.

Comment 40

[David :Bienvenu]

Mathias, this fix was checked in about a month ago, so any Mozilla nightly build
since then should have this fix in it. The soon-to-be-release 1.7a will also
have the fix in it.

Comment 41

[Christian Eyrich]

Erm David, my patch only applies to SMTP and POP3, not IMAP. The IMAP code is
still somewhat strange to me, but I guess it should be quite easy for you to
implement it (the flow is also described here:
http://davenport.sourceforge.net/ntlm.html#ntlmImapAuthentication).

Matthias, this NTLM implementation still hasn't been tested against a M$ server
(only a third party server implementation), so help is very appreciated. If not
IMAP you could test sending mails (SMTP) with the current 1.7 trunk-builds.
Could you post the supported authentication mechanisms of your server?
Just type "telnet yourserveraddress 25" and then "EHLO test" in response to the
servers greeting.

Comment 42

[MatthiasElmo]

I used to work at an ISP so I do know a little about telneting to a mail server
on the various services.  Heck I can still even remember how to send an email
via telnet, :-).  I'll do a little editing of the text for security reasons but
here's some info....  I'll put *'s wherever it is necessary to cover something
up that shouldn't be disclosed for security reasons, eg the name of the server,
etc....

220 ************************* ESMTP Server (Microsoft Exchange Internet Mail
Service 5.5.2653.13) ready
ehlo test
250-************************* Hello [*********]
250-XEXCH50
250-HELP
250-ETRN
250-DSN
250-SIZE 0
250-AUTH LOGIN
250 AUTH=LOGIN

So SMTP doesn't even support NTLM.  However IMAP does..... see below....

* OK Microsoft Exchange IMAP4rev1 server version 5.5.2653.23
(*************************) ready
111 CAPABILITY
* CAPABILITY IMAP4 IMAP4rev1 IDLE LITERAL+ LOGIN-REFERRALS MAILBOX-REFERRALS NAM
ESPACE AUTH=NTLM
111 OK CAPABILITY completed.

Comment 43

[MatthiasElmo]

I have varified that IMAP does work in Outlook Express and Outlook but it does
not work in Mozilla 1.6 or in the latest Thunderbird nightly (tested last night).

Comment 44

[MatthiasElmo]

I also tested POP3 on the same server yesterday with Mozilla 1.6 and the latest
Thunderbird daily.  It too failed.

here is the data from our server's POP3 service...
+OK Microsoft Exchange POP3 server version 5.5.2653.23 ready
auth
+OK The operation completed successfully.
NTLM

Comment 45

[MatthiasElmo]

Oh.... one more question.  I didn't even think about this but if I test a daily
build which daily builds is this code being built into?  Mozilla, Thunderbird,
or both?  Then I know which dailys to download and test.

Comment 46

[David :Bienvenu]

oh, sorry, I didn't notice the IMAP part - I should not have marked this fixed,
since we haven't done IMAP. I'll try to do an imap implementation. A test
account to try this against would be tremendously helpful!

Seamonkey and Thunderbird nightly builds both get the same backend fixes, and
many of the same front end fixes. This is a backend fix.

Re POP3, can you generate a POP3 protocol log by following these instructions
for POP3? I'm not sure we generate log the authentication process, but it might
help.

http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap

Comment 47

[Christian Eyrich]

Matthias, thank you for the informations provided.
It's a pity that NTLM isn't supported for SMTP, but as you wrote, you can test
it with POP3.

NTLM is also in TB 0.5 final that has just been released.

It's a little bit strange that POP3 didn't work when you tested it. 
To use NTLM you have to check "Use secure authentication" in the POP server
settings. But if this isn't checked the standard authentication via plain
USER/PASS is used. Though it's possible that the admin disabled it, I never have
seen this.

If you've any problems, just comment here and attach a POP3 communications log
(see the URL David posted).

Comment 48

[Martin Kutschker]

Added NTLM to the list of protocols for the sake of completeness.

Don't know if this protocol should be listed here or in the existing bug 119694.
Don't know how many sites use NTLM to protect internal news server. They
probably use IMAP.

Comment 49

[MatthiasElmo]

Weird.  I totally swear I put everything in correctly in the daily I downloaded
but I got home today and tried it again.  IMAP still doesn't work but POP3
totally works.

Last night when I checked "Secure Password Authentication" it said this server
does not support "Secure Password Authentication".......

Anyway.... POP3 seem to be working.
I can almost feel IMAP working.... :-)  At least I get the impression that going
from POP3 to IMAP working shouldn't be much of a streach....

Thank you all who are working on this.

Comment 50

[MatthiasElmo]

Are there any new builds for me to test?  Specifically that have IMAP implimented?

Comment 51

[David :Bienvenu]

Not yet, IMAP is on my list but I haven't gotten to it yet.

Comment 52

[Graham King]

I'm very eager to test IMAP too. Exchange server inside firewall so I can't make
accounts available but I'll report back as best I can.

Comment 53

[Christian Eyrich]

Graham, the SMTP server you're using also doesn't support NTLM?

Comment 54

[Graham King]

Yes, it does:

220 EXLN1-VS1.london.kbcfp.com Microsoft ESMTP MAIL Service, Version: 6.0.3790.0
ready at  Fri, 20 Feb 2004 10:01:33 +0000 
ehlo test
250-EXLN1-VS1.london.kbcfp.com Hello [194.203.211.16]
250-TURN
250-SIZE 10485760
250-ETRN
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-8bitmime
250-BINARYMIME
250-CHUNKING
250-VRFY
250-X-EXPS GSSAPI NTLM LOGIN
250-X-EXPS=LOGIN
250-AUTH GSSAPI NTLM LOGIN
250-AUTH=LOGIN
250-X-LINK2STATE
250-XEXCH50
250 OK

 but it also supports cleartext (LOGIN is cleartext right ?). Our IMAP server is
NTLM only (go figure).

With the Thunderbird build of today (20th Feb 2004) I can send a mail using in
'Outgoing Server (SMTP)', 'Use secure connection: No'.
With 'TLS' I get 'Sending of message failed. An error occured sending mail:
Unable to connect to SMTP server EXLN1-VS1. The server may be down or may be
incorrectly configured ...'

 Is TLS related to NTLM ? If not how do I test NTLM on SMTP ?

 Thanks,
 Graham.

Comment 55

[Christian Eyrich]

LOGIN is base64 encoded cleartext. So the data is obscured, but easy to decode.

As your server doesn't support STARTTLS, it's ok you're getting an error when
you set TLS (the error message is not clear and misleading, but that's another bug).
No, TLS or any connection type is independent from the authentication mechanism
(LOGIN, NTLM a.s.o.).

In the standard configuration we try NTLM if available as your server says. But
from here I can't say if it works since we fall back silently to the next weaker
(LOGIN in this case) if one mechanism fails.
A short log (see comment #46 for the howto url) would make it sure.

Comment 56

[Graham King]

Attachment: Thunderbird nightly SMTP to Exchange server

Log uploaded.

Comment 57

[Christian Eyrich]

Hey, thanks a lot, Graham. That's the first evidence it works with a M$ server.

Comment 58

[MatthiasElmo]

Hey?  I showed that POP3 worked with a MS server days ago.  :-(.

Comment 59

[Christian Eyrich]

Sorry Matthias, firstly I meant NTLM on SMTP in comment #57 (I use initial
response there which could have been problematic). And secondly you only wrote
you can login to the POP3 server. But I don't know if it really used NTLM to
authenticate.

Comment 60

[MatthiasElmo]

Any updates for IMAP functionality?  I haven't seen any activity in quite a
while.  Just curious since I see that 1.7b was released and I thought this was a
blocker for it....

Comment 61

[Christian Eyrich]

No, no updates here. And blocker? It never was - and I don't think an RFE should be.
But though not having deeper knowledge of IMAP I'll try to port my fix over.

Comment 62

[David :Bienvenu]

Christian, I'm going to do this for 1.8, and it's going to require changing
nsImapProtocol to inherit from nsMsgProtocol, which I don't think you want to
attempt.

Comment 63

[Christian Eyrich]

Yep, that would be to much for the start. If you've it on your list, I can
resist from digging in.

Comment 64

[David :Bienvenu]

I have this coded up for IMAP, but I don't have anything to test against. Does
anyone have an imap test account I could try? thx.

Comment 65

[MatthiasElmo]

I can test it but I don't have a public server I can give you.  It is my work
email.  Is there a place I can download a windows build of this version so I can
test it?

Comment 66

[David :Bienvenu]

no, there's no test build - I just have the code written but haven't found
anything to run it against yet.

Comment 67

[Graham King]

My IMAP (Exchange) is internal to the company too, but I found this server:
http://www.stalker.com/cpro/

The two key features being
"
The CommuniGate Pro Server supports the non-standard NTLM and MSN SASL methods
used in Microsoft® products. 
"
and
"
We offer fully functioning, fully supported evaluation versions for you to
download and install
"

 Maybe you could use that ?

Comment 68

[David :Bienvenu]

thx, that's helpful. Communigate does AUTH=MSN, not AUTH=NTLM, so as Christian
found earlier, I'll need to handle both.

Comment 69

[David :Bienvenu]

OK, I'm not convinced that Communigate's NTLM/MSN authentication works with
Mozilla. It didn't work with my IMAP code, and then I tried it as a pop3 server,
and it failed there too. Normal authentication works fine. It could be problem
with our ntlm implementation -  m_authModule->GetNextToken(inBuf, inBufLen,
&outBuf, &outBufLen) did not seem to give an outbufLen that was correct, and the
string returned in step 2 seemed awfully generic. Maybe I didn't set up the
server correctly. But I suspect that the IMAP code should work now, asusming the
basic stuff works. I do have an issue with thread stuff - I don't know if PSM
code can run on different threads.  I do know that I need to make sure PSM gets
loaded before the imap thread tries to use it, because the psm loading code
asserts if it's not called from the UI thread. I probably need to pre-flight it.
I could proxy the ntlm calls to the UI thread too...

Comment 70

[David :Bienvenu]

I take it back about m_authModule->GetNextToken - I stepped through it, and it
looks like it was doing the right thing...and there is a configuration option to
get the communigate server to advertise NTLM. But I'm still getting an error
from the server that the username or password is incorrect, when I do auth=msn.
I'll try this with OE.

Comment 71

[David :Bienvenu]

OK, I know why OE works and we don't with this server - I looked at the
NTLM_TYPE1_FLAGS flags we were sending, and compared them to the ones Outlook
Express was sending - OE was not sending NTLM_NegotiateUnicode,
NTLM_RequestTarget, or NTLM_NegotiateNTLM2Key. If I change our code not to set
those flags, we work with this server. I'll try to narrow it down to fewer
flags. Darin would know better than I do, but the problem is either with our
implementation of one or more of these flags, or with the communigate server.

Comment 72

[David :Bienvenu]

the problem is with NTLM_NegotiateNTLM2Key - if I turn off that flag, we work
with this server. But I don't know if it's a problem with our code or that server

Comment 73

[Christian Eyrich]

David, I installed CommuniGate too to see what I can do about the problem. But I
didn't find the switch to advertise NTLM - where is it?

I did the same work with The Bat instead of Outlook. And with MSN I can
reproduce the problem. It seems to be triggered by the flag NegotiateAlwaysSign
together with NegotiateNTLM2Key - it doesn't occur with only one of them set.
The question is, has the server a problem with it or Mozilla?

Darin, why do we set NegotiateNTLM2Key at all? Comment in ParseType2Msg() says
// we currently do not implement LMv2/NTLMv2 or NTLM2 responses,

Comment 74

[David :Bienvenu]

if you use the web admin page, under "Obscure", there's a page where you can
tell it to advertise MSN and/or NTLM. 

Comment 75

[Christian Eyrich]

Ah, thanks, indeed obscure. :)

Regarding not doing NTLM2, we seem to do and the comment is wrong.

Comment 76

[Darin Fisher]

(In reply to comment #73)
> Darin, why do we set NegotiateNTLM2Key at all? Comment in ParseType2Msg() says
> // we currently do not implement LMv2/NTLMv2 or NTLM2 responses,

we set the NegotiateNTLM2Key so that we can optionally issue a "NTLM2 session
response" in our type2 message.  this is not the same thing as LMv2/NTLMv2.
LMv2/NTLMv2 cannot be negotiated, so it is not very useful.  both the server and
client must be configured to use LMv2/NTLMv2 instead of LMv1/NTLMv1.  NTLM2
session response is just another beast altogether ;-)

Comment 77

[Christian Eyrich]

Ok, I must have made a mistake by evaluating the cause of the authentication
failure with CG Pro. It's really only the use of NegotiateNTLM2Key.
I contacted the makers of CG Pro - their answer:
"CommuniGate Pro 4.1x definitely does not support NTLM2 (it ignores that 
flag). And it does not support it "incorrectly", i.e. it does not return the 
proper capabilities flagset, inidicating that it does not support NTLM2. 
The 4.2 versions (currently in beta) should return the capability word 
correctly."

So our problem but not our fault.

Comment 78

[David :Bienvenu]

I think I finally parsed that :-) I don't think we can do anything about that
problem, especially if they're going to fix it. But I have a related question -
currently, in the pop3 code, we prefer CRAM-MD5 over NTLM - in other words, if
both capabilities are set, we'll use CRAM-MD5, and if that fails, I guess we
fall back to NTLM. Do you think I should do the same in the IMAP code? 

Comment 79

[Darin Fisher]

failover is probably a good idea.

as for NTLM protocols, i had toyed with adding a hidden preference to control
the version used by Mozilla.  we would need to add something like that if we
ever wanted to make Mozilla support LMv2/NTLMv2.

Comment 80

[Christian Eyrich]

David, I think IMAP should also prefer CRAM-MD5 (and yes, POP falls back).
It not only doesn't expose CG's problem, CRAM-MD5 should also be more secure
than NTLM (particularly as long we don't support NTLMv2 or if the server doesn't
do NTLM2).

Comment 81

[David :Bienvenu]

ah, thx, Christian, that's exactly what I wanted to know. And good job getting
hold of the CG Pro folks. I will fix the imap code to do fallback from cram-md5
to  MSN/NTLM

Comment 82

[David :Bienvenu]

Attachment: proposed fix for imap

This implements NTLM/MSN for IMAP. It makes it so we inherit from nsMsgProtocol
so we can use the ntlm utilities there. I should remove some of the
nsImapProtocol member variables that also exist in nsMsgProtocol...

Comment 83

[Christian Eyrich]

> I should remove some of the nsImapProtocol member variables that also exist in
> nsMsgProtocol...

And I vote for FormatStringWithHostNameByID().

Comment 84

[David :Bienvenu]

Attachment: proposed fix

cleaned up patch, remove duplicate stuff in nsImapProtocol that's already in
nsMsgProtocol - should be a small amount of code savings...

Comment 85

[Scott MacGregor]

Comment on attachment 145765 [details] [diff] [review]
proposed fix

I assume you expect both of these strings to be longer than 64 bytes most of
the time:

+	     nsCString challengeStr(GetServerStateParser().fAuthChallenge);
+	     nsCString response;

Comment 86

[David :Bienvenu]

those strings tend to be pretty long so I think nsCString is probably the way to go.

Comment 87

[David :Bienvenu]

fixed for imap

Comment 88

[Christian :Biesinger (don't email me, ping me on IRC)]

this caused bustage on some tinderboxes in nsImapCore.h:
+    kHasAuthMSNCapability = 0x00200000,  /* AUTH MSN extension */
 } eIMAPCapabilityFlag;

the comma here caused the error. I checked in a fix.

Comment 89

[Frederic Bezies]

Looks like this patch had broken thunderbird 0.6 branch :[

I opened bug 241849 for thunderbird.

Comment 90

[Graham King]

I'm happy to report that Thunderbird 0.6 works for NTLM with IMAP on MS Exchange
2003. Extracts from my log include:

2132[1f52e20]: 1f49de8:exln1-vs1:NA:CreateNewLineFromSocket: * OK Microsoft
Exchange Server 2003 IMAP4rev1 server version 6.5.6944.0
(EXLN1-VS1.london.kbcfp.com) ready.
(...)
2132[1f52e20]: 1f49de8:exln1-vs1:NA:CreateNewLineFromSocket: * CAPABILITY IMAP4
IMAP4rev1 IDLE LOGIN-REFERRALS MAILBOX-REFERRALS NAMESPACE LITERAL+ UIDPLUS
CHILDREN AUTH=NTLM
2132[1f52e20]: ReadNextLine [stream=1f53188 nb=28 needmore=0]
2132[1f52e20]: 1f49de8:exln1-vs1:NA:CreateNewLineFromSocket: 1 OK CAPABILITY
completed.
2132[1f52e20]: 1f49de8:exln1-vs1:NA:SendData: 2 authenticate NTLM
(...)
2132[1f52e20]: 1f49de8:exln1-vs1:NA:CreateNewLineFromSocket: 2 OK AUTHENTICATE
completed.

 which to my untrained eye looks good. I can post the whole log if asked.

 Thanks David.

Comment 91

[Rich Unger]

Does it work for NNTP as well?

Comment 92

[David :Bienvenu]

er, no, it doesn't work for NNTP - at least, I didn't do anything for NNTP. Any
references to docs as to how it works with NNTP?

Comment 93

[Rich Unger]

The only info I could find on SPA and NNTP was the issue filed against pan for
the same thing: http://bugzilla.gnome.org/show_bug.cgi?id=105050

Seems they haven't solved it yet, either.  At least they've found a server to
test against, though.