162752 - IMAP/SSL Message Authentication Code Error

Status: VERIFIED FIXED

(Core Graveyard :: Security: UI, defect, P2)

Description

[Chuck Kuczaj]

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1b) Gecko/20020814
BuildID:    2002081409

When establishing an IMAP/SSL connection, the following popup
appears:
"<IMAP server> received a message with incorrect
Message Authentication Code.  If the error occurs frequently,
contact the website administrator."

When connecting with Outlook, there is the normal warning
about the self-signed server certificate, but the connection
works fine and I'm able to see my main via IMAP/SSL.

The following was observed in the /var/log/syslog file on
the Solaris 7 server box:

Aug 14 15:33:14 server imapd[20483]: imaps SSL service init from
X.X.X.X
Aug 14 15:33:14 server imapd[20483]: Unable to accept SSL connection, host=[X.X.X.X]
Aug 14 15:33:14 server imapd[20483]: SSL error status: error:1408F455:SSL
routines:SSL3_GET_RECORD:decryption failed or bad record mac



Reproducible: Always
Steps to Reproduce:
1. Create IMAP/SSL (port 993) account
2. Connect to remove IMAP/SSL server
3.

Comment 1

[Chuck Kuczaj]

Just FYI, my server is a Solaris 7 SPARC box running
UW IMAP 2002-RC2, and OpenSSL 0.9.7 Beta 2.

I used openssl verify and openssl s_client to verify
my self-signed certifications in imapd.pem and ipop3d.pem
in /usr/local/ssl/certs.

Using Outlook 2000 SR-1, I am able to connect.

Please let me know if I can be of any further help.

Comment 2

[John Unruh]

Reporter, you may have a corrupted certificate DB. Can you try this with a new
profile? related: bug 156341

Comment 3

[Jonas Linde]

I suspect I have the same bug, but it has nothing to do with IMAP. When I try to
view some of the electronic invoices at my online bank I get the popup error
"Mozilla received a message with incorrect Message Authentication Code. If the
error occurs frequently, contact the website administrator."

The URL that is failing is https://www.e-girot.net/ewh/visafaktura. Actually
there's a POST to this with a few hidden fields named cert and ticket but the
error pops up whith the naked URL as well.

I'm running Mozilla 1.1b 2002082708 on Linux and, yes, it is the same with a new
profile.

Is this the same bug or should I file a new one?

Comment 4

[John Unruh]

Regarding Jonas' website, https://www.e-girot.net/ewh/visafaktura , I can reach 
it if I disable TLS.
Here are the server's particulars - IBM_HTTP_Server/1.3.6.2 Apache/1.3.7-dev 
(Unix)
Supported SSL ciphers: 
DES with MD5 
RC4 with MD5 (export version restricted to 40-bit key) 
Nelsonb, I cannot reach this site with your tstclnt app in any of the 5 
combinations.

Comment 5

[Nelson Bolyard (seldom reads bugmail)]

John,  Please repeat the test with the 5 uses of tstclnt, using ssltap to 
record the connection activity.  If you have any questions about how to 
do that, please send me email or call me.  

Comment 6

[John Unruh]

Nelson, never mind about tstclnt failing. I was typing ">stdin.txt" instead of 
"<stdin.txt".
This bug appeared between these two builds - 2002-08-14-09-trunk/ and 
2002-08-15-04-trunk/                                                   

Kaie, this may have something to do with your checkins on 08/14/2002 16:43.
With the earlier build, I get a warning about low grade encryption, and can 
reach the site. With the next build, I cannot reach the site, and get the 
Message Authentication Code error. https://www.e-girot.net/ewh/visafaktura

Comment 7

[Kai Engert [:KaiE:]]

So you suspect the checkin from bug 87902. I'll have a look.

Comment 8

[Nelson Bolyard (seldom reads bugmail)]

John, I'd really like to see the ssltap output for this site.
Without that output, we don't know what's really going wrong.
We shouldn't "fix" this problem until we know what the real problem is.

Comment 9

[Sean Harding]

I'm seeing this problem when I try to follow the link
http://www.alsoft.com/cgi-bin/BrowserCheck.pl, which appears to redirect to
https://www.asecureserver.com/cgi-bin/makepage.cgi?orderforma.html

Both URLs work fine in IE on the same machine. 

This is build 2002082909 on Mac OS X 10.2.

Comment 10

[Nelson Bolyard (seldom reads bugmail)]

Attachment: ssltap output shows failed attempts to do https request

The ssltap output clearly shows the problem.  The browser is NOT attempting to
do any SSL.  It is opening the connection and then sending the http request in
the clear without doing any SSL first.	

So, I'd guess that the problem is that when an http URL redirects to an https
URL, the socket for the new request (to the https server) is being set up 
incorrectly, so that it doesn't attempt to do SSL.

Comment 11

[Sean Harding]

I don't think that's exactly it, because I get the same error when I go directly
to the HTTPS URL in Comment #9

Comment 12

[Kai Engert [:KaiE:]]

At least I'm now confused, since the summary says IMAP/SSL (which does not use
redirection), but you now talk about https.

Is this all the same problem, or should we have two separate bugs?

Comment 13

[Sean Harding]

Well, Comment #3 and on seemed to indicate that we were discussing both here
(or, more accurately, that these are two symptoms of the same problem). I'd be
happy to open a new bug if that's what folks think needs to be done.

Comment 14

[Nelson Bolyard (seldom reads bugmail)]

There are at least 2 separate bugs here, and separate bug reports should exist 
for each.  

There are (or, may be):
1. an imap/ssl bug resulting in a MAC error
2. an https bug, resulting in a MAC error
3. an https bug, which may occur on redirection only, where no SSL is done.
   I suspect that there is no BAD MAC error reported for this last case,
   because a BAD MAC error would only occur if the browser was attempting to
   do SSL, and in this case, it was not attempting to do SSL.

John, when you were producing that ssltap output, is it possible that you used 
an http URL instead of an https URL?

Comment 15

[Sean Harding]

I've opened Bug 166931 for the HTTP issue.

Comment 16

[Nelson Bolyard (seldom reads bugmail)]

Comment on attachment 98004 [details]
ssltap output shows failed attempts to do https request

The ssltap attachment above apparently is unrelated to the subject of this bug
report.  It was apparently produced using an http URL, not https.

Comment 17

[John Unruh]

Attachment: Corrected SSLTAP output

Comment 18

[Nelson Bolyard (seldom reads bugmail)]

In the initial report, and in comment 1, an IMAPS server reports receiving an 
SSL record with a BAD MAC.  Mozilla reports that the server has reported that
it (the server) received a record with a bad mac.  That's problem #1, 
ostensibly the subject of this bug report.  We don't know yet whether this
is related to TLS intolerance, or not.

In comment 3, Jona Linde reported that Mozilla complains that it has received 
a record with a BAD MAC from a certain https server, www.e-girot.net.  
This is problem #2, separate from #1 in two ways: 
a) it involves https, not imaps, and
b) mozilla (not the remote server) received the BAD MAC.  
In comment 4, John Unruh notes that problem #2 is apparently a TLS intolerant 
server  problem.  

In comment 9, Sean Harding reports a problem with another https server, 
www.asecureserver.com.  This https server is reporting that it received 
a BAD MAC from mozilla.  So, this is a third different set of symptoms.  
It is like problem 1 in that the server receives the (allegedly) bad mac, 
but unlike problem 1, it is an https problem.  It is like problem 2 in 
that the server is https, but unlike problem 2, the server receives the 
bad mac, not the browser.  Call this problem #3.  Another bug report has
now been opened for that problem: see bug 166931.  AFAIK, it has not
yet been determined whether this server is TLS intolerant, or not.

Comments 10 through 13 are the result of analyzing an ssltap output that
was reported to show the behavior of problem #3.  Unfortunately, it was
not actually showing any of the above 3 problems, and is irrelevant.
I wish there was a way to delete these comments and that first attachment
from this bug report.

Comment 17 (which accompanies the second attachment) shows the ssl protocol 
behavior of problem #3.  It probably should be attached to bug 166931.

It may be that these problems are all related.  It has been suggested that all 
these problems began recently.  

One observation about the ssltap output in 
http://bugzilla.mozilla.org/attachment.cgi?id=98027&action=view is that 
there is no attempt to retry the connection using SSL3 (rather than TLS).  

I can think of two potential explanations.  One of these is a testable 
hypothesis.  They are:
a) some change in NSS is causing NSS to now compute bad MACs on occasion.
b) some change in PSM now prevents PSM from falling back from TLS to SSL3 
when certain errors occur that are caused by TLS intolerant servers.  

I tend to doubt hypothesis a, because if NSS was computing macs incorrectly,
we'd see this problem a lot more than has been reported so far.

If it can be shown that all 3 servers are TLS intolerant, then hypothesis
b seems the most likely explanation for all the problems.  If repeating the 
tests with TLS disabled fixes the problem in all cases, then the problems are 
certainly related to TLS intolerant servers.

Comment 19

[John Unruh]

This command fails for both www.e-girot.net and www.asecureserver.com
/u/junruh/tls >./tstclnt -h www.e-girot.net -d . -o < stdin.txt
---------------------------------------------------------------------
This command passes for both www.e-girot.net and www.asecureserver.com
/u/junruh/tls >./tstclntx -h www.e-girot.net -d . -o < stdin.txt
So, the two servers are TLS intolerant, Flavor B. They both show a warning of 
Message Authentication Code failure, but they both can be reached if TLS is 
turned off.

Comment 20

[Kai Engert [:KaiE:]]

With your latest comments, you convinced me that it is very likely that all
reported problems are caused by the same problem.


I'm able to confirm that:
- this new behaviour is not caused by changes in NSS
- the new behaviour it caused by changes in PSM
- the culprit is the patch in bug 87902, which contained the logic to fix bug 149910

In bug 149910 it was requested to avoid unnecessary connection retries when it
is clear that the cause can not be a TLS intolerant server.

However, at the time of discussion in bug 149910, no complete list of error
codes were given, that would justify an automatic retry.

Because of the lack of knowledge which error codes justify an automatic retry, I
chose to reduce the list of such error codes to the bare minimum. I found that
list by experimenting with other known TLS intolerant servers. (That is
currently only: PR_CONNECT_RESET_ERROR when no proxy is being used, and
PR_END_OF_FILE_ERROR when a proxy is being used.)

I can confirm that backing out the patch from 87902 fixes this bug (although we
don't want to back out that patch).

A test showed: Adding SSL_ERROR_BAD_MAC_ALERT to the list of error codes where
we assume (on first write) an TLS intolerant server and make Mozilla
automatically retry, Mozilla is again able to access
https://www.asecureserver.com/cgi-bin/makepage.cgi?orderforma.html


Question 1:
Is it reasonable, when TLS is enabled, and first write fails with
SSL_ERROR_BAD_MAC_ALERT, to always automatically retry without TLS?

If yes, question 2:
Are there more SSL error codes, where PSM should assume it is worth retrying
without TLS, when there is a failure on first write? Is that list of such error
codes already known? Or is that list not yet known, and should we extend that
list of error codes later, as we find other TLS intolerant that cause other SSL
error codes?

Comment 21

[Nelson Bolyard (seldom reads bugmail)]

The answer to question 1 is Yes.

It is very likely that there are other error codes that also signify TLS 
intolerant servers on the first I/O attempt (read or write).  We do not 
have a complete list at this time.  I expect the list will grow as we 
get more experience with this.  

I doubt that we can determine by code examination alone the complete set 
of errors that indicate TLS intolerance.  However, I think there is a 
fairly large list of errors that we can be certain do NOT indicate TLS
intolerance.  Perhaps we should attempt to enumerate those, and then 
change the code to retry with SSL3 for any SSL or SEC errors not in that
list.  

What do you think of that idea?

Comment 22

[John Unruh]

*** Bug 166931 has been marked as a duplicate of this bug. ***

Comment 23

[Stephane Saux]

How do we get the list of errors that do not indicate a TLS issue?

We can also just add the error that's causing this bug to the current list and
open a bug for Nelson's solution (two step fix).

Comment 24

[Kai Engert [:KaiE:]]

I agree with the proposal of adding more error codes to "cause autoamtic non-TLS
retry" as we go.

As a result, I think we should create a patch within this bug, to add the
automatic retry for the SSL_ERROR_BAD_MAC_ALERT error only.

Comment 25

[Kai Engert [:KaiE:]]

Attachment: Patch v1

Make PSM retry on SSL_ERROR_BAD_MAC_ALERT on first write with -1 bytes written.

Comment 26

[Kai Engert [:KaiE:]]

Javi, can you please review?

Comment 27

[Javier Delgadillo]

Comment on attachment 101974 [details] [diff] [review]
Patch v1

r=javi

Comment 28

[Kai Engert [:KaiE:]]

What did I do? Why does the patch suddenly no longer work?

For some reason, today NSS does not report SSL_ERROR_BAD_MAC_ALERT, but
SSL_ERROR_BAD_MAC_READ.

Is it reasonable to retry on either error code?

I'm attaching a new patch that checks for both errors.

Comment 29

[Kai Engert [:KaiE:]]

Oh well. In addition, this patch is also incompatible with the required changes
done in bug 163605. Because of the introduction of the non-blocking mode used in
bug 163605, the error can show up at other places. Right now my testing showed
the error happens when necko calls our read function.

Comment 30

[Kai Engert [:KaiE:]]

The patch for this bug is now integrated into bug 163605.

Comment 31

[Nelson Bolyard (seldom reads bugmail)]

Kai,  In answer to your question in comment 28,

SSL_ERROR_BAD_MAC_ALERT means that the other end detected that it received
a bad mac and told us about it via an alert record.  SSL_ERROR_BAD_MAC_READ
means that we detected a bad mac in a record we received from the other end.

Either way, a bad MAC indicates that either:
a) some "man in the middle" is interfering with the traffic, or 
b) the two ends are not in agreement about the exact values of the keys 
they're using, which can be the result of TLS intolerance.  

So, yes, I'd say it's reasonable to treat both of those errors as reasons
to retry without TLS.

Comment 32

[Kai Engert [:KaiE:]]

Thanks for the explanation, the patch in bug 163605 will use both error codes.

Comment 33

[Kai Engert [:KaiE:]]

If you have time today and want to help test the fix for this bug, please grab a
build from http://www.kuix.de/mozilla/163605/ and test whether it works, and
there are no regressions in SSL functionality. Please give feedback today or
early tomorrow. Thanks!

Comment 34

[Kai Engert [:KaiE:]]

This should be fixed starting with tomorrow's build, by the patch for bug
163605. Please reopen if you still can reproduce the problem.

Comment 35

[John Unruh]

Verified.