278499 - has received an incorrect or inexpected message. Error Code: -12227 ("Padding length invalid" or "Invalid SSLv3 padding")

Status: RESOLVED INVALID

(MailNews Core :: Networking, defect)

Description

[Ralf Hauser]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0

when using Thunderbird version 1.0 (20041206) on a apache james secure Pop Site
the sun Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0-b64)
Java HotSpot(TM) Client VM (build 1.5.0-b64, mixed mode, sharing)
fails with the following error message:
Caused by: javax.crypto.BadPaddingException: Padding length invalid: 241
        at com.sun.net.ssl.internal.ssl.CipherBox.removePadding(CipherBox.java:442)
        at com.sun.net.ssl.internal.ssl.CipherBox.decrypt(CipherBox.java:290)
        at com.sun.net.ssl.internal.ssl.InputRecord.decrypt(InputRecord.java:144)
        at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:762)
        ... 16 more


Reproducible: Always

Steps to Reproduce:
the result is varying all the time --> see grep from the log file:
Caused by: javax.crypto.BadPaddingException: Padding length invalid: 81
Caused by: javax.crypto.BadPaddingException: Padding length invalid: 212
Caused by: javax.crypto.BadPaddingException: Padding length invalid: 252
Caused by: javax.crypto.BadPaddingException: Padding length invalid: 246
Caused by: javax.crypto.BadPaddingException: Invalid SSLv3 padding: 24
Caused by: javax.crypto.BadPaddingException: Padding length invalid: 190
Caused by: javax.crypto.BadPaddingException: Padding length invalid: 179
Caused by: javax.crypto.BadPaddingException: Padding length invalid: 193
Caused by: javax.crypto.BadPaddingException: Padding length invalid: 167
Caused by: javax.crypto.BadPaddingException: Padding length invalid: 241



Expected Results:  
outlook2003 and opera 7.5 have no problem

bouncycastle.org 1.2.5 is used to avoid the problem of bug 249223

is this related to Bug 166655 ?

Comment 1

[Ralf Hauser]

see also: http://issues.apache.org/jira/browse/JAMES-350

Comment 2

[David :Bienvenu]

this would seem more like a core ssl/security problem than a thunderbird problem
per se. Cc'ing Nelson. Does Seamonkey have the same problem?

Comment 3

[Ralf Hauser]

Apologies, worked on it a little bit more and it appears that if I start the
server with jdk1.4, the problem is not there

But anyway, wouldn't it be possible to return something more self-explanatory
than error "-12227"?

Comment 4

[Nelson Bolyard (seldom reads bugmail)]

This was yet another example of a misconfigured James server being unable
to negotiate a handshake with a valid client.  The only thing about this
incident that indicates mozilla's behavior that needs improvement is the
poor error message.  So, I am dup'ing this bug to 107491.

*** This bug has been marked as a duplicate of 107491 ***

Comment 5

[Ralf Hauser]

not totally convinced that the error is on the james side:
- Thunderbird and jdk1.4 negotiate TLS_DHE_RSA_WITH_AES_256_CBC_SHA and all
works fine
- Thunderbird and jdk1.5 negotiate TLS_DHE_RSA_WITH_AES_128_CBC_SHA and this it
fails with a fatal handshake alert
- Outlook and jdk1.5 use TLS_RSA_WITH_RC4_128_MD5 and it works with no problem

I'll attach the ethereal dumps of these negotiations to JAMES-350

Comment 6

[Nelson Bolyard (seldom reads bugmail)]

Above it has been indicated that changing the James software 
configuration enables James to negotiate the handshake.  
James is known to have problems with certain ciphersuites, ones
that IE does not implement.  So, I do not consider the statement
that it works with IE to indicate anything about the cause of 
the problem.  

Don't attach Ethereal output.  I won't look at it.
Attach output from ssltap or ssldump, if you must.
Please do not add any such output as a bug comment, but rather attach it.

Comment 7

[Ralf Hauser]

Nelson,
You are referring to james configuration errors "above" - I also checked through
the mentioned 107491 and there the word "james" doesn't appear to exist. Do you
have any specific reference for this applicable to the issue at hand?
I have the suspicion that James isn't the culprit because AFAIK to date, james
officially only supports JDK1.3. So, if it worked with jdk1.4 and no longer now,
I have the suspicion that the new sunjce_provider.jar of Sun's jdk1.5 has its
troubles with PSM in general and not just james.
ok, I'll try to attach the ssltap or ssldump you asked for next.

Comment 8

[Ralf Hauser]

as per docu RFE bug 281161, I didn't get ssltap to do what you wished - apologies

there seem to be subtle issues with the newest sun jce:
http://forums.java.sun.com/thread.jspa?messageID=3067152

Comment 9

[Ralf Hauser]

had difficulties with ssldump as well
(http://sourceforge.net/tracker/index.php?func=detail&aid=1116814&group_id=68993&atid=523055)
if anybody is willing to share a winXP binary or either one, it would be great!

Comment 10

[Nelson Bolyard (seldom reads bugmail)]

Ralf, Another example is your bug 249223

ssltap binaries for Windows are downloadable from 
ftp://ftp.mozilla.org/pub/mozilla.org/security/nss/releases/NSS_3_9_2_RTM/WIN954.0_OPT.OBJ/nss-3.9.2.zip

Comment 11

[Ralf Hauser]

Nelson, if you are referring to bug 249223, I can assure you that this bug here
is not the same, so that doesn't give any additional insights how to solve the
problem at hand.
Also, as per bug 281161, the binaries you cite don't appear to work on WindowsXP.
Do you have any other hints? Many thanks in advance!  Ralf

Comment 12

[Ralf Hauser]

still happens with 1.5.0_03-b07, java.class.version, : 49.0
will add the ssltap for this next


javax.net.ssl.SSLException: Connection has been shutdown:
javax.net.ssl.SSLHandshakeException: Invalid padding
        at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1154)
        at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:65)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
        at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
        at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
        at java.io.InputStreamReader.read(InputStreamReader.java:167)
        at java.io.BufferedReader.fill(BufferedReader.java:136)
        at java.io.BufferedReader.read(BufferedReader.java:157)
        at
org.apache.james.util.CRLFTerminatedReader.readLine(CRLFTerminatedReader.java:98)
        at ... .JamesPOP3Handler.readCommandLine(JamesPOP3Handler.java:523)
        at ... .JamesPOP3Handler.handleConnection(JamesPOP3Handler.java:313)
        at
org.apache.james.util.connection.ServerConnection$ClientConnectionRunner.run(ServerConnection.java:417)
        at
org.apache.james.util.thread.ExecutableRunnable.execute(ExecutableRunnable.java:55)
        at org.apache.james.util.thread.WorkerThread.run(WorkerThread.java:90)
Caused by: javax.net.ssl.SSLHandshakeException: Invalid padding
        at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:150)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1476)
        at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:774)
        at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1025)
        at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:619)
        at
com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
        at sun.nio.cs.StreamEncoder$CharsetSE.implFlush(StreamEncoder.java:410)
        at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:152)
        at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:213)
        at java.io.BufferedWriter.flush(BufferedWriter.java:236)
        at java.io.PrintWriter.flush(PrintWriter.java:270)
        at
org.apache.james.util.InternetPrintWriter.println(InternetPrintWriter.java:90)
        at
org.apache.james.util.InternetPrintWriter.println(InternetPrintWriter.java:187)
        at
com.privasphere.privalope.mail.JamesPOP3Handler.handleConnection(JamesPOP3Handler.java:306)
        ... 3 more
Caused by: javax.crypto.BadPaddingException: Padding length invalid: 212
        at com.sun.net.ssl.internal.ssl.CipherBox.removePadding(CipherBox.java:442)
        at com.sun.net.ssl.internal.ssl.CipherBox.decrypt(CipherBox.java:290)
        at com.sun.net.ssl.internal.ssl.InputRecord.decrypt(InputRecord.java:144)
        at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:762)
        ... 16 more

Comment 13

[Ralf Hauser]

Attachment: pop3sJdk1.5.ssltap.htm

Comment 14

[Ralf Hauser]

Attachment: pop3sJdk1.4.ssltap.htm

and for comparison the ssltap output when starting james with jdk1.4

Comment 15

[Nelson Bolyard (seldom reads bugmail)]

So TLS/DHE-RSA/AES256-CBC/SHA seems to work OK with JDK 1.4 but not with JDK 1.5 ?

You've demonstrated and documented that mozilla works fine with some server 
component versions, but when you change some server components to other 
versions, it stops working.  That's not indicative of a bug in the client.  

Sounds to me like you need to file a bug against some component of your server 
just as James or JDK, not mozilla.

Comment 16

[Ralf Hauser]

Perhaps it is a mozilla problem after all:

<<From: David Hook [[e-mail]]
Sent: Monday, July 18, 2005 8:58 AM
To: [e-mail]
Subject: TLS/DHE-RSA/AES256-CBC/SHA not working under jdk1.5


The exception's being caused because whatever it is that's generating the
encrypted data is violating section 6.2.3.2 of the TLS protocol (RFC 2246).

The problem is that rather than padding a block with say 6 bytes to make it a
multiple of the block length of the cipher by adding

0x06 0x06 0x06 0x06 0x06 0x06

they're probably adding

some random 5 bytes 0x06

The SSL API has obviously being "fixed" in JDK 1.5 to recognize this as an
error, so it's complaining when it finds that the last byte of padding hasn't
been repeated. My guess is the JDK 1.4 version of the SSL API just grabbed the
last byte of the padding and chopped of that many bytes. Whoever is producing
the offending cipher text is most probably responsible for the error - they've
padded it in a lazy fashion.

Regards,

David>>

Comment 17

[Nelson Bolyard (seldom reads bugmail)]

SSL3 and TLS both define the plaintext used with block ciphers to end in 
an array of "padding", followed by a single padding_length byte.  They 
define the term "padding" such that it does not include the padding_length 
byte itself.  With block ciphers, there is always a padding byte, but the
amount of padding (preceeding that byte) may be zero.

In both SSL3 and TLS, the padding_length byte contains a number which is the
number of bytes of padding that preceed it (that preceed the padding_length
byte).  If there are no bytes of padding preceeding it, the padding_length
byte contains 00.

SSL 3.0 allows the array of padding bytes (which preceed the padding_length
byte) to contain any data.  TLS requires every byte in the padding array to
contain the same value as the padding length byte.  

So, for example, in the case where there are 5 bytes of padding, followed 
by 1 byte of padding length, the last 6 bytes of the TLS plaintext block 
all contain 05, e.g.      05 05 05 05 05 05 
Notice that's 6 bytes of 05, not 6 bytes of 06.  

Both SSL3 and TLS require the total length of the plaintext (including the
contents, the MAC, the padding and the padding_length) must be a multiple 
of the block size in bytes. 

SSL3 requires that the number of bytes of padding, plus the number of bytes 
of padding_length (which is always 1), must add up to no more than the block 
cipher's block size, e.g. 8 for DES, 16 for AES.  Another way to say this is
that the padding array must be less bytes than the block size.  

TLS allows the total amount of padding (not including the padding_length 
byte) to be up to 255 bytes regardless of the block cipher's block length, 
provided it meets that total length constraint mentioned above.

NSS uses one common set of code to create padding for SSL3 and TLS.  It 
produces padding such that the length of the padding (exclusing the
padding_length byte) is always less than the block size.  The result is 
legal for both SSL3 and TLS.  It fills all bytes of padding with the same 
value as the padding_length byte, which is required for TLS and permitted 
by SSL3.  

When block enciphered data is received with invalid padding, that usually 
indicates that the key used to decrypt the data was not the same key used 
to encrypt the data, and all the decrypted data in the decrypted ciphertext 
record is actually garbage.

One way this could happen would be if your server is using different 
private keys, one with JDK 1.4 and another with JDK 1.5, but the same
cert in both cases.  Alternatively, it could be using different certs
(with different public keys) but the same private key, or any other cause
of a mismatch between the server's private key and the public key in the
server's cert.  The counter-measure to the Bleichenbacher attack 
necessitates that all master secret decryption failures (failures of the 
decryptions done using the RSA private key) result in the use of wrong
keys for actual bulk-data encryption and decryption (and MACing too).

With versions of NSS before 3.10, this problem (of a public key being
used that did not match the server's private key) was known to be caused 
by a server using several different certs over time that had the same 
issuer name and serial number, but different public keys.  These were 
usually self-issued certs.  That problem is now detected by current 
versions of NSS (e.g. NSS 3.10).  But I think you may be using a slightly 
older version of NSS.  

If your server uses two different certs, with different public keys but 
the same issuer name and serial numbers, one cert with JDK 1.4 and another
with JDK 1.5, then that's your problem.  The solution is to use certs 
with different, unique serial numbers, and never reuse serial numbers, ever.

Comment 18

[Nelson Bolyard (seldom reads bugmail)]

Based on the comments above, I think it is now apparent that the problem was 
due to a server issue, block cipher padding being implemented incorrectly.

I see no evidence of any mozilla bug here.  So, I am marking this bug 
invalid.  Please do not reopen this bug without evidence that mozilla has 
failed to properly implement the SSL3/TLS specs with regard to padding or
MACing.

Comment 19

[Andreas Sterbenz]

This is almost certainly a bug in the BouncyCastle Diffie-Hellman
implementation. It just returns y.modPow(x, p).toByteArray() as the secret,
which means the byte[] is in 2's complement format. This is wrong as it will
prefix a 0x00 byte if the MSB is set. (This is from looking at the BouncyCastle
1.23 source, which is what I happen to have on my disk).

Remove the BouncyCastle provider and use the default SunJCE provider and
everything should work fine. Or fix BouncyCastle.

Comment 20

[Nelson Bolyard (seldom reads bugmail)]

(Note that, at this point, we're discussing an issue in another product or 
set of products, not one in mozilla.  Is there a better forum for these 
non-mozilla issues?  mozilla's bug tracking system seems the wrong place.)

The NSS and OpenSSL implementations of the _DHE_ cipher suites, and all 
other implementations that interoperate with them (which means nearly all
implementations) put no leading zero bytes in the result of the DHE 
computation (which is the premaster secret in _DHE_ cipher suites).  
In other words, the _DHE_ output has all leading zero bytes removed.
So the value used as the premaster secret in DHE cipher suites 
effectively has all leading zero bytes removed.

That behavior is not explicitly called out in the SSL3 or TLS 1.0 specs, 
but is called out in the TLS 1.1 draft, and (as I wrote above) all currently 
interoperating SSL3/TLS implementations do this (remove leading zero
bytes).  The TLS 1.1 draft is not attempting to change the protocol in 
this regard, but merely to be a more complete specification that agrees with
all currently interoperating implementations.  

The PKCS#3 standard for Diffie-Hellman computations requires the output 
to be the same length (same number of bytes) as the prime P, which often 
means that leading zeros are required in order to meet that standard. 
Based on what Andreas wrote above, (Thanks, Andreas) I'd guess that 
Bouncy Castle's DHE implementation is trying to produce PKCS#3 compliant
output.  

So, for the SSL3/TLS _DHE_ cipher suites to interoperate, either
a) the DHE implementation must remove the leading zero bytes before 
outputting the result (which doesn't conform to PKCS#3), 
OR
b) the code that inputs the DHE result and uses it as a premaster secret,
from which to derive the master secret, must remove those leading zero 
bytes.  
OR 
c) both of the above.

Given that many Diffie-Hellman implementations attempt to be PKCS#3 
compliant, it is probably best for SSL/TLS implementations to implement 
choice (b) above.  Note that this removal of leading zero bytes is done 
only for DHE cipher suites, and not for ECDHE ones!