385471 - Problem with HP iLO2: No HTTPS connection possible due to duplicate certificate serial number from same CA

Status: RESOLVED DUPLICATE of bug 435013

(Firefox :: Security, defect)

Description

[Ulrich Windl]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.4) Gecko/20061107 SUSE/1.1.2-1.1 SeaMonkey/1.1.2
Build Identifier: 

When trying to connect to HP'S iLOM2 (Integrated Lights Out Management 2) via HTTPS, Firefox 2.0.0.4 (on Windows/XP) brings up a windows saying that there was a certificate with a duplicate serial number from the same CA detected, and that I should get a different certificate.

This is bad because:
1) The iLOM2 firmware creates the certificates locally, allowing only limited user influence.
2) The user cannot decide to "connect anyway" (like for other dubious certificates)
3) There's no way to display the details of the certificates that Firefox claims to have duplicate serial numbers (the average user might not care, but the IT professional will)

iLOM firmware in question is "E.03.30" (latest AFAIK) on a HP Integrity (Itanium2) rx4640 server. Certificate can be viewed from command line's "SO" (=Security Options) and "L" (display SSL certificate). Unfortunately no serial number is displayed there. Probably it's zero all the time. Neither can I see the issuing CA, but I guess the certificate is self-signed.

Reproducible: Always

Steps to Reproduce:
1. connect to HP's iLOM2 that has created at least two certificates during its lifetime
2. A pop-up appears saying that the certificate is not acceptable
3. The user only has [OK] as choice to continue; no connection is established
Actual Results:  
No encrypted connection is possible

Expected Results:  
An encrypted connection should be possible, at least temporarily accepting the dubious certificate

Display (or allow to display) details of the certificate being in question, in comparison to the conflicting certificate.
Offer a user's choice of what to do: Cancel connection, Accept temporarily, Accept permanently, maybe replacing the conflicting certificate

Comment 1

[Jerry Nilson]

I think this is major concern -- Firefox used to be the only fully reliable web browser for configuring SonicWall appliances -- now it is hardly usable thanks to this "bug-feature" you have introduced!

This has apparently generated quite a lot of headache out on all kinds of forums too ... . I have not found a work-around yet myself -- deleting certificates does not help. /Jerry

Comment 2

[Ted Owens]

I can confirm this bug.  I have the same problem with Astaro Security Gateway products.  I have a handful of these devices that I cannot connect to using Firefox.

My current work-around is to temporarily accept and then restart Firefox to get to the next device.

I can confirm that the serials are in fact different for the certificates,  I even changed the Certificate Authority for each device and it didn't make a difference.

Comment 3

[aWs]

Using Firefox 2.0.0.9, Win2k. My mailserver has a self-signed certificate that expires every year. I make a new one. This is my first year with FireFox. New certificate fails this year, FF says "invalid cert, duplicate serial number". However the old certificate expired yesterday... should allow new certificate to be used. Certificate Manager shows old one as expired.

Odd, Mozilla 1.7.13 says the same thing. Was last year a leap year or something ? Certificate Manager shows old one as expired. I was using Moz last year when certificate was accepted. FF inherited cert from Moz. If I delete old cert, of course it works.

I hate to admit it, but MSIE knows that the old cert has expired, and asks if you want to accept the new one. Firefox doesn't recognize that the old cert has expired - even tho it shows the expiry date of yesterday... I really don't want to post on my web site - that users should just use MSIE .. .. ..

Comment 4

[Sean Greenhow]

Since being upgraded to Firefox 2.0.0.12 (both Linux and Windros) I have the same problem when trying to access my Lynksys WRT54G router. The only way I can log is to use IE- that sucks! Please always allow the user a choice of accepting a suspect certificate. In the mean time back to bad old IE ....   

Comment 5

[Martin Meyer]

This bug seems to be much worse in FF3 - there is no way whatsoever to purge the old certificate. Even after it has been removed from the stored server certificate list it seems to linger on. The only workaround I have found it to remove cert8.db.

I work with HP's iLO and OA products all day and I frequently upgrade the firmware, which regenerates their certificate. After I upgrade the firmware I can't access that iLO or OA from that system. A complete purge of my "private data" did nothing, and my ultimate solution was to remove cert8.db.

Is anyone looking into this? Is there any chance of it being addressed before the release of Firefox 3?

Comment 6

[Dustin Hartman]

I am a Firewall admin, and with Cisco, CheckPoint, Nokia, Netscren, etc... firewalls the secondary in an High Availability pair always has the same Serial Number, which is necessary by design.

With FF3 this new security check makes it impossible to connect to both the primary and secondary firewall because they have the same serial number on the cert.  I understand the need for general users, which is the vast majority of the firefox community, to be protected from duplicate serial numbers, but surely there should be some advanced option that can be manually changed for IT users?  If we accept the risk, we should be able to disable that security check.

Comment 7

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

1. Let's keep this bug focused on the HP iLO/iLOM products. Bug 435013 can be used for tracking the more general problem.

2. A workaround: the the HP iLO documentation clearly states that it is possible to replace the certificate used on the device with your own, which could be your own self-signed certificate or a certificate issued by a certificate authority. Installing your own certificates should solve this problem for you, and it is highly recommended to do so. See:

   http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00212796/c00212796.pdf
   http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00209014/c00209014.pdf
   http://www.vcritical.com/2010/11/automating-ssl-certificate-deployments-for-hp-ilo/
   http://www.rtfm-ed.co.uk/2007/06/11/using-self-signed-ssl-certificates-with-hp-ilo/

3. Another, worse, workaround: When you add the certificate exceptions, make sure you uncheck the checkbox in the override dialog box that makes the exception permanent. This will significantly reduce the chances of getting the duplicate serial number error, and usually you can resolve it by restarting the browser, when the override isn't permanent.

4. Regardless of what mitigations that we may (or may not) implement in Gecko, this is clearly a bug in HP's iLO product. It should NOT be generating certificates with duplicate issuer/serial-number combinations. We should file a support request with HP about the issue. If any users have filed a support request with HP about the issue already, please add your HP issue number to the bug or email me ([e-mail]) with it. (Kev, is resolving this with help on HP's end something in your department?)

5. There are many possible things we could do in Gecko which may or may not work around the issue. But, before we could even come up with an automatic workaround, it would be very helpful to have the full certificate chain from one of the devices. Having the cert chain will be very useful in determining exactly what the devices are doing wrong. To get the full certificate chain, click on the site identity block (or the lock icon, in Nightly/Aurora) > More Information > View Certificate > Details tab > Export. For the file type, choose "X.509 Certificate with Chain". Since this certificate may contain information about the hostname of the server. Then attach it to the bug; when you add the attachment, you will have the option of marking it as security-sensitive before uploading it so other users will not be able to access it.

Comment 8

[Andreas van dem Helge]

Is there any step to reproduce these errors? I have various iLO and iLO2 machines and at this time I am unable to reproduce.

Comment 9

[Kaspar]

iLO and ILOM are different beasts, actually. The former is from HP, the latter from Sun^WOracle.

(In reply to Andreas van dem Helge from comment #8)
> Is there any step to reproduce these errors? I have various iLO and iLO2
> machines and at this time I am unable to reproduce.

Judging from http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c02944964, it seems that this has been fixed for iLO 3, at least.

Comment 10

[Andreas van dem Helge]

Yes but specifically what I have access to play with is iLO (1) on DL360 G3 and iLO2 on DL360 G5. I have multiple of each machines. I am looking on the iLO2 right now and one is reporting the certificate serial number of 69:4C:4F:00, while the other has a different serial with a much longer length.

Comment 11

[Kaspar]

(In reply to Andreas van dem Helge from comment #10)
> Yes but specifically what I have access to play with is iLO (1) on DL360 G3
> and iLO2 on DL360 G5.

Then I assume it has been fixed in these versions in the meantime, too. For iLO2, see e.g.

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?swItem=MTX-4ecf8f7064f74603a86eec2077&mode=5

v2.01 and v2.06 look like a possible candidates (they include certificate related fixes), though they don't mention this issue explicitly.

What's the validity of your iLO certs, BTW? For people still running into the issue even after the firmware has been upgraded, perhaps it's necessary to explicitly trigger a regeneration of the certificate?

Comment 12

[Andreas van dem Helge]

I've managed to inadvertently reproduce this under Nightly 16.

I visit both ILO (1) sites no issues. Now the idea was to reset the config to defaults, but I need to ensure before I do that I have under control how to reset the passwords, since I assume reset to defaults resets the password to the one printed on the server's sticker and as much as I want to help I can't suffer downtime or be locked out of the ILO despite these servers seem reliable and it's not really needed.

Anyways I read the ILO config as follows (this is on a RHEL 5 system with the HP management software Proliant Support Pack (PSP) installed) so in the system with 192.168.3.165 ILO I follow:

# hponcfg -w /root/ilo.config
Please note it appears the current password is not retrivied

Then I write back the same configuration:
# hponcfg -f /root/ilo.cfg


After this process firefox is unable to access the ILO (1) https site. Via internet explorer I am able to view the certificate serial number as: ‎69 4c 4f 00

The certificate details are:

Issued by: [ILO Hostname]
Issued to: [ILO Hostname]
Validity: 12/05/2002 to 12/05/2022

In internet explorer I can also view the other ILO (1) at 192.168.3.170 indicates the same certificate serial number. Likewise the other certificate details are identical. In Firefox I can access the 192.168.3.170 ILO but I can no longer access the 192.168.3.165 ILO. The one I can access is indicates a serial number of 69:4C:4F:00 in Firefox.

Common between the system is the Firmware Version:   1.94   03/19/2009

This has me thinking this is bad behaviour as if the first seen duplicate certificate is honored. Worst case, if Firefox chooses to keep this horrible behaviour, it should block any certificate with a duplicate serial number and not allow access to any site including the first seen one.

Comment 13

[Andreas van dem Helge]

I believe I have devised a scenario to reproduce this Firefox bug with the use of a single iLO (1) host.

We use a combination of Internet Explorer and Firefox for this, as the entire process (verifying the 2nd certificate serial number) is not possible in Firefox and we want to keep this consistent so we always get the serial numbers with the browser that does not experience this bug

1) Visit the site in Internet Explorer to obtain the certificate serial number and SHA1. In this case it is:
SHA1 ea 93 ff 07 e2 58 0e 82 44 bf f5 f7 df 18 00 1f 15 32 4e 2c
Serial 69 4c 4f 00
2) Visit the site in Firefox to verify that the site is viewable
3) We use RedHat Enterprise Linux 5 with an installed HP Proliant Support Pack (PSP.) This software is available for other operating systems and the process should be similar, if not the same. hponcfg could be installed separately but in this case we use PSP for simplicity's sake as this is already way off scope. Using the hponcfg tool the iLO is reset to defaults:

# hponcfg -r
Firmware Revision = 1.94 Device type = iLO Driver name = hpilo
Resetting to Factory Defaults...This takes 15-30 seconds.
The RILOE II/iLO has been rebooted.

4) Wait for the iLO server to become pingable again and then visit the site in Internet Explorer noting:
a) access to the site is still possible, it hos not been blocked by Internet Explorer
b) the certificate serial number remains as: 69 4c 4f 00
c) the certificate SHA1 has changed to: a6 cd 10 35 c8 a4 b2 a6 27 f8 22 5a 93 02 35 8d 0d 45 ee 59

Since this process resets the iLO server to defaults the iLO hostname and this the certificate issuer name is changed to the default. Therefore, if the system you are testing with starts off with a non-default hostname you will need to reset iLO to defaults TWICE, as well as accessing the iLO HTTPS page TWICE to experience this Firefox bug. This further explains why some people may not be experiencing this issue or may perceive this issue to be non-existent. The default hostname of the system is ILO-[server serial number]

Comment 14

[Andreas van dem Helge]

Sorry missed 5) which is obviously:

Attempt to access iLO server from Firefox and note that is has been blocked with a "sec_error_reused_issuer_and_serial" error.

Comment 15

[Kaspar]

(In reply to Andreas van dem Helge from comment #12)
> Common between the system is the Firmware Version:   1.94   03/19/2009

This is not iLO 2, for the records - it's the latest available release for iLO version 1, where the issue hasn't been addressed AFAICT.

> Since this process resets the iLO server to defaults

It's actually sufficient to change one of the (port) settings under Administration -> Global Settings or Administration -> Network Settings, after which iLO will need to restart.

When iLO version 1 is restarted and a self-signed certificate is used, this is what happens to the SSL cert: iLO generates a new key pair, but will *only* replace the subjectPublicKeyInfo field in the cert (and update the signature, of course). I.e., it leaves serial number, issuer and subject DN, and even the validity of the cert completely unchanged (yuck!).

NSS will subsequently reject the new cert in CERT_NewTempCertificate(), and unless the browser is restarted, this will happen, too, if there is no permanent exception for the old cert. I.e., even if the old cert is not in the on-disk DB, this will unavoidably lead to the sec_error_reused_issuer_and_serial error, and there's no way other than a browser shutdown to get out of this jam.

Comment 16

[Andreas van dem Helge]

What can I do to help get a fix on the Firefox end for this bug?