436870 - FF3 RC1 loses SSL certificate exceptions

Status: VERIFIED FIXED

(Core :: Security: PSM, defect)

Description

[todesschaf]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9) Gecko/2008051202 Firefox/3.0
Build Identifier: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9) Gecko/2008051202 Firefox/3.0

When I first installed FF3 RC1, it lost all the exceptions I had made for SSL certificates that have, for example, mismatched names in the cert vs. the domain name (internal hosts where I work often have certificates with this condition, because of CNAMEs or other strange configurations). Thinking it was some error upgrading from B5 to RC1, I ignored it and grudgingly re-added all my SSL cert exceptions. Things went along fine until today, when I got to work in the morning, and found that my firefox instance (which has been running at least all weekend) had lost ALL the exceptions yet again, even for hosts that I had in an open tab all weekend! Additionally, I now have two copies of exceptions listed in my certificate manager (for the hosts I have tried since firefox lost all my exceptions)

Reproducible: Couldn't Reproduce

Steps to Reproduce:
1. Add a permanent exception for an SSL certificate where the host in the certificate doesn't match the host you're actually connecting to
2. Wait some indeterminate period of time (could be weeks)
3. Go to the same host from step 1 and notice that your supposedly permanent exception no longer exists.
Actual Results:  
SSL certificate exceptions are randomly lost, but when one is lost, all are lost

Expected Results:  
Permanent SSL certificate exceptions should, in fact, be permanent

Comment 1

[Elmar Ludwig]

I think I am seeing this as well, so I'm confirming this now. Also moving to Core/PSM since I believe that is the right component.

I upgraded my trunk build to 2008051604 about two weeks ago and see most of the same kinds of problems here:

- all previously stored certificate exceptions were lost
- adding a new exception works, but the exception is lost again after about a week (the exact time seems to vary a bit, not sure how to reproduce this)
- but I do not have duplicate exceptions listed in the certificate manager (

I just didn't report this earlier because I do not have clear steps to reproduce.

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9pre) Gecko/2008051604 Minefield/3.0pre ID:2008051604

Comment 2

[Kai Engert [:KaiE:]]

Sounds strange.

I guess you are sure that at time of exception creation the "permanent" checkbox in the dialog's lower area was checked?

You could try to look at file cert_override.txt in your Firefox profile directory. Is it really using entries? Or are the entries are still there, and Firefox is no longer loading them correctly?

I think in theory the following could happen:
- you add an exception for a site that somehow we fail to save correctly into the file
- the next time you start firefox we attempt to load the file. Maybe we are unable to load it, because saving created an incorrect file
- therefore you get an empty list and your exceptions are gone

But right now I can not think of a site that would cause us to produce a corrupt file.

Comment 3

[Johnathan Nightingale [:johnath]]

If you don't mind doing so, it would be very helpful if you could attach, or paste the contents, of your cert_override.txt file in this bug.  It does sound like you may be getting some corruption there, but as Kai says, it's not clear why that should be.

Comment 4

[todesschaf]

Attachment: cert_override.txt from one affected machine

This is the cert_override.txt from one of the machines this has happened on (which is currently in a normal working state, so who knows if the file will actually be helpful). After the last time the loss of cert overrides happened, the only entries in the file were the ones I had added since the loss of overrides. In reference to comment #2, the idea that it gets cleared on FF startup because the file is corrupt seems suspicious to me, as like I said, this happened most recently to me on a running firefox (I left work on Friday and it was all fine, then returned to work on Monday, had a tab open to a machine that should have had a cert override, refreshed the content of that tab, and was presented with the 'invalid cert' FF page instead of the refreshed content). Of course, maybe the file went funky while FF was running and it wasn't until I restarted FF (which I did in an attempt to get my cert overrides back) that the file was actually cleared out.

Comment 5

[Honza Bambas (:mayhemer)]

Isn't there a chance that the certificate(s) where changed/regenerated by web administrators of those pages during weekend? Are those pages on a very different servers or are those some intranet pages of your company? As seen from the file I would say so.

Comment 6

[Kai Engert [:KaiE:]]

Nick, every time the server changes to use a different certificate, the stored exception no longer matches the server, and you'll get warned again (and need to add an exception again).

Thanks to Honza for reminding us about this.

Comment 7

[Kai Engert [:KaiE:]]

I remember we have a bug filed that requests to improve the user experience when a server changes its cert. The error message should mention that you do have an exception, which doesn't match the new bad cert.

Comment 8

[todesschaf]

I know for a fact that the tab I reloaded did not have a changed certificate over the weekend, so that's not the issue with that machine, and I'm 99% certain that none of the other certificates changed. All the same, like I said, cert_override.txt had definitely been cleared of its contents, as the only overrides in that file after the weekend were the ones I had added that day (there were other servers at work that I know I had overrides for that I hadn't hit that day).

Comment 9

[Elmar Ludwig]

Attachment: my cert_override.txt file

This is the cert_override.txt from my affected profile. It currently contains only four entries, although I see several more in the Servers tab of the Certificate Manager. All of the others have "*" listed in the Server column, so I guess those were added before the new way of handling certificate exceptions was added and are probably not affected.

(In reply to comment #2)
> I guess you are sure that at time of exception creation the "permanent"
> checkbox in the dialog's lower area was checked?

Yes. I rarely uncheck that box and the exceptions tend to survive several reboots for me until they start to disappear again (some days to one week).

> Isn't there a chance that the certificate(s) where changed/regenerated by web
> administrators of those pages during weekend?

For some of the sites affected in my case, I am the web administrator and definitely did not replace the certificates.

Comment 10

[Elmar Ludwig]

(In reply to comment #6)
> Nick, every time the server changes to use a different certificate, the stored
> exception no longer matches the server, and you'll get warned again (and need
> to add an exception again).

Just as another data point:

When I started to see this, I tested it with the SSL certificate of www.amazon.de by manually marking the entries for Verisign as not trusted in the Cert Manager for my profile and creating a certificate exception for www.amazon.de. This entry disappeared two or three times over the last two weeks. I don't think amazon.de changes its certificate on a weekly basis.

Comment 11

[Johnathan Nightingale [:johnath]]

It seems like Iang might be seeing a similar problem, here.

  https://financialcryptography.com/mt/archives/001054.html

(see 5th paragraph)

Comment 12

[Honza Bambas (:mayhemer)]

To summarize:
- cert exceptions was lost during uninterrupted firefox run
- server certificates was not changed

A possible way how it could be lost under these circumstances is here:
http://lxr.mozilla.org/mozilla/source/security/manager/ssl/src/nsCertOverrideService.cpp#186

But only if "profile-before-change" (possibly with "shutdown-cleanse") is called during runtime. I could not find such call in firefox its self. Isn't there any extension that could do this?

Comment 13

[Ian Grigg]

In my case (as reported by Jonath #11), it happened while I was working on that blog post, doing some changes to it (which was why I wrote it in there!).  So it was quite dynamic, one moment clicks to do things like save were ok, and the next moment, the exception was dead.  I do not recall whether other exceptions were gone as well.

I've done a few restarted, etc since, and it has not returned.  I took a copy of the cert_override.txt just now and we'll see if it happens again.

It is a new clean RC1 install, no previous download, on a new account on a Mac/Leopard.
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-GB; rv:1.9) Gecko/2008051202 Firefox/3.0

Comment 14

[todesschaf]

In reference to comment #12, the only mention of either profile-before-change *or* shutdown-cleanse I could find in any of my extensions was two references to profile-before-change in the adblockplus extension (version 0.7.5.4), however, those two references seem to be comparisons, not calls (perhaps someone more acquainted with ff extensions can confirm if I'm right on the first instance):

in prefs.js, line 308 (prefs.addObservers):
observerService.addObserver(this, "profile-before-change", false);

in prefs.js, line 1305 (prefs.observe):
else if (this.initialized && topic == "profile-before-change")

this file was extracted from the adblockplus.jar

Comment 15

[Honza Bambas (:mayhemer)]

Nick, thanks you took a look. This is only listening to the event, not call. So, it's not the cause. However, thanks for your time.

Comment 16

[Elmar Ludwig]

Ok, I'm definitely seeing corruption of the cert_override.txt here.

I changed my firefox startup script to make a backup of the cert_override.txt of the current profile on each browser start. It looks like somehow the host names get wiped out from most of the entries when new exceptions are added. Alas, I still don't have a clear way to reproduce the problem. I'm going to attach the relevant copies of the file.

Comment 17

[Elmar Ludwig]

Attachment: cert_override.txt.2008-06-05 19:03:59 (ok)

This is the last intact copy of my cert_override.txt.

Comment 18

[Elmar Ludwig]

Attachment: cert_override.txt.2008-06-05 19:30:16 (corrupted)

This is the corrupted file that is missing host names on all entries except one.

Comment 19

[Elmar Ludwig]

Attachment: cert_override.txt.2008-06-06 08:27:38 (current)

This is the current version of my cert_override.txt. When (re)adding the exception for bugzilla.mozilla.org, all entries without host names were purged from the file.

Note that all versions of the cert_override.txt are from a profile that has many built-in CAs marked as untrusted in the hope of reproducing the problem more easily (just to explain why there is an exception for bugzilla.mozilla.org).

Comment 20

[Honza Bambas (:mayhemer)]

Comment on attachment 324005 [details]
cert_override.txt.2008-06-05 19:30:16 (corrupted)

>	OID.2.16.840.1.101.3.4.2.1	staffa.virtuos.uni-osnabrueck.de:443	OID.2.16.840.1.101.3.4.2.1	39:5A:7D:81:73:96:16:62:08:60:E8:4D:10:30:AC:AF:7F:2D:08:81:F6:34:B5:0D:5A:F7:23:1C:90:BF:1C:55	U	AAAAAAAAAAAAAAAJAAAAlgDRi5q+HIKgsDCBkzELMAkGA1UEBhMCREUxIDAeBgNV  BAoTF1VuaXZlcnNpdGFldCBPc25hYnJ1ZWNrMRgwFgYDVQQLEw9aZW50cnVtIHZp  cnRVT1MxJDAiBgNVBAMUGyoudmlydHVvcy51bmktb3NuYWJydWVjay5kZTEiMCAG  CSqGSIb3DQEJARYTZWxtYXIubHVkd2lnQHVvcy5kZQ==

Just for curiosity: this is entry that only one have a host+port in the corrupted file. But, the same entry (same fingerprint) without a host+port is, besides, twice contained in the file. The remaining entries are all entries from the original, uncorrupted, file but w/o a host+port. Look like a hash table has been corrupted after a new entry was added. I cannot reproduce it on win nor mac with FF 3.1pre.

Comment 21

[Elmar Ludwig]

(In reply to comment #20)
> But, the same entry (same fingerprint) without a host+port is, besides,
> twice contained in the file. The remaining entries are all entries from the
> original, uncorrupted, file but w/o a host+port. Look like a hash table has
> been corrupted after a new entry was added.

I'm not sure about that. This particular cert is a wildcard certificate (its CN is "*.virtuos.uni-osnabrueck.de") and the same cert is used on multiple servers in this domain. If I recall correctly I also added an exception for one or two of the other servers that have the same certificate.

Comment 22

[Honza Bambas (:mayhemer)]

Elmar, do you remembered which servers you added? I will try it again with exact steps as you did to reproduce. Thanks.

Comment 23

[Elmar Ludwig]

I think staffa.virtuos.uni-osnabrueck.de, lewis.virtuos.uni-osnabrueck.de and mainland.virtuos.uni-osnabrueck.de. I don't remember exactly, but I'll look into my browser history when I'm back home today.

Most of the actual contents on these servers is password protected, but I guess this does not matter for adding certificate exceptions.

Comment 24

[Honza Bambas (:mayhemer)]

Elmar, thank you. I can reproduce the problem on Mac. 

Steps to reproduce this bug:
1. Use attachment 324004 [details] as your cert_override.txt file in your profile, no matter the certs are not stored in the cert database
2. Add exception for https://staffa.virtuos.uni-osnabrueck.de
3. Restart FF (not sure this step is really necessary)
4. Add exception for https://lewis.virtuos.uni-osnabrueck.de
5. Add exception for https://mainland.virtuos.uni-osnabrueck.de => the file is corrupted now

Comment 25

[Honza Bambas (:mayhemer)]

Attachment: Fix

I broke this. I forget to copy the hostname:port string in entry copy constructor. When the hash table is about to grow it deep-copies all entries to a newly allocated table. Hostname:port was not carried to a new entry and therefor it were lost (missing in the table to lookup during cert override check and also in the file).

Comment 26

[Kai Engert [:KaiE:]]

Comment on attachment 324050 [details] [diff] [review]
Fix

r=kaie

Comment 27

[Andy Hey]

This still appears broken in RC3?

Comment 28

[Kai Engert [:KaiE:]]

Andy, which RC3 are you referring to? (I don't think there is one at this time).

While this bug has a patch attached, it's not yet checked in.
RC2 does not have this patch, unfortunately.

Comment 29

[Jo Hermans]

(In reply to comment #28)
> Andy, which RC3 are you referring to? (I don't think there is one at this
> time).

http://blog.mozilla.com/blog/2008/06/11/third-firefox-3-release-candidate-available-for-download/

Comment 30

[Kai Engert [:KaiE:]]

Thanks for the clarification, I had missed that there was an RC3 (for Mac only).

But what I said remains true, it does not contain the fix for this bug.

We'll have to address that in the first bugfix release (hopefully 3.0.1)

Comment 31

[Honza Bambas (:mayhemer)]

May I ask why so late? This is one line fix that fixes wrong patch that has already been in RC1. It IMHO should be landed ASAP.

Comment 32

[Kai Engert [:KaiE:]]

Because it was too late, the decision to ship it had already been made (so I have been told on IRC by beltzner).

Unfortunately we had failed to request blocking1.9 on this bug.

Comment 33

[todesschaf]

Is there an estimate yet on when 3.0.1 (or whatever release will have this fix) is going to be released? It's getting rather annoying to lose my cert exceptions now on pretty much a weekly basis, and I'm trying to decide if it would be easier for me to take the patch above and make my own build, or just wait until this fix is available in an official release.

Comment 34

[Elmar Ludwig]

You can grab one of the 3.0.1pre nightly builds once the fix has landed on that branch. Update releases for Firefox 3.0 are planned every 6 to 8 weeks, IIRC.

Comment 35

[Kai Engert [:KaiE:]]

Comment on attachment 324050 [details] [diff] [review]
Fix

clearing approval1.9 request as it's released.

requesting 1.9.0.1 approval for firefox 3.0.1

Comment 36

[Dão Gottwald [:dao]]

This is ready to land on trunk, right?

Comment 37

[Kai Engert [:KaiE:]]

(In reply to comment #36)
> This is ready to land on trunk, right?

Good point!
Checked in, 921a583d2c0f
Marking fixed.

Comment 39

[Mike Beltzner [:beltzner, not reading bugmail]]

Comment on attachment 324050 [details] [diff] [review]
Fix

a=beltzner

Comment 40

[Dave Camp (:dcamp)]

Checking in nsCertOverrideService.h;
/cvsroot/mozilla/security/manager/ssl/src/nsCertOverrideService.h,v  <--  nsCertOverrideService.h
new revision: 1.5; previous revision: 1.4
done

Comment 41

[Carsten Book [:Tomcat]]

verified fixed using Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1 and the steps to reproduce from comment #24

-> Verified 1.9.01