Closed Bug 870329 Opened 12 years ago Closed 3 years ago

hotfixes should be able to be signed with a timestamped signature, so that they're valid forever

Categories

(Firefox :: General, defect)

Product:
Firefox
Component:
General
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED INVALID
Tracking Flags:
Tracking Status
firefox24 - ---

People

(Reporter: bhearsum, Unassigned)

Details

Twice now we've hit issues with distributing hotfixes to older versions of Firefox due to those versions not recognizing newer certificates: bug 865873 and bug 803596 (or one of the bugs related to that - I can't find the one I wanted to...) The way this is generally dealt with in the signing world is to use a timestamped signature. This is what we do for our Windows builds, for example. Some info on timestamping: https://www.eldos.com/sbb/articles/5731.php http://msdn.microsoft.com/en-us/library/windows/desktop/bb931395%28v=vs.85%29.aspx.
I thought we had established that we already do this?
I dug up mail on this and found that we only use the local time of the signing machine: Signtool seems to just use PR_Now() as the signing time, so basically the local time on the computer. That seems pretty worrying as it effectively means our certificates never expire. On 2 November 2012 12:07, Ben Hearsum <bhearsum@mozilla.com> wrote: > Unless signtool is talking to a hardcoded timestamp server, I don't > think it counts....time on the local machine doesn't count! That's why > timestamp servers exist, aiui! > > On 11/02/12 03:05 PM, Dave Townsend wrote: > > Ok, it looks like signtool already embeds the signing time into the XPI, > > since I set my clock forwards 3 months and our existing hotfix still > shows > > as signed. So I guess we already have this solved? What I wonder is where > > is signtool getting its time from and isn't that easy to spoof? > > > > > > On 2 November 2012 11:57, Brian Smith <bsmith@mozilla.com> wrote: > > > >> Dave Townsend wrote: > >>> Brian, we're looking to see whether there is a way to sign hotfix > >>> XPIs before the expiry of the certificate we use and then Firefox > >>> still treat that as valid after the certificate expires. catlee and > >>> bhearsum have suggested adding a timestamp signature as they do for > >>> our installers already but I'm not sure what NSS supports. > >> > >> > >> > http://mxr.mozilla.org/mozilla-central/source/security/nss/lib/pkcs7/p7decode.c#1421 > >> > >> Our verification code already uses the signing time if the signing time > is > >> included in the signature. It is likely that the problem is caused by > the > >> tool that generates the signature--it probably isn't including the > signing > >> time in the signature. Send me the XPI and I will run through it in the > >> debugger. > >> > >> Cheers, > >> Brian Also of note: Dave Townsend wrote: > Signtool seems to just use PR_Now() as the signing time, so basically > the local time on the computer. That seems pretty worrying as it > effectively means our certificates never expire. Unexpiring signatures is the model we have for Firefox updates (the MAR files) too. AFAICT, the assumptions we are making are: 1. The transport-layer security between the browser and AMO, plus AMO's internal security, ensure that we only get the latest version of the hotfix patch from AMO. 2. It is always safe to install the latest hotfix patch. Perhaps we would like the ability to revoke a hotfix that we mistakenly sent out. This is easy to do: just create a new version of the hotfix patch that is a no-op, and then distribute it via AMO. This is the same model we are using for Mozilla Marketplace app trust revocations, BTW. Cheers, Brian
(In reply to Ben Hearsum [:bhearsum] from bug 865873 comment #28) > This is a known issue with hotfixes. Our certificates are only valid for one > year, and the current one is only valid for another ~6 months. This means > that any Firefoxes older than ~6 months are unable to receive hotfixes. That is surprising to me. Why? Is Firefox hard-coding the certificate it uses to verify the signature of the hotfix addon? Please provide a link to the code that does the extra signature verification for hotfix addons. See bug 858832 about removing support for signed extensions in general. Once we do that, we will be able to handle the special considerations of the hotfix addon any way we choose. We would have two signing models in use: the MAR signing model and the Firefox Marketplace JAR signing model. It would be best to just have the Firefox Marketplace sign the hotfix JAR the same way that it signs apps, and then have the client verify that signature of the hotfix extension in the same way that it verifies signed apps. Note that the Firefox Marketplace model still uses the expiration date of the certificate as the expiration date of the signature. But, that is by design: we should just never sign anything with a certificate that is valid for less than the lifetime that we want the signature to have, because honoring certificate expiration is required for proper certificate revocation checking. (In the case of the Firefox Marketplace signing model, certificate expiration *is* the revocation checking scheme.) A hotfix should not need to live much past six weeks after it was created because a normal Firefox update that incorporates the real update will be released within that timeframe, so it should be sufficient to just verify that the certificate would not expire less six weeks (plus some slack) after the signature is generated. Anyway, I suspect that the problem that is trying to be solved is different from the problem than what the proposed solution fixes, so I would like to see the special code for handling the addon hotfix's signature before proceeding.
(In reply to Brian Smith (:bsmith) from comment #3) > (In reply to Ben Hearsum [:bhearsum] from bug 865873 comment #28) > > This is a known issue with hotfixes. Our certificates are only valid for one > > year, and the current one is only valid for another ~6 months. This means > > that any Firefoxes older than ~6 months are unable to receive hotfixes. > > That is surprising to me. Why? Is Firefox hard-coding the certificate it > uses to verify the signature of the hotfix addon? Please provide a link to > the code that does the extra signature verification for hotfix addons. We do the same kinds of additional certificate checks that we do for Firefox updates: http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/extensions/AddonManager.jsm#1008 implemented in bug 704988 at the request of the security team.
(In reply to Brian Smith (:bsmith) from comment #3) > (In reply to Ben Hearsum [:bhearsum] from bug 865873 comment #28) > > This is a known issue with hotfixes. Our certificates are only valid for one > > year, and the current one is only valid for another ~6 months. This means > > that any Firefoxes older than ~6 months are unable to receive hotfixes. > > That is surprising to me. Why? Is Firefox hard-coding the certificate it > uses to verify the signature of the hotfix addon? Please provide a link to > the code that does the extra signature verification for hotfix addons. I believe so. The last time this came up we shipped a hotfix to update the fingerprints of the cert that we validate hotfixes against: https://bug803596.bugzilla.mozilla.org/attachment.cgi?id=673488 I'll admit that I don't fully understand this though - I just know that we can't ship hotfixes to Firefoxes that were built prior to us getting our current signing cert.
(In reply to Dave Townsend (:Mossop) from comment #4) > (In reply to Brian Smith (:bsmith) from comment #3) > > (In reply to Ben Hearsum [:bhearsum] from bug 865873 comment #28) > > > This is a known issue with hotfixes. Our certificates are only valid for one > > > year, and the current one is only valid for another ~6 months. This means > > > that any Firefoxes older than ~6 months are unable to receive hotfixes. > > > > That is surprising to me. Why? Is Firefox hard-coding the certificate it > > uses to verify the signature of the hotfix addon? Please provide a link to > > the code that does the extra signature verification for hotfix addons. > > We do the same kinds of additional certificate checks that we do for Firefox > updates: > http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/extensions/ > AddonManager.jsm#1008 implemented in bug 704988 at the request of the > security team. Specifically, we verify the fingerprint of the signing cert.
Would be great if we could get a resolution moving forward in the FF24 timeframe so that the next ESR has this functionality too.
tracking-firefox24: --- → ?
(In reply to Dave Townsend (:Mossop) from comment #6) > Specifically, we verify the fingerprint of the signing cert. (In reply to Alex Keybl [:akeybl] from comment #7) > Would be great if we could get a resolution moving forward in the FF24 > timeframe so that the next ESR has this functionality too. It seems like this bug is proposing an overly complicated solution to a simple problem. The hotfix mechanism is not sufficient, on its own, as a security mechanism. But, maintenance to ESR are primarily security fixes. If a security fix pushed to ESR causes a problem that needs an emergency fix, it seems likely that emergency fix is similarly security-critical and so it should be done with a real update. Consequently, there should be very little need for hotfix addon updates on ESR. The ESR branch is for managed deployments. Arguably, such deployments would prefer real Firefox updates over hotfix updates anyway and probably would want to turn off the hotfix mechanism, so that they can maintain the control over their deployments that is the whole reason they are using ESR in the first place. So, again, the hotfix seems particularly bad for ESR. The core issue is that the ESR branch contains a hash of a certificate that isn't the certificate that we use to issue a hotfix. So, you know, why don't we solve that by making sure that the ESR branch always has the correct hotfix SHA1 hash? In the event that we forget to update that hash, then we simply issue a real update instead of a hotfix (see the previous two paragraphs for why we should anyway), which fixes the hash along with the problem we were intending to fix. That seems *much* easier for us to do than to (a) set up and maintain a timestamping server, and (b) modify Gecko to use timestamp-based signatures. Anyway, we are working on new certificate verification code that will, in Q3, allow us to change the way we validate the signatures of the hotfix addon so that we can sign/verify the hotfix addon with a certificate issued by the internal Mozilla CA, using exactly the same design we use for Firefox Marketplace apps. This will allow us to update the end-entity certificate any time we want, without reducing safety, without needing to deploy a timestamping server, without adding timestamping logic to Gecko, and without designing an THIRD, completely different, update/app signing mechanism. So, even if we're not thrilled with the idea in the previous paragraph, if we can at least do that through Q3, then we will be able to better fix the underlying issue very straightforwardly in Q3/Q4.
(In reply to Brian Smith (:bsmith) from comment #8) > (In reply to Dave Townsend (:Mossop) from comment #6) > > Specifically, we verify the fingerprint of the signing cert. > > (In reply to Alex Keybl [:akeybl] from comment #7) > > Would be great if we could get a resolution moving forward in the FF24 > > timeframe so that the next ESR has this functionality too. > > It seems like this bug is proposing an overly complicated solution to a > simple problem. > > The hotfix mechanism is not sufficient, on its own, as a security mechanism. > But, maintenance to ESR are primarily security fixes. If a security fix > pushed to ESR causes a problem that needs an emergency fix, it seems likely > that emergency fix is similarly security-critical and so it should be done > with a real update. Consequently, there should be very little need for > hotfix addon updates on ESR. What if we bork updates somehow and strand users? That's one example of how it's important for us to be able to hotfix our ESR population.
The second reason we'd like this is because we keep having to update our cert for old versions of Firefox, in case we ever figure out why we stranded them and can help them update forward through a hotfix. That's not specific to FF24, but does make this something we want in the medium term.
As far as the feasibility of actually fixing this bug is concerned: Timestamping mechanisms require us to choose a CA that we trust to issue timestamping certificates that sign the timestamps. Right now, we don't trust any CA for timestamping and NSS doesn't have a timestamping trust bit. If we still want to consider fixing this bug (i.e. implement timestamping support), then we need to figure out (a) which timestamping servers we would trust, and (b) how to validate the timestamp signature against only those CAs. AFAICT, that likely would depend on us implementing the same stuff that we would need to implement in order to support the Mozilla root CA. But, if we were to do that, then we could just switch to the Mozilla root CA. I know it kind of sucks to wait until the end of Q3 for a proper fix, but AFAICT it seems like we have sufficient workarounds for now to hold until then. Also, in the interim, can't anybody give us a code signing cert that is valid for two years instead of just one? I find it hard to believe that all CAs are limiting these certs to one-year terms.
Sorry, comment #11 was assuming that my previous comment made it to the bug, but it got blocked due to mid-air collision. Here it is: There are certain things that we really have to be sure never break: startup and updates are pretty much "cannot fail" things. So, I am skeptical about how necessary a hotfix to fix updates would be, and/or how successful we would generally be in fixing updates with a hotfix. Anyway, I think we can mitigate that by making sure (e.g. in a release checklist) that we always have the latest SHA1 hash in all branches and that the signing cert isn't close to expiring--especially if we would only need to do so for ~4-6 months before we can switch the cert verification to the internal Mozilla root CA and completely avoid the issue using the same design we're using for apps now (and hopefully AMO-hosted addons in the future). Also, ESR releases are for managed environments and if we push out a bad release, the sysadmins people managing those managed environments can help mitigate the impact of broken updates.
(In reply to Brian Smith (:bsmith) from comment #11) > As far as the feasibility of actually fixing this bug is concerned: > > Timestamping mechanisms require us to choose a CA that we trust to issue > timestamping certificates that sign the timestamps. Right now, we don't > trust any CA for timestamping We use Verisign's timestamp server as part of our Authenticode signing for Windows builds. > Also, in the interim, can't anybody give us a code signing cert that is > valid for two years instead of just one? I find it hard to believe that all > CAs are limiting these certs to one-year terms. Policy, (which isn't to say that it can't be changed (https://mana.mozilla.org/wiki/display/SECURITY/Key+management+policy#Keymanagementpolicy-Windowssigning). (In reply to Brian Smith (:bsmith) from comment #12) > Sorry, comment #11 was assuming that my previous comment made it to the bug, > but it got blocked due to mid-air collision. Here it is: > > There are certain things that we really have to be sure never break: startup > and updates are pretty much "cannot fail" things. So, I am skeptical about > how necessary a hotfix to fix updates would be, and/or how successful we > would generally be in fixing updates with a hotfix. > > Anyway, I think we can mitigate that by making sure (e.g. in a release > checklist) that we always have the latest SHA1 hash in all branches and that > the signing cert isn't close to expiring--especially if we would only need > to do so for ~4-6 months before we can switch the cert verification to the > internal Mozilla root CA and completely avoid the issue using the same > design we're using for apps now (and hopefully AMO-hosted addons in the > future). Part of the problem here is that we have lots of people that fall behind. It's easy to make sure that the latest version of Firefox has the latest fingerprints. It's not possible to make sure that older versions (that for some unknown reason, haven't been updating), have the latest fingerprints. Adding a timestamp (or something else that lets the signatures be valid in perpetuity) works around this issue. If the Mozilla root CA solves this issue (by letting us trust any certificate signed by the root rather than specific fingerprints, I presume), it's coming in the next year, and we know we can use it to generate at list XPI signing certificates...I agree that this probably isn't worth the effort. This is the first time I've heard about the root CA, though.
(In reply to Brian Smith (:bsmith) from comment #8) > The hotfix mechanism is not sufficient, on its own, as a security mechanism. > But, maintenance to ESR are primarily security fixes. If a security fix > pushed to ESR causes a problem that needs an emergency fix, it seems likely > that emergency fix is similarly security-critical and so it should be done > with a real update. Consequently, there should be very little need for > hotfix addon updates on ESR. I don't think this is a sound conclusion. The things we use hotfixes for are usually just as relevant to ESR as they are to release builds. We can push hotfixes more quickly, and in some cases we can reach more users. (In reply to Brian Smith (:bsmith) from comment #12) > There are certain things that we really have to be sure never break: startup > and updates are pretty much "cannot fail" things. So, I am skeptical about > how necessary a hotfix to fix updates would be, and/or how successful we > would generally be in fixing updates with a hotfix. I don't really share your skepticism. Given the severity of the "we broke updates" problem, having an independent defense-in-depth mechanism that could potentially address it seems sound. There exists ways that updates could break that couldn't be solved by a hotfix, but there also exists ways that updates could break that could.
(In reply to Ben Hearsum [:bhearsum] from comment #13) > (In reply to Brian Smith (:bsmith) from comment #11) > > As far as the feasibility of actually fixing this bug is concerned: > > > > Timestamping mechanisms require us to choose a CA that we trust to issue > > timestamping certificates that sign the timestamps. Right now, we don't > > trust any CA for timestamping > > We use Verisign's timestamp server as part of our Authenticode signing for > Windows builds. OK. That sounds reasonable. Who is the backup CA? > > > Also, in the interim, can't anybody give us a code signing cert that is > > valid for two years instead of just one? I find it hard to believe that all > > CAs are limiting these certs to one-year terms. > > Policy, (which isn't to say that it can't be changed > (https://mana.mozilla.org/wiki/display/SECURITY/ > Key+management+policy#Keymanagementpolicy-Windowssigning). I can't read the policy because it was moved to the RelEng Wiki that I don't have access to. However, I suggest that we simply change the policy to (a) buy 3-year certificates with 2048+-bit RSA keys, (b) put those keys in an HSM, and (c) replace the SHA1 hash we currently have hard-coded into the binaries with the new cert's hash. > > Anyway, I think we can mitigate that by making sure (e.g. in a release > > checklist) that we always have the latest SHA1 hash in all branches and that > > the signing cert isn't close to expiring--especially if we would only need > > to do so for ~4-6 months before we can switch the cert verification to the > > internal Mozilla root CA and completely avoid the issue using the same > > design we're using for apps now (and hopefully AMO-hosted addons in the > > future). > > Part of the problem here is that we have lots of people that fall behind. > It's easy to make sure that the latest version of Firefox has the latest > fingerprints. It's not possible to make sure that older versions (that for > some unknown reason, haven't been updating), have the latest fingerprints. > Adding a timestamp (or something else that lets the signatures be valid in > perpetuity) works around this issue. Timestamp signatures still require that the certificate is valid at the time of the timestamp. So, if we were to hard-code the SHA-1 hash of a one-year certificate into ESR-24, then 366 days after ESR-24 that certificate will have expired, but we need to be able to issue hotfixes that get signed later than 366 days after ESR-24 (potentially much later, to accommodate users stranded on ESR-24 even after the next major ESR update). > If the Mozilla root CA solves this issue (by letting us trust any > certificate signed by the root rather than specific fingerprints, I > presume), it's coming in the next year, and we know we can use it to > generate at list XPI signing certificates...I agree that this probably isn't > worth the effort. This is the first time I've heard about the root CA, > though. The root CA I am talking about is the one that signs apps for marketplace.firefox.com for B2G (and now Android) packaged apps. It is definitely plausible that we could have the new app signature verification done in 2013Q4. However, somebody needs to advocate for it to get on the SecEng team agenda in order for that to happen.
(In reply to Brian Smith (:bsmith) from comment #15) > (In reply to Ben Hearsum [:bhearsum] from comment #13) > > (In reply to Brian Smith (:bsmith) from comment #11) > > > As far as the feasibility of actually fixing this bug is concerned: > > > > > > Timestamping mechanisms require us to choose a CA that we trust to issue > > > timestamping certificates that sign the timestamps. Right now, we don't > > > trust any CA for timestamping > > > > We use Verisign's timestamp server as part of our Authenticode signing for > > Windows builds. > > OK. That sounds reasonable. Who is the backup CA? If the question is "who do we go to if Verisign is down", the answer is "no one". Obviously, this is a SPOF (though, one that we've never had fail on us). > > > > > Also, in the interim, can't anybody give us a code signing cert that is > > > valid for two years instead of just one? I find it hard to believe that all > > > CAs are limiting these certs to one-year terms. > > > > Policy, (which isn't to say that it can't be changed > > (https://mana.mozilla.org/wiki/display/SECURITY/ > > Key+management+policy#Keymanagementpolicy-Windowssigning). > > I can't read the policy because it was moved to the RelEng Wiki that I don't > have access to. However, I suggest that we simply change the policy to (a) > buy 3-year certificates with 2048+-bit RSA keys, (b) put those keys in an > HSM, and (c) replace the SHA1 hash we currently have hard-coded into the > binaries with the new cert's hash. HSMs will be coming at some point, no firm timeline yet. Kang, any thoughts on using a 3 year 2048 bit cert? > > > Anyway, I think we can mitigate that by making sure (e.g. in a release > > > checklist) that we always have the latest SHA1 hash in all branches and that > > > the signing cert isn't close to expiring--especially if we would only need > > > to do so for ~4-6 months before we can switch the cert verification to the > > > internal Mozilla root CA and completely avoid the issue using the same > > > design we're using for apps now (and hopefully AMO-hosted addons in the > > > future). > > > > Part of the problem here is that we have lots of people that fall behind. > > It's easy to make sure that the latest version of Firefox has the latest > > fingerprints. It's not possible to make sure that older versions (that for > > some unknown reason, haven't been updating), have the latest fingerprints. > > Adding a timestamp (or something else that lets the signatures be valid in > > perpetuity) works around this issue. > > Timestamp signatures still require that the certificate is valid at the time > of the timestamp. So, if we were to hard-code the SHA-1 hash of a one-year > certificate into ESR-24, then 366 days after ESR-24 that certificate will > have expired, but we need to be able to issue hotfixes that get signed later > than 366 days after ESR-24 (potentially much later, to accommodate users > stranded on ESR-24 even after the next major ESR update). Yeah, I understand that they need to be valid at the time of signing. The reason that timestamped signatures help is that it means we could ship a hotfix with the new fingerprints up to the day before the current certificate expires, and users would still be able to apply it in perpetuity. This avoids the need to have a long overlap between certificate validity, which saves money and time.
Flags: needinfo?(gdestuynder)
hello there! ;-) So.. - the policy can be indeed changed and feedback/comments are more than welcome. I did note the releng link is not reachable and will address that in a separate bug or irc conversation - Mozilla Root CA, as in the software-generated CA on the ssl1 server should not be used for Firefox, at least not in it's current state, it's in need of a sec-review AFAIR. - The Root CA generation mechanism and machines we use for marketplace can potentially be used to generate a new sub-CA or direct certificates. It is on a HSM right now (its also operating 100% offline and a few security assurance members are needed to generate new signatures). With a sub-CA we can make it online/automated if necessary, or delegate access. - "Complete" use of HSMs by Releng is on the table but this is a joes/joduinn decision/discussion. AFAIK this would require quite some time. Using the existing HSM-hosted CA is certainly faster. - Using 3 year 2048 RSA cert sounds reasonable and within the norm.
Flags: needinfo?(gdestuynder)
I'm no longer sure what this bug is for. We've established that when signing an XPI the time of signing is stored and as long as it is before the certificate expiry then Firefox treats the signature as valid. That seems to meet the criteria for the current summary of this bug. But the time of signing currently comes only from the signing computer's clock. This isn't very trustworthy so perhaps that should be changed. In which case we need NSS's signtool to be able to consult a timestamp server and the certificate verification code to be able to handle that. Either that or we change things so we never trust anything signed by an expired certificate regardless of when the signing claims to have happened. Either way it seems like we need to have different bugs filed to this one.
(In reply to Dave Townsend (:Mossop) from comment #18) > I'm no longer sure what this bug is for. > > We've established that when signing an XPI the time of signing is stored and > as long as it is before the certificate expiry then Firefox treats the > signature as valid. That seems to meet the criteria for the current summary > of this bug. Sounds like there's no problem here.
tracking-firefox24: + → -

I don't think we distribute hotfixes like this anymore?

Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → INVALID
You need to log in before you can comment on or make changes to this bug.