Closed Bug 412031 Opened 17 years ago Closed 15 years ago

Purchase "Mozilla Foundation" code-signing cert and establish policy

Categories

(mozilla.org :: Governance, task)

Product:
mozilla.org
Component:
Governance
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: samuel.sidler+old, Assigned: mrz)

References

(
URL
)

Details

(Whiteboard: [stalled, KaiRo]) 

This probably isn't the appropriate component, but I'm filing here for now and Frank can move this as he sees fit.

This is a spin off of bug 409459 comment 9.

Camino would like to code-sign its releases. Presumably, other non-MoCo projects will want the same including SeaMonkey and Sunbird/Lightning. To do this, we need a code-signing cert, probably one from Verisign. As Frank said in that bug, this is the first and easiest step.

Second, and related, we need to deal with Verisign to prove "organizational identity." I can work with Frank on this since he thinks it'll be easier working with someone in Mountain View to get this done.

Finally, we need to establish a process and procedures for using the cert. Quoting Frank:

> 3. Technical protection of the private key and maintaining secure use of the
> key when signing code. This is the biggest issue as far as I'm concerned.
> Before approving something like this I would need to have a firm plan for how
> you plan to do this: who are the key people responsible (for accountability I'd
> want one primary person plus backup), what are the technical measures, what are
> the administrative procedures to get something signed, and so on. This is
> especially important if the same cert/key is going to be used with other
> projects (i.e., not just Camino).

There are basically two ways to do this:

  1. Have a machine, inside a firewall with the cert on it. Sign builds on
     that machine and *only* on that machine.
  2. Distribute the cert on a usb key (or something similar) to those
     trusted to have it.

The disadvantages to #1 are mainly procedural: the person signing the builds has to get the build in, sign it, get it out, then get it to staging. Right now we can do that straight from the tinderbox (minus the signing part, of course).

The disadvantages to #2 are mainly in trust. We'll have a physical device that can get into the wrong hands. It saves a lot of team in procedure though since the signer wouldn't need to go in and out of remote sessions. Getting a person access requires no new accounts, just physical transfer of the cert.

Talking with Mark, I think #2 is better for most people, but I can see the desire to use #1. I'm not sure what other think, however.

Thoughts?
Assignee: hecker → server-ops
Severity: enhancement → normal
Component: CA Certificates → Server Operations
OS: Mac OS X → All
QA Contact: ca-certificates → justin
Hardware: Macintosh → All
Assignee: server-ops → zak
Component: Server Operations → Governance
QA Contact: justin → governance
(In reply to comment #0)
>   1. Have a machine, inside a firewall with the cert on it. Sign builds on
>      that machine and *only* on that machine.
> 
> The disadvantages to #1 are mainly procedural: the person signing the builds
> has to get the build in, sign it, get it out, then get it to staging. Right now
> we can do that straight from the tinderbox (minus the signing part, of course).

#1 would probably also end up being 2-3 machines, given that Mac releases must be signed on 10.5 and presuming SeaMonkey and Sunbird/Lightning want to sign their Windows releases (like Fx/Tb do now) as well as their Mac ones (I don't know if Linux has any comprable mechanism available, but if so, I assume the cross-platform projects will want to avail themselves of it).
I suggest that you should shop around for the best bargain you can get on 
an object signing certificate.  If Verisign is the best deal, then choose
that, but I wouldn't suggest that you limit the search to just one CA, 
the one with the biggest name.  
(In reply to comment #2)
> I suggest that you should shop around for the best bargain you can get on 
> an object signing certificate.  If Verisign is the best deal, then choose
> that, but I wouldn't suggest that you limit the search to just one CA, 
> the one with the biggest name.  

The only reason I specified the VeriSign one is because this is what Microsoft recommends -- specifically, a "VeriSign ‘Microsoft Authenticode’ Code Signing Digital ID" -- and I don't know enough about various certs to know if that matters in Windows code-signing. If it doesn't, then great, we can shop elsewhere. But if it does, we should probably get the code signing cert from VeriSign. You know better than I. :)
Another option is to purchase one certificate for each machine which is doing the signing. This reduces the impact of a security breach, because instead of needing to revoke the certificate which signed _all_ our builds, we only need to revoke the one which (for example) signed Camino builds for Mac OS 10.5, or whatever.

This is obviously a bit more expensive, but that shouldn't be an issue. We may well find CAs who are happy to donate certificates to us; whether we would accept that is a diplomatic question. (I'm not sure if we currently pay for our SSL certs.)

Gerv
(In reply to comment #4)
> Another option is to purchase one certificate for each machine which is doing
> the signing. This reduces the impact of a security breach, because instead of
> needing to revoke the certificate which signed _all_ our builds, we only need
> to revoke the one which (for example) signed Camino builds for Mac OS 10.5, or
> whatever.

Frank mentioned this in commenting in the other bug, but added "there might be a
good reason to get just one." I don't know enough about certs to know what that reason might be, so I'll let him comment on that. That'd definitely be easier overall, I think. And the expense really isn't that exuberant.
I don't recall exactly what I was thinking of when I wrote ""there might be
a good reason to get just one [certificate]". But let me ask a different question: What is the existing practice with regard to signing of Firefox and Thunderbird releases? Do they have just one Mozilla Corporation certificate and private key? Is is replicated on multiple build systems? It seems that it might make sense to adapt existing practices unless there's a good reason not to.

Other comments: A code signing certificate is typically associated with an organizational entity, in this case the Mozilla Foundation. So all other things being equal, it seems logical to have just one cert.

On the subject of which CA to get the cert from, I agree with Nelson that we should not limit ourselves to just VeriSign in terms of choices; there are other perfectly good CAs we could use. One obvious factor is that the CA needs to be already in NSS as a trusted root and bundled with whatever releases need to be recognizing and verifying the cert. Less obvious, but implied by some of the Camino-related comments, is that the CA's root cert needs to be recognized and trusted by OS X; is this the case? What about Windows; does the CA need to be in the Microsoft list as well? Also, is it OK if the CA were only recently added, or do we need a CA that is bundled with older versions of either Mozilla software or Microsoft or Apple software (e.g., older versions of XP)? Once these questions are answered we can generate a list of candidate CAs.

Finally, I would prefer to pay list price for the cert (or certs) as opposed to accepting complimentary certs from a CA, to avoid any conflict of interest issues with respect to future decisions about that CA.
(In reply to comment #6)
> What is the existing practice with regard to signing of Firefox and
> Thunderbird releases? Do they have just one Mozilla Corporation certificate and
> private key? Is is replicated on multiple build systems? It seems that it might
> make sense to adapt existing practices unless there's a good reason not to.

Currently, to my knowledge, the Mozilla Corporation has one code-signing cert that it uses for both Firefox and Thunderbird. I'm not sure if this will be changing with MailCo, however. (My assumption is "yes" since it's a "Mozilla Corporation" cert.) They have it on one machine that they sign builds with, but don't yet sign Mac builds. In that case, they would need to have it on two machines. It's not at all integrated with the build system, though I'm not sure if there are intentions to do so given the automation push.

> On the subject of which CA to get the cert from, I agree with Nelson that we
> should not limit ourselves to just VeriSign in terms of choices; there are
> other perfectly good CAs we could use. One obvious factor is that the CA needs
> to be already in NSS as a trusted root and bundled with whatever releases need
> to be recognizing and verifying the cert. Less obvious, but implied by some of
> the Camino-related comments, is that the CA's root cert needs to be recognized
> and trusted by OS X; is this the case? What about Windows; does the CA need to
> be in the Microsoft list as well? Also, is it OK if the CA were only recently
> added, or do we need a CA that is bundled with older versions of either Mozilla
> software or Microsoft or Apple software (e.g., older versions of XP)? Once
> these questions are answered we can generate a list of candidate CAs.

Camino currently supports Mac OS X 10.3 "Panther" and above, however we'll be dropping 10.3 support and only support 10.4 "Tiger" and above for Camino 2.0. We haven't yet released Camino 1.6, but given its timeframe (a month, maybe two from now), I'm not sure we'll have a code-signing cert in time for that release. That being said, Apple didn't offer the ability to sign code -- and presumably verify signed code -- until 10.5 "Leopard" so that should be our target.

My guess is that since SeaMonkey and other projects support Windows 2000 and up on the trunk, they'll want a CA that supports OSes in that range as well.
This bug also desires to "establish policy".  
I imagine the policy might govern such things as:
- the number of signing certs
- the criteria on which CAs are to be selected
- any cost criteria
- what files Mozilla is going to sign (installers? zips? installed files?)
and last but not least
- what are the objectives of this signing, that is, what different pieces of 
software that verify signatures (OSes, or other) are we trying to satisfy?

If that last bullet will include software produced by Mozilla, then IMO the 
policy also needs to address when (at what times, under what circumstances) 
Mozila software will verify signatures, and what it will do if signatures 
fail to verify.  If past/present behavior is any predictor, I expect the 
answer to that last question to be: nothing of any significance or effect. :(
The policy might also wish to address operational issues surrounding the use
of the private key for the signing cert, such as
- who will have access to the signing key?  (what job positions?)
- will signing be automated as part of any build or packaging process? 
  or will it require manual intervention (such an entering a password) to 
  control use of that key?
- Will the key be generated and kept in a hardware gizmo from which it 
  cannot be extracted, to eliminate the possibility of it being copied?
  or will it be kept in an ordinary disk file (e.g. key3.db, or OS key store)?
Something tells me Zak is no longer working on this...
Assignee: zak → nobody
Your intuition serves you well. I'd bet that Frank or Gerv is best situated to take this over.
So I would say that what remains to be done includes some or all of the following:

- Whoever does Camino releases needs to state they are happy from a technical standpoint with this - for example, that release automation would still work.

- If Camino releases are currently done on MoCo hardware by the MoCo team, then we can reuse a lot of their policies and procedures for key management. If not, we need to look at what's in place.

- The Camino team need to choose a suitable certificate provider and ascertain the cost of an appropriate certificate. My feeling is that this should be at their discretion, balancing the ubiquity of the relevant root, cost and other factors.

- The Foundation needs to decide whether this is something we just pay for or whether it would be more appropriate to take the funds from the Camino-specific donations pool.

Have I missed anything?

Gerv
Is this now affecting Camino only? I was under the impression that at least SeaMonkey also fits into what's requested here.
I have set up everything for SeaMonkey release automation to parallel (and/or reuse) what's currently used for Firefox 3.5, but the signing step is missing, as we don't have certificates or signing infrastructure.
I have the impression though that currently even Firefox only signs Windows builds and not Mac (or even Linux).
(In reply to comment #13)
> Is this now affecting Camino only? I was under the impression that at least
> SeaMonkey also fits into what's requested here.

No, it's now, and has always been, for all mozilla.org projects (Calendar, Camino, SeaMonkey); Camino was initially the product driving the issue, though. When the discussion last ended, it sounded like all would be sharing the same cert, presuming MoFo could get one that is in all the OS roots for all the versions of all the (tier-1?) OSes the non-Fx/Tb products currently support.

> I have the impression though that currently even Firefox only signs Windows
> builds and not Mac (or even Linux).

There's a bug somewhere for code-signing Firefox Mac builds, but no, I don't think anyone has done the build or release infrastructure work to implement it.  When these bugs (this one, bug 409459, and the Firefox one) were filed initially, it was expected that we'd really need to get code-signing working on the double on Mac OS X, but that has turned out not to be the case.
KaiRo: is the SeaMonkey release infrastructure controlled by you rather than MoCo? Smokey: what about the Camino infrastructure? Clearly, the Foundation needs to make some effort to check that keys in its name will be stored securely, so what can you tell us about the arrangements?

Do you guys have particular certs and providers in mind?

Gerv
Camino will figure this out on our own.
(In reply to comment #15)
> KaiRo: is the SeaMonkey release infrastructure controlled by you rather than
> MoCo? Smokey: what about the Camino infrastructure? Clearly, the Foundation
> needs to make some effort to check that keys in its name will be stored
> securely, so what can you tell us about the arrangements?

The SeaMonkey 2.x build infrastructure is hosted by MoCo IT and controlled by me remotely, and for older versions we don't seek being able to sign them.

(In reply to comment #15)
> Do you guys have particular certs and providers in mind?

I don't have anything particular in mind, though we're trying to stay as closely as possible to what MoCo and MoMo are doing on their ends.
Apologies for losing focus here; I'll CC myself.

KaiRo: would there be any technical issues (let's ask that question before we ask the question about political issues :-) with the SeaMonkey builds just being signed with the same key used for Firefox and Thunderbird?

Gerv
(In reply to comment #18)
> KaiRo: would there be any technical issues (let's ask that question before we
> ask the question about political issues :-) with the SeaMonkey builds just
> being signed with the same key used for Firefox and Thunderbird?

I don't see any issues with that, as long as there's some way to ensure I can get signed builds done in a timely manner.
I think, both for that and for not putting the burden of doing SeaMonkey work on the MoCo releng team, I'd need to have a way of triggering signing myself remotely.


For the political issues, I'm just unsure if it's good to have non-MoCo binaries signed with a cert that says "Mozilla Corporation", we should have a "Mozilla Foundation" label on them.
I apologise for the delay here.

OK, I've checked and it's fine to have SeaMonkey binaries signed with a "Mozilla Foundation" cert and to get hold of one. However, a better course, given the "One Mozilla" strategy, is to see if we can get a "Mozilla" cert and use that for everything.

I've filed bug 528504 on that issue. Let's see if we can get some traction there.

Gerv
OK, bug 528504 didn't work out. we're back to this plan.

mrz: what do you need from MoFo to start the procurement process?

Gerv
Summary: Purchase code-signing cert for use by mozilla.org projects and establish policy → Purchase "Mozilla Foundation" code-signing cert and establish policy
(In reply to comment #21)
> OK, bug 528504 didn't work out. we're back to this plan.
> 
> mrz: what do you need from MoFo to start the procurement process?

This is difficult for me to drive - whoever needs this key should drive the Windows host that does the application.

Key and CSR generation is done within an ActiveX component and the same Windows host that applied for the key needs to retrieve it which dumps the cert into IE's certificate store.  I had to export all three variations for RelEng and then RelEng had to convert.

Who eventually needs to have this key?
Whoever runs the SeaMonkey 2 build process, which I understood from KaiRo happened on boxes under the control of Mozilla IT. Have I misunderstood?

Gerv
On boxes Mozilla IT hosts - we're hands off on anything other than maintaining the hardware or VM hardware.
I can handle getting this.  I need:

1. CN - Mozilla Foundation (?)
2. OU - SeaMonkey Project (?)
3. process to hand off key (& cert)

Key size is fixed at 2048.
Assignee: nobody → mrz
(KaiRo is out until next month, aiui. Might be worth coordinating with him when he's back from vacation.)
CN and OU look good to me. I'll let KaiRo take it from here when he gets back.

Gerv
Whiteboard: [stalled, KaiRo]
Well, that puts us right at the point where we actually need to find out how to get a signing host for SeaMonkey, then - up to now I wasn't even sure if we need an own one, now it looks like we need our own solution, if I read all this correctly - and actually, I don't even have a clue right now what exactly we need and how we would get it. :(
OK, So the point is: Kairo, if and when you are technically in a position to sign SeaMonkey builds, the Mozilla Foundation is happy to obtain a cert for you. When that happens, either reopen this bug or file a new one.

Gerv
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → INCOMPLETE
Depends on: 537324
You need to log in before you can comment on or make changes to this bug.