840368 - Test Key Signing after ceremony

Status: RESOLVED FIXED

(Cloud Services :: Operations: Marketplace, task)

Description

[Caitlin Galimidi [:c8o]]

Test of new key signing service on Tuesday Feb 12th. 
TOMORROW (see blocking bug 819053)

Please communicate with Joe Stevensen and Guillaume DeS. on any issues experienced as quickly as possible. All key signing resources are arranged onsite.

Comment 1

[Guillaume Destuynder [:kang] [this account is no longer active]]

New certs are attached to bug https://bugzilla.mozilla.org/show_bug.cgi?id=819053#c22

Comment 2

[Caitlin Galimidi [:c8o]]

rtilder: i believe this is your bug to close... did you actually confirm testing already in bug number 819053?

Comment 3

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

I started to do my own testing of the app1.reviewer.hsm cert. It should not be issued from the same root as the app1.hsm cert, because we don't want the production devices to trust it. Instead, it should be issued from its own root.

There are various options. The cleanest option is to re-generate the root cert and then re-issue the app1.hsm cert from the new cert. Then, we don't have to worry about the wrongly-issued app1.reviewer.hsm cert.

Another option would be to revoke the app1.reviewer.hsm cert and preload that revocation into NSS.

I recommend that we go with the first option (re-gen the root cert), so we don't have to start out worrying about revocation. 

Also, I really recommend that, as part of this, we sign test apps with both the prod cert and the reviewer cert so that we can have a complete test of each.

Comment 4

[Guillaume Destuynder [:kang] [this account is no longer active]]

Regarding the question in email:
"does the decision " impact work that serverops and netops have downstream to implement?"
No.

We can generate a 2nd root ca and sign the reviewers with it without impacting netops and server ops.

As per :joes we might have to do this tomorrow.

Comment 5

[Guillaume Destuynder [:kang] [this account is no longer active]]

Ok, I checked with joes.

I want to be sure that if we regenerate the certificates tomorrow (15 Feb 2013) those are the FINAL certificates (this was already requested for the previous generation).

In this case, we would:

- generate a root CA for reviewers, this will be stored on the offline machine and generated using the SAME generation script and procedure as the regular root CA
- the root CA reviewers will use the SAME HSM protected by the SAME ACS, but using a DIFFERENT private key. The private key usage will be protected by the SAME OCS as the original root CA.
- generate a new CSR for reviewers and sign it with the reviewers root CA, using the SAME generation script and procedure as the regular certificate

ie all that will change are:
- the root ca name (root ca reviewers)
- the private keys for the root ca  reviewers and the reviewers certificate

what will not change:
- current root ca
- current app signing signed cert
- ACS
- OCS

Definitions:
ACS= Admin Card Set, used to administrate HSMs
OCS= Operator Card Set, used to sign

The scripts are available at https://mana.mozilla.org/wiki/download/attachments/26416648/hsm_scripts-marketplace-2.tar.gz.gpg?version=1&modificationDate=1360627052958&api=v2 and signed by our key (those are identical at the previous script, as we'll just change the key name during generation).

[e-mail] and [e-mail], please confirm that this is what is requested and correct.

Comment 6

[Guillaume Destuynder [:kang] [this account is no longer active]]

After discussing with bsmith on IRC, it's better to also regenerate the root ca (and app signing cert) again, as there's no check that the old reviewer certificate is revoked on clients.

Also, since the certificates are not required to be ready tomorrow, we'll make sure everything is documented and correct (certificate/PKI wise) before generating new ones.

Will update on this bug and we'll create new bugs as necessary. Hope that is ok with everyone.

Comment 7

[Guillaume Destuynder [:kang] [this account is no longer active]]

Here's our list of questions, and answers when available:

https://etherpad.mozilla.org/dLWLvIJr4o

Please have a look (:rtilder, :joes, :jthomas primarily)  and complete what's missing/ask more questions if necessary. This should help making a complete documentation and hopefully make sure there is nothing forgotten/needs to regenerate/incomprehension, etc.

Thanks!

Comment 8

[Ryan Tilder [:rtilder]]

Responded to a few questions as well as added a couple questions.

Comment 9

[Guillaume Destuynder [:kang] [this account is no longer active]]

We still have a few questions that need to be replied to, in particular regarding testing.
I have sets "FIXMEs" in order to quickly identify all these.

In particular line 88 and 44 DIRECTLY block the new CA/Certs generation.

https://etherpad.mozilla.org/dLWLvIJr4o

Comment 10

[Joe Stevensen [:joe]]

Ryan,

The sooner you get this info the better. Most of us are in SF next week for the BsidesSF and RSA conference, therefore won't be in MTV.

Comment 11

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

I updated the wiki and discussed this on IRC. AFAICT, we're not blocked on any input from me any longer. Let me know if there is more I can do to help.

My most immediate remaining unaddressed concern is about the testing--in particular, how we're going to sign the test apps that go into mozilla-central's test suite.

Comment 12

[Guillaume Destuynder [:kang] [this account is no longer active]]

I've modified the current generation scripst to support what has been written in the etherpad. I'll put those in a wiki page once we're 100% sure they're ok.

Mainly, I found a couple of unaddressed points:

* cRLSign in CAs. We might want it for the future, or should it be removed?

* Subject name conventions. I use: 
- For reviewers:
  + Root CA:
     /C=US/ST=CA/L=Mountain View/O=Mozilla Marketplace Reviewers /CN=root.reviewers.sign.svc.mozilla.com
  + Cert:
    /C=US/ST=CA/L=Mountain View/O=Mozilla Marketplace Reviewers /OU=Marketplace Reviewers App Signing /CN=app1.reviewers.sign.svc.mozilla.com

For regular users:
  + Root CA:
    /C=US/ST=CA/L=Mountain View/O=Mozilla Marketplace /CN=root.sign.svc.mozilla.com
  + Cert:
    /C=US/ST=CA/L=Mountain  View/O=Mozilla Marketplace /OU=Marketplace App Signing /CN=app1.sign.svc.mozilla.com

Rationale:
- For C, ST, L:
This is the current place where certs are generated _from_ (for the Roots its the real physical place) and Mozilla's headquarters
- For O:
This is the unit for this certificate/root ca. Those clearly indicate the purpose of the certs, and are different between CAs. This also ensure that CSRs signed are signed with the correct root as openssl otherwise fails.
- For CN:
This is made up to give an idea of what the certificate is and which teams handles it.

Feel free to correct if not ok.

Comment 13

[Guillaume Destuynder [:kang] [this account is no longer active]]

Attachment: hsm scripts

I'm attaching the current version of the scripts to this bug, hope that's ok. I'll modify them if necessary tho.

I'll need ack on those if we decide to go with the above subject names and crlsign enabled on CA, to be able to generate on friday.

Comment 14

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Guillaume Destuynder [:kang] from comment #12)
> Mainly, I found a couple of unaddressed points:
> 
> * cRLSign in CAs. We might want it for the future, or should it be removed?

You can leave it in. It is unlikely we'll ever use CRLs but this is not harmful to have.

>      /C=US/ST=CA/L=Mountain View/O=Mozilla Marketplace Reviewers
> /CN=root.reviewers.sign.svc.mozilla.com
>   + Cert:
>     /C=US/ST=CA/L=Mountain View/O=Mozilla Marketplace Reviewers
> /OU=Marketplace Reviewers App Signing /CN=app1.reviewers.sign.svc.mozilla.com
> 
> For regular users:
>   + Root CA:
>     /C=US/ST=CA/L=Mountain View/O=Mozilla Marketplace
> /CN=root.sign.svc.mozilla.com
>   + Cert:
>     /C=US/ST=CA/L=Mountain  View/O=Mozilla Marketplace /OU=Marketplace App
> Signing /CN=app1.sign.svc.mozilla.com
> 
> Rationale:
> - For C, ST, L:
> This is the current place where certs are generated _from_ (for the Roots
> its the real physical place) and Mozilla's headquarters
> - For O:
> This is the unit for this certificate/root ca. Those clearly indicate the
> purpose of the certs, and are different between CAs. This also ensure that
> CSRs signed are signed with the correct root as openssl otherwise fails.
> - For CN:
> This is made up to give an idea of what the certificate is and which teams
> handles it.
> 
> Feel free to correct if not ok.

O stands for Organization, which is usually the company name. See the https://addons.mozilla.org EV certificate for an example.
OU stands for Organizational Unit (i.e. department), which seems to be what you are using O for.

However, I am not sure you really need or even want all of that junk. Up to you guys really. The main purpose of these complex names is to make them unambiguous.

HOWEVER, I do recommend one important change: do not make the CN field look like a domain name. Some software will interpret that to be the hostname for which the certificate is valid. (Firefox won't, because the cert doesn't have the right EKU and because Firefox doesn't trust it for SSL anyway.) I suggest you try "app-reviewers-1" and "app-production-1" instead.

Comment 15

[Ryan Tilder [:rtilder]]

I agree that cRLSign is unlikely to cause any harm.

As I said in the email thread on the topic of SubjectName and CommonName several weeks ago, I view the selection of these fields as entirely an operational decision: Ops has to support it so they should go with what makes doing so easiest for them.  At the time a CN value that appeared to be an FQDN was most preferable.  However, bsmith's comment about random software improperly interpreting the CN's FQDN formatting as valid for a given host is a cogent point that had not occurred to me.  Avoiding an FQDN in this instance may prevent confusion with both 3rd party software and individuals who see the hostname and assume that the certificate(s) are for SSL use.

Comment 16

[Guillaume Destuynder [:kang] [this account is no longer active]]

I also didn't think of any software that misinterpret the CN. I will use:
app-reviewers-marketplace-1 and app-production-marketplace-1 if you're ok with that.
We can set the other fields to optional, too.

Comment 17

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Guillaume Destuynder [:kang] from comment #16)
> I also didn't think of any software that misinterpret the CN. I will use:
> app-reviewers-marketplace-1 and app-production-marketplace-1 if you're ok
> with that. We can set the other fields to optional, too.

I do think that would be clearer. And, to be clear, any of the proposed names are OK as far as the Firefox(OS) client is concerned. Replacing dots with dashes, and rearranging the text in "O=" and "OU=" is about making things clear to us (people), and guarding against potential flaws in other peoples' software.

Comment 18

[Guillaume Destuynder [:kang] [this account is no longer active]]

Attachment: hsm scripts

Updated the scripts, i packed my git repository so you can see changes a little more easily. The change author is my local git name, I plan to put that in a more official repository with a more official author name and process when we have one up (pending bug 843421). Changes start at revisions a92bb999458ba293665395877c9469be0ae0ec2f

Main changes:
- made everything but the CN optional in the SSL policy
- changed CN as per above model for CSRs and CA


Here is a resulting TEST cert with the included scripts, for convenience (I removed the actual cert data to be sure its not used):
CA production example:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
        Signature Algorithm: sha384WithRSAEncryption
        Issuer: CN=root-ca-production-marketplace EXAMPLE
        Validity
            Not Before: Feb 21 23:14:09 2013 GMT
            Not After : Feb 19 23:14:09 2023 GMT
        Subject: CN=root-ca-production-marketplace
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (2048 bit)
                Modulus (2048 bit):
                    [...]
                Exponent: 415031 (0x65537)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:TRUE
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Extended Key Usage: critical
                Code Signing
            X509v3 Authority Key Identifier: 
                DirName:/CN=root-ca-production-marketplace
                serial:01
            X509v3 Subject Key Identifier: 
                [...]
    Signature Algorithm: sha384WithRSAEncryption



Signed production cert example:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
        Signature Algorithm: sha384WithRSAEncryption
        Issuer: CN=root-ca-production-marketplace EXAMPLE
        Validity
            Not Before: Feb 21 23:24:13 2013 GMT
            Not After : Feb 20 23:24:13 2018 GMT
        Subject: CN=app-production-marketplace-1
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (2048 bit)
                Modulus (2048 bit):
                    [..]
                Exponent: 415031 (0x65537)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage: critical
                Code Signing
            X509v3 Subject Key Identifier: 
                [..]
            X509v3 Authority Key Identifier: 
                keyid:[..]
                DirName:/CN=root-ca-production-marketplace
                serial:01
    Signature Algorithm: sha384WithRSAEncryption

Comment 19

[Jason Thomas [:jason]]

(In reply to Guillaume Destuynder [:kang] from comment #16)
> I also didn't think of any software that misinterpret the CN. I will use:
> app-reviewers-marketplace-1 and app-production-marketplace-1 if you're ok
> with that.
> We can set the other fields to optional, too.

I believe these CN should be okay.

Comment 20

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Guillaume Destuynder [:kang] from comment #18)
> CA production example:
>         Version: 3 (0x2)
>         Serial Number: 1 (0x1)
>         Signature Algorithm: sha384WithRSAEncryption
>         Issuer: CN=root-ca-production-marketplace EXAMPLE

Why no OU,O,L,ST,C?

>                 Exponent: 415031 (0x65537)

This is a mistake. The clear intent is that the exponent should be 65537, not 415031. i.e. something is interpreting a decimal value to be a hex value. That should be investigated and corrected. Is the problem with the tool printing the key, or the tool generating the key?

> Signed production cert example:

Same issues as above.

Comment 21

[Ryan Tilder [:rtilder]]

I'll echo Brian's comments on why such a truncated SubjectName for both certs.  We should have O and OU at the very least and might as well toe the line on what fields are generally expected in a certificate subject.

The exponent issue seems to be a problem with hsm_scripts/secworld/4_generate_key.sh:

    generatekey -c ${OCS_NAME} ${TYPE} protect=${PROTECT} \
                        recovery=yes \
                        type=RSA size=2048 pubexp=65537 \
                        ident=${KEY_NAME} \
                        nvram=${NVRAM}

Looks like it interprets the pubexp parameter as a hexadecimal value automagically.

Comment 22

[Jason Thomas [:jason]]

How about O=Mozilla Corporation, OU=Mozilla Marketplace [App, Reviewer] Signing Service?

Comment 23

[Guillaume Destuynder [:kang] [this account is no longer active]]

Attachment: hsm scripts

Damn, I really should have caught the pubexp. Thanks for reviewing. I agreed on subject with :jason again, and made it as per comment 22 except App is replaced by Production for consistency. Also checked with him on IRC.
I need review+ from all 3 of you. Lets see if this one passes.

ROOT CA EXAMPLE
Certs:
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
        Signature Algorithm: sha384WithRSAEncryption
        Issuer: O=Mozilla Corporation , OU=Mozilla Marketplace Production Signing Service , CN=root-ca-production-marketplace
        Validity
            Not Before: Feb 22 18:43:14 2013 GMT
            Not After : Feb 20 18:43:14 2023 GMT
        Subject: O=Mozilla Corporation , OU=Mozilla Marketplace Production Signing Service , CN=root-ca-production-marketplace
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (2048 bit)
                Modulus (2048 bit):
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:TRUE
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Extended Key Usage: critical
                Code Signing
            X509v3 Authority Key Identifier: 
                DirName:/O=Mozilla Corporation /OU=Mozilla Marketplace Production Signing Service /CN=root-ca-production-marketplace
                serial:01
            X509v3 Subject Key Identifier: 
    Signature Algorithm: sha384WithRSAEncryption

CERT SIGNED BY ROOT CA EXAMPLE:
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
        Signature Algorithm: sha384WithRSAEncryption
        Issuer: O=Mozilla Corporation , OU=Mozilla Marketplace Production Signing Service , CN=root-ca-production-marketplace
        Validity
            Not Before: Feb 22 18:44:59 2013 GMT
            Not After : Feb 21 18:44:59 2018 GMT
        Subject: O=Mozilla Corporation , OU=Mozilla Marketplace Production Signing Service , CN=app-production-marketplace-1
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (2048 bit)
                Modulus (2048 bit):
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage: critical
                Code Signing
            X509v3 Subject Key Identifier: 
            X509v3 Authority Key Identifier: 
                DirName:/O=Mozilla Corporation /OU=Mozilla Marketplace Production Signing Service /CN=root-ca-production-marketplace
                serial:01
    Signature Algorithm: sha384WithRSAEncryption

Comment 24

[Jason Thomas [:jason]]

Looks okay to me with regards to O, OU and CN.

Comment 25

[Ryan Tilder [:rtilder]]

Comment on attachment 717205 [details]
hsm scripts

The Subject Key Identifier field is being reported but is empty.  From the previous version generated I'm going to assume that is a copy'n'paste'n'edit artifact.

Looks good to me as long as bsmith doesn't foresee any issues with attribs being absent.

Comment 26

[Guillaume Destuynder [:kang] [this account is no longer active]]

the values were removed in the pasting

Comment 27

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Comment on attachment 717205 [details]
hsm scripts

It also looks OK to me.

Exponent 3 (0x03) would result in faster performance for verifying the signatures on the phone but 65537 is also OK.

IMO, if we're going to include O and OU then we might as well include at least C=US too. (I can understand why we might not want to include L=Mountain View and/or ST=California.)

Please also post the actual generated certificates for review.

Comment 28

[Guillaume Destuynder [:kang] [this account is no longer active]]

I can change this and generate final ones.
I added C=US after checking with jason on IRC
I also changed the exponent to 0x03:
                Exponent: 3 (0x3)
Issuer: C=US, O=Mozilla Corporation , OU=Mozilla Marketplace Production Signing Service , CN=root-ca-production-marketplace

(this is the result from a new test I generated)

I'm not going to require another round of reviews as I believe those changes are minimal. 

Next review will be for the real, final ones. This will take some time to generate due to the key ceremony.

Comment 29

[Guillaume Destuynder [:kang] [this account is no longer active]]

Attachment: final certs

Attached are the "final" certificates, pending approval.

/C=US was NOT added. The reason is that the subject length would be longer than the maximum allowed by either OpenSSL or the HSM, and that everyone were ok with it being left out. Since we were already setup for the ceremony, we decided to go ahead.

Summary (hex values removed for pasting purposes, see attached files for the complete certs, signed by opsec's gpg key):

**** ROOT CA PRODUCTION
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
    Signature Algorithm: sha384WithRSAEncryption
        Issuer: O=Mozilla Corporation , OU=Mozilla Marketplace Production Signing Service , CN=root-ca-production-marketplace
        Validity
            Not Before: Feb 23 01:50:11 2013 GMT
            Not After : Feb 21 01:50:11 2023 GMT
        Subject: O=Mozilla Corporation , OU=Mozilla Marketplace Production Signing Service , CN=root-ca-production-marketplace
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                   [..]
                Exponent: 3 (0x3)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:TRUE
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Extended Key Usage: critical
                Code Signing
            X509v3 Authority Key Identifier: 
                DirName:/O=Mozilla Corporation /OU=Mozilla Marketplace Production Signing Service /CN=root-ca-production-marketplace
                serial:01

            X509v3 Subject Key Identifier: 
[..]
    Signature Algorithm: sha384WithRSAEncryption
   [..]

**** APP SIGNING PRODUCTION CERTIFICATE

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
    Signature Algorithm: sha384WithRSAEncryption
        Issuer: O=Mozilla Corporation , OU=Mozilla Marketplace Production Signing Service , CN=root-ca-production-marketplace
        Validity
            Not Before: Feb 23 02:05:46 2013 GMT
            Not After : Feb 22 02:05:46 2018 GMT
        Subject: O=Mozilla Corporation , OU=Mozilla Marketplace Production Signing Service , CN=app-production-marketplace-1
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    [...]
                Exponent: 3 (0x3)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage: critical
                Code Signing
            X509v3 Subject Key Identifier: 
[...]
            X509v3 Authority Key Identifier: 
[...]
                DirName:/O=Mozilla Corporation /OU=Mozilla Marketplace Production Signing Service /CN=root-ca-production-marketplace
                serial:01

    Signature Algorithm: sha384WithRSAEncryption


**** ROOT CA REVIEWERS CERTIFICATE
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
    Signature Algorithm: sha384WithRSAEncryption
        Issuer: O=Mozilla Corporation , OU=Mozilla Marketplace Reviewers Signing Service , CN=root-ca-reviewers-marketplace
        Validity
            Not Before: Feb 23 01:55:31 2013 GMT
            Not After : Feb 21 01:55:31 2023 GMT
        Subject: O=Mozilla Corporation , OU=Mozilla Marketplace Reviewers Signing Service , CN=root-ca-reviewers-marketplace
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    [..]
                    5a:ff
                Exponent: 3 (0x3)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:TRUE
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Extended Key Usage: critical
                Code Signing
            X509v3 Authority Key Identifier: 
                DirName:/O=Mozilla Corporation /OU=Mozilla Marketplace Reviewers Signing Service /CN=root-ca-reviewers-marketplace
                serial:01

            X509v3 Subject Key Identifier: 
[...]
    Signature Algorithm: sha384WithRSAEncryption

**** APP SIGNING REVIEWERS CERTIFICATE
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
    Signature Algorithm: sha384WithRSAEncryption
        Issuer: O=Mozilla Corporation , OU=Mozilla Marketplace Reviewers Signing Service , CN=root-ca-reviewers-marketplace
        Validity
            Not Before: Feb 23 02:08:14 2013 GMT
            Not After : Feb 22 02:08:14 2018 GMT
        Subject: O=Mozilla Corporation , OU=Mozilla Marketplace Reviewers Signing Service , CN=app-reviewers-marketplace-1
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    [...]
                Exponent: 3 (0x3)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage: critical
                Code Signing
            X509v3 Subject Key Identifier: 
[..]
            X509v3 Authority Key Identifier: 
[..]
                DirName:/O=Mozilla Corporation /OU=Mozilla Marketplace Reviewers Signing Service /CN=root-ca-reviewers-marketplace
                serial:01

    Signature Algorithm: sha384WithRSAEncryption



If approved, the HSM world will be securely transfered from HSM1 to HSM2 (i.e. only HSM1 has the correct private keys until this transfer occurs - well, HSM1 and our security world backup [which can't be used without a quorum])

Thanks for your help so far. Hope nothing is missing and that the subject an exponentials are OK for everyone.

Comment 30

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Comment on attachment 717434 [details]
final certs

export NSS=<path to NSS>
export PATH=$NSS/lib:$NSS/bin

$ certutil -d . -N
$ certutil -d . -A -n root-ca-production-marketplace -i root-ca-production-marketplace.crt -t ",,C"
$ certutil -d . -L
$ certutil -d . -L -n root-ca-production-marketplace

Here, I see what appears to be trailing whitespace in the components of the name. i.e. "Mozilla Corporation " instead of "Mozilla Corporation". Also, I see that C=US is missing, like kang stated above. I suspect that kang's conclusion that OpenSSL has an arbitrary limit in the name length is wrong. We should figure out the real reason that OpenSSL rejected the C=US and then add it back. (IIRC, there is some maximum name length limit in the HSM generation scripts.)

$ vfychain -d . -u 6 -a app-production-marketplace-1.crt

Couldn't import app-production-marketplace-1.crt, -8054 = You are attempting to import a cert with the same issuer/serial as an existing cert, but that is not the same cert.

Because, the EE certs have serial number 1 instead of serial number 100001 or whatever we chose.

Comment 31

[Guillaume Destuynder [:kang] [this account is no longer active]]

The whitespace actually makes C (country) 3 characters, that would also trigger the openssl length limit for that very field (max 2 characters).

I'm going to generate new certificates from a test HSM,  ie not final certificates.
Those should be THOROUGHLY tested WITH the final code and libraries, as initially planned when we made the first run  of test certificates.
These tests show that this has never been done.

I can regenerate the test certificates quickly, while generating final ones takes a long time due to the key ceremony. 

Whenever those test certificates has been actually test-loaded and test signed, then we can make a final one.

Comment 32

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Guillaume Destuynder [:kang] from comment #31)
> The whitespace actually makes C (country) 3 characters, that would also
> trigger the openssl length limit for that very field (max 2 characters).
> 
> I'm going to generate new certificates from a test HSM,  ie not final
> certificates.
> Those should be THOROUGHLY tested WITH the final code and libraries, as
> initially planned when we made the first run  of test certificates.
> These tests show that this has never been done.

I agree. This requires a test app to be signed with each cert (production and reviewer).

Comment 33

[Guillaume Destuynder [:kang] [this account is no longer active]]

Attachment: TEST certificates and reports

I have added automatic NSS and OpenSSL checking to the generated certificate. I ran NSS checks with hg mozilla-central HEAD 119065:69eec5e8a477  (last commit Wed Jan 16 15:51:00 2013 -0500).

Please find attached the resulting TEST certificate (the OU has TEST in the name).

Those were generated on a real HSM, but using private keys that were generated without following the key ceremony policy. Those should otherwise be exactly identical to the final certs output.

The archive also contains quick reports (the .txt files) in order to spot issues easily.

I believe the next step is then to test sign an app with them.

Comment 34

[Guillaume Destuynder [:kang] [this account is no longer active]]

Attachment: TEST certificates and reports

If  I upload the proper archive, it might be better. Sorry.

Comment 35

[Guillaume Destuynder [:kang] [this account is no longer active]]

After speaking with :joes and :wclouser

We're going to setup (with :jlaz) the test hsm the same way as the prod hsm and attempt to test signing with trunion on it. That test will be enough for us to go on and provide a new "final" certificate (mostly due to time constraints).

We will most likely need :rtilder to help doing the actual validation testing with trunion.

Comment 36

[Guillaume Destuynder [:kang] [this account is no longer active]]

Note, to clarify, the next steps are:

- setup a test host with HSM+puppetization that is similar to the production HSM host
- use the above TEST certificates with trunion (the signing service) to sign an app
- repeat the process until the certificate works
- generate a real final certificate using the same settings
- provide the certificates to everyone

Comment 37

[Guillaume Destuynder [:kang] [this account is no longer active]]

Thanks to the awesome help of :jason and :jlaz we setup the stage-like test HSM in MTV.
We set it up with the certs from comment 34.
We then performed a test signing by following instructions from https://wiki.mozilla.org/Apps/PrivilegedApplication/SigningService

The test worked :)
The test can be reproduced by running /root/req | nc localhost 10001 for the reviewer cert (change port in the req file and nc to 10000 for prod test).

Req does the test request from https://wiki.mozilla.org/Apps/PrivilegedApplication/SigningService and the reply is the one documented:
HTTP/1.1 200 OK
Server: gunicorn/0.15.0
Date: Tue, 26 Feb 2013 03:13:06 GMT
Connection: close
Content-Type: application/json; charset=UTF-8
Content-Length: 2439

{"zigbert.rsa": "MIIHEgYJKoZIhvcNAQcCoIIHAzCCBv8CAQExCzAJBgUrDgMCGgUAMAsGCSqGSIb3DQEHAaCCBLowggS2MIIDnqADAgECAgMQAAEwDQYJKoZIhvcNAQEMBQAwgZExCzAJBgNVBAYTAlVTMRwwGgYD[...]}

Ryan, can you confirm this test procedure looks ok and *reasonably* valid to you? If so I'll generate new "final" prod certs tomorrow.

Thanks.

Comment 38

[Guillaume Destuynder [:kang] [this account is no longer active]]

Note, to clarify: if we can generate a valid app that would be good (this is _anyway_ necessary to close this bug, at some point).
Due to the time constraints, and people ability to generate final certs, we'll probably generate the "final" certs tomorrow nevertheless, as we won't be able to do so wednesday.

Comment 39

[Guillaume Destuynder [:kang] [this account is no longer active]]

I have tested the TEST cert on the device as well, following's bsmith instructions:

    Generate the new certificate according to above specifications
    Sign a test app with the new cert.
    Follow the instructions here to create a Gecko patch:
    https://mxr.mozilla.org/mozilla-central/source/security/build/b2g-certdata.mk
    Note: The instructions there are out of date. bsmith will update them
    Need to add/remove blank line at end of security/coreconf/coreconf.dep file because of broken build system dependencies.
    Test out the changes:
    touch security/coreconf/coreconf.def
    MOZCONFIG=$YOUR_MOZCONFIG make -j9 -f client.mk
    SOLO_FILE=test_signed_apps-marketplace.js make -C $OBJDIR/security/manager/ssl/tests check-one

This returned:
INFO | Result summary:
INFO | Passed: 1
INFO | Failed: 0
INFO | Todo: 0


I'm now going to generate a final certificate and produce the same test,  and submit it for review.

Comment 40

[Guillaume Destuynder [:kang] [this account is no longer active]]

Attachment: "final" prod & review certs, scripts, reports

Attached new final certs (created using the key ceremony procedure).

You'll find:
ROOT CA for Production and Reviewers (PEM and DER)
App Signing certs for Production and Reviewers (PEM)

in hsm_scripts/certs/ca:
production.txt, reviewers.txt
Those 2 files have all the certificate info, OpenSSL checks and NSS checks (all passed).

What's missing:
B2G NSS test and app signing test, and resulting patch with the new certificates (need:/data/www/{reviewer,production}-app-signer.addons.phx1.mozilla.com/keys/appstore.{key,jwk,chain} setup by Jason)

Let's hope this is the last run

Comment 41

[Jason Smith [:jsmith]]

Comment on attachment 718773 [details]
"final" prod & review certs, scripts, reports

I probably can't help with review on the certs - Ryan and Brian are your best candidates. If QA testing is needed with the certs after review+ is given, let me know.

Comment 42

[Guillaume Destuynder [:kang] [this account is no longer active]]

Comment on attachment 718773 [details]
"final" prod & review certs, scripts, reports

Sorry, autocomplete tricked me, I meant to add the other Jason. Corrected!

Comment 43

[Jason Thomas [:jason]]

(In reply to Guillaume Destuynder [:kang] from comment #40)
> What's missing:
> B2G NSS test and app signing test, and resulting patch with the new
> certificates
> (need:/data/www/{reviewer,production}-app-signer.addons.phx1.mozilla.com/
> keys/appstore.{key,jwk,chain} setup by Jason)
> 
> Let's hope this is the last run

The trunion service has been configured with the following certs on app1.hsm.phx1.svc.mozilla.com.

Comment 44

[Guillaume Destuynder [:kang] [this account is no longer active]]

Signed test app manually using production HSM. Ran tests:

kang@kang-z21:~/B2G/gecko$ export OBJDIR=/home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/
(reverse-i-search)`M': cd ^CTA-INF/
(reverse-i-search)`OS': git grep AppNeedsInherited^CPrivileges(reverse-i-search)`': ^C
kang@kang-z21:~/B2G/gecko$ SOLO_FILE=test_signed_apps-marketplace.js make -C $OBJDIR/security/manager/ssl/tests check-one
make: Entering directory `/home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/security/manager/ssl/tests'
/home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/_virtualenv/bin/python -u /home/kang/B2G/gecko/config/pythonpath.py \
	  -I/home/kang/B2G/gecko/build \
      -I../../../../_tests/mozbase/mozinfo \
	  /home/kang/B2G/gecko/testing/xpcshell/runxpcshelltests.py \
	  --symbols-path=../../../../dist/crashreporter-symbols \
	  --build-info-json=../../../../mozinfo.json \
	  --test-path=test_signed_apps-marketplace.js \
	  --testing-modules-dir=../../../../_tests/modules \
	  --profile-name=firefox \
	  --test-plugin-path=../../../../dist/plugins \
	  --verbose \
	   \
	  /home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/dist/bin/xpcshell \
	  ../../../../_tests/xpcshell/security/manager/ssl/tests/unit
TEST-INFO | profile dir is /tmp/firefox/xpcshellprofile
TEST-INFO | /home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/_tests/xpcshell/security/manager/ssl/tests/unit/test_signed_apps-marketplace.js | running test ...
TEST-INFO | /home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/_tests/xpcshell/security/manager/ssl/tests/unit/test_signed_apps-marketplace.js | full command: ['/home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/dist/bin/xpcshell', '-g', '/home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/dist/bin', '-a', '/home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/dist/bin', '-r', '/home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/dist/bin/components/httpd.manifest', '-m', '-n', '-s', '-e', 'const _HTTPD_JS_PATH = "/home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/dist/bin/components/httpd.js";', '-e', 'const _HEAD_JS_PATH = "/home/kang/B2G/gecko/testing/xpcshell/head.js";', '-e', 'const _TESTING_MODULES_DIR = "/home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/_tests/modules/";', '-f', '/home/kang/B2G/gecko/testing/xpcshell/head.js', '-p', '/home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/dist/plugins', '-e', 'const _SERVER_ADDR = "localhost"', '-e', 'const _HEAD_FILES = [];', '-e', 'const _TAIL_FILES = [];', '-e', 'const _TEST_FILE = ["/home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/_tests/xpcshell/security/manager/ssl/tests/unit/test_signed_apps-marketplace.js"];', '-e', '_execute_test(); quit(0);']
TEST-INFO | /home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/_tests/xpcshell/security/manager/ssl/tests/unit/test_signed_apps-marketplace.js | current directory: '/home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/_tests/xpcshell/security/manager/ssl/tests/unit'
TEST-INFO | /home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/_tests/xpcshell/security/manager/ssl/tests/unit/test_signed_apps-marketplace.js | environment: ['LD_LIBRARY_PATH=/home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/dist/bin', 'XPCSHELL_TEST_PROFILE_DIR=/tmp/firefox/xpcshellprofile', 'NS_TRACE_MALLOC_DISABLE_STACKS=1', 'XPCOM_MEM_LEAK_LOG=/tmp/firefox/xpcshellprofile/runxpcshelltests_leaks.log', 'XPCOM_DEBUG_BREAK=stack-and-abort', 'MOZ_CRASHREPORTER_NO_REPORT=1']
TEST-PASS | /home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/_tests/xpcshell/security/manager/ssl/tests/unit/test_signed_apps-marketplace.js | test passed (time: 716.077ms)
>>>>>>>

TEST-INFO | (xpcshell/head.js) | test 1 pending

TEST-INFO | (xpcshell/head.js) | test 2 pending

TEST-INFO | (xpcshell/head.js) | test 2 finished

TEST-INFO | (xpcshell/head.js) | running event loop

TEST-INFO | (xpcshell/head.js) | test 2 pending
TEST-INFO | /home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/_tests/xpcshell/security/manager/ssl/tests/unit/test_signed_apps-marketplace.js | Starting 

TEST-INFO | (xpcshell/head.js) | test 2 finished

TEST-INFO | /home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/_tests/xpcshell/security/manager/ssl/tests/unit/test_signed_apps-marketplace.js | openSignedJARFileCallback called for privileged-app-test-1.0

TEST-PASS | /home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/_tests/xpcshell/security/manager/ssl/tests/unit/test_signed_apps-marketplace.js | [openSignedJARFileCallback : 41] 2153390067 == 2153390067

TEST-PASS | /home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/_tests/xpcshell/security/manager/ssl/tests/unit/test_signed_apps-marketplace.js | [openSignedJARFileCallback : 42] false == false

TEST-PASS | /home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/_tests/xpcshell/security/manager/ssl/tests/unit/test_signed_apps-marketplace.js | [openSignedJARFileCallback : 43] false == false

TEST-INFO | (xpcshell/head.js) | test 2 pending

TEST-INFO | (xpcshell/head.js) | test 2 finished

TEST-INFO | (xpcshell/head.js) | test 1 finished

TEST-INFO | (xpcshell/head.js) | exiting test

TEST-PASS | (xpcshell/head.js) | 3 (+ 0) check(s) passed

TEST-INFO | (xpcshell/head.js) | 0 check(s) todo
<<<<<<<
INFO | Result summary:
INFO | Passed: 1
INFO | Failed: 0
INFO | Todo: 0
make: Leaving directory `/home/kang/B2G/gecko/obj-x86_64-unknown-linux-gnu/security/manager/ssl/tests'

Added patch to bug 845642. When this bug is closed (ie Marketplace confirms this is ok/tested) the patch can also be reviewed.

Comment 45

[Ryan Tilder [:rtilder]]

Comment on attachment 718773 [details]
"final" prod & review certs, scripts, reports

Looks good.