1346425 - Generate new Developer ID for signing Mac Desktop builds

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, enhancement, P1)

Description

[Hal Wine [:hwine] use NI!]

Current Developer ID expires on 2017-05-23 which is between the FF 53 & FF 54 releases.

We need to generate a new Developer ID and deploy it to production before then.

However, we ALSO need to test update behavior as described in bug 1346422 for the reasons given in bug 1346414.

Therefore, this bug is only for the generation of the new Developer ID, and deployment will be in a separate bug.

Comment 1

[Chris AtLee [:catlee]]

Do we know if we need a new Developer ID for Nightly/Aurora and one for Beta/Release? Or are the keys themselves separate somehow?

Comment 2

[Hal Wine [:hwine] use NI!]

(In reply to Chris AtLee [:catlee] from comment #1)
> Do we know if we need a new Developer ID for Nightly/Aurora and one for
> Beta/Release? Or are the keys themselves separate somehow?

We do need to replace both key pairs, and both certificates. (i.e. we have 2 expiring on May 23)

Comment 3

[Aki Sasaki (not active)]

New Developer ID means we should be going to https://appleid.apple.com/ to create a new account?
Or do you mean a new Developer ID Certificate per https://mana.mozilla.org/wiki/display/RelEng/Signing#Signing-OSXSigning-DeveloperIDcertificate ?

Comment 4

[Hal Wine [:hwine] use NI!]

Well, to be precise:
 1. new key pairs
 2. new certificates for that key pair

You may also want to review the DR instructions at:
  https://mana.mozilla.org/wiki/display/RelEng/OS+X+Code+Signing+Key

They point to the instructions you cited, so there _should_ be no conflicts. But if you spot any, please fix - this is a workflow we don't often execute.

Comment 5

[Aki Sasaki (not active)]

(In reply to Hal Wine [:hwine] (use NI) from comment #4)
> Well, to be precise:
>  1. new key pairs
>  2. new certificates for that key pair

Hal: afaict, https://mana.mozilla.org/wiki/display/RelEng/Signing#Signing-OSXSigning-DeveloperIDcertificate only describes how to request a new Developer ID Certificate from an existing key.  I'm trying to find if/where the keypair expires and am not finding it; it looks like the keypair has no expiration.  Are you assuming the keypair needs to be rotated due to expiration, or requesting it because it's better security to rotate the keypair?

Ben: do you have any opinions about the above? If we were to create a new keypair, would I proceed with creating a self-signed keypair through openssl and then import to the keychain, and then proceed with the above DeveloperIDcertificate how to?

I do think I need to amend my patch in bug 1346422 to possibly update these locations...

https://dxr.mozilla.org/mozilla-central/source/browser/app/macbuild/Contents/Info.plist.in
https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/updater/macbuild/Contents/Info.plist.in

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Aki Sasaki [:aki] from comment #5)
> (In reply to Hal Wine [:hwine] (use NI) from comment #4)
> > Well, to be precise:
> >  1. new key pairs
> >  2. new certificates for that key pair
> 
> Hal: afaict,
> https://mana.mozilla.org/wiki/display/RelEng/Signing#Signing-OSXSigning-
> DeveloperIDcertificate only describes how to request a new Developer ID
> Certificate from an existing key.  I'm trying to find if/where the keypair
> expires and am not finding it; it looks like the keypair has no expiration. 
> Are you assuming the keypair needs to be rotated due to expiration, or
> requesting it because it's better security to rotate the keypair?
> 
> Ben: do you have any opinions about the above? If we were to create a new
> keypair, would I proceed with creating a self-signed keypair through openssl
> and then import to the keychain, and then proceed with the above
> DeveloperIDcertificate how to?

I think it's probably good to rotate the key if we're already doing the certificate.

It looks like it doesn't really matter how the keypair is created. On my local mac, Keychain Access will let me specify one, but if I don't, it will create one for me and stuff it in the "login" Keychain. Probably it's best to create it by hand, simply so we can control the algorithms, and put it into a blank Keychain from the start.

Note that there's no concept of "signing" on the keypair - they just exist. It's the Developer ID certificate that we get back from Apple (after giving them a CSR created from our keypair) that gets signed.

> I do think I need to amend my patch in bug 1346422 to possibly update these
> locations...
> 
> https://dxr.mozilla.org/mozilla-central/source/browser/app/macbuild/Contents/Info.plist.in
> https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/updater/macbuild/Contents/Info.plist.in

Yeah, I think so. Stephen Pohl is the resident expert on that pinning, might be worth pinging him.

Comment 7

[Hal Wine [:hwine] use NI!]

:hwine agrees with :bhearsum -- there is no value that I can think of in re-using the existing keypair -- validity is controlled by the certificate.

Comment 8

[Aki Sasaki (not active)]

There are no instructions on how to generate / export the key, just the developer ID.
I can play with it a bit more, and then I might lean towards having it generate its own key (if I nuke the login keychain?)
Not entirely clear here.

If either of you know more than the documentation says, or if I'm missing something, please let me know.  The new keypair seems like an undocumented and non-hard-blocker requirement, and the expiration date is coming closer.

Comment 9

[Aki Sasaki (not active)]

- Using mac-v2-signing7, which has been pulled out of the pool. logs only show token requests.
- vpn; finder -> go -> connect to server -> vnc://mac-v2-signing7
    - cltsign/pw to connect
    - I've been scrolling all the way right and choosing "Other", then cltsign/pw to log in, but then it looks like cltsign is named 'Builder' even though it's not cltbld, so maybe that works
        - yes
        - this is where copy/paste stops working, and I have to type in all the passphrases in manually over vnc, which makes me stabby
- open keychain access
- try to back up <key> keypair from login keychain
    - successful with public pem
    - unsuccessful with private p12
        - successful the 2nd private p12 export attempt; evidently i mistyped the login pw
- look into generating new keypair via gpg
- searching makes me think openssl instead
    - https://rietta.com/blog/2012/01/27/openssl-generating-rsa-key-from-command/ shows how
        - trying to import the privkey.pem into a new keychain gives "an error has occurred. unable to import an item. the contents of this item cannot be retrived."
            - didn't ask for passphrase?
    - the above export gives a p12 instead of a pem; maybe that's it?
        - this looks to be a pkcs#12 file
        - pkcs#12 file generation seems to involve a cert http://help.globalscape.com/help/secureserver3/Generating_a_PKCS_12_private_key_public_certificate.htm

- decide that guessing how to generate a proper key by googling and guessing is not ideal
    - looks like i can reset the keychain. is this a better solution? https://kb.mit.edu/confluence/display/istcontrib/Create+a+new+login+or+reset+Keychain+for+Mac+OS+X

Comment 10

[[e-mail] (:bhearsum)]

(In reply to Aki Sasaki [:aki] from comment #9)
> - Using mac-v2-signing7, which has been pulled out of the pool. logs only
> show token requests.
> - vpn; finder -> go -> connect to server -> vnc://mac-v2-signing7
>     - cltsign/pw to connect
>     - I've been scrolling all the way right and choosing "Other", then
> cltsign/pw to log in, but then it looks like cltsign is named 'Builder' even
> though it's not cltbld, so maybe that works
>         - yes
>         - this is where copy/paste stops working, and I have to type in all
> the passphrases in manually over vnc, which makes me stabby
> - open keychain access
> - try to back up <key> keypair from login keychain
>     - successful with public pem
>     - unsuccessful with private p12
>         - successful the 2nd private p12 export attempt; evidently i
> mistyped the login pw
> - look into generating new keypair via gpg
> - searching makes me think openssl instead
>     -
> https://rietta.com/blog/2012/01/27/openssl-generating-rsa-key-from-command/
> shows how
>         - trying to import the privkey.pem into a new keychain gives "an
> error has occurred. unable to import an item. the contents of this item
> cannot be retrived."
>             - didn't ask for passphrase?
>     - the above export gives a p12 instead of a pem; maybe that's it?
>         - this looks to be a pkcs#12 file
>         - pkcs#12 file generation seems to involve a cert
> http://help.globalscape.com/help/secureserver3/
> Generating_a_PKCS_12_private_key_public_certificate.htm
> 
> - decide that guessing how to generate a proper key by googling and guessing
> is not ideal
>     - looks like i can reset the keychain. is this a better solution?
> https://kb.mit.edu/confluence/display/istcontrib/
> Create+a+new+login+or+reset+Keychain+for+Mac+OS+X

Sorry for the lack of documentation here, but it sounds like you're on the right track. I tried to dig up information in old bugs, but it looks like this was simply never written down.

IIRC, you *do* need to generate a key with OpenSSL ("openssl genrsa"), then create the CSR by hand ("openssl req"). Once you submit that to Apple and get back the Developer ID certificate, you can create the p12 from the key+cert. I did a quick test of this locally and AFAICT there's nothing special about the CSR created from Keychain Access vs one I created by hand.

Comment 11

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #9)
>         - this is where copy/paste stops working, and I have to type in all
> the passphrases in manually over vnc, which makes me stabby

Locally,

System Preferences -> General -> Allow Handoff between this Mac and your iCloud devices (unclick this box!)

The login screen still requires me to type in the password, but once I'm logged in, Screen Sharing allows me to copy/paste!!

Comment 12

[Aki Sasaki (not active)]

To enable copy/paste,

(locally) System Preferences -> General -> Allow Handoff between this Mac and your iCloud devices (unclick this box!)

- Using mac-v2-signing7, which has been pulled out of the pool. logs only show token requests.
- vpn; finder -> go -> connect to server -> vnc://mac-v2-signing7
    - cltsign/pw to connect
    - Builder/pw to log in; also can scroll all the way to the right to choose "Other" to type in cltsign/pw
- open keychain access
- try to back up <key> keypair from login keychain
    - successful with public pem
    - successful with private p12; use login pw
        - there will be another pw to save it
- create new pvt key
    - `openssl genrsa -des3 -out private.pem 2048`
    - `openssl rsa -in private.pem -outform PEM -pubout -out public.pem`
    - visually inspect the above pem files
- create new csr
    - `openssl req -new -key private.pem -out my_request.csr`
        - previous instructions under `Create a CSR` here: https://mana.mozilla.org/wiki/display/RelEng/Signing#Signing-OSXSigning-DeveloperIDcertificate
        - compared against the GUI-generated csr; best I could do was `C=US, CN=MozillaCorporation/emailAddress=[e-mail]`
        - did so via cmdln to specify the privkey to use
    - verify with `openssl req -in my_request.csr -noout -text`
- get developer ID per https://mana.mozilla.org/wiki/display/RelEng/Signing#Signing-OSXSigning-DeveloperIDcertificate
- create a keychain per https://mana.mozilla.org/wiki/display/RelEng/Signing#Signing-OSXSigning-DeveloperIDcertificate
    - should this keychain have a pw?  Should it match the private key pw?
    - drag developer id in, works!  Didn't ask for a pw
    - drag private pem in, fails!
    - try to create a pkcs12 file -- should i include more than one cert?  intermediate cert?
    - `openssl pkcs12 -export -out certificate.pfx -inkey private.pem -in developerID_application.cer`
    - try to import certificate.pfx, fails!
- debug by adding nightly signing.keychain to Keychain Access
    - leaving things here til tomorrow, but if you remember anything, please let me know?



# Links
- https://developer.apple.com/library/content/documentation/IDEs/Conceptual/AppDistributionGuide/MaintainingCertificates/MaintainingCertificates.html
- https://developer.apple.com/library/content/documentation/Security/Conceptual/CodeSigningGuide/Introduction/Introduction.html
- https://developer.apple.com/library/content/documentation/IDEs/Conceptual/AppDistributionGuide/MaintainingCertificates/MaintainingCertificates.html#//apple_ref/doc/uid/TP40012582-CH31-SW15
- https://developer.apple.com/library/content/documentation/IDEs/Conceptual/AppDistributionGuide/MaintainingCertificates/MaintainingCertificates.html#//apple_ref/doc/uid/TP40012582-CH31-SW6
- https://developer.apple.com/library/content/documentation/IDEs/Conceptual/AppDistributionGuide/MaintainingCertificates/MaintainingCertificates.html#//apple_ref/doc/uid/TP40012582-CH31-SW34
- https://developer.apple.com/library/content/documentation/IDEs/Conceptual/AppDistributionGuide/MaintainingCertificates/MaintainingCertificates.html#//apple_ref/doc/uid/TP40012582-CH31-SW29
- https://developer.apple.com/library/content/documentation/Security/Conceptual/SecTransformPG/SigningandVerifying/SigningandVerifying.html
- https://www.ssl.com/how-to/create-a-pfx-p12-certificate-file-using-openssl/

Comment 13

[Aki Sasaki (not active)]

https://hg.mozilla.org/projects/pine/rev/df0cf9a32b4bca1a7d3e24b0db7acc962fb3891b
bug 1346425 - turn off extraneous tasks on pine.

Comment 14

[Aki Sasaki (not active)]

I haven't had time to focus on key generation the past few work days because of releaseduty and mergeduty. Hopefully I get time this week, otherwise I'll have to get back to it after the mergeday/rc/release/beta1 tasks quiet down.

Comment 15

[Aki Sasaki (not active)]

I decided to give up on manual key generation and try nuking / resetting the login keychain to get an auto-generated key.

http://kb.mit.edu/confluence/display/istcontrib/Create+a+new+login+or+reset+Keychain+for+Mac+OS+X has some steps, which I repeated several times because it looked like the keychain kept getting restored.  When I exported the pubkey it appeared to be different, so it may have worked.

I exported the pub and priv keys (exporting/importing from the keychain seems to resist copy/paste, grr).  Then I was going to proceed, but then I decided to try to import the privkey in a new keychain to make sure it works, first.  Of course, since I had just exported it, right?

I was hitting errors trying to import the priv + pub keys exported from the login keychain, into a new signing.keychain.  So I've ruled out bad openssl commands, I believe.  "An error has occurred. Unable to import an item. The contents of this item cannot be retrieved."

!!!
http://stackoverflow.com/a/18319568
    security import priv_key.p12 -k ~/Library/Keychains/login.keychain
    security import pub_key.pem -k ~/Library/Keychains/login.keychain
^^^^^ This works!!  Replacing {pub,priv}_key.{pem,p12} with the filename of the keys, and replacing the last arg with the full path to the keychain.

Comment 16

[Aki Sasaki (not active)]

Latest:

We hit the max of 5 developer IDs.  We're in contact with Apple to try to either revoke some, or add additional.

Comment 17

[Aki Sasaki (not active)]

Latest:

- Apple gave us 2 additional Developer ID slots, because of urgency.  This unblocks me.
- hwine is in contact with Apple to revoke the 3 Developer IDs that are no longer under our control.
- I generated a new nightly cert, and was able to switch mac-v2-signing7 over.  If we want to use this nightly key, we can switch our signing format to 'macapp' instead of 'dmg' in-tree.

Pinged the Oak folk and would like to switch Oak over to macapp signing.
If and when we think this is good, I should generate a new release cert and do the same thing, then start switching signing servers and trees over to 'macapp' instead of 'dmg' signing.

Comment 18

[Aki Sasaki (not active)]

Attachment: nightly-cer.txt

Output from `openssl x509 -inform der -in developerID_application.cer -noout -text`

Comment 19

[Aki Sasaki (not active)]

Output from `openssl dgst -sha1 developerID_application.cer`:
SHA1(developerID_application.cer)= 31680f7404a844874e8f7ff007554678a4c1d0c4

Comment 20

[Aki Sasaki (not active)]

Attachment: prev nightly cer vs new nightly cer

Output from `diff -U3 ~/Desktop/prev.txt ~/Desktop/nightly-cer.txt`

Comment 21

[Aki Sasaki (not active)]

Attachment: release-cer.txt

Output from `openssl x509 -inform der -in developerID_application.cer -noout -text` attached.

Output from `openssl dgst -sha1 developerID_application.cer` :
SHA1(developerID_application.cer)= 266aa401a13906b0423b5332364f840587fd7a36

Comment 22

[Aki Sasaki (not active)]

Attachment: prev release cer vs new release cer

Output from `diff -U3 ~/Desktop/old_release.txt ~/Desktop/release-cer.txt`