Generate OSX builds signed with new key, and verify update from prior build works okay.

RESOLVED FIXED

Status

enhancement
P1
normal
RESOLVED FIXED
2 years ago
a year ago

People

(Reporter: hwine, Assigned: aki)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments)

As discussed in bug 1346414, we want to demonstrate that the update process between a build signed with the current key, and a build signed with the new key, perform normally.

The key transition will affect the following products which have updates scheduled:
  Firefox 53
  Firefox 54
  Esr 52
  Thunderbird 52

Steps we believe are needed:
 - develop a plan to safely sign specific binaries with the new production key
 - work with QA & Dev to determine what sort of scenarios are needed
 - produce and deliver those updaters to QA
(Assignee)

Comment 1

2 years ago
This probably involves turning off nightly updates on balrog (pinning), changing the key, signing builds, and verifying updates before unpinning.  In the future, ideally we have automatic update tests for every nightly update before pointing users at it.
(Assignee)

Updated

2 years ago
Assignee: nobody → aki
(Assignee)

Comment 2

2 years ago
I'm currently thinking we should create a new "dmgv3" signing format:

_ Patches to specify certain mac signing server(s) with the new dmgv3 key format, but not dmgv2, in non-signing-scriptworker signing (buildbot, funsize?)
_ Create the new keys on those signing servers
_ Patch to use dmgv3 signing in the appropriate place(s): buildbot, funsize?
_ Turn off balrog nightlies/aurora, plan for testing during a non-release window
_ land/roll out patch to use dmgv3
 _ build nightly against dmgv3 format
 _ test updates to new dmgv3 signed dmg
 _ go/no-go: start over or fix if broken, otherwise proceed
_ roll out dmgv3 keys + signing configs everywhere, retire dmgv2
This might be a good opportunity to rename the poorly named "dmg" signing. We actually sign the .app, not the .dmg, and signing of the actual dmg is apparently happening. Maybe "macapp" or something like that for the new format?
(Assignee)

Comment 4

2 years ago
(In reply to Ben Hearsum (:bhearsum) from comment #3)
> This might be a good opportunity to rename the poorly named "dmg" signing.
> We actually sign the .app, not the .dmg, and signing of the actual dmg is
> apparently happening. Maybe "macapp" or something like that for the new
> format?

Hm. signingscript + signtool might be a good place to do that rename, but we could add a macapp on the buildbot side as well.
(Assignee)

Comment 5

2 years ago
Attachment #8852110 - Flags: review?(rail)
(Assignee)

Updated

2 years ago
Attachment #8852110 - Attachment description: mac-v2-signing-7_macapp.patch → [puppet] mac-v2-signing-7_macapp.patch
(Assignee)

Comment 6

2 years ago
Attachment #8852111 - Flags: review?(rail)
(Assignee)

Updated

2 years ago
Keywords: leave-open
Comment hidden (mozreview-request)

Updated

2 years ago
Attachment #8852110 - Flags: review?(rail) → review+

Updated

2 years ago
Attachment #8852111 - Flags: review?(rail) → review+

Comment 8

2 years ago
mozreview-review
Comment on attachment 8852150 [details]
bug 1346422 - switch mac signing format to macapp.  a=release

https://reviewboard.mozilla.org/r/124368/#review126928
Attachment #8852150 - Flags: review?(rail) → review+
(Assignee)

Comment 12

2 years ago
- the developer cert at [1] looks like it is valid through 2027.  I'm guessing either the key or the CSR expires soon.

    openssl x509 -in DeveloperIDCA.cer -inform der -text -noout

- bug 1346425 specifies a new Developer ID.  [2] seems to tell me this is tied to the apple id.  When I log into developer.apple.com, I'm faced with an MFA screen, which is good, but I'm uncertain how best to proceed here.  It might make sense for me to wait for someone with an MFA device and log in with that.

[1] https://hg.mozilla.org/build/puppet/file/tip/modules/signingserver/files/DeveloperIDCA.cer
[2] https://mana.mozilla.org/wiki/display/RelEng/Signing#Signing-OSXSigning-DeveloperIDcertificate

Updated

2 years ago
Priority: -- → P1
(Assignee)

Comment 13

2 years ago
Current status:
- I have access to the account now.
- I have a question for hwine and bhearsum about key rotation in https://bugzilla.mozilla.org/show_bug.cgi?id=1346425#c5 .
(Assignee)

Comment 14

2 years ago
I was able to generate a new nightly signing cert in bug 1346425.  I need to test this somewhere.

I'm currently thinking:

- land patch(es) in Oak
- test on Oak

When ready,
- generate release cert
- update patches to accept new release cert
- move a percentage of mac signing servers to new certs
- uplift needed patches to m-c
    - if paranoid, freeze balrog updates and test after building nightlies
    - unfreeze when ready
- uplift to release branches
(Assignee)

Comment 15

2 years ago
Rob, Matt, would it be ok to land https://bugzilla.mozilla.org/attachment.cgi?id=8852150 on Oak to test mac updates with the new nightly signing cert?
Flags: needinfo?(robert.strong.bugs)
Flags: needinfo?(mhowell)
Sounds good to me. I'd say you need rstrong's OK more than mine though.
Flags: needinfo?(mhowell)
(In reply to Aki Sasaki [:aki] from comment #15)
> Rob, Matt, would it be ok to land
> https://bugzilla.mozilla.org/attachment.cgi?id=8852150 on Oak to test mac
> updates with the new nightly signing cert?
That's fine with me.
Flags: needinfo?(robert.strong.bugs)
(Assignee)

Comment 18

2 years ago
https://hg.mozilla.org/projects/oak/rev/8d5a51df682a8666571ffa66d1602d2b98e4480f
bug 1346422 - switch mac signing format to macapp. r=rail a=release
(Assignee)

Comment 19

2 years ago
I installed Mac Oak Nightly (previous cert) yesterday, and updated to Mac Oak Nightly (current cert) today.

I used RB App Checker Lite [1] to verify the signature on the new Mac Oak Nightly.  We're good.
I think we're ready to test on m-c, but am open to more testing if we want.

[1] https://itunes.apple.com/us/app/rb-app-checker-lite/id519421117?mt=12
(Assignee)

Updated

2 years ago
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED

Comment 20

2 years ago
Pushed by asasaki@mozilla.com:
https://hg.mozilla.org/mozilla-central/rev/4b8939ed1281
switch mac signing format to macapp. r=rail a=release
wsmwk - FYI, if you want to unbreak mac on esr45 you need to merge https://hg.mozilla.org/releases/mozilla-esr45/rev/56539f165df3 to the THUNDERBIRD_45_VERBRANCH.
(In reply to Nick Thomas [:nthomas] from comment #21)
> wsmwk - FYI, if you want to unbreak mac on esr45 you need to merge
> https://hg.mozilla.org/releases/mozilla-esr45/rev/56539f165df3 to the
> THUNDERBIRD_45_VERBRANCH.

AFAICT that bug landed on mozilla-esr45 default, so it should be picked up by a merge to default which we typically do before a release.
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.