Mac notarization is slow
Categories
(Release Engineering :: Release Automation: Signing, defect)
Tracking
(firefox-esr60 unaffected, firefox-esr68 fixed, firefox68 fixed, firefox69 fixed, firefox70 fixed)
| Tracking | Status | |
|---|---|---|
| firefox-esr60 | --- | unaffected |
| firefox-esr68 | --- | fixed |
| firefox68 | --- | fixed |
| firefox69 | --- | fixed |
| firefox70 | --- | fixed |
People
(Reporter: mhentges, Unassigned)
References
Details
Attachments
(1 file)
in general, looks like macosx64 signing is holding things up badly right now :(
~ RyanVM July 17th at 6:50PM PST
Sounds like Mac Notarization has significantly affected the build speed, we should investigate why it's taking longer than expected
|
||
Comment 1•5 years ago
•
|
||
We're going to look at:
- chunking partner signing
- potentially running 2 scriptworker instances per prod host, rather than 1
- adding more hosts when we have hardware (consuming the mac signing server pool would help here; that's blocked by esr60 signing needs atm.)
- adding nagios checks which should give us a better idea when and how much the queue gets overloaded
|
||
Comment 2•5 years ago
|
||
(In reply to Aki Sasaki [:aki] (he/him) (UTC-7) from comment #1)
- adding more hosts when we have hardware (consuming the mac signing server pool would help here; that's blocked by esr60 signing needs atm.)
These hosts are Macmini7,1 (r7) like the mac-v3-signing pool, and can be converted over:
mac-v2-signing8.srv.releng.mdc1
mac-v2-signing9.srv.releng.mdc1
mac-v2-signing10.srv.releng.mdc1
mac-v2-signing11.srv.releng.mdc2
mac-v2-signing12.srv.releng.mdc2
mac-v2-signing13.srv.releng.mdc2
These ones are older Macmini5,3 which don't support running Mojave:
mac-v2-signing1.srv.releng.mdc1
[mac-v2-signing2.srv.releng.mdc1 is being decommed in bug 1564151]
mac-v2-signing3.srv.releng.mdc1
mac-v2-signing4.srv.releng.mdc2
mac-v2-signing5.srv.releng.mdc2
mac-v2-signing6.srv.releng.mdc2
If we just have ESR60 using mac-v2 then we can probably start the conversion process for 8-13 now. The logs are pretty quiet on mac-v2-signing1.srv.releng.mdc1.
|
|
||
Comment 3•5 years ago
|
||
|
|
||
Comment 4•5 years ago
|
||
Supporting testing for D39482:
- taskgraph-diff shows changes in mb/mr for release phases only (not in me because params don't enable partners)
- staging release at https://tools.taskcluster.net/groups/FsZLAXgWR-2xTqa-flc95Q (earlier code, but same graph generated). partner config set up to make sure 8 eme-free locales, and 5-10 partner combos across two configs, both get chunked properly
- picked a partner repack and verified expected changes from repack to signing to repackage (ie builds are hooked up correctly)
Pushed by nthomas@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/a2aa55fa742a notarize 5 partner & eme-free builds per task instead of individually, r=aki
|
|
||
Comment 6•5 years ago
|
||
| uplift | ||
|
||
Comment 7•5 years ago
|
||
I guess the ESR branches aren't affected since we don't run these repacks on those branches.
|
||
Comment 8•5 years ago
|
||
| bugherder | ||
|
|
||
Comment 9•5 years ago
|
||
mac-v3-signing{7,10,11,12,13} are now also live.
|
|
||
Comment 10•5 years ago
|
||
| uplift | ||
(In reply to Ryan VanderMeulen [:RyanVM] from comment #7)
I guess the ESR branches aren't affected since we don't run these repacks on those branches.
We have a very small number of repacks on esr60 and 68 so this code will have no real win, but I'll land this for consistency of the taskcluster code enabling easier uplifts later. Was only blocked yesterday by a tree closure.
https://hg.mozilla.org/releases/mozilla-esr68/rev/503689d2dec56fc9bbf47da4ae781630b5082ee9 (68.1.0esr)
https://hg.mozilla.org/releases/mozilla-esr68/rev/22b3a1995129a3234922af5bdc65ac536ed5f6e1 (68.0.2esr)
Description
•