Closed Bug 1567235 Opened 4 months ago Closed 4 months ago

Mac notarization is slow

Categories

(Release Engineering :: Release Automation: Signing, defect)

defect
Not set

Tracking

(firefox-esr60 unaffected, firefox-esr68 fixed, firefox68 fixed, firefox69 fixed, firefox70 fixed)

RESOLVED FIXED
Tracking Status
firefox-esr60 --- unaffected
firefox-esr68 --- fixed
firefox68 --- fixed
firefox69 --- fixed
firefox70 --- fixed

People

(Reporter: mhentges, Unassigned)

References

(Depends on 1 open bug)

Details

Attachments

(1 file)

in general, looks like macosx64 signing is holding things up badly right now :(
~ RyanVM July 17th at 6:50PM PST

Sounds like Mac Notarization has significantly affected the build speed, we should investigate why it's taking longer than expected

We're going to look at:

  • chunking partner signing
  • potentially running 2 scriptworker instances per prod host, rather than 1
  • adding more hosts when we have hardware (consuming the mac signing server pool would help here; that's blocked by esr60 signing needs atm.)
  • adding nagios checks which should give us a better idea when and how much the queue gets overloaded

(In reply to Aki Sasaki [:aki] (he/him) (UTC-7) from comment #1)

  • adding more hosts when we have hardware (consuming the mac signing server pool would help here; that's blocked by esr60 signing needs atm.)

These hosts are Macmini7,1 (r7) like the mac-v3-signing pool, and can be converted over:
mac-v2-signing8.srv.releng.mdc1
mac-v2-signing9.srv.releng.mdc1
mac-v2-signing10.srv.releng.mdc1
mac-v2-signing11.srv.releng.mdc2
mac-v2-signing12.srv.releng.mdc2
mac-v2-signing13.srv.releng.mdc2

These ones are older Macmini5,3 which don't support running Mojave:
mac-v2-signing1.srv.releng.mdc1
[mac-v2-signing2.srv.releng.mdc1 is being decommed in bug 1564151]
mac-v2-signing3.srv.releng.mdc1
mac-v2-signing4.srv.releng.mdc2
mac-v2-signing5.srv.releng.mdc2
mac-v2-signing6.srv.releng.mdc2

If we just have ESR60 using mac-v2 then we can probably start the conversion process for 8-13 now. The logs are pretty quiet on mac-v2-signing1.srv.releng.mdc1.

Supporting testing for D39482:

  • taskgraph-diff shows changes in mb/mr for release phases only (not in me because params don't enable partners)
  • staging release at https://tools.taskcluster.net/groups/FsZLAXgWR-2xTqa-flc95Q (earlier code, but same graph generated). partner config set up to make sure 8 eme-free locales, and 5-10 partner combos across two configs, both get chunked properly
  • picked a partner repack and verified expected changes from repack to signing to repackage (ie builds are hooked up correctly)
Pushed by nthomas@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a2aa55fa742a
notarize 5 partner & eme-free builds per task instead of individually, r=aki

I guess the ESR branches aren't affected since we don't run these repacks on those branches.

Status: NEW → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED

mac-v3-signing{7,10,11,12,13} are now also live.

(In reply to Ryan VanderMeulen [:RyanVM] from comment #7)

I guess the ESR branches aren't affected since we don't run these repacks on those branches.

We have a very small number of repacks on esr60 and 68 so this code will have no real win, but I'll land this for consistency of the taskcluster code enabling easier uplifts later. Was only blocked yesterday by a tree closure.

https://hg.mozilla.org/releases/mozilla-esr68/rev/503689d2dec56fc9bbf47da4ae781630b5082ee9 (68.1.0esr)
https://hg.mozilla.org/releases/mozilla-esr68/rev/22b3a1995129a3234922af5bdc65ac536ed5f6e1 (68.0.2esr)

You need to log in before you can comment on or make changes to this bug.