1500323 - decomm partner-repack-1.srv.releng.mdc{1,2}, mac-v2-signing{1,2,3}.srv.releng.mdc1, mac-v2-signing{4,5,6}.srv.releng.mdc2

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Hardware, enhancement)

Description

[Nick Thomas [:nthomas] (UTC+12)]

We no longer use mac minis to create one-off partner builds so we can decommission these two machines. No credentials need removal/scrubbing.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Actually .... catlee, do we want to repurpose these as something else ?

Comment 2

[Maryann Cristofi [:mcristofi]]

(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #0)
> We no longer use mac minis to create one-off partner builds so we can
> decommission these two machines. No credentials need removal/scrubbing.

Nick, do you want us to decom them still or should we wait per catlee, if you want to repurpose them?

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

I'd like to wait for catlee's input so I'll move this out of your queue until then.

Comment 4

[Kendall Libby [:fubar] (he/him)]

If RelEng is done with them, we'll probably have a use for them. :-)

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

catlee doesn't have any strong opinions about this. Aki, do you know if we'd benefit from an extra mac signing server in mdc[12] ?

Comment 6

[Aki Sasaki (not active)]

- Datadog currently looks decent for signing times on mac.
- I wouldn't mind more mac signing bandwidth, but
  a) it does look like we have some OSX test bandwidth issues, and
  b) we may have other signing time optimizations we can still make, e.g. preferring mdc1 from usw2 and mdc2 from use1.

If I am correct in thinking the OSX test pool is overloaded, it might make sense to decomm/use for the test pool. If the test pool is in good shape, I'd be happy with extra mac signing servers.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Apparently these are r5 machines and wouldn't match the r7 test pool, and extra signing capacity is likely to help, so lets repurpose these as mac-signing-v2's.

partner-repack-1.srv.releng.mdc1  -->  mac-v2-signing14.srv.releng.mdc1
partner-repack-1.srv.releng.mdc2  -->  mac-v2-signing15.srv.releng.mdc2

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

To Do
* DNS: add new hostnames, remove partner-repack-1's
* Puppet: remove partner-repack-1's at https://dxr.mozilla.org/build-central/source/puppet/manifests/moco-nodes.pp#179
* Reimage/re-puppetize
* Set up signing server processes (shortlist people)
* Puppet: add to signing pools

Comment 9

[Aki Sasaki (not active)]

We probably want to do something with these boxes related to mac notarization.

Comment 10

[Aki Sasaki (not active)]

Wontfix? We're retiring the signing servers; we're using partner-repack-1.mdc1 for notarization dev work, and partner-repack-1.mdc2 is sitting idle.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Makes sense. We should decomm both partner-repack-1 machines at some point when we're done with dev for notarization.

Comment 12

[Aki Sasaki (not active)]

We've retired the mac signing pool. I think there's something preventing us from using these as prod/dep notarization boxes? If so, we may want to retire them.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

mac-v2-signing{1,2,3,4,5,6} are also r5 minis that we can't use for mac-v3-signing. They can be decommissioned/repurposed too.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

I've purged secrets from mac-v2-signing{1,3,4,5,6}, and disabled puppet with launchctl. Can't reach mac-v2-signing2 because it's sick (bug 1564151). None of these are in nagios any more.

Over to relops to organise the rest of the decomm (disk wipe etc).

Comment 16

[Jake Watkins [:dividehex]]

I've shutdown mac-v2-signing{1,3,4,5,6} since they were still hitting the puppetmasters and possible causing OOM due to launchd's over zealous retries. Next step is to remove them from DNS and file a ticket with qst to remove the hardware from the racks.

Comment 18

[Jake Watkins [:dividehex]]

Inventory is missing all but mac-v2-signing2 and :rtucker believes the entries might have been removed during a recent hardware audit. The hardware definitely exists since I just logged in and shut them all down.

Comment 19

[Greg Cox [:gcox]]

(In reply to Jake Watkins [:dividehex] from comment #18)

Inventory is missing all but mac-v2-signing2 and :rtucker believes the entries might have been removed during a recent hardware audit. The hardware definitely exists since I just logged in and shut them all down.

Back in bug 1494066, 6 'depsigning' minis were re-DNS'ed to mac-v2-signing{1,6}. For whatever reason, the inventory entries were not updated (except for number 2). Shout-out to :bhearsum for even calling out "Do we need to rename anything anywhere to avoid future confusion" back then. (Whoops.)

I updated inventory entries for those hosts... just in time for y'all to kill them.

Comment 20

[Jake Watkins [:dividehex]]

Thanks, Greg for finding those!

I've removed DNS records for mac-v2-signing{1,3,4,5,6} and partner-repack-1.srv.releng.mdc{1,2}.