1049546 - please re-image two machines as 10.9.0 for signing

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[[e-mail] (:bhearsum)]

Per e-mail, we're going to use https://inventory.mozilla.org/en-US/systems/show/10993/ and https://inventory.mozilla.org/en-US/systems/show/4558/. Let's go with mac-v2-signing[12] for names. The currenty r5-mini-002 will become mac-v2-signing3, but only after I finish my manual testing on it, so we can track that in a follow-up bug or whatever.

They should be in srv.releng.scl3.mozilla.com. I assume we'll need a separate bug for netflows - I might need a hand in figuring out the right way to file that.

These won't puppetize yet - I'll be adding them to the nodes file as soon as I sort out the password hashes.

Comment 1

[[e-mail] (:bhearsum)]

Puppet has been updated - these should puppetize correctly when they come up.

Comment 2

[Amy Rich [:arr] [:arich]]

I turned mac-signing1 into mac-v2-signing1 and reimaged it.
I turned the casper machine into mac-v2-signign2, but I think that the netboot image we're using for the install server can't support hardware that new. When I tried to image it, it just never came back online.

For expediency sake, I recommend using one of the snowleopard machines if we need 3 of these asap. 

I'll be on PTO till mid next week, so I'll hand this off to dustin to wrangle (either getting a new netboot image that works with the 6,2 hardware or getting a snow machine moved and reimaged) while I'm out.

Comment 3

[[e-mail] (:bhearsum)]

Let's repurpose t-snow-r4-0166 -- it's shut down buildbot-wise and ready to be reimaged. I've updated buildbot-configs and slavealloc - haven't touched inventory or anything like that.

Comment 4

[[e-mail] (:bhearsum)]

OK, I updated inventory and filed bug 1049859 for the netflows. Dustin said he'll fill in the details there.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

So, to be clear on what's where:

mac-v2-signing1.srv.releng.scl3.mozilla.com has address 10.26.48.33
  old mac-signing1
  https://inventory.mozilla.org/en-US/systems/show/4558/

mac-v2-signing2.srv.releng.scl3.mozilla.com has address 10.26.49.25
  old casper host; didn't image
  https://inventory.mozilla.org/en-US/systems/show/10993/

mac-v2-signing3.srv.releng.scl3.mozilla.com has address 10.26.49.19
  old t-snow-r4-0166
  https://inventory.mozilla.org/en-US/systems/show/3951/

We're married to those IPs now that we've requested flows, so there are two remaining steps:

1. Get t-snow-r4-0166 moved to the srv VLAN, adjust inventory, and reimage (ASAP)
2. Adjust r5-mini-002's inventory to turn it into mac-v2-signing2 with the IP above, and reimage (Ben, let me know when you're done with it and I can take care of this).

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

and,

1.5 move the current mac-v2-signing2 to some temporary name in inventory, with notes, so as not to clash or confuse the issue.  I'll take care of this now.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Step 1.5 is complete, and step 1 is pending completion of bug 1049934.

I added the host to install.build by MAC address, so it should image right up once dcops netboots it in the new VLAN.

Comment 8

[david garvey:dgarvey]

Hi,

Please see bug 1049956. I commented this out in svn, I hope you don't mind:


dgarvey@dgarvey-mozilla:~/svn/sysadmins/puppet/trunk/modules/nagios$ svn diff -r PREV:HEAD manifests/releng/scl3.pp
Index: manifests/releng/scl3.pp
===================================================================
--- manifests/releng/scl3.pp	(revision 91641)
+++ manifests/releng/scl3.pp	(revision 91643)
@@ -1756,12 +1756,12 @@
                 'mac-signing-servers'
             ]
         },
-        'mac-v2-signing2.srv.releng.scl3.mozilla.com' => {
-            contact_groups => 'build',
-            hostgroups => [
-                'mac-signing-servers'
-            ]     
-        },    
+#        'mac-v2-signing2.srv.releng.scl3.mozilla.com' => {
+#            contact_groups => 'build',
+#            hostgroups => [
+#                'mac-signing-servers'
+#            ]     
+#        },    
         'mac-signing2.srv.releng.scl3.mozilla.com' => {
             contact_groups => 'build',
             hostgroups => [
dgarvey@dgarvey-mozilla:~/svn/sysadmins/puppet/trunk/modules/nagios$

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

OK, mac-v2-signing{1,3} are up and in nagios.  mac-v2-signing2 is still commented out in nagios because it's still r5-mini-005 (pending step 2).

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

And via bug 1050522, I've enabled NRPE to actually work for these two hosts.  The 'signing server' check for both is critical.  I've acked it.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

I renamed r5-mini-002's system and SREG records to mac-v2-signing2 in inventory.  It was already renamed in DS.  I'll add nagios once the reimage is complete.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

The reimage actually worked :D

Added to nagios.  Someone with signing secrets will need to bring this up.