742045 - add new known_host via OPSI/puppet

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

Build systems will now need to upload symbols to relengweb1.dmz.scl3.mozilla.com.  They'll need a known_hosts entry for that.

The public key is

dm-wwwbuild01.mozilla.org,10.2.74.128 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAqypyDVhommeNS4BH/iSk+PKGuq0pna39/xzaomq43NMoVhv+k/XNUDuwKZLc9R7mQ91VSeqOkFPziupfDliQRb7pLBMQN3hR412yUlzh7RghySnNZc1OnwYKtJ0I+Ysf4tTtxo1tcUouacnGWHHbCjpVfCgdtJeo3T/KtQjrhMk=
mobile-dashboard1.build.mtv1.mozilla.com,10.250.48.16 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwydkyMMieRxDQWoSpx1Xfl9fdZwGzi5DcxUGLK5TQHG77Qe77KGIoOomyPPC0/knz96UuuJ/vSAo/JAkWIdVFnbs6pG2wmQBW4X2t4nLUu5iiEeub0TTGN0GqLd2p48wIoEJmTlc2r/aXzDjbwxUBV4P07YtL/oyFolu02WS4avTYJs4O84LuMiBnaLX4vb4baV77L0756SnOYbZejHXiw+BH+QioYUAeWSf6NfX/RKyuvpM+xrzb8t4+Ad9289WsCAYWBkVIi1zKKwDA372jTg5Iy8u4f4RfR4cgVSJRbZGBMJdYtlrrs6EIp6Yb2QRdsTpkvkDifVD3t9mYPqvtw==

We'll need this deployed before we switch to uploading stuff to relengweb1.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

that was the wrong key (the IPs are eerily similar!)

> relengweb1.dmz.scl3.mozilla.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwmkEfrYo/5IBx0lKuEc6ZrzX7vRStEuL4XlxX8kCld0nA4J783HBTSXSSk8tlttYXsKkpgVHknGEvF9K27X09ihXxygkrfqiR8szxgKgeDMqVGFBmOKMgvWrOHX8pzA+gry3riUeWIDaEmsxSIWbO1lpaeLddQYkx2zBdGlZiqqyGJ2oBCYxnzXk/bJ/1O1O4oynLvoFQ9LWkWG75qM9DQGRDpq+KuBtJ08yn7s3ij32+1P7Y9ncSyYUezNF0dxfv5dLOaTKHBDhSc2j5ZuZ/d7XneXP66EQjZDIe5h5Rgyg0uzxQRs/akUy36mHBFIbfzAcRo/RyNUChFGgdoaVBw==

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

RelEng, how do we handle review when deploying changes to files distributed to puppet, but not in hg ? I'll need to update these files for mac/linux:

./darwin10-i386/build/local/Users/cltbld/.ssh/known_hosts
./centos5-x86_64/build/local/home/cltbld/.ssh/known_hosts
./darwin9-i386/build/local/Users/cltbld/.ssh/known_hosts
./centos5-i686/build/local/home/cltbld/.ssh/known_hosts

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

My technique was always to describe the changes textually in the attachment description.  If there's no patch, I just requested r+ in text.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [opsi]

I've added two entries for each of the new machines, the actual FQDN and the CNAME because I don't know how we're going to handle the cutover. On linux at least ssh is quiet happy to have records with different IPs for the same hostname, and I'll double check on win/mac.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [opsi] v2

I dropped dp-ausstage01.phx.mozilla.com because aus3-staging.m.o is working now.

Comment 6

[Rail Aliiev [:rail]]

Comment on attachment 612495 [details] [diff] [review]
[opsi] v2

lgtm

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

This may need relengweb1 added as well?

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 612495 [details] [diff] [review]
[opsi] v2

I dropped the dm-symbolpush01.mozilla.org line since that doesn't actually make sense (the m in dm means mpt, so we're not gonna change DNS, just swap the buildbot config).

http://hg.mozilla.org/build/opsi-package-sources/rev/392902e8e9d8

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

I've deployed the known_hosts change to 
  {,mv-,scl-,scl3-}production-puppet
updating 
  /N/production/centos5-i686/build/local/home/cltbld/.ssh/known_hosts
  /N/production/centos5-x86_64/build/local/home/cltbld/.ssh/known_hosts
  /N/production/darwin10-i386/build/local/Users/cltbld/.ssh/known_hosts
  /N/production/darwin9-i386/build/local/Users/cltbld/.ssh/known_hosts

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Somehow lost my comment from yesterday, but ...

I updated production-opsi and set v1.2 of ssh-config to 'setup' on 
 mw32-ix-slave02 through 26, except 19 and 21 which are staging but still in prod opsi
 w32-ix-slave02 through 44
 win32-ix-ref.uib.local

They've all updated except w32-ix-slave06, w32-ix-slave12, and w32-ix-slave16. The first two got turned into seamonkey machines, and the last isn't talking to opsi properly since it got reimaged (bug 720167).

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

The following slave sets (from buildbot-configs/mozilla/production_config.py) are verified updated (plus the new ssh key deployed):
MAC_SNOW_MINIS = ['moz2-darwin10-slave%02i' % x for x in range(5,10) + \
                 range(15,30) + range(40,57) if x not in (52,)] # bug683792
MAC_MINIS      = ['moz2-darwin9-slave%02i' % x for x in range(1,27) + range(38,55) \
                 if x not in (3,4,5,8,10,20,40)] # bug629763
XSERVES        = ['bm-xserve%02i' % x for x in [7,9,11,12,15,16,17,18,19,22]] # bug700705
LINUX_VMS      = ['moz2-linux-slave%02i' % x for x in [1,2] + range(5,47) \
                 if x not in (10,17)]
LINUX_IXS      = ['mv-moz2-linux-ix-slave%02i' % x for x in range(2,22)] + \
                 ['linux-ix-slave%02i' % x for x in [1,2,6] + range(12,43)]
LINUX64_VMS    = ['moz2-linux64-slave%02i' % x for x in range(1,13) \
                 if x not in (7,10)]
LINUX64_IXS    = ['linux64-ix-slave%02i' % x for x in range(3,22)]
WIN32_IXS      = ['mw32-ix-slave%02i' % x for x in range(2,16) + [20, 26]] + \
                 ['w32-ix-slave%02i' % x for x in range(24,45)]
except:
 moz2-darwin10-slave26, moz2-darwin10-slave46 - currently down
 mv-moz2-linux-ix-slave02 - loaned to rjesup

 moz2-darwin10-slave24  - decommissioned
 moz2-darwin9-slave17   - decommissioned
 moz2-darwin9-slave53   - no response, gonna die anyway
 bm-xserve18            - connection timeout during banner exchange, gonna die

These ones aren't yet:
WIN64_IXS      = ['w64-ix-slave%02i' % x for x in [2] + range(6,25)]
Missed them in the net flow request, reopened bug 742083

MOCK_DL120G7   = ['bld-centos6-hp-%03d' % x for x in range(6,24)] # 5 staging, 17 prod, 17 try
Using a different puppet server, but not doing symbols or updates yet so not a problem right now.

MAC_LION_MINIS = ['bld-lion-r5-%03d' % x for x in range(41,81)]
Doesn't appear to be managed, ie no
 scl3-production-puppet:/N/production/darwin11-x86_64/build/local/Users/cltbld/.ssh
at all (in fact nothing from the depth of Users) where the darwin10 and darwin9 dirs do have those. I think this is just a matter of copying the files over but need to confirm with puppet people.

The bld-lion-r5 are the major thing left here.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [puppet-manifests] manage ssh known_hosts and config on r5 machines

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 614171 [details] [diff] [review]
[puppet-manifests] manage ssh known_hosts and config on r5 machines

http://hg.mozilla.org/build/puppet-manifests/rev/68d54e9023e1

Deployed to {scl3,scl,mpt,mv}-p-p after setting up the files in scl3 only. The config on the slaves had /home/cltbld prefixes for the IdentityFile declarations, which I fixed to say /User/cltbld like darwin10.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Nick Thomas [:nthomas] from comment #11)
> These ones aren't yet:
> WIN64_IXS      = ['w64-ix-slave%02i' % x for x in [2] + range(6,25)]
> Missed them in the net flow request, reopened bug 742083

Waiting on the flow ...
 
> MOCK_DL120G7   = ['bld-centos6-hp-%03d' % x for x in range(6,24)] # 5
> staging, 17 prod, 17 try
> Using a different puppet server, but not doing symbols or updates yet so not
> a problem right now.

The first person who needs to upload can deal with these.

> MAC_LION_MINIS = ['bld-lion-r5-%03d' % x for x in range(41,81)]
Done now.

Just Win64 left now, and that needn't block us.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Did moz2-darwin10-slave46. Still to do moz2-darwin10-slave26, win64.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

win64 are done. I've left a comment on bug 731294 for moz2-darwin10-slave26 so lets call this done.