518652 - rollout puppet on production mac machines

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

This bug is to track to the deployment of Puppet on the production mac machines (moz2-darwin9-* and try-mac-*).

Comment 1

[[e-mail] (:bhearsum)]

Attachment: moz2-darwin9-* deployment script

Comment 2

[[e-mail] (:bhearsum)]

Attachment: better version, using the right plist file

Comment 3

[[e-mail] (:bhearsum)]

Attachment: deployment script, try version

Comment 4

[[e-mail] (:bhearsum)]

Deployment instructions here: https://wiki.mozilla.org/User:Bhearsum/Scratchpad

Comment 5

[[e-mail] (:bhearsum)]

(In reply to comment #0)
> This bug is to track to the deployment of Puppet on the production mac machines
> (moz2-darwin9-* and try-mac-*).

Nick points out that many xserves need including here, specifically: 11, 12, 16, 17, 18, 19, 22.

Comment 6

[John Ford [:jhford] CET/CEST Berlin Time]

this didn't work for me on moz2-darwin9-slave11.  I assume that I'd want to be using production-puppet not staging-puppet.  Is this the case?  Any ideas on what is going wrong?

[jhford@maemo-flashing ~]$ ssh [e-mail]
The authenticity of host 'moz2-darwin9-slave11.build (10.2.71.92)' can't be established.
RSA key fingerprint is 32:bd:ba:d6:d7:b7:0b:eb:ec:ff:fe:b0:be:f9:a2:a3.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'moz2-darwin9-slave11.build,10.2.71.92' (RSA) to the list of known hosts.
Password:
Password:
Password:
Last login: Thu Sep 24 06:15:04 2009
moz2-darwin9-slave11:~ cltbld$ wget --no-check-certificate -Opuppet-deploy.sh https://bug518652.bugzilla.mozilla.org/attachment.cgi?id=402641
--2009-09-24 18:40:34--  https://bug518652.bugzilla.mozilla.org/attachment.cgi?id=402641
Resolving bug518652.bugzilla.mozilla.org... 63.245.209.86
Connecting to bug518652.bugzilla.mozilla.org|63.245.209.86|:443... connected.
WARNING: cannot verify bug518652.bugzilla.mozilla.org's certificate, issued by `/C=US/O=Equifax/OU=Equifax Secure Certificate Authority':
  Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 200 OK
Length: 702 [text/plain]
Saving to: `puppet-deploy.sh'

100%[=======================================>] 702         --.-K/s   in 0s     

2009-09-24 18:40:35 (167 MB/s) - `puppet-deploy.sh' saved [702/702]

moz2-darwin9-slave11:~ cltbld$ chmod +x puppet-deploy.sh
moz2-darwin9-slave11:~ cltbld$ sudo ./puppet-deploy.sh
Password:
Sorry, try again.
Password:
Sorry, try again.
Password:
sudo: 2 incorrect password attempts
moz2-darwin9-slave11:~ cltbld$ sudo ./puppet-deploy.sh
Password:
The authenticity of host 'bm-xserve05.build.mozilla.org (10.2.71.61)' can't be established.
RSA key fingerprint is 91:21:2f:36:55:b2:2d:f4:62:21:40:bf:33:9a:1a:2d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'bm-xserve05.build.mozilla.org' (RSA) to the list of known hosts.
Password:
puppet-0.24.8.gem                             100% 1075KB   1.1MB/s   00:00    
Bulk updating Gem source index for: http://gems.rubyforge.org
Successfully installed facter-1.5.6
1 gem installed
Installing ri documentation for facter-1.5.6...
Installing RDoc documentation for facter-1.5.6...
Could not find main page README
Could not find main page README
Could not find main page README
Could not find main page README
Bulk updating Gem source index for: http://gems.rubyforge.org
Successfully installed puppet-0.24.8
1 gem installed
Installing ri documentation for puppet-0.24.8...
Installing RDoc documentation for puppet-0.24.8...
Could not find main page README
Could not find main page README
Could not find main page README
Could not find main page README
The authenticity of host 'staging-puppet.build.mozilla.org (10.2.71.91)' can't be established.
RSA key fingerprint is 56:fa:76:dd:d1:87:3c:e1:2f:a2:41:c7:f7:29:f2:4e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'staging-puppet.build.mozilla.org,10.2.71.91' (RSA) to the list of known hosts.
Connection closed by 10.2.71.91
[e-mail]'s password: 
auto_master                                   100%  278     0.3KB/s   00:00    
[e-mail]'s password: 
com.reductivelabs.puppet.plist-staging        100% 1021     1.0KB/s   00:00    
mount: /N: unknown special file or file system.

Comment 7

[[e-mail] (:bhearsum)]

Attachment: production deployment script, build network

Comment 8

[[e-mail] (:bhearsum)]

Attachment: deployment script, try version

Comment 9

[[e-mail] (:bhearsum)]

John volunteered for this.

Comment 10

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: manifests patch for new slaves

update to add production mac machines to site-production.pp

Comment 11

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: manifests patch for new slaves + try slaves

i didn't add the try slaves in the original patch

Comment 12

[[e-mail] (:bhearsum)]

Comment on attachment 403568 [details] [diff] [review]
manifests patch for new slaves + try slaves

changeset:   54:b0e277a19358

Comment 13

[John Ford [:jhford] CET/CEST Berlin Time]

Looking at bm-xserve19, it looks like this hasn't gone horribly wrong.  I am now going to do the same on bm-xserve17 and moz2-darwin9-slave02 for verification.

Comment 14

[John Ford [:jhford] CET/CEST Berlin Time]

Rolled out to 11,12,16,18.

Comment 15

[John Ford [:jhford] CET/CEST Berlin Time]

That last comment should read bm-xserveN.  I have rolled out to 22 as well now.  As it stands, bm-xserve11,12,16,17,18,19,22 have puppet.  19 is still in staging.

Comment 16

[John Ford [:jhford] CET/CEST Berlin Time]

deployed on moz2-darwin9-slave05,06,12,13,15,17,18

Comment 17

[John Ford [:jhford] CET/CEST Berlin Time]

deployed on moz2-darwin9-slave07,09,10,11,14,17  This should be the last of the production ones.  I am going to start on the staging try slaves next

Comment 18

[John Ford [:jhford] CET/CEST Berlin Time]

try-mac-slave01,03,06 are completely done.

try-mac-slave11,15,16,17,19 have the software set up but require the puppetca --sign step and reloading of the service.

try-mac-slave02,04,06,08,10,12,13,14,18 have not been touched yet

Comment 19

[[e-mail] (:bhearsum)]

(In reply to comment #17)
> deployed on moz2-darwin9-slave07,09,10,11,14,17  This should be the last of the
> production ones.  I am going to start on the staging try slaves next

you missed 16 :-)

Comment 20

[John Ford [:jhford] CET/CEST Berlin Time]

moz2-darwin9-slave16 is set up.

try-mac-slave01-04,05-08,10-19 are set up and back in the pool

Comment 21

[[e-mail] (:bhearsum)]

Attachment: deployment script for build network, final version

Comment 22

[[e-mail] (:bhearsum)]

Attachment: deployment script for try, final version

Comment 23

[[e-mail] (:bhearsum)]

Attachment: deployment script for build network, including sleep-and-run-puppet.sh

Comment 24

[[e-mail] (:bhearsum)]

Attachment: deployment script for try, including sleep-and-run-puppet.sh