827079 - Setup SeaMonkey PuppetMaster

Status: RESOLVED FIXED

(SeaMonkey :: Release Engineering, defect)

Description

[Justin Wood (:Callek)]

This is to track setting up the SeaMonkey puppetmaster

Comment 1

[Justin Wood (:Callek)]

For SeaMonkey we're also setting up the CA on the same machine as our master.

I'm following the instructions at:

https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Set_up_a_standalone_puppetmaster

openssl is already installed:
openssl-1.0.0-20.el6_2.5.x86_64

Procmail installed, with ver 3.22-25.1.el6

Installed:
  wget.x86_64 0:1.12-1.4.el6

 mkdir /root/puppetca
 cd /root/puppetca
 wget http://hg.mozilla.org/build/puppet/raw-file/tip/setup/ca-scripts/{ca_config.sh,openssl.conf,generate-cert.sh,do-revocations.sh}
 vi ca_config.sh
 # set master_ssldir to /root/puppetca
 # set crl_push_dest=/tmp and make a note to figure out the CRL stuff later
 mkdir -p client_certs/revoke generated certdir ca/{private,certs}
 mv openssl.conf ca
 vi ca/openssl.conf
 # replace /var/lib/puppet/ssl-master with /root/puppetca
 touch ca/inventory.txt
 echo 01 > ca/serial

openssl req  -new -newkey rsa:2048 -keyout ca/ca_key.pem \
    -days 3650 -x509 -out ca/ca_crt.pem \
    -subj '/CN=ca.seamonkey.mozilla.org'

# Used CN of ca.seamonkey.mozilla.org so its obvious if there is ever a leak/swap who the CN belongs to.

Installed:
  rsync.x86_64 0:3.0.6-9.el6

I'm now running:

rsync -a --exclude=lost+found --delete rsync://puppetagain.pub.build.mozilla.org/data/ /data/

...and waiting

Comment 2

[Justin Wood (:Callek)]

sooo. dustin, I know we said on IRC that the HPs had "plenty of space" but that appears to be wrong:

[root@sea-puppet puppetca]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_seapuppet-lv_root
                       50G   50G     0 100% /
tmpfs                 1.9G     0  1.9G   0% /dev/shm
/dev/sda1             485M   32M  429M   7% /boot
/dev/mapper/vg_seapuppet-lv_home
                      176G  188M  167G   1% /home
[root@sea-puppet puppetca]# rsync -a --exclude=lost+found --delete rsync://puppetagain.pub.build.mozilla.org/dat
a/ /data/
rsync: write failed on "/data/repos/yum/mirrors/centos/6/2012-09-06/os/i386/Packages/gnome-power-manager-2.28.3-6
.el6.i686.rpm": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(301) [receiver=3.0.6]
rsync: connection unexpectedly closed (65 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [generator=3.0.6]
[root@sea-puppet puppetca]#

What is my choices here? Should I ask IT to give us a VM instead?

Comment 3

[Justin Wood (:Callek)]

dustin, see c#2

Comment 4

[Robert Kaiser]

Umm, 50G *used* for the root partition? That sounds awfully like something has strangely filled this up. 50G ought to be enough for any system installation, I'd guess - and /home has plenty more for doing actual work with a user account.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

oh, why do you have that formatted with a separate /home partition?  Put it all on one partition, or rename /home to /data (which will still be really tight)

:kairo, the puppet master has mirrors of a whole bunch of yum, deb, and dmg repos.

Comment 6

[Justin Wood (:Callek)]

Ok, since we expanded the /data partition already and have 250GB here now, but with the recent addition of apt repos (for Ubuntu) on the MoCo side, which would put us over 250GB needed, I opted to not mirror ubuntu to this setup yet.

Verified command with:
[root@sea-puppet ~]# rsync -n -v -a --exclude=lost+found --exclude=repos/apt/ --delete rsync://puppetagain.pub.build.mozilla.org/data/ /data/ | grep /apt

Then ran:
[root@sea-puppet ~]# rsync -a --exclude=lost+found --exclude=repos/apt/ --delete rsync://puppetagain.pub.build.mozilla.org/data/ /data/


Then again the No Space Left on Device... I thought :dustin said he was installing this the "right" way this time :-)  And since we installed from scratch, I lost the magic commands on how to do it from the detached screen session that *was* here last time.

[root@sea-puppet ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_seapuppet-lv_root
                       50G   50G     0 100% /
tmpfs                 1.9G     0  1.9G   0% /dev/shm
/dev/sda1             485M   36M  424M   8% /boot
/dev/mapper/vg_seapuppet-lv_home
                      174G  188M  165G   1% /home

:dustin can you help

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Probably screen sessions aren't the best way to document things!

I forgot that the default partitioning scheme involves a /home.  Here's what you want to do to combine the whole thing into one LV:

First, unmount /home.  You'll need to make sure you're only logged in as root, and not chdir'd to /home.  Then remove the /dev/mapper/vg_seapuppet-lv_home logical volume.  Then, resize the /dev/mapper/vg_seapuppet-lv_root with an additional 174G, using lvresize.  Lvresize has a handy "-r" option I just noticed which will automatically resize the filesystem, too.  Then you're done!

Comment 8

[Justin Wood (:Callek)]

[root@sea-puppet ~]# mount -v
/dev/mapper/vg_seapuppet-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/sda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
[root@sea-puppet ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_seapuppet-lv_root
                       50G   50G     0 100% /
tmpfs                 1.9G     0  1.9G   0% /dev/shm
/dev/sda1             485M   36M  424M   8% /boot
[root@sea-puppet ~]# lvdisplay
  --- Logical volume ---
  LV Name                /dev/vg_seapuppet/lv_root
  VG Name                vg_seapuppet
  LV UUID                TY6BwI-Leit-r4vI-yeVa-ZXM8-X7dN-1ZnqY8
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                50.00 GiB
  Current LE             12800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

  --- Logical volume ---
  LV Name                /dev/vg_seapuppet/lv_home
  VG Name                vg_seapuppet
  LV UUID                h813W6-4AXb-tOBz-fp3e-dJ9f-WuIO-QRgkoU
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                176.57 GiB
  Current LE             45201
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

  --- Logical volume ---
  LV Name                /dev/vg_seapuppet/lv_swap
  VG Name                vg_seapuppet
  LV UUID                33lavW-2ecc-0hJZ-2EYA-LY58-6xDF-hpGXi1
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                5.83 GiB
  Current LE             1492
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1
[root@sea-puppet ~]# lvremove /dev/vg_seapuppet/lv_home
Do you really want to remove active logical volume lv_home? [y/n]: y
  Logical volume "lv_home" successfully removed

[root@sea-puppet ~]# lvresize -L +174G -r /dev/vg_seapuppet/lv_root
  Extending logical volume lv_root to 224.00 GiB
  Logical volume lv_root successfully resized
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/mapper/vg_seapuppet-lv_root is mounted on /; on-line resizing required
old desc_blocks = 4, new_desc_blocks = 14
Performing an on-line resize of /dev/mapper/vg_seapuppet-lv_root to 58720256 (4k) blocks.
The filesystem on /dev/mapper/vg_seapuppet-lv_root is now 58720256 blocks long.

[root@sea-puppet ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_seapuppet-lv_root
                      221G   50G  161G  24% /
tmpfs                 1.9G     0  1.9G   0% /dev/shm
/dev/sda1             485M   36M  424M   8% /boot


-- Noticed when pasting the above that I had a bit more wiggle room:

[root@sea-puppet ~]# lvresize -L +0.57G -r /dev/vg_seapuppet/lv_root
  Rounding up size to full physical extent 584.00 MiB
  Extending logical volume lv_root to 224.57 GiB
  Logical volume lv_root successfully resized
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/mapper/vg_seapuppet-lv_root is mounted on /; on-line resizing required
old desc_blocks = 14, new_desc_blocks = 15
Performing an on-line resize of /dev/mapper/vg_seapuppet-lv_root to 58869760 (4k) blocks.
The filesystem on /dev/mapper/vg_seapuppet-lv_root is now 58869760 blocks long.

[root@sea-puppet ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_seapuppet-lv_root
                      222G   50G  161G  24% /
tmpfs                 1.9G     0  1.9G   0% /dev/shm
/dev/sda1             485M   36M  424M   8% /boot


--- Successfully expanded

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Nicely done :)

Comment 10

[Justin Wood (:Callek)]

...finished the rsync:
[root@sea-puppet ~]# rsync -a --exclude=lost+found --exclude=repos/apt/ --delete rsync://puppetagain.pub.build.mozilla.org/data/ /data/

And :dustin suggested in IRC that there may be more room (vgs command), indeed there was. Expanded a bit more!

[root@sea-puppet ~]# vgs
  VG           #PV #LV #SN Attr   VSize   VFree
  vg_seapuppet   1   2   0 wz--n- 232.39g 2.00g
[root@sea-puppet ~]# lvresize -L +2G -r /dev/vg_seapuppet/lv_root
  Extending logical volume lv_root to 226.57 GiB
  Insufficient free space: 512 extents needed, but only 511 available
[root@sea-puppet ~]# lvresize -L +1.99G -r /dev/vg_seapuppet/lv_root
  Rounding up size to full physical extent 1.99 GiB
  Extending logical volume lv_root to 226.56 GiB
  Logical volume lv_root successfully resized
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/mapper/vg_seapuppet-lv_root is mounted on /; on-line resizing required
old desc_blocks = 15, new_desc_blocks = 15
Performing an on-line resize of /dev/mapper/vg_seapuppet-lv_root to 59392000 (4k) blocks.
The filesystem on /dev/mapper/vg_seapuppet-lv_root is now 59392000 blocks long.

[root@sea-puppet ~]# vgs
  VG           #PV #LV #SN Attr   VSize   VFree
  vg_seapuppet   1   2   0 wz--n- 232.39g 4.00m

Comment 11

[Justin Wood (:Callek)]

So I'm working through https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Set_up_a_standalone_puppetmaster#Clone_and_configure_puppetagain_repo

puppet CA cert passed to ewong.

Choosing to use puppet320 [from http://hg.mozilla.org/users/dmitchell_mozilla.com/puppet320/] even though its not fully proven yet, but to prevent us having to redo this work later.

Resynced, and created/setup the config's and ca, per the first linked doc.

Setup secrets by following https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Secrets some secrets were not doc'd, and others we don't have machines yet to set them up on, so as it remains

current unset secrets:
============
[root@sea-puppet production]# grep \"\" manifests/extlookup/secrets.csv
android_tests_password,""
balrog_password,""
balrog_username,""
bors_servo_gh_user,""
bors_servo_gh_pass,""
buildbot_schedulerdb_database,""
buildbot_schedulerdb_hostname,""
buildbot_schedulerdb_password,""
buildbot_schedulerdb_username,""
buildbot_statusdb_database,""
buildbot_statusdb_hostname,""
buildbot_statusdb_password,""
buildbot_statusdb_username,""
builder_pw_pbkdf2,""
builder_pw_pbkdf2_iterations,""
builder_pw_pbkdf2_salt,""
jetperf_oauth_key,""
jetperf_oauth_secret,""
linux_tests_password,""
mac_tests_password,""
mozpool_db_database,""
mozpool_db_hostname,""
mozpool_db_password,""
mozpool_db_username,""
mozpool_inventory_password,""
mozpool_inventory_url,""
mozpool_inventory_username,""
prod_build_password,""
pulse_exchange,""
pulse_password,""
pulse_username,""
root_pw_pbkdf2,""
root_pw_pbkdf2_iterations,""
root_pw_pbkdf2_salt,""
signing_server_dep_password,""
signing_server_nightly_password,""
signing_server_release_password,""
signing_server_username,""
talos_oauth_key,""
talos_oauth_secret,""
try_build_password,""
tuxedo_password,""
tuxedo_username,""
win_tests_password,""
puppetsync_pubkey,""
puppetmaster_deploy_htpasswd,""

Comment 12

[Justin Wood (:Callek)]

Attachment: [puppet320] seamonkey-nodes/config

ewong does this look right?
dustin any concerns?

(notes):
* Left IT/Relops in keys list as per moco setup
* Did not add ewong to keys list yet, as he is not in the keys.pp file


Dustin, as an aside I presume https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Set_up_a_standalone_puppetmaster#Clone_and_configure_puppetagain_repo and on has no [known] problems when run against 320?

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

No, that page is outdated.  I removed it.  See https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Bootstrap_a_Puppetmaster

Comment 14

[Justin Wood (:Callek)]

After some trials and tribulations this is up!