Closed Bug 827079 Opened 7 years ago Closed 6 years ago

Setup SeaMonkey PuppetMaster

Categories

(SeaMonkey :: Release Engineering, defect)

x86_64
Windows 7
defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: Callek, Assigned: Callek)

References

Details

Attachments

(1 file)

This is to track setting up the SeaMonkey puppetmaster
For SeaMonkey we're also setting up the CA on the same machine as our master.

I'm following the instructions at:

https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Set_up_a_standalone_puppetmaster

openssl is already installed:
openssl-1.0.0-20.el6_2.5.x86_64

Procmail installed, with ver 3.22-25.1.el6

Installed:
  wget.x86_64 0:1.12-1.4.el6

 mkdir /root/puppetca
 cd /root/puppetca
 wget http://hg.mozilla.org/build/puppet/raw-file/tip/setup/ca-scripts/{ca_config.sh,openssl.conf,generate-cert.sh,do-revocations.sh}
 vi ca_config.sh
 # set master_ssldir to /root/puppetca
 # set crl_push_dest=/tmp and make a note to figure out the CRL stuff later
 mkdir -p client_certs/revoke generated certdir ca/{private,certs}
 mv openssl.conf ca
 vi ca/openssl.conf
 # replace /var/lib/puppet/ssl-master with /root/puppetca
 touch ca/inventory.txt
 echo 01 > ca/serial

openssl req  -new -newkey rsa:2048 -keyout ca/ca_key.pem \
    -days 3650 -x509 -out ca/ca_crt.pem \
    -subj '/CN=ca.seamonkey.mozilla.org'

# Used CN of ca.seamonkey.mozilla.org so its obvious if there is ever a leak/swap who the CN belongs to.

Installed:
  rsync.x86_64 0:3.0.6-9.el6

I'm now running:

rsync -a --exclude=lost+found --delete rsync://puppetagain.pub.build.mozilla.org/data/ /data/

...and waiting
Assignee: nobody → bugspam.Callek
sooo. dustin, I know we said on IRC that the HPs had "plenty of space" but that appears to be wrong:

[root@sea-puppet puppetca]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_seapuppet-lv_root
                       50G   50G     0 100% /
tmpfs                 1.9G     0  1.9G   0% /dev/shm
/dev/sda1             485M   32M  429M   7% /boot
/dev/mapper/vg_seapuppet-lv_home
                      176G  188M  167G   1% /home
[root@sea-puppet puppetca]# rsync -a --exclude=lost+found --delete rsync://puppetagain.pub.build.mozilla.org/dat
a/ /data/
rsync: write failed on "/data/repos/yum/mirrors/centos/6/2012-09-06/os/i386/Packages/gnome-power-manager-2.28.3-6
.el6.i686.rpm": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(301) [receiver=3.0.6]
rsync: connection unexpectedly closed (65 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [generator=3.0.6]
[root@sea-puppet puppetca]#

What is my choices here? Should I ask IT to give us a VM instead?
dustin, see c#2
Umm, 50G *used* for the root partition? That sounds awfully like something has strangely filled this up. 50G ought to be enough for any system installation, I'd guess - and /home has plenty more for doing actual work with a user account.
oh, why do you have that formatted with a separate /home partition?  Put it all on one partition, or rename /home to /data (which will still be really tight)

:kairo, the puppet master has mirrors of a whole bunch of yum, deb, and dmg repos.
Ok, since we expanded the /data partition already and have 250GB here now, but with the recent addition of apt repos (for Ubuntu) on the MoCo side, which would put us over 250GB needed, I opted to not mirror ubuntu to this setup yet.

Verified command with:
[root@sea-puppet ~]# rsync -n -v -a --exclude=lost+found --exclude=repos/apt/ --delete rsync://puppetagain.pub.build.mozilla.org/data/ /data/ | grep /apt

Then ran:
[root@sea-puppet ~]# rsync -a --exclude=lost+found --exclude=repos/apt/ --delete rsync://puppetagain.pub.build.mozilla.org/data/ /data/


Then again the No Space Left on Device... I thought :dustin said he was installing this the "right" way this time :-)  And since we installed from scratch, I lost the magic commands on how to do it from the detached screen session that *was* here last time.

[root@sea-puppet ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_seapuppet-lv_root
                       50G   50G     0 100% /
tmpfs                 1.9G     0  1.9G   0% /dev/shm
/dev/sda1             485M   36M  424M   8% /boot
/dev/mapper/vg_seapuppet-lv_home
                      174G  188M  165G   1% /home

:dustin can you help
Flags: needinfo?(dustin)
Probably screen sessions aren't the best way to document things!

I forgot that the default partitioning scheme involves a /home.  Here's what you want to do to combine the whole thing into one LV:

First, unmount /home.  You'll need to make sure you're only logged in as root, and not chdir'd to /home.  Then remove the /dev/mapper/vg_seapuppet-lv_home logical volume.  Then, resize the /dev/mapper/vg_seapuppet-lv_root with an additional 174G, using lvresize.  Lvresize has a handy "-r" option I just noticed which will automatically resize the filesystem, too.  Then you're done!
Flags: needinfo?(dustin)
[root@sea-puppet ~]# mount -v
/dev/mapper/vg_seapuppet-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/sda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
[root@sea-puppet ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_seapuppet-lv_root
                       50G   50G     0 100% /
tmpfs                 1.9G     0  1.9G   0% /dev/shm
/dev/sda1             485M   36M  424M   8% /boot
[root@sea-puppet ~]# lvdisplay
  --- Logical volume ---
  LV Name                /dev/vg_seapuppet/lv_root
  VG Name                vg_seapuppet
  LV UUID                TY6BwI-Leit-r4vI-yeVa-ZXM8-X7dN-1ZnqY8
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                50.00 GiB
  Current LE             12800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

  --- Logical volume ---
  LV Name                /dev/vg_seapuppet/lv_home
  VG Name                vg_seapuppet
  LV UUID                h813W6-4AXb-tOBz-fp3e-dJ9f-WuIO-QRgkoU
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                176.57 GiB
  Current LE             45201
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

  --- Logical volume ---
  LV Name                /dev/vg_seapuppet/lv_swap
  VG Name                vg_seapuppet
  LV UUID                33lavW-2ecc-0hJZ-2EYA-LY58-6xDF-hpGXi1
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                5.83 GiB
  Current LE             1492
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1
[root@sea-puppet ~]# lvremove /dev/vg_seapuppet/lv_home
Do you really want to remove active logical volume lv_home? [y/n]: y
  Logical volume "lv_home" successfully removed

[root@sea-puppet ~]# lvresize -L +174G -r /dev/vg_seapuppet/lv_root
  Extending logical volume lv_root to 224.00 GiB
  Logical volume lv_root successfully resized
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/mapper/vg_seapuppet-lv_root is mounted on /; on-line resizing required
old desc_blocks = 4, new_desc_blocks = 14
Performing an on-line resize of /dev/mapper/vg_seapuppet-lv_root to 58720256 (4k) blocks.
The filesystem on /dev/mapper/vg_seapuppet-lv_root is now 58720256 blocks long.

[root@sea-puppet ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_seapuppet-lv_root
                      221G   50G  161G  24% /
tmpfs                 1.9G     0  1.9G   0% /dev/shm
/dev/sda1             485M   36M  424M   8% /boot


-- Noticed when pasting the above that I had a bit more wiggle room:

[root@sea-puppet ~]# lvresize -L +0.57G -r /dev/vg_seapuppet/lv_root
  Rounding up size to full physical extent 584.00 MiB
  Extending logical volume lv_root to 224.57 GiB
  Logical volume lv_root successfully resized
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/mapper/vg_seapuppet-lv_root is mounted on /; on-line resizing required
old desc_blocks = 14, new_desc_blocks = 15
Performing an on-line resize of /dev/mapper/vg_seapuppet-lv_root to 58869760 (4k) blocks.
The filesystem on /dev/mapper/vg_seapuppet-lv_root is now 58869760 blocks long.

[root@sea-puppet ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_seapuppet-lv_root
                      222G   50G  161G  24% /
tmpfs                 1.9G     0  1.9G   0% /dev/shm
/dev/sda1             485M   36M  424M   8% /boot


--- Successfully expanded
Nicely done :)
...finished the rsync:
[root@sea-puppet ~]# rsync -a --exclude=lost+found --exclude=repos/apt/ --delete rsync://puppetagain.pub.build.mozilla.org/data/ /data/

And :dustin suggested in IRC that there may be more room (vgs command), indeed there was. Expanded a bit more!

[root@sea-puppet ~]# vgs
  VG           #PV #LV #SN Attr   VSize   VFree
  vg_seapuppet   1   2   0 wz--n- 232.39g 2.00g
[root@sea-puppet ~]# lvresize -L +2G -r /dev/vg_seapuppet/lv_root
  Extending logical volume lv_root to 226.57 GiB
  Insufficient free space: 512 extents needed, but only 511 available
[root@sea-puppet ~]# lvresize -L +1.99G -r /dev/vg_seapuppet/lv_root
  Rounding up size to full physical extent 1.99 GiB
  Extending logical volume lv_root to 226.56 GiB
  Logical volume lv_root successfully resized
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/mapper/vg_seapuppet-lv_root is mounted on /; on-line resizing required
old desc_blocks = 15, new_desc_blocks = 15
Performing an on-line resize of /dev/mapper/vg_seapuppet-lv_root to 59392000 (4k) blocks.
The filesystem on /dev/mapper/vg_seapuppet-lv_root is now 59392000 blocks long.

[root@sea-puppet ~]# vgs
  VG           #PV #LV #SN Attr   VSize   VFree
  vg_seapuppet   1   2   0 wz--n- 232.39g 4.00m
So I'm working through https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Set_up_a_standalone_puppetmaster#Clone_and_configure_puppetagain_repo

puppet CA cert passed to ewong.

Choosing to use puppet320 [from http://hg.mozilla.org/users/dmitchell_mozilla.com/puppet320/] even though its not fully proven yet, but to prevent us having to redo this work later.

Resynced, and created/setup the config's and ca, per the first linked doc.

Setup secrets by following https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Secrets some secrets were not doc'd, and others we don't have machines yet to set them up on, so as it remains

current unset secrets:
============
[root@sea-puppet production]# grep \"\" manifests/extlookup/secrets.csv
android_tests_password,""
balrog_password,""
balrog_username,""
bors_servo_gh_user,""
bors_servo_gh_pass,""
buildbot_schedulerdb_database,""
buildbot_schedulerdb_hostname,""
buildbot_schedulerdb_password,""
buildbot_schedulerdb_username,""
buildbot_statusdb_database,""
buildbot_statusdb_hostname,""
buildbot_statusdb_password,""
buildbot_statusdb_username,""
builder_pw_pbkdf2,""
builder_pw_pbkdf2_iterations,""
builder_pw_pbkdf2_salt,""
jetperf_oauth_key,""
jetperf_oauth_secret,""
linux_tests_password,""
mac_tests_password,""
mozpool_db_database,""
mozpool_db_hostname,""
mozpool_db_password,""
mozpool_db_username,""
mozpool_inventory_password,""
mozpool_inventory_url,""
mozpool_inventory_username,""
prod_build_password,""
pulse_exchange,""
pulse_password,""
pulse_username,""
root_pw_pbkdf2,""
root_pw_pbkdf2_iterations,""
root_pw_pbkdf2_salt,""
signing_server_dep_password,""
signing_server_nightly_password,""
signing_server_release_password,""
signing_server_username,""
talos_oauth_key,""
talos_oauth_secret,""
try_build_password,""
tuxedo_password,""
tuxedo_username,""
win_tests_password,""
puppetsync_pubkey,""
puppetmaster_deploy_htpasswd,""
ewong does this look right?
dustin any concerns?

(notes):
* Left IT/Relops in keys list as per moco setup
* Did not add ewong to keys list yet, as he is not in the keys.pp file


Dustin, as an aside I presume https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Set_up_a_standalone_puppetmaster#Clone_and_configure_puppetagain_repo and on has no [known] problems when run against 320?
Attachment #747798 - Flags: review?(ewong)
Attachment #747798 - Flags: feedback?(dustin)
Attachment #747798 - Flags: review?(ewong) → review+
Attachment #747798 - Flags: feedback?(dustin) → feedback+
After some trials and tribulations this is up!
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Blocks: 884454
You need to log in before you can comment on or make changes to this bug.