787607 - sync: server-storage Production deploy: server_storage -> 1.13-8, server_core -> 2.10-7

Status: RESOLVED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[James Bonacci [:jbonacci]]

See bug 786864 for full details on the deployment to the Sync Stage environment.

Please deploy server-storage 1.13-8 and server-core 2.10-7 to sync server stage and dev environments. Build command:

make build PYPI=http://pypi.build.mtv1.svc.mozilla.com/simple PYPIEXTRAS=http://pypi.build.mtv1.svc.mozilla.com/extras PYPISTRICT=1 SERVER_STORAGE=rpm-1.13-8 SERVER_CORE=rpm-2.10-7 CHANNEL=prod RPM_CHANNEL=prod build_rpms


Please add Production-specific details to this bug as we move forward.

Comment 1

[James Bonacci [:jbonacci]]

Description above was pulled directly from bug 786864.
The requirements need to be adjusted to fit Production (rather than Stage or Dev).
Sorry for the confusion.

Comment 2

[Bob Micheletto [:bobm]]

Current differences between Stage and Production sync-storage:
Packages installed on sync2.web.scl2.stage.svc.mozilla.com not on sync2.web.scl2.svc.mozilla.com
perl-ExtUtils-MakeMaker
libX11-common
perl-devel
libXtst
perl-IO-Compress-Zlib
perl-Archive-Zip
python26-zope.deprecation
perl-HTML-Tagset
perl-Compress-Zlib
libXi
perl-Module-Load
perl-Module-Load-Conditional
python26-metlog-py
zeromq
perl-HTML-Parser
python26-repoze.who
java-1.6.0-sun
perl-Compress-Raw-Zlib
logstash-metlog
python26-umemcache
libICE
libXau
perl-IPC-Cmd
jpackage-utils
pyzmq
uuid
perl-ExtUtils-ParseXS
perl-Test-Harness
python26-beaker
libxcb
perl-Archive-Extract
logstash
system-config-firewall-base
kernel-devel
perl-Locale-Maketext-Simple
libXext
perl-IO-Compress-Base
mercurial
libX11
gdb
python26-argparse
libXt
libSM
perl-Params-Check
perl-libwww-perl
python26-repoze.lru
perl-URI

Packages installed on sync2.web.scl2.svc.mozilla.com not on sync2.web.scl2.stage.svc.mozilla.com
m4
freetype
libusb1
fontconfig
libcgroup
pixman
atk
ustr
gnutls
libtasn1

Packages with differing versions:
Package:        sync2.web.scl2.stage.svc.mozilla.com    sync2.web.scl2.svc.mozilla.com

python26-pastescript    1.7.5   1.7.3
python26-gevent 0.13.7  0.13.6
facter  1.5.9   1.5.8
elfutils-libs   0.152   0.148
yum     3.2.29  3.2.27
python26-zope.interface 3.8.0   3.6.1
python26-greenlet       0.4.0   0.3.1
elfutils-libelf 0.152   0.148
python26-markupsafe     0.15    0.12
gpg-pubkey      2fa658e0        fd431d51
python26-recaptcha-client       1.0.6   1.0.5
yum-plugin-security     1.1.30  1.1.26
python26-routes 1.13    1.12.3
python26-cef    0.3     0.2
python26-simplejson     2.5.2   2.1.6
python26-syncstorage    1.13    1.10
elfutils        0.152   0.148
tzdata  2011g   2011l
python26-pylibmc        1.2.3   1.1.1
python26-pastedeploy    1.5.0   1.3.4
python26-memcached      1.48    1.47
python26-services       2.10    2.6.5
yum-utils       1.1.30  1.1.26
python26-mako   0.7.0   0.4.1

Comment 3

[Bob Micheletto [:bobm]]

The /etc/sync/sync.conf file will require the following additions:
[app]
modules = metlog_loader
 
[metlog_loader]
backend = services.metrics.MetlogLoader
config = %(here)s/sync.conf
 
[metlog]
logger = syncstorage
sender_class = metlog.senders.udp.UdpSender
sender_host = 127.0.0.1
sender_port = 5566

Because of the inclusion of metlog sync-storage in production will also require an /etc/logstash.conf file.

The staging environment has this configured through puppet, so it should be a simple matter of implementing the staging configuration in production.  However, there is a problem with the %config directive in the .spec file which will cause the sync configuration files to be overwritten, and copied into an .rpmsave file after installation.  The syncpush script will need a minor modification to support an immediate puppet run before restarting the application.

Comment 4

[Bob Micheletto [:bobm]]

Scheduling this change for 9/10/2012 between 14:00 - 16:00 PDT.  The change will be documented here: https://intranet.mozilla.org/Services/Ops/ChangeWindow_20120910

Comment 5

[Bob Micheletto [:bobm]]

Changing schedule to 9/12/2012 between 14:00 - 16:00 PDT.

Comment 6

[Bob Micheletto [:bobm]]

Attachment: Link to Change Deployment Plan

Adding the link inside of an attachment in order to add the review option.  For convenience please see https://intranet.mozilla.org/Services/Ops/ChangeWindow_20120912 for the deployment plan.

Comment 7

[:Atoll]

sync{1..8} includes sync5, but sync{2,3,4,6,7,8} excludes sync5. r+ once this inconsistency is addressed (or justified).

Comment 8

[Bob Micheletto [:bobm]]

Resolved by removing sync8 from the PHX1 deploy, and leaving SCL2 the same.  The intention was to exclude sync5 from the SCL2 deploy.

Comment 9

[Bob Micheletto [:bobm]]

Moving step 5 to PHX1 from SCL2, because new node assignment is frozen in SCL2.  Changes documented here: https://intranet.mozilla.org/index.php?title=Services%2FOps%2FChangeWindow_20120912&action=historysubmit&diff=150653&oldid=150617 have been reviewed by Richard Soderberg.  He has provided verbal approval.

Comment 10

[Bob Micheletto [:bobm]]

The production deploy to Sync has been rolled back due to problems with an errant puppet manifest.  No impact was observed.  Services Ops will try again tomorrow.  

A new change window will be added to this Bug.

Comment 11

[Bob Micheletto [:bobm]]

Attachment: Link to Change Deployment Plan

Due to roll back, a new change window has been created.  The included link shows the changes between the 20120912 change window, which was rolled back, and the 20120913 window being proposed.  See the following: https://intranet.mozilla.org/index.php?title=Services%2FOps%2FChangeWindow_20120913&action=historysubmit&diff=150663&oldid=150655

Comment 12

[Bob Micheletto [:bobm]]

Sync deploy completed.