862613 - sync: server-storage stage deploy: server_storage -> 1.15-8, server_core -> 2.12-8

Status: VERIFIED INCOMPLETE

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[Ryan Kelly [:rfkelly]]

Please deploy server-storage 1.15-8 and server-core 2.12-8 to sync server stage environment. Build command:

make build PYPI=http://pypi.build.mtv1.svc.mozilla.com/simple PYPIEXTRAS=http://pypi.build.mtv1.svc.mozilla.com/extras PYPISTRICT=1 SERVER_STORAGE=rpm-1.15-8 SERVER_CORE=rpm-2.12-8 CHANNEL=prod RPM_CHANNEL=prod build_rpms


This includes a single bugfix over the previous release, for Bug 853032.  But since this bug deals with mysql connection-handling, I would still recommend a full 24-hour loadtest before rolling it out to prod.

To test whether the bug has been resolved, restart mysql on one of the db machines while a loadtest is in progress.  We should see a brief spike of 503s until the database comes back up, then a return to successful request handling.  If the bug is not fixed, the failures will continue even after mysql comes back up.

Since there are no user-facing problems associated with this bug, there's no great urgency from my end to get this out to production.  Ops have requested a new tag so it can be rolled out at their convenience.

Comment 1

[James Bonacci [:jbonacci]]

Yes. QA will do the usual harm to Stage before this goes to Production.
I will coordinate with OPs on the testing since we need to restart mysql on a db.

Comment 2

[Ryan Kelly [:rfkelly]]

Oh, yikes, I linked the wrong bug here.  Copy-paste fail!  It should be Bug 857323.

Comment 3

[Bob Micheletto [:bobm]]

This version has been pushed to Stage, and is ready for QA.

Comment 4

[James Bonacci [:jbonacci]]

Verified the following on the webheads:
processes
logging
versions:
python26-services-2.12-8.noarch
python26-syncstorage-1.15-8.noarch

Moving on to a 24-hour load test...

Comment 5

[James Bonacci [:jbonacci]]

Results of load test were mixed. 
Nothing new to report, but nothing really to fix or break the release.
So, not worth rolling out to Production.

:rfkelly to continue investigations through https://bugzilla.mozilla.org/show_bug.cgi?id=857323

Comment 6

[James Bonacci [:jbonacci]]

Verified as finished, but not a complete push to Production.