1317532 - Please deploy syncstorage 1.6.0 to Production and verify

Status: RESOLVED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[Karl Thiessen [:kthiessen, he/him]]

+++ This bug was initially created as a clone of Bug #1315425 +++

I've made a new version of server-syncstorage with preffed-off-by-default support for batch uploads.  It's in the "1.6.0" tag:

  https://github.com/mozilla-services/server-syncstorage/releases/tag/1.6.0

The biggest change here is the batch-upload stuff, including feature flag described in Bug 1304627.  But it has also picked up a couple of unrelated changes in the merge to master:

  Bug 1062120 - Fail the whole POST request if we get a DB error
  Bug 1314171 - Reintroduce 'older' query parameter

Barring any issues discovered, this will be the tag with which we roll out the batch-upload work according to the deployment plan in [1].  So let's have at it :-)

[1] https://docs.google.com/document/d/1JHZZ-6ceQAWctopJydRiRS8mdMxQbn1qHnPQBrlbR3I/edit?ts=581d0ef0

Comment 1

[Bob Micheletto [:bobm]]

Deployed a single 1.6.0 node in production which completes "Canary Production Deployment 1".

Comment 2

[Bob Micheletto [:bobm]]

(In reply to Bob Micheletto [:bobm] from comment #1)
> Deployed a single 1.6.0 node in production which completes "Canary
> Production Deployment 1".

Another node was deployed at 1.6.0 to bring the count up to two.  Those nodes have been running smoothly.   "Production Deployment of Code" is now under way.

Comment 3

[Bob Micheletto [:bobm]]

(In reply to Bob Micheletto [:bobm] from comment #2)
> Another node was deployed at 1.6.0 to bring the count up to two.  Those
> nodes have been running smoothly.   "Production Deployment of Code" is now
> under way.

"Production Deployment of Code" phase is now complete with the following exception: the purge_ttl job has not been re-enabled on hosts where it was disabled.  It will be re-enabled separately to avoid confusing errors related to the code deployment from errors due to re-enabling the job.

After deployment 404 errors dropped from 90% of 4xx traffic to 25%.  :rfkelly pointed out that this is because the /info/configuration is now live and clients are finding it.

Comment 4

[Bob Micheletto [:bobm]]

(In reply to Bob Micheletto [:bobm] from comment #3)
"Canary Production Deployment 2" phase is now complete with a single 1.6.0 node with the batch upload API enabled having been deployed.

Comment 5

[Bob Micheletto [:bobm]]

The batch API has been enabled on ~50% of production Sync servers.

Comment 6

[:Grisha Kruglov]

(In reply to Bob Micheletto [:bobm] from comment #5)
> The batch API has been enabled on ~50% of production Sync servers.

What are the per-batch record count limits? 10,000?

Comment 7

[Bob Micheletto [:bobm]]

(In reply to :Grisha Kruglov from comment #6)
> (In reply to Bob Micheletto [:bobm] from comment #5)
> > The batch API has been enabled on ~50% of production Sync servers.
> 
> What are the per-batch record count limits? 10,000?

That is correct.

Comment 8

[Bob Micheletto [:bobm]]

Batch uploads have been enabled on all servers, and will be enabled on all new nodes as deployed.  Closing this bug.