Please deploy syncstorage 1.6.0 to Production and verify

RESOLVED FIXED

Status

RESOLVED FIXED
2 years ago
a year ago

People

(Reporter: kthiessen, Assigned: bobm)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

2 years ago
+++ This bug was initially created as a clone of Bug #1315425 +++

I've made a new version of server-syncstorage with preffed-off-by-default support for batch uploads.  It's in the "1.6.0" tag:

  https://github.com/mozilla-services/server-syncstorage/releases/tag/1.6.0

The biggest change here is the batch-upload stuff, including feature flag described in Bug 1304627.  But it has also picked up a couple of unrelated changes in the merge to master:

  Bug 1062120 - Fail the whole POST request if we get a DB error
  Bug 1314171 - Reintroduce 'older' query parameter

Barring any issues discovered, this will be the tag with which we roll out the batch-upload work according to the deployment plan in [1].  So let's have at it :-)

[1] https://docs.google.com/document/d/1JHZZ-6ceQAWctopJydRiRS8mdMxQbn1qHnPQBrlbR3I/edit?ts=581d0ef0
(Assignee)

Comment 1

2 years ago
Deployed a single 1.6.0 node in production which completes "Canary Production Deployment 1".
Status: NEW → ASSIGNED
(Assignee)

Comment 2

2 years ago
(In reply to Bob Micheletto [:bobm] from comment #1)
> Deployed a single 1.6.0 node in production which completes "Canary
> Production Deployment 1".

Another node was deployed at 1.6.0 to bring the count up to two.  Those nodes have been running smoothly.   "Production Deployment of Code" is now under way.
(Assignee)

Comment 3

2 years ago
(In reply to Bob Micheletto [:bobm] from comment #2)
> Another node was deployed at 1.6.0 to bring the count up to two.  Those
> nodes have been running smoothly.   "Production Deployment of Code" is now
> under way.

"Production Deployment of Code" phase is now complete with the following exception: the purge_ttl job has not been re-enabled on hosts where it was disabled.  It will be re-enabled separately to avoid confusing errors related to the code deployment from errors due to re-enabling the job.

After deployment 404 errors dropped from 90% of 4xx traffic to 25%.  :rfkelly pointed out that this is because the /info/configuration is now live and clients are finding it.
(Assignee)

Comment 4

2 years ago
(In reply to Bob Micheletto [:bobm] from comment #3)
"Canary Production Deployment 2" phase is now complete with a single 1.6.0 node with the batch upload API enabled having been deployed.
(Assignee)

Comment 5

2 years ago
The batch API has been enabled on ~50% of production Sync servers.

Comment 6

2 years ago
(In reply to Bob Micheletto [:bobm] from comment #5)
> The batch API has been enabled on ~50% of production Sync servers.

What are the per-batch record count limits? 10,000?
(Assignee)

Comment 7

2 years ago
(In reply to :Grisha Kruglov from comment #6)
> (In reply to Bob Micheletto [:bobm] from comment #5)
> > The batch API has been enabled on ~50% of production Sync servers.
> 
> What are the per-batch record count limits? 10,000?

That is correct.
(Assignee)

Comment 8

2 years ago
Batch uploads have been enabled on all servers, and will be enabled on all new nodes as deployed.  Closing this bug.
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Reporter)

Updated

a year ago
QA Contact: kthiessen
You need to log in before you can comment on or make changes to this bug.