Please deploy syncstorage-rs 0.2.8 to prod
Categories
(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)
Tracking
(Not tracked)
People
(Reporter: pjenvey, Assigned: eolson)
Details
Changes from current prod 0.2.1 (previously 0.2.2, rolled back to 0.2.1):
0.2.8 (2020-03-26)
Bug Fixes
0.2.7 (2020-03-24)
Chore
- adapt googleapis-raw dep to 0.2 branch (58f2051f)
Refactor
0.2.5 (2020-03-11)
Bug Fixes
0.2.4 (2020-03-10)
Bug Fixes
- GETs with a limit and no sort never advance X-Weave-Next-Offset (c95f2ff2)
0.2.2 (2020-02-12)
Chore
Performance
- Port get_bsos' pagination optimization (9266f753)
Features
- restrict release mode logging to ERROR (#427) (9ab20845, closes #426)
- recategorize logging messages into appropriate states (d8aeb3ee, closes #416)
- script to count total users in spanner (13d2490d)
- User migration scripts (3500b9b9)
Refactor
Bug Fixes
Assignee | ||
Comment 1•5 years ago
|
||
I rolled this out to the canary instance and saw some very odd behavior. The most concerning is the data storage in spanner grew very quickly. From metrics, the canary instance was writing much more data than the other 3 instances (still running 0.2.1). I stopped the canary instance and the data slowed back to a more typical growth rate.
The 5 hours leading up to the deploy, spanner grew by 3.4 GB/hr. After the deployment, it grew 31 GB/hr. After I killed the canary instance, the data growth resumed its previous growth rate.
Can we get some testing on this in stage? What could it be doing - writing that much data?
Reporter | ||
Comment 2•5 years ago
•
|
||
Documented this here: https://github.com/mozilla-services/syncstorage-rs/issues/554
The data growth's likely due to the form fixed. We've released 0.2.9 to double check, without the sorting differences that increased Spanner's CPU, closing this out for that: https://bugzilla.mozilla.org/show_bug.cgi?id=1627038
Description
•