Closed Bug 1625301 Opened 5 years ago Closed 5 years ago

Please deploy syncstorage-rs 0.2.8 to prod

Categories

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: pjenvey, Assigned: eolson)

Details

Changes from current prod 0.2.1 (previously 0.2.2, rolled back to 0.2.1):

0.2.8 (2020-03-26)

Bug Fixes

0.2.7 (2020-03-24)

Chore

  • adapt googleapis-raw dep to 0.2 branch (58f2051f)

Refactor

0.2.5 (2020-03-11)

Bug Fixes

0.2.4 (2020-03-10)

Bug Fixes

  • GETs with a limit and no sort never advance X-Weave-Next-Offset (c95f2ff2)

0.2.2 (2020-02-12)

Chore

  • revert temp. sentry tags for the mutation limit issue (f213a79c, closes #389)

Performance

  • Port get_bsos' pagination optimization (9266f753)

Features

  • restrict release mode logging to ERROR (#427) (9ab20845, closes #426)
  • recategorize logging messages into appropriate states (d8aeb3ee, closes #416)
  • script to count total users in spanner (13d2490d)
  • User migration scripts (3500b9b9)

Refactor

  • kill unnecessary copies from protobuf Values (0de96712, closes #422)

Bug Fixes

  • filter out variable data from URI metric (#421) (3986c451, closes #420)

I rolled this out to the canary instance and saw some very odd behavior. The most concerning is the data storage in spanner grew very quickly. From metrics, the canary instance was writing much more data than the other 3 instances (still running 0.2.1). I stopped the canary instance and the data slowed back to a more typical growth rate.

The 5 hours leading up to the deploy, spanner grew by 3.4 GB/hr. After the deployment, it grew 31 GB/hr. After I killed the canary instance, the data growth resumed its previous growth rate.

Can we get some testing on this in stage? What could it be doing - writing that much data?

Flags: needinfo?(pjenvey)

Documented this here: https://github.com/mozilla-services/syncstorage-rs/issues/554

The data growth's likely due to the form fixed. We've released 0.2.9 to double check, without the sorting differences that increased Spanner's CPU, closing this out for that: https://bugzilla.mozilla.org/show_bug.cgi?id=1627038

Status: NEW → RESOLVED
Closed: 5 years ago
Flags: needinfo?(pjenvey)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.