972609 - Fuzz the x-weave-backoff header

Status: VERIFIED FIXED

(Cloud Services Graveyard :: Server: Sync, defect, P3)

Description

[Ryan Kelly [:rfkelly]]

We currently apply a few seconds of "fuzzing" around the value of the Retry-After header, to help mitigate a dogpile of clients all retrying at the same time.  We should probably do the same for the X-Weave-Backoff and X-Backoff headers across all our properties.

Comment 1

[Richard Newman [:rnewman]]

fwiw, clients also fuzz.

Comment 2

[Ryan Kelly [:rfkelly]]

Attachment: mozsvc-fuzz-backoff-headers.diff

This patch moves the fuzzing logic into mozsvc so that it can be shared with tokenserver, and extends it to apply to other backoff-related headers.

Comment 3

[Ryan Kelly [:rfkelly]]

Attachment: sync15-remove-fuzz-headers.diff

And a corresponding patch to remove it from syncstorage.

The existing fuzzing logic added a random number between 0 and 5 seconds.  In the new version, I went with a "5% or 5 seconds, whichever is greater" approach because ISTM that if clients have to wait longer, we should also spread them out over a longer period of time.

Comment 4

[Ryan Kelly [:rfkelly]]

https://github.com/mozilla-services/mozservices/commit/344449e3d9dad7e930c2defb617833903d118228
https://github.com/mozilla-services/server-syncstorage/commit/05a36115c104efb152702d9c43354ae43a977aa4

Comment 5

[James Bonacci [:jbonacci]]

Verified in code.