981974 - Deploy server-syncstorage rpm-1.5.1-5

Status: VERIFIED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[Ryan Kelly [:rfkelly]]

There's some good stuff in this tag, but it means the deploy requires a config change.  In particular we get:

  * Bug 668664: kill pending queries on thread exit
  * Bug 878668: try to enforce a timeout on each individual request

This introduces a custom gunicorn worker with per-request timeout handling, which needs to be enabled in deployment.  The existing gunicorn command-line needs to change from using "-k gevent" to using "-k mozsvc.gunicorn_worker.MozSvcGeventWorker".

IIRC we are currently using this in sync1.1 deployment under a slightly different name, so it should be safe to copy the setup from there with the name modified as above.

Other good stuff includes a fix for an error in the error-reporting routines (which e.g. is masking the real error in Bug 981294) and the ability to easily put the server into a backoff state per Bug 975306.  None of these require any config changes.

Comment 1

[Ryan Kelly [:rfkelly]]

Ahem, I mean "rpm-1.5.1-5"

Comment 2

[Ryan Kelly [:rfkelly]]

We're seeing some unexpected tracebacks/timeouts in the app logs when running with the new mozsvc gunicorn worker.  We've decided to disable the new worker but keep the code changes for now, and we'll do some follow-up debugging once it's out the door.

Comment 3

[Bob Micheletto [:bobm]]

This has been deployed to stage, but without the mozsvc gunicorn worker as mentioned in comment 2.

Comment 4

[Ryan Kelly [:rfkelly]]

I'm running the API testsuite as verification, like this from a built server-syncstorage checkout:

    ./local/bin/python ./syncstorage/tests/functional/test_storage.py --use-token-server https://token.stage.mozaws.net/1.0/sync/1.5

We should try to incorporate such a run into our standard verification process to prevent bugs like Bug 980776 sneaking up on us in future.

Comment 5

[Ryan Kelly [:rfkelly]]

With latest server-syncstorage master, you should also be able to use the loadtest syntax for hitting a specific node:

   ./local/bin/python ./syncstorage/tests/functional/test_storage.py [node-name]#[secret]

Comment 6

[James Bonacci [:jbonacci]]

:rfkelly
+1000 for the above comments
We already have something good for FxA-Auth-Server (remote integration tests)
I think it is still: $ PUBLIC_URL=https://api-accounts.stage.mozaws.net npm run test-remote

For https://github.com/mozilla-services/tokenserver/tree/master/tokenserver/tests,
which of the test_*.py would be appropriate for a quick test?

Comment 7

[Ryan Kelly [:rfkelly]]

> For https://github.com/mozilla-services/tokenserver/tree/master/tokenserver/tests,
> which of the test_*.py would be appropriate for a quick test?

I don't think they can be used in this way, yet.  I need to add that.

Comment 8

[James Bonacci [:jbonacci]]

Already deploying. Doing a quick verification now.

Comment 9

[James Bonacci [:jbonacci]]

Quick verification is a pass.