Closed Bug 981974 Opened 10 years ago Closed 10 years ago

Deploy server-syncstorage rpm-1.5.1-5

Categories

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

task
Not set
normal

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: rfkelly, Unassigned)

References

Details

(Whiteboard: [qa+])

There's some good stuff in this tag, but it means the deploy requires a config change.  In particular we get:

  * Bug 668664: kill pending queries on thread exit
  * Bug 878668: try to enforce a timeout on each individual request

This introduces a custom gunicorn worker with per-request timeout handling, which needs to be enabled in deployment.  The existing gunicorn command-line needs to change from using "-k gevent" to using "-k mozsvc.gunicorn_worker.MozSvcGeventWorker".

IIRC we are currently using this in sync1.1 deployment under a slightly different name, so it should be safe to copy the setup from there with the name modified as above.

Other good stuff includes a fix for an error in the error-reporting routines (which e.g. is masking the real error in Bug 981294) and the ability to easily put the server into a backoff state per Bug 975306.  None of these require any config changes.
Whiteboard: [qa+]
Summary: Deploy server-syncstorage rpm-1.15.1-5 → Deploy server-syncstorage rpm-1.5.1-5
Ahem, I mean "rpm-1.5.1-5"
Blocks: 981294
We're seeing some unexpected tracebacks/timeouts in the app logs when running with the new mozsvc gunicorn worker.  We've decided to disable the new worker but keep the code changes for now, and we'll do some follow-up debugging once it's out the door.
This has been deployed to stage, but without the mozsvc gunicorn worker as mentioned in comment 2.
I'm running the API testsuite as verification, like this from a built server-syncstorage checkout:

    ./local/bin/python ./syncstorage/tests/functional/test_storage.py --use-token-server https://token.stage.mozaws.net/1.0/sync/1.5

We should try to incorporate such a run into our standard verification process to prevent bugs like Bug 980776 sneaking up on us in future.
With latest server-syncstorage master, you should also be able to use the loadtest syntax for hitting a specific node:

   ./local/bin/python ./syncstorage/tests/functional/test_storage.py [node-name]#[secret]
:rfkelly
+1000 for the above comments
We already have something good for FxA-Auth-Server (remote integration tests)
I think it is still: $ PUBLIC_URL=https://api-accounts.stage.mozaws.net npm run test-remote

For https://github.com/mozilla-services/tokenserver/tree/master/tokenserver/tests,
which of the test_*.py would be appropriate for a quick test?
> For https://github.com/mozilla-services/tokenserver/tree/master/tokenserver/tests,
> which of the test_*.py would be appropriate for a quick test?

I don't think they can be used in this way, yet.  I need to add that.
Already deploying. Doing a quick verification now.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Quick verification is a pass.
Status: RESOLVED → VERIFIED
Blocks: 982967
Blocks: 987455
No longer blocks: 987455
You need to log in before you can comment on or make changes to this bug.