1363322 - Please deploy tokenserver 1.2.27

Status: RESOLVED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[Ryan Kelly [:rfkelly]]

This version includes support for handling a new message format that FxA will be using for its SQS notifications:

  https://github.com/mozilla-services/tokenserver/pull/107
 
Tagged at:

  https://github.com/mozilla-services/tokenserver/releases/tag/1.2.25

Comment 1

[Ryan Kelly [:rfkelly]]

Hi :jrgm, do I recall correctly that you're now the primary for tokenserver deploys?  This one blocks some cleanup work on the FxA side of things, ref https://github.com/mozilla/fxa-auth-server/pull/1879

Comment 2

[Rebecca Billings [:rbillings]]

FWIW I did run the tokenserver tests on stage, test results were good.

Comment 3

[Ryan Kelly [:rfkelly]]

Hi :jrgm, I've made a 1.2.26 release to incorporate the fix for Bug 1365328:

  https://github.com/mozilla-services/tokenserver/releases/tag/1.2.26

Since there were no updates here I figure we can derail the 1.2.25 deploy and replace it with the new one, but let me know if you prefer a fresh bug.

Comment 4

[Ryan Kelly [:rfkelly]]

Hi :jrgm, any update on this one?  It's blocking some merges on the FxA side of things.

Comment 5

[Karl Thiessen [:kthiessen, he/him]]

Bob, Rebecca and I ran the tokenserver load tests against stage, and discovered that the stage tokenserver is not configured to auto-scale.  Otherwise, the tests were within normal ranges of performance, etc.

Do we need to verify auto-scaling behaviour before releasing this tag?

Comment 6

[Ryan Kelly [:rfkelly]]

I don't see any particular server-load-related risks with this change, so if it passes a basic smoketest, I think it's safe to ship.

Comment 7

[John Morrison [:jrgm]]

Docker deploy in stage fails with 
Traceback (most recent call last):
File "/usr/local/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/local/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/app/tokenserver/scripts/purge_old_records.py", line 163, in <module>
tokenserver.scripts.run_script(main)
File "tokenserver/scripts/__init__.py", line 19, in run_script
exitcode = main()
File "/app/tokenserver/scripts/purge_old_records.py", line 146, in main
request_timeout=opts.request_timeout)
File "/app/tokenserver/scripts/purge_old_records.py", line 46, in purge_old_records
config = tokenserver.scripts.load_configurator(config_file)
File "tokenserver/scripts/__init__.py", line 33, in load_configurator
config.include(tokenserver)
File "/usr/local/lib/python2.7/site-packages/pyramid/config/__init__.py", line 805, in include
c(configurator)
File "tokenserver/__init__.py", line 38, in includeme
load_and_register("tokenserver", config)
File "/usr/local/lib/python2.7/site-packages/mozsvc/plugin.py", line 83, in load_and_register
plugin = load_from_config(section_name, settings["config"])
File "/usr/local/lib/python2.7/site-packages/mozsvc/plugin.py", line 113, in load_from_config
return klass(**kwargs)
File "tokenserver/assignment/sqlnode/sql.py", line 193, in __init__
logging_name='tokenserver.assignment.sqlnode'
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 387, in create_engine
return strategy.create(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 56, in create
entrypoint = u._get_entrypoint()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/url.py", line 139, in _get_entrypoint
cls = registry.load(name)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 212, in load
(self.group, name))
sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:pymysql

A couple of questions:
- is this docker image tested when using mysql
- has the startup command changed? Current under docker is `/usr/bin/docker-current run --net host -m 0b -v \
    /etc/tokenserver/settings.ini:/app/settings.ini --log-driver=none --name tokenserver hub.prod.mozaws.net/mozilla \
    /tokenserver:1.2.26 gunicorn --worker-class gevent --paste settings.ini --workers 5 --max-requests 20000 --log-config \ 
    settings.ini

Comment 8

[Ryan Kelly [:rfkelly]]

Hmm, somehow `PyMySQL==0.6.2` and `pymysql_sa==1.0` seem to have disappeared from the dependency list in this latest version :-/

> - is this docker image tested when using mysql

So I'm going to bet "no", or at least, not with a config that's sufficiently close to production :-(

Comment 9

[Ryan Kelly [:rfkelly]]

Yeah, those deps went missing here: https://github.com/mozilla-services/tokenserver/commit/dfb7eebde0361fcefaf732749ef3887fe0210ef0

Comment 10

[Ryan Kelly [:rfkelly]]

Hopeful fix here, pending review: https://github.com/mozilla-services/tokenserver/pull/109

ni? myself for a fresh tag once this lands

Comment 11

[Ryan Kelly [:rfkelly]]

I've made a 1.2.27 tag with the above fixes:

  https://github.com/mozilla-services/tokenserver/releases/tag/1.2.27

Comment 12

[Ryan Kelly [:rfkelly]]

from sync standup today, it sounds like this went ok in loadtest; :jrgm please let me know when this goes live so we can unblock the dependant fxa-side work.

Comment 13

[John Morrison [:jrgm]]

This 1.2.27 is live now in production.