996915 - Deploy tokenserver 1.2.4-2 to stage

Status: VERIFIED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[Ryan Kelly [:rfkelly]]

(This is not for deployment until after the current one in Bug 993537.  I just want to get it prepped while I've still got the notes clear in my head)

This version of tokenserver includes database migration scripting, and some important additions/removals of database indexes:

    Bug 777650 - add db migration scripting for tokenserver
    Bug 988134 - X-Client-State tracking prevents proper node-reassignment
    Bug 988643 - add index for looking up assignments by node node

For proper deployment and testing of the db migrations, we need to get stage into the same state as prod.  IIUC it's currently slightly different due to an aborted deployment.  The procedure should be:

  * Build the rpm and deploy to all webheads
  * From one of the webheads, run the migration tool as follows:

      ./bin/alembic stamp 17d209a72e2f
      ./bin/alembic downgrade base
      ./bin/alembic upgrade head

This will mark it as its current schema version, *downgrade* it to match what's currently in production, then *upgrade* the schema to the desired state.

If we want to be really clever we could run a loadtest while doing the migration, to see how it affects performance.  But probably we don't want to muck around with it that much.

Note that as described in Bug 988643 Comment 11, we will need a second deployment to safely bring the deployments up to current head.  I've prepped rpm-1.2.2 for this purpose but let's cross that bridge when we get to it.

Comment 1

[Ryan Kelly [:rfkelly]]

Morphing this into an actual deploy bug.  Benson, please put this through to stage and try out the db schema migrations as described above.

Comment 2

[Ryan Kelly [:rfkelly]]

Please bump the urgency of this, it should be a pretty painless deploy (heh, famous last words?) and is now a blocker for getting Bug 996751 out the door.

Comment 3

[Ryan Kelly [:rfkelly]]

I've made a new tag that includes the purge_old_records changes from Bug 996751, and thus requires the config changes in https://github.com/mozilla-services/puppet-config/pull/396.

It would be great if we can get this pushed live before FF29 release on the 29th.

Comment 4

[Ryan Kelly [:rfkelly]]

For completeness, I'll note that the process described in the initial comment above it still valid for deploying this new tag to stage, including running the different alembic commands to twiddle the state of the db.

When deploying to prod, we don't need to do the downgrade/upgrade dance, we just deploy the new code and do:

      ./bin/alembic upgrade head

Comment 5

[Benson Wong [:mostlygeek]]

I got the stack deployed to stage. However, when I run: 

   > ./bin/alembic stamp 17d209a72e2f

I get: 

Traceback (most recent call last):
  File "./bin/alembic", line 12, in <module>
    load_entry_point('alembic==0.6.4', 'console_scripts', 'alembic')()
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/config.py", line 298, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/config.py", line 293, in main
    self.run_cmd(cfg, options)
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/config.py", line 279, in run_cmd
    **dict((k, getattr(options, k)) for k in kwarg)
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/command.py", line 258, in stamp
    script.run_env()
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/script.py", line 203, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/util.py", line 212, in load_python_file
    module = load_module_py(module_id, path)
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/compat.py", line 58, in load_module_py
    mod = imp.load_source(module_id, path, fp)
  File "tokenserver/assignment/sqlnode/migrations/env.py", line 7, in <module>
    from tokenserver.util import find_config_file
ImportError: No module named tokenserver.util

Comment 6

[Ryan Kelly [:rfkelly]]

Possibly some jank with python paths and virtualenvs, I'll take a look...

Comment 7

[Ryan Kelly [:rfkelly]]

Attachment: ts-run-alembic-in-virtualenv.diff

Yep, there was some sys.path weirdness in the virtualenv.  I've patched the alembic script to work around it (see attached) and made a new 1.2.4 tag with the changes.  Benson, please try again with this new tag.

Comment 8

[Benson Wong [:mostlygeek]]

I'm on it!

Comment 9

[Benson Wong [:mostlygeek]]

Still have the same issue: 

[bwong@ip-10-187-27-74 tokenserver]$ ./bin/alembic stamp 17d209a72e2f
Traceback (most recent call last):
  File "./bin/alembic", line 12, in <module>
    load_entry_point('alembic==0.6.4', 'console_scripts', 'alembic')()
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/config.py", line 298, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/config.py", line 293, in main
    self.run_cmd(cfg, options)
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/config.py", line 279, in run_cmd
    **dict((k, getattr(options, k)) for k in kwarg)
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/command.py", line 258, in stamp
    script.run_env()
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/script.py", line 203, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/util.py", line 212, in load_python_file
    module = load_module_py(module_id, path)
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/compat.py", line 58, in load_module_py
    mod = imp.load_source(module_id, path, fp)
  File "tokenserver/assignment/sqlnode/migrations/env.py", line 7, in <module>
    from tokenserver.util import find_config_file
ImportError: No module named tokenserver.util



I'll leave the staging servers up stage: ts-s-2014-04-24

Comment 10

[Benson Wong [:mostlygeek]]

The problem was not with the code, but with the RPM build. The tag 1.2.4 was repointed at a new commit hash and git doesn't update local tags when that happens. Fixed in the RPM build script.

Comment 11

[Benson Wong [:mostlygeek]]

Ran the DB migration commands. One error, but maybe a mid-air collision with something else that was done before. 

Note: The add nodeid column took < 2minutes. 

Output dump

[bwong@ip-10-150-62-67 tokenserver]$ ./bin/alembic stamp 17d209a72e2f
INFO  [alembic.migration] Context impl MySQLImpl.
INFO  [alembic.migration] Will assume non-transactional DDL.

[bwong@ip-10-150-62-67 tokenserver]$ ./bin/alembic downgrade base
INFO  [alembic.migration] Context impl MySQLImpl.
INFO  [alembic.migration] Will assume non-transactional DDL.
INFO  [alembic.migration] Running downgrade 17d209a72e2f -> None, add replaced_at_idx
Traceback (most recent call last):
  File "./bin/alembic", line 12, in <module>
    load_entry_point('alembic==0.6.4', 'console_scripts', 'alembic')()
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/config.py", line 298, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/config.py", line 293, in main
    self.run_cmd(cfg, options)
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/config.py", line 279, in run_cmd
    **dict((k, getattr(options, k)) for k in kwarg)
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/command.py", line 151, in downgrade
    script.run_env()
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/script.py", line 203, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/util.py", line 212, in load_python_file
    module = load_module_py(module_id, path)
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/compat.py", line 58, in load_module_py
    mod = imp.load_source(module_id, path, fp)
  File "tokenserver/assignment/sqlnode/migrations/env.py", line 75, in <module>
    run_migrations_online()
  File "tokenserver/assignment/sqlnode/migrations/env.py", line 68, in run_migrations_online
    context.run_migrations()
  File "<string>", line 7, in run_migrations
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/environment.py", line 681, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/migration.py", line 225, in run_migrations
    change(**kw)
  File "tokenserver/assignment/sqlnode/migrations/versions/17d209a72e2f_add_replaced_at_idx.py", line 24, in downgrade
    op.drop_index('replaced_at_idx', 'users')
  File "<string>", line 7, in drop_index
  File "<string>", line 1, in <lambda>
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/util.py", line 329, in go
    return fn(*arg, **kw)
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/operations.py", line 793, in drop_index
    self._index(name, table_name, ['x'], schema=schema)
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/ddl/impl.py", line 164, in drop_index
    self._exec(schema.DropIndex(index))
  File "/data/tokenserver/lib/python2.6/site-packages/alembic/ddl/impl.py", line 76, in _exec
    conn.execute(construct, *multiparams, **params)
  File "/data/tokenserver/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 664, in execute
    return meth(self, multiparams, params)
  File "/data/tokenserver/lib/python2.6/site-packages/sqlalchemy/sql/ddl.py", line 67, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/data/tokenserver/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 718, in _execute_ddl
    compiled
  File "/data/tokenserver/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 874, in _execute_context
    context)
  File "/data/tokenserver/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1023, in _handle_dbapi_exception
    exc_info
  File "/data/tokenserver/lib/python2.6/site-packages/sqlalchemy/util/compat.py", line 185, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/data/tokenserver/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 867, in _execute_context
    context)
  File "/data/tokenserver/lib/python2.6/site-packages/sqlalchemy/engine/default.py", line 388, in do_execute
    cursor.execute(statement, parameters)
  File "/data/tokenserver/lib/python2.6/site-packages/pymysql/cursors.py", line 102, in execute
    result = self._query(query)
  File "/data/tokenserver/lib/python2.6/site-packages/pymysql/cursors.py", line 202, in _query
    conn.query(q)
  File "/data/tokenserver/lib/python2.6/site-packages/pymysql/connections.py", line 734, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/data/tokenserver/lib/python2.6/site-packages/pymysql/connections.py", line 845, in _read_query_result
    result.read()
  File "/data/tokenserver/lib/python2.6/site-packages/pymysql/connections.py", line 1049, in read
    first_packet = self.connection._read_packet()
  File "/data/tokenserver/lib/python2.6/site-packages/pymysql/connections.py", line 826, in _read_packet
    packet.check_error()
  File "/data/tokenserver/lib/python2.6/site-packages/pymysql/connections.py", line 373, in check_error
    raise_mysql_exception(self.__data)
  File "/data/tokenserver/lib/python2.6/site-packages/pymysql/err.py", line 117, in raise_mysql_exception
    _check_mysql_exception(errinfo)
  File "/data/tokenserver/lib/python2.6/site-packages/pymysql/err.py", line 113, in _check_mysql_exception
    raise InternalError(errno, errorvalue)
sqlalchemy.exc.InternalError: (InternalError) (1091, u"Can't DROP 'replaced_at_idx'; check that column/key exists") '\nDROP INDEX replaced_at_idx ON users' ()

[bwong@ip-10-150-62-67 tokenserver]$ ./bin/alembic upgrade head
INFO  [alembic.migration] Context impl MySQLImpl.
INFO  [alembic.migration] Will assume non-transactional DDL.
INFO  [alembic.migration] Running upgrade 17d209a72e2f -> 3d5af3924466, drop clientstate_idx
INFO  [alembic.migration] Running upgrade 3d5af3924466 -> 846f28d1b6f, add nodeid column

Comment 12

[Ryan Kelly [:rfkelly]]

It looks like stage did not have the expected "replaced_at_idx" that I thought was created in Bug 986204, perhaps it was manually removed.  Otherwise this all seems to be working as expected.

Benson, can you please check on the prod db and confirm what indexes is has currently?  "show indexes from users" should give me the info I need.

Comment 13

[Ryan Kelly [:rfkelly]]

> Note: The add nodeid column took < 2minutes. 

The real fun will be in the next deploy, where we populate this column for all existing records...

Comment 14

[Ryan Kelly [:rfkelly]]

I did the following to fixup the migration state on stage:

    # Un-apply the two successfully-applied migrations
    ./bin/alembic downgrade 17d209a72e2f 

    # Update version number to reflect actual state of db
    ./bin/alembic stamp base

    # Re-apply all db migrations
    ./bin/alembic upgrade head

All seems to be well in the db.

Comment 15

[James Bonacci [:jbonacci]]

30 minute load test was successful.
No unusual issues or results.

Comment 16

[Benson Wong [:mostlygeek]]

:rfkelly

+-------+------------+-----------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name        | Seq_in_index | Column_name  | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------+------------+-----------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| users |          0 | PRIMARY         |            1 | uid          | A         |       80569 |     NULL | NULL   |      | BTREE      |         |               |
| users |          0 | clientstate_idx |            1 | email        | A         |       80569 |     NULL | NULL   |      | BTREE      |         |               |
| users |          0 | clientstate_idx |            2 | service      | A         |       80569 |     NULL | NULL   |      | BTREE      |         |               |
| users |          0 | clientstate_idx |            3 | client_state | A         |       80569 |     NULL | NULL   |      | BTREE      |         |               |
| users |          1 | lookup_idx      |            1 | email        | A         |       80569 |     NULL | NULL   |      | BTREE      |         |               |
| users |          1 | lookup_idx      |            2 | service      | A         |       80569 |     NULL | NULL   |      | BTREE      |         |               |
| users |          1 | lookup_idx      |            3 | created_at   | A         |       80569 |     NULL | NULL   |      | BTREE      |         |               |
+-------+------------+-----------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+

No replaced_at_idx

Comment 17

[James Bonacci [:jbonacci]]

Marking as Verified. This is now out in Production.