1118455 - Deploy relengapi-2.0.0

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

This includes a bunch of supporting features for porting things like BuildAPI into RelengAPI.

https://github.com/mozilla/build-relengapi/milestones/v1.2.0

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

This already involves some breaking changes from bug 1132957

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

2.0.0 is almost completely reviewed and ready for deployment.  I'll try this in staging, first.  The complications I can see are:
 * addition of three columns is required to the auth_tokens table, but it should be possible to do this beforehand without impacting operation of relengapi-1.1.1.
 * deployment of the new version requires a concurrent change to the permission names in the configuration file

A few minor things we don't need to worry about during transition, but should address:
 * mapper is using v1 tokens, which will still work, but it'd be nice to upgrade
 * mapper is using the incorrect 'Authentication' header, and should be converted to use 'Authorization'

All in all, we *should* be able to do this without downtime.

Comment 3

[Justin Wood (:Callek)]

:dustin,

One thing I hit in my local testing when I upgraded relengapi-core from a 1.1 based auth to 2.0 based auth was that a permission that I had while logged in (one of the token ones), once upgraded the session I had was still valid, however the token was not, so I was getting server errors loading pages and had to *manually* nav to the logout screen to support things.

I suggest a plan for that case before we move too far forward with deployment here, whether that be invalidating *all* session data (so all users need to re-log in), or if its something like "if a user has perms that don't actually exist in the code, we force a logout and then they have to log in again to get updated perms in their session) or something ;-)

I'm not too picky on the solution as long as we don't break the state for those users.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

The permissions have a lifetime, currently set to 1h, before they are refreshed.  I'll dial that down to 60s now.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

In staging, before pushing the update:

mysql> alter table auth_tokens add column `typ` varchar(4) NOT NULL default 'prm';
Query OK, 0 rows affected (0.15 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> alter table auth_tokens add column `user` text;
Query OK, 0 rows affected (0.02 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> alter table auth_tokens add column `disabled` tinyint(1) NOT NULL default 0;
Query OK, 0 rows affected (0.03 sec)
Records: 0  Duplicates: 0  Warnings: 0

** deploy here -- remainder should happen afterward or issuing tokens will fail **

mysql> alter table auth_tokens alter column `disabled` drop default;
Query OK, 0 rows affected (0.01 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> alter table auth_tokens alter column `typ` drop default;
Query OK, 0 rows affected (0.01 sec)
Records: 0  Duplicates: 0  Warnings: 0

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Also (missed this in release notes, but it's fairly minor) badpenny_jobs.task_id is now NOT NULL:

mysql> alter table badpenny_jobs modify column task_id int(11) not null;
Query OK, 0 rows affected (0.08 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> alter table badpenny_jobs alter column task_id drop default;
Query OK, 0 rows affected (0.00 sec)
Records: 0  Duplicates: 0  Warnings: 0

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

I had to upgrade Sphinx due to the theme change, and that pulled in a bunch of other Python requirements.  I'm verifying that badpenny is working in staging now, which is a bit of a hairy yak.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

We're going to need to fix https://github.com/mozilla/build-relengapi/issues/202 before deploying.

Comment 9

[Justin Wood (:Callek)]

(In reply to Dustin J. Mitchell [:dustin] from comment #7)
> I had to upgrade Sphinx due to the theme change, and that pulled in a bunch
> of other Python requirements.  I'm verifying that badpenny is working in
> staging now, which is a bit of a hairy yak.

for sanity can we pip freeze current production vs staging.

So we can identify what deps upgraded.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

--- /root/stage.txt     2015-04-02 07:02:07.668740812 -0700
+++ /root/prod.txt      2015-04-02 07:02:22.233986713 -0700
@@ -1,8 +1,6 @@
-alabaster (0.7.3)
 amqp (1.4.6)
 anyjson (0.3.3)
 argparse (1.1)
-Babel (1.3)
 billiard (3.3.0.18)
 blinker (1.3)
 boto (2.32.1)
@@ -14,6 +12,7 @@
 Flask (0.10.1)
 Flask-BrowserID (0.0.4)
 Flask-Login (0.2.10)
+Flask-Principal (0.4.0)
 ipaddr (2.1.11)
 IPy (0.75)
 itsdangerous (0.23)
@@ -22,13 +21,14 @@
 MarkupSafe (0.18)
 MySQL-python (1.2.5)
 newrelic (2.46.0.37)
+oauthlib (0.6.1)
 pip (1.5.4)
-Pygments (2.0.2)
+Pygments (1.6)
 python-dateutil (1.5)
 python-ldap (2.4.15)
 python-memcached (1.53)
 pytz (2014.1)
-relengapi (2.0.1)
+relengapi (1.1.1)
 relengapi-clobberer (0.5.1)
 relengapi-mapper (0.2.2)
 requests (2.4.3)
@@ -36,9 +36,7 @@
 simplegeneric (0.8)
 simplejson (3.3.0)
 six (1.6.1)
-snowballstemmer (1.2.0)
-Sphinx (1.3.1)
-sphinx-rtd-theme (0.1.7)
+Sphinx (1.2.2)
 SQLAlchemy (0.9.4)
 WebOb (1.2.3)
 Werkzeug (0.9.3)

some of those are leftovers, but better safe than sorry.  BTW I'm going to look at deploying this using a PaaS tool - Heroku, Deis, etc. - which would (I think) eliminate some of these concerns.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Woo, with relengapi-2.0.2 (coincidentally including only PR 202), badpenny works fine in staging.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

waiting for hal at 10:15am pacific before deploying to prod

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

Deployed, including the cleanup DB tasks.