1508401 - Please deploy Firefox Monitor v3.1.3 to stage then prod

Status: RESOLVED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[Lesley Norton [:lnorton]]

Please deploy the v3.1.2 tag of the blurts-server repo to http://blurts-server.stage.mozaws.net/ and then to monitor.firefox.com.

Need to run db migrations after the deploy.

Code change-list is here:

https://github.com/mozilla/blurts-server/compare/v3.1.1...v3.1.2

Comment 1

[Luke Crouch [:groovecoder]]

:bobm - the command to run knex migrations is:

knex --knexfile db/knexfile.js migrate:latest

This will add the index to the created_at column, so we can finish https://bugzilla.mozilla.org/show_bug.cgi?id=1497693

Comment 2

[Lesley Norton [:lnorton]]

:bobm Please deploy v3.1.3 tag of the blurts-server repo to http://blurts-server.stage.mozaws.net/ and then to monitor.firefox.com. 

Code change-list is here:
https://github.com/mozilla/blurts-server/compare/v3.1.2...v3.1.3

Comment 3

[Bob Micheletto [:bobm]]

3.1.3 has been deployed to stage.

Comment 4

[Bob Micheletto [:bobm]]

Knex migration completed in stage:

$ ./node_modules/knex/bin/cli.js --knexfile db/knexfile.js migrate:latest
Working directory changed to ~/db
Using environment: stage
Batch 5 run: 1 migrations
/app/db/migrations/20181108151941_add_created_at_index.js

Comment 5

[Bob Micheletto [:bobm]]

Stage QA ok'd by :lesleynorton over IRC.  Production deployment complete.  

Knex migration output in production:

$ ./node_modules/knex/bin/cli.js --knexfile db/knexfile.js migrate:latest
Working directory changed to ~/db
Using environment: prod
Batch 5 run: 1 migrations
/app/db/migrations/20181108151941_add_created_at_index.js

Comment 6

[Luke Crouch [:groovecoder]]

Thanks :bobm. Can you try this in stage to spot-check the script before we make it a cron job?


SELECT COUNT(id), verified FROM subscribers GROUP BY verified;

node scripts/node scripts/delete-unverified-subscribers.js

SELECT COUNT(id), verified FROM subscribers GROUP BY verified;


After the script runs, we should see the same number of `t` records, and no `f` records.

Comment 7

[Luke Crouch [:groovecoder]]

:bobm - can you also deploy this to https://stage.firefoxmonitor.nonprod.cloudops.mozgcp.net/ so that :ciprian_georgiu can QA the GCP environment with the latest branch of the code?

Comment 8

[Jeremy Orem [:oremj]]

v3.1.3 has been deploy to https://stage.firefoxmonitor.nonprod.cloudops.mozgcp.net/

Comment 9

[Jeremy Orem [:oremj]]

fxmonitor=> SELECT COUNT(id), verified FROM subscribers GROUP BY verified;
 count | verified
-------+----------
  6369 | f
    84 | t
(2 rows)

/usr/bin/docker run -it --env-file /etc/dockerflow/blurts-server.txt hub.prod.mozaws.net/pipelines/blurts-server:v3.1.3-105 node scripts/delete-unverified-subscribers.js

fxmonitor=> SELECT COUNT(id), verified FROM subscribers GROUP BY verified;
 count | verified
-------+----------
     3 | f
    84 | t
(2 rows)

I ran it a few times and there are always 3 left unverified.

Comment 10

[Luke Crouch [:groovecoder]]

Can you check if those 3 records have created_at values? If they are NULL values that may not match the SQL used in the script.

Comment 11

[Bob Micheletto [:bobm]]

(In reply to Luke Crouch [:groovecoder] from comment #10)
> Can you check if those 3 records have created_at values? If they are NULL
> values that may not match the SQL used in the script.

fxmonitor=> SELECT id, created_at FROM subscribers WHERE verified is FALSE;
  id  |          created_at
------+-------------------------------
 6467 | 2018-11-20 23:56:07.525889+00
 6469 | 2018-11-20 23:56:17.207854+00
 6470 | 2018-11-20 23:56:18.739375+00
(3 rows)

Comment 12

[Luke Crouch [:groovecoder]]

Oh, that makes sense then. Those records are not past the 24-hour expiration time, so they should be preserved.

It should be fine to run the script on production, and to set it up to run on a cron job. Every hour should be sufficient.

Comment 13

[Luke Crouch [:groovecoder]]

https://stage.firefoxmonitor.nonprod.cloudops.mozgcp.net/?breach=xHamster has an error. Does the GCP environment bounce the server during deploy? That should trigger a reload of breach data from HIBP.

Comment 14

[Luke Crouch [:groovecoder]]

:bobm - it looks like https://stage.firefoxmonitor.nonprod.cloudops.mozgcp.net/?breach=xHamster has no "subscribers" table and that's causing many failures. Can we snapshot the db from aws stage environment over to GCP?

Comment 15

[Luke Crouch [:groovecoder]]

Ping :bobm - can we get the subscribers table snapshot over to GCP ASAP so :ciprian_georgiu can run the GCP environment thru the QA test plan?

Comment 16

[Bob Micheletto [:bobm]]

(In reply to Luke Crouch [:groovecoder] from comment #15)
> Ping :bobm - can we get the subscribers table snapshot over to GCP ASAP so
> :ciprian_georgiu can run the GCP environment thru the QA test plan?

Done.

Comment 17

[Luke Crouch [:groovecoder]]

Hmm ... still getting an error on https://stage.firefoxmonitor.nonprod.cloudops.mozgcp.net/?breach=xHamster.

At least I don't see any new sentry errors for the pglib connection. [1]

Now it looks like the node process is unable to load breaches from HIBP.

Did the node process restart so it could reload HIBP breaches and (re)connect to the db? Can you check that the HIBP_API_ROOT, HIBP_API_TOKEN, HIBP_KANON_API_ROOT, and HIBP_KANON_API_TOKEN match our AWS stage environment too?

https://sentry.prod.mozaws.net/operations/firefox-monitor/?query=is%3Aunresolved+environment%3Astage

Comment 18

[Luke Crouch [:groovecoder]]

:cpirian_georgiu - I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1510349 for the meta tracking bug to migrate to GCP and cc'd you there.

:bobm - to close this bug out, can we verify that we've run this script on the production environment?

node scripts/delete-unverified-subscribers.js

Please paste the before + after query results as in https://bugzilla.mozilla.org/show_bug.cgi?id=1508401#c9.

And verify the production environment has this job enabled on an hourly schedule?

Comment 19

[Bob Micheletto [:bobm]]

(In reply to Luke Crouch [:groovecoder] from comment #18)

fxmonitor=> SELECT COUNT(id), verified FROM subscribers GROUP BY verified;
 count  | verified
--------+----------
  50497 | f
 251426 | t
(2 rows)


/usr/bin/docker run -it --env-file /etc/dockerflow/blurts-server.txt hub.prod.mozaws.net/pipelines/blurts-server:v3.1.3-105 node scripts/delete-unverified-subscribers.js

fxmonitor=> SELECT COUNT(id), verified FROM subscribers GROUP BY verified;
 count  | verified
--------+----------
    552 | f
 251426 | t
(2 rows)

> And verify the production environment has this job enabled on an hourly schedule?

Not yet, we'll need a new deployment for that.