1494367 - socorro deploy: 337

Status: RESOLVED FIXED

(Socorro :: Infra, task, P1)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

We want to do a Socorro -prod deploy today tagged 337.

It consists of the following:

(current tag: 336 - 416d9d8)
0dd29b6: fix bug 1346883: remove postgres from processor (willkg)
13dc5d1: refactor python3 test targets to match recent updates to primary test targets (lonnen)
dd46663: Clean up Makefile; add lint3 rule (willkg)
5530a46: fix bug 1468815: switch to a single docker image (willkg)
9540fb4: Fix lint in CI (willkg)
1c18f11: Fix lint in CI to run with the right service (willkg)
fec00c4: Remove old comments (willkg)
43f7fb5: Downgrade to same image everything else uses (willkg)
a7c6af0: Add note about gawk being required for sentry-cli (willkg)
5ae9c40: fix bug 1466974: set COMPOSE_PROJECT_NAME in .env (willkg)
1d86c70: fix bug 1488774: remove cv/ref qualifiers in function names (willkg)
046aa45: fix bug 1470704: add timing metrics for processor rules (willkg)
34b30bf: fix bug 1470704: change key into a tag (willkg)
f4b26d0: fix bug 1478080: vendor crontabber (willkg)
eee85d1: Remove boundary lines (willkg)
207cae4: Merge app into crontabber_app (willkg)
921d0fb: Cosmetic style fixing (willkg)
12fd5e4: fix bug 1470703: add crontabber job failure/success metrics (willkg)
4b9e557: fix bug 1493200: fix empty string signature generation (willkg)
e88fa32: Fix comment (willkg)
de1a0ff: fixi bug 1389197: fix os_name to be "Unknown" and not empty string (willkg)
b4e9438: fix bug 1459216: drop signature_products and signature_products_rollup (willkg)
f967daa: fix bug 1435076: remove Correlations model (willkg)
563b49d: fix bug 1494136: switch to postgres for webapp tests (willkg)
51b1030: fix bug 1470704: fix rule metrics key (willkg)
a17bb72: fix bug 1457747: move graphics_device to Django (willkg)
20530e0: Remove edit/lookup form and add Django model admin (willkg)
6d6a7c5: Fix table and migration after testing with prod data (willkg)
(next tag: 337 - 2c53e98)

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Grabbing this to do now.

There's a *LOT* of stuff here.

0dd29b6: fix bug 1346883: remove postgres from processor (willkg)

This has been on stage for a while and it's fine.

13dc5d1: refactor python3 test targets to match recent updates to primary test targets (lonnen)
dd46663: Clean up Makefile; add lint3 rule (willkg)
5530a46: fix bug 1468815: switch to a single docker image (willkg)
9540fb4: Fix lint in CI (willkg)
1c18f11: Fix lint in CI to run with the right service (willkg)
fec00c4: Remove old comments (willkg)
43f7fb5: Downgrade to same image everything else uses (willkg)
a7c6af0: Add note about gawk being required for sentry-cli (willkg)

This switches Socorro from one docker image per service to a single docker image that runs all the services (except the collector--that's still separate). This has been on stage for a while and it's fully tested: local dev environment, CI, deploys, running migrations, crontabber jobs, and running services.

5ae9c40: fix bug 1466974: set COMPOSE_PROJECT_NAME in .env (willkg)

This fixes the tests to run when the repo directory isn't "socorro".

1d86c70: fix bug 1488774: remove cv/ref qualifiers in function names (willkg)

When looking into whether this is working on stage, I discovered that I fixed one cause of the issue, but there must be another cause. I reopened the bug to look into it further.

046aa45: fix bug 1470704: add timing metrics for processor rules (willkg)
34b30bf: fix bug 1470704: change key into a tag (willkg)
51b1030: fix bug 1470704: fix rule metrics key (willkg)

Timing rules for the processor are fine. I added a dashboard graph.

f4b26d0: fix bug 1478080: vendor crontabber (willkg)
eee85d1: Remove boundary lines (willkg)
207cae4: Merge app into crontabber_app (willkg)
921d0fb: Cosmetic style fixing (willkg)

Vendoring crontabber blocked a bunch of other work. This has been on stage for a while and looks great.

12fd5e4: fix bug 1470703: add crontabber job failure/success metrics (willkg)

I added two dashboard graphs for this.

4b9e557: fix bug 1493200: fix empty string signature generation (willkg)
e88fa32: Fix comment (willkg)

This works on stage.

de1a0ff: fixi bug 1389197: fix os_name to be "Unknown" and not empty string (willkg)

Looks good.

b4e9438: fix bug 1459216: drop signature_products and signature_products_rollup (willkg)
f967daa: fix bug 1435076: remove Correlations model (willkg)

This is effectively dead code removal. It's fine.

563b49d: fix bug 1494136: switch to postgres for webapp tests (willkg)

This only affects tests and CI--looks good.

a17bb72: fix bug 1457747: move graphics_device to Django (willkg)
20530e0: Remove edit/lookup form and add Django model admin (willkg)
6d6a7c5: Fix table and migration after testing with prod data (willkg)

This looks good. I can see the data in the table using the Django admin. It's showing up in the signature report. Comparing the "OOM | small" signature report between stage and prod suggests that the code rewrite fixes an issue that's been out for a long time and it's actually showing lookups now. So that's cool.

I want to look at that signature generation issue. If I can fix it quickly today, I will. Otherwise I'll push it off until after a prod deploy.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

I fixed the outstanding issue. New list of commits to go out:

(current tag: 336 - 416d9d8)
0dd29b6: fix bug 1346883: remove postgres from processor (willkg)
13dc5d1: refactor python3 test targets to match recent updates to primary test targets (lonnen)
dd46663: Clean up Makefile; add lint3 rule (willkg)
5530a46: fix bug 1468815: switch to a single docker image (willkg)
9540fb4: Fix lint in CI (willkg)
1c18f11: Fix lint in CI to run with the right service (willkg)
fec00c4: Remove old comments (willkg)
43f7fb5: Downgrade to same image everything else uses (willkg)
a7c6af0: Add note about gawk being required for sentry-cli (willkg)
5ae9c40: fix bug 1466974: set COMPOSE_PROJECT_NAME in .env (willkg)
1d86c70: fix bug 1488774: remove cv/ref qualifiers in function names (willkg)
046aa45: fix bug 1470704: add timing metrics for processor rules (willkg)
34b30bf: fix bug 1470704: change key into a tag (willkg)
f4b26d0: fix bug 1478080: vendor crontabber (willkg)
eee85d1: Remove boundary lines (willkg)
207cae4: Merge app into crontabber_app (willkg)
921d0fb: Cosmetic style fixing (willkg)
12fd5e4: fix bug 1470703: add crontabber job failure/success metrics (willkg)
4b9e557: fix bug 1493200: fix empty string signature generation (willkg)
e88fa32: Fix comment (willkg)
de1a0ff: fixi bug 1389197: fix os_name to be "Unknown" and not empty string (willkg)
b4e9438: fix bug 1459216: drop signature_products and signature_products_rollup (willkg)
f967daa: fix bug 1435076: remove Correlations model (willkg)
563b49d: fix bug 1494136: switch to postgres for webapp tests (willkg)
51b1030: fix bug 1470704: fix rule metrics key (willkg)
a17bb72: fix bug 1457747: move graphics_device to Django (willkg)
20530e0: Remove edit/lookup form and add Django model admin (willkg)
6d6a7c5: Fix table and migration after testing with prod data (willkg)
b8108d7: fix bug 1488774: fix another cause of & signatures (willkg)
(next tag: 337 - f919b16)

I'll wait for a stage deploy and then verify the new fix. Then do a prod deploy.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

$ git show 337
tag 337
Tagger: Will Kahn-Greene <[e-mail]>
Date:   Wed Sep 26 15:54:58 2018 -0400

Tag 337 for deploy
-----BEGIN PGP SIGNATURE-----

iQEzBAABCgAdFiEEzY3I6/E2Vh8o9nPoGL3RpvBm7BEFAlur5BYACgkQGL3RpvBm
7BG5Igf/d3veZmrA5Nh+p+WFhdpmfkF0hvsQNTHBenl68H0OQgf0co+zj40+jkA8
hSRCRZUQlccgN9aojhus1Ir7S/o5KWevNMblf0iKqF6g6RFvv6Z+XEyGDY1jFREE
/PMOJ26UgnJKoETdq3KL2PE4b4s81tPPyVAuLkVMULZDYkUkk7yBXugtz6l8rT/O
j1L8OvKROqFPxz4mKKhwAp6e44zF/fDh6mlV8Xu68Yi3I5/bAkPYrEzJFh/A+eqJ
6B+ByHhW5vmMgKkW/YNE2THCCSoRzAvNngJ9cMR8AzuxkjohjFwUF7ZWCS6Ce4mh
U0/NBY+fatnMVRPV6Pk5DVOJQ24fGA==
=XXo+
-----END PGP SIGNATURE-----

commit f919b1690f2c4f3b42deba749ffbe98ebeb5b6b6 (HEAD -> master, tag: 337, official/master)
Merge: 2c53e987a b8108d71c
Author: Will Kahn-Greene <[e-mail]>
Date:   Wed Sep 26 14:28:55 2018 -0400

    Merge pull request #4614 from willkg/1488774-fix-amp-2
    
    fix bug 1488774: fix another cause of & signatures


After deploy, we need to do the following:

1. check to see if the data migration for graphics adapters worked
2. reprocess crash reports that have a signature of "" and "&"
3. figure out better dashboard graphs for processor rule timings and crontabber job timings

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

1. Data migration went fine. Signature reports look better than they did before. Yay!
2. I reprocessed crash reports.

I'll talk with Brian about making better dashboard graphs. I can do that outside of this bug.

I checked graphs and sentry and verything looks good.

Marking as FIXED.