1496559 - Deploy Autograph 2.4.0 to cloudservices stage and prod

Status: RESOLVED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[Greg G]

Placeholder for a deploy the week of Oct. 15th 2018.

Please deploy Autograph {{TAG}} to staging and perform QA testing.
https://mana.mozilla.org/wiki/display/SVCOPS/Autograph#Autograph-QA

Comment 1

[Greg G]

tag: https://github.com/mozilla-services/autograph/releases/tag/2.4.0

CI jobs: 

* https://circleci.com/gh/mozilla-services/autograph/514
* https://circleci.com/gh/mozilla-services/autograph/515
* https://circleci.com/gh/mozilla-services/autograph/516

docker image: https://hub.docker.com/r/mozilla/autograph/tags/

code changes from 2.3.2: 

https://github.com/mozilla-services/autograph/compare/2.3.2..2.4.0

* add ulfr's gpg signer https://github.com/mozilla-services/autograph/pull/140
* wrap APK manifest lines per the JAR spec https://github.com/mozilla-services/autograph/pull/145 (XPI manifests are unchanged)
* add an option for zip compression settings https://github.com/mozilla-services/autograph/pull/153  (NB: don't use this unless the current "compress everything" strategy is breaking media loading like it does for FxR sound files since it can output unaligned ZIPs https://github.com/mozilla-services/autograph/issues/154)
* remove MD5 hashes in XPI manifest and signature files https://github.com/mozilla-services/autograph/pull/142

Comment 2

[Greg G]

Generated and published a 4096-bit RSA gpg to https://gpg.mozilla.org/pks/lookup?search=0x45009525A93D2F49&op=vindex

Added it to the stage config, tested autograph against the config, added revocation cert to hiera sops, and deleted the keys from my local keyring.

Comment 3

[Greg G]

Updated the monitor deployment code and redeployed it to stage and prod (to accommodate the new gpg signer in stage). Logs look OK.

Comment 4

[Greg G]

Renewed content sig certs and tested the updated configs run with autograph and the monitor runs against them (haven't gotten the updated monitor to build locally but the other stage signers pass).

Deploying stage.

Comment 5

[Greg G]

Stage is deployed. Starting QA.

Comment 6

[Greg G]

Stage QA:

b. content signature expirations; dummy gpg signer is present; other signers OK.

c. Invoking from the stage AWS console with the test event, I get the error "Calling the invoke API action failed with this message: [object Object]". Invoking from the CLI yields:

> aws lambda invoke --function-name kinto-lambda-refresh_signature --payload '{"server": "https://settings-writer.stage.mozaws.net/v1"}' kinto-refresh-call.out
{
    "StatusCode": 200,
    "FunctionError": "Unhandled",
    "ExecutedVersion": "$LATEST"
}

where the output file contains permissions errors:

> cat kinto-refresh-call.out
{"errorMessage": "\nGET /v1/buckets/main-workspace/collections/sites-classification - 403 403 - {'code': 403, 'errno': 121, 'error': 'Forbidden', 'message': 'This user cannot access this resource.'}\n\nGET /v1/buckets/main-workspace/collections/personality-provider-models - 403 403 - {'code': 403, 'errno': 121, 'error': 'Forbidden', 'message': 'This user cannot access this resource.'}\n\nGET /v1/buckets/main-workspace/collections/personality-provider-recipe - 403 403 - {'code': 403, 'errno': 121, 'error': 'Forbidden', 'message': 'This user cannot access this resource.'}", "errorType": "RefreshError", "stackTrace": [["/var/task/aws_lambda.py", 260, "refresh_signature", "raise RefreshError(\"\\n\" + \"\\n\\n\".join(error_messages))"]]}


:leplatrem does that error ^ look familiar? is https://settings-writer.stage.mozaws.net/v1 the right server to call?

Not sure who owns those buckets, so this could also be a permissions error on our side too.

d. signed the test addon and it looks fine

e. bpitts can you run "./manage.py update_signatures --force" on the SHIELD stage admin sometime today?

Comment 7

[Greg G]

The logs for that kinto refresh lambda invocation do show a bunch of collections being signed successfully:

START RequestId: c266e4c7-d089-11e8-b78f-a1c5370abcbf Version: $LATEST
Looking at /buckets/monitor/collections/changes:
Looking at /buckets/main-workspace/collections/sites-classification: GET /v1/buckets/main-workspace/collections/sites-classification - 403 403 - {'code': 403, 'errno': 121, 'error': 'Forbidden', 'message': 'This user cannot access this resource.'}
Looking at /buckets/main-workspace/collections/personality-provider-models: GET /v1/buckets/main-workspace/collections/personality-provider-models - 403 403 - {'code': 403, 'errno': 121, 'error': 'Forbidden', 'message': 'This user cannot access this resource.'}
Looking at /buckets/main-workspace/collections/personality-provider-recipe: GET /v1/buckets/main-workspace/collections/personality-provider-recipe - 403 403 - {'code': 403, 'errno': 121, 'error': 'Forbidden', 'message': 'This user cannot access this resource.'}
Looking at /buckets/main-workspace/collections/product-integrity: Refresh signature: status= signed at 2018-10-15 14:51:36 UTC ( 1539615096905 )
Looking at /buckets/main-workspace/collections/language-dictionaries: Refresh signature: status= signed at 2018-10-15 14:51:37 UTC ( 1539615097625 )
Looking at /buckets/main-workspace/collections/focus-experiments: Refresh signature: status= signed at 2018-10-15 14:51:38 UTC ( 1539615098405 )
Looking at /buckets/main-workspace/collections/cfr: Refresh signature: status= None at 2018-10-15 14:51:39 UTC ( 1539615099061 )
Looking at /buckets/main-workspace/collections/onboarding: Refresh signature: status= signed at 2018-10-15 14:51:39 UTC ( 1539615099763 )
Looking at /buckets/main-workspace/collections/tippytop: Refresh signature: status= signed at 2018-10-15 14:51:40 UTC ( 1539615100421 )
Looking at /buckets/security-state-staging/collections/cert-revocations: Refresh signature: status= None at 2018-10-15 14:51:41 UTC ( 1539615101259 )
Looking at /buckets/main-workspace/collections/rocket-prefs: Refresh signature: status= None at 2018-10-15 14:51:41 UTC ( 1539615101900 )
Looking at /buckets/main-workspace/collections/rocket-releases: Refresh signature: status= None at 2018-10-15 14:51:42 UTC ( 1539615102541 )
Looking at /buckets/staging/collections/qa: Refresh signature: status= to-review at 2018-10-15 14:51:43 UTC ( 1539615103144 )
Looking at /buckets/security-state-staging/collections/intermediates: Refresh signature: status= work-in-progress at 2018-10-15 14:51:43 UTC ( 1539615103854 )
Looking at /buckets/pinning-staging/collections/pins: Refresh signature: status= signed at 2018-10-15 14:51:44 UTC ( 1539615104573 )
Looking at /buckets/staging/collections/plugins: Refresh signature: status= signed at 2018-10-15 14:51:45 UTC ( 1539615105224 )
Looking at /buckets/staging/collections/certificates: Refresh signature: status= to-review at 2018-10-15 14:51:45 UTC ( 1539615105983 )
Looking at /buckets/staging/collections/addons: Refresh signature: status= to-review at 2018-10-15 14:51:47 UTC ( 1539615107005 )
Looking at /buckets/staging/collections/gfx: Refresh signature: status= signed at 2018-10-15 14:51:47 UTC ( 1539615107983 )

GET /v1/buckets/main-workspace/collections/sites-classification - 403 403 - {'code': 403, 'errno': 121, 'error': 'Forbidden', 'message': 'This user cannot access this resource.'}

GET /v1/buckets/main-workspace/collections/personality-provider-models - 403 403 - {'code': 403, 'errno': 121, 'error': 'Forbidden', 'message': 'This user cannot access this resource.'}

GET /v1/buckets/main-workspace/collections/personality-provider-recipe - 403 403 - {'code': 403, 'errno': 121, 'error': 'Forbidden', 'message': 'This user cannot access this resource.'}: RefreshError
Traceback (most recent call last):
File "/var/task/aws_lambda.py", line 260, in refresh_signature
raise RefreshError("\n" + "\n\n".join(error_messages))
aws_lambda.RefreshError:
GET /v1/buckets/main-workspace/collections/sites-classification - 403 403 - {'code': 403, 'errno': 121, 'error': 'Forbidden', 'message': 'This user cannot access this resource.'}

GET /v1/buckets/main-workspace/collections/personality-provider-models - 403 403 - {'code': 403, 'errno': 121, 'error': 'Forbidden', 'message': 'This user cannot access this resource.'}

GET /v1/buckets/main-workspace/collections/personality-provider-recipe - 403 403 - {'code': 403, 'errno': 121, 'error': 'Forbidden', 'message': 'This user cannot access this resource.'}

END RequestId: c266e4c7-d089-11e8-b78f-a1c5370abcbf
REPORT RequestId: c266e4c7-d089-11e8-b78f-a1c5370abcbf Duration: 13375.29 ms Billed Duration: 13400 ms Memory Size: 128 MB Max Memory Used: 53 MB

Comment 8

[:wezhou]

The 403 errors should have gone now now that what's in https://bugzilla.mozilla.org/show_bug.cgi?id=1498664 is applied to -stage.

Comment 9

[Greg G]

(In reply to :wezhou from comment #8)
> The 403 errors should have gone now now that what's in
> https://bugzilla.mozilla.org/show_bug.cgi?id=1498664 is applied to -stage.

Thanks :wezhou! I kicked off the refresh lambda from the command line again and just see 403s for two collections now:

collections/personality-provider-models
collections/personality-provider-recipe

{"errorMessage": "\nGET /v1/buckets/main-workspace/collections/personality-provider-models - 403 403 - {'code': 403, 'errno': 121, 'error': 'Forbidden', 'message': 'This user cannot access this resource.'}\n\nGET /v1/buckets/main-workspace/collections/personality-provider-recipe - 403 403 - {'code': 403, 'errno': 121, 'error': 'Forbidden', 'message': 'This user cannot access this resource.'}", "errorType": "RefreshError", "stackTrace": [["/var/task/aws_lambda.py", 261, "refresh_signature", "raise RefreshError(\"\\n\" + \"\\n\\n\".join(error_messages))"]]}

:wezhou do you know if those errors should block a prod deploy?

Comment 10

[Miles Crabill [:miles]]

I ran "./manage.py update_signatures --force" on the SHIELD stage admin node successfully.

Comment 11

[:wezhou]

> collections/personality-provider-models
> collections/personality-provider-recipe

Someone must have created those collections *AFTER* we cleaned up last 403 errors.

This proved again the point :leplatrem made in [1] where he said, "I am more and more convinced that letting users create collections is not entirely a good idea", which I agree, for it looks like users may leave behind uncleaned data which break other tests.


> :wezhou do you know if those errors should block a prod deploy?

From operations perspective, probably no, because these two collections don't even exist on -prod. But let's wait to see if :leplatrem agrees.


[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1495492#c4

Comment 12

[Greg G]

Thanks :wezhou and :miles!

In about:remotesettings both stage and stage preview envs sync and poll data without sign errors for all collections now, so I think we're fine to deploy to prod.

I pinged :leplatrem on irc about the 403s just to be sure.

Comment 13

[Greg G]

Deploying to prod.

Comment 14

[Greg G]

Deployed successfully. Starting QA.

Comment 15

[Greg G]

Prod QA:

b. logs from the monitor showing updated 2019-01-13 expiration for content sigs and other sigs OK.

c. the kinto refresh lambda succeeded from CLI with return status:

{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}

and file output: "null"

Logs show no errors.

Polling and refreshing sync data with about:remotesettings for prod and prod preview envs does not show any errors.

d. signed the addon and its signature is correct.

e. :bpitts or :miles can one of you run "./manage.py update_signatures --force" on the shield prod admin?

Comment 16

[Miles Crabill [:miles]]

I've run "./manage.py update_signatures --force" and things look good on the prod admin node.

Comment 17

[Greg G]

Thanks :miles!