1090412 - Move Tokenserver+Verifier Stage to new development IAM

Status: VERIFIED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task, P1)

Description

[James Bonacci [:jbonacci]]

Because we need an official bug for QA to track.

Comment 1

[James Bonacci [:jbonacci]]

REF: https://github.com/mozilla-services/svcops/pull/286

Comment 2

[Benson Wong [:mostlygeek]]

token.stage.mozaws.net points at the new stack. This also has a new RDS server so we might need to do some work if we need the data from the old RDS server.

Comment 3

[James Bonacci [:jbonacci]]

I can't imagine why we would. No active testing is happening in Stage that would require transfer of data...
Anyway, cool. I will start verifying this asap...

Comment 4

[James Bonacci [:jbonacci]]

Still in the process of checking things out, but just pointing out a few changes:

Instance size is m3.medium here. I think the last deploy to old Dev IAM it was a c3.large.
There are fewer "token-prod.ini" processes running: 4 instead of 6
I could not find this token process:
token	local/bin/python -m tokenserver.scripts.process_account_deletions -v ./token-prod.ini ts-account-change-stage

The verifier_out.log shows a couple of these from today:
{"op":"bid.v2","name":"bid.v2","time":"2014-10-29T20:07:59.786Z","pid":3015,"v":1,"hostname":"ip-172-31-39-215","message":"verify","result":"failure","reason":"timeout trying to load well-known for api-accounts.stage.mozaws.net","assertion":"...etc...,"trustedIssuers":["api.accounts.firefox.com","api-accounts.stage.mozaws.net","api-accounts-dev.stage.mozaws.net","api-accounts.dev.lcip.org","api-accounts-latest.dev.lcip.org","nightly.dev.lcip.org","latest.dev.lcip.org"],"rp":"https://token.stage.mozaws.net"}

The process_account_deletions.error.log shows a lot of traceback from today and earlier:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/runpy.py", line 122, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.6/runpy.py", line 34, in _run_code
    exec code in run_globals
  File "/data/tokenserver/tokenserver/scripts/process_account_deletions.py", line 121, in <module>
    tokenserver.scripts.run_script(main)
  File "tokenserver/scripts/__init__.py", line 19, in run_script
    exitcode = main()
  File "/data/tokenserver/tokenserver/scripts/process_account_deletions.py", line 116, in main
    opts.aws_region, opts.queue_wait_time)
  File "/data/tokenserver/tokenserver/scripts/process_account_deletions.py", line 81, in process_account_deletions
    queue.delete_message(msg)
  File "/data/tokenserver/local/lib/python2.6/site-packages/boto/sqs/queue.py", line 299, in delete_message
    return self.connection.delete_message(self, message)
  File "/data/tokenserver/local/lib/python2.6/site-packages/boto/sqs/connection.py", line 208, in delete_message
    params = {'ReceiptHandle' : message.receipt_handle}
AttributeError: 'str' object has no attribute 'receipt_handle'


Other than that, everything checks out ok.
I can load test once the loads cluster is back up and running (it is being migrated right now I think)...

:mostlygeek please check the logs and let me know if that is all expected...

Comment 5

[Benson Wong [:mostlygeek]]

Hmm... seems like it is a bug in boto. Tokenserver lists boto=2.27, boto=2.34 is out. Maybe upgrading to that will fix this issue. 

Also we're on tokenserver 1.2.9, and there is a 1.2.10 with what looks like some pretty extensive changes. Maybe should roll 1.2.11 with the more recent boto and give it a test. 

:rfkelly what do you think?

Comment 6

[Ryan Kelly [:rfkelly]]

>  Maybe should roll 1.2.11 with the more recent boto and give it a test. 

Yes, we should definitely do this.  I'll prep it today.

Comment 7

[James Bonacci [:jbonacci]]

OK, so POC that the migration was a happy one.
Next deployment ticket for TS+Verifier will go to new Dev IAM.