Closed Bug 1148545 Opened 10 years ago Closed 10 years ago

Deploy build-tooltool to relengapi

Categories

(Infrastructure & Operations :: RelOps: General, task)

Product:
Infrastructure & Operations
Component:
RelOps: General
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dustin, Assigned: dustin)

References

Details

This involves: * landing a few more things: * https://github.com/mozilla/build-tooltool/pull/10 * https://github.com/mozilla/build-relengapi/pull/196 * https://github.com/mozilla/build-relengapi/pull/187 * releasing a new version of relengapi * building a cloudformation script to create the S3 buckets * setting up badpenny (running badpenny-cron in a crontask on webheads) * deploying to the web cluster * documenting the new upload process (it's pretty simple but requires a token) * disabling the existing upload mechanism * uploading the existing tooltool data to the new tooltool * I'll try to determine appropriate visibility levels and filenames * converting build automation to use the new client, including using a permanent token on each host
Depends on: 1118455
Assignee: relops → dustin
(In reply to Dustin J. Mitchell [:dustin] from comment #0) > This involves: > * landing a few more things: (DONE) > * releasing a new version of relengapi > * building a cloudformation script to create the S3 buckets (DONE) > * setting up badpenny (running badpenny-cron in a crontask on webheads) (DONE - Bug 1118455) > * deploying to the web cluster > * documenting the new upload process (it's pretty simple but requires a > token) (DONE) > * disabling the existing upload mechanism > * uploading the existing tooltool data to the new tooltool > * I'll try to determine appropriate visibility levels and filenames > * converting build automation to use the new client, including using a > permanent token on each host Add: * Allow all devs to upload to tooltool, generate user tokens
(In reply to Dustin J. Mitchell [:dustin] from comment #0) > This involves: > * releasing a new version of relengapi (DONE) > * deploying to the web cluster > * disabling the existing upload mechanism > * uploading the existing tooltool data to the new tooltool > * I'll try to determine appropriate visibility levels and filenames > * converting build automation to use the new client, including using a > permanent token on each host > * Allow all devs to upload to tooltool, generate user tokens
* I need to use bucket policies to better lock down access to the buckets to something narrower than just the AWS account. I should probably create the relengapi users with cloudformation too
The staging server appears not to be accepting the "Authorization" header, and even after switching the client to use "Authentication": INFO - LICENSE.txt: starting upload ERROR - LICENSE.txt: failed Traceback (most recent call last): File "tooltool.py", line 778, in _s3_upload (resp.status, resp.reason, resp_body)) RuntimeError: Non-200 return from AWS: 403 Forbidden <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>B315E8AFFCAB59B0</RequestId><HostId>FG5dturWy2xb7OqzKwrHdd9j/2uoiP9BdbwFNs+S7KwFko4iHstCuChLSv3fb+WlLdSKp+1TC6Q=</HostId></Error> (sandbox) Which suggests I haven't set up the user policy correctly.
For the Authorization/Authentication issue, https://github.com/mozilla/build-relengapi/issues/208
Auth issue required an upgrade to Flask-Login. Fixed. Now, on to the AWS permissions.
I addd a usre policy via cloudformation. That changed the error from RuntimeError: Non-200 return from AWS: 403 Forbidden <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>56A9F1DD4087560B</RequestId><HostId>YjhTGmmHWvr6SOTOpa482sjaZg526GUB6wcHB+KzozOmeU9rNiab+QQsBjF6jin6</HostId></Error> to INFO - LICENSE.txt: starting upload https://mozilla-releng-staging-usw1-tooltool.s3-us-west-1.amazonaws.com/sha512/054fdbe8cb55d1f7592871311c5a9da76710c7a085fd24457644d80aa0ea3c344c57e99aab3a6fb2ec7ed93c15c8f997d5b7b60692c318f9cacad6418bb359e1?Signature=9KGrgB8jWVaZ%2BY7jsD82UNwcIhE%3D&Expires=1428085058&AWSAccessKeyId=AKIAILABTOAMWCDH5EBA ERROR - LICENSE.txt: failed Traceback (most recent call last): File "tooltool.py", line 778, in _s3_upload (resp.status, resp.reason, resp_body)) RuntimeError: Non-200 return from AWS: 403 Forbidden <?xml version="1.0" encoding="UTF-8"?> <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>AKIAILABTOAMWCDH5EBA</AWSAccessKeyId><StringToSign>PUT application/octet-stream 1428085058 /mozilla-releng-staging-usw1-tooltool/sha512/054fdbe8cb55d1f7592871311c5a9da76710c7a085fd24457644d80aa0ea3c344c57e99aab3a6fb2ec7ed93c15c8f997d5b7b60692c318f9cacad6418bb359e1</StringToSign><SignatureProvided>9KGrgB8jWVaZ+Y7jsD82UNwcIhE=</SignatureProvided><StringToSignBytes>50 55 54 0a 0a 61 70 70 6c 69 63 61 74 69 6f 6e 2f 6f 63 74 65 74 2d 73 74 72 65 61 6d 0a 31 34 32 38 30 38 35 30 35 38 0a 2f 6d 6f 7a 69 6c 6c 61 2d 72 65 6c 65 6e 67 2d 73 74 61 67 69 6e 67 2d 75 73 77 31 2d 74 6f 6f 6c 74 6f 6f 6c 2f 73 68 61 35 31 32 2f 30 35 34 66 64 62 65 38 63 62 35 35 64 31 66 37 35 39 32 38 37 31 33 31 31 63 35 61 39 64 61 37 36 37 31 30 63 37 61 30 38 35 66 64 32 34 34 35 37 36 34 34 64 38 30 61 61 30 65 61 33 63 33 34 34 63 35 37 65 39 39 61 61 62 33 61 36 66 62 32 65 63 37 65 64 39 33 63 31 35 63 38 66 39 39 37 64 35 62 37 62 36 30 36 39 32 63 33 31 38 66 39 63 61 63 61 64 36 34 31 38 62 62 33 35 39 65 31</StringToSignBytes><RequestId>ADAED72E986778B3</RequestId><HostId>iLPz6jhTyVr7auNEfDt+g8pUlResOM47dvbpzW7E0VYrLYvmcrdxsr+5mTqgLxNT</HostId></Error> Which doesn't make much sense -- in my dev environment I've repeatedly signed real AWS URLs. I'm going to try updating boto first before digging further.
Same error with boto-2.37.0.
Derp, I had a space in the secret key :(
The verification task is failing: File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/relengapi/blueprints/tooltool/grooming.py", line 112, in verify_file_instance if key.storage_class != 'STANDARD': File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/boto/s3/key.py", line 199, in _get_storage_class list_items = list(self.bucket.list(self.name.encode('utf-8'))) File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/boto/s3/bucketlistresultset.py", line 34, in bucket_lister encoding_type=encoding_type) File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/boto/s3/bucket.py", line 472, in get_all_keys '', headers, **params) File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/boto/s3/bucket.py", line 410, in _get_all response.status, response.reason, body) S3ResponseError: S3ResponseError: 403 Forbidden <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>16EB688054EC9362</RequestId><HostId>L/STz309bVUqzrT+WmXusPUBi/ssR1QY4dqUnAdBWbxo+z1YffvHaEPswJj+IzoJ</HostId></Error> looks like the policy is wrong (listing keys is a bucket operation, but I've only applied it to object ARNs)
I'm not going to push this to production right now, though.
A successful fetch from production: dustin@euclid ~/code/relengapi/t/tooltool/x [master] $ python ../tooltool.py fetch --verbose DEBUG - processing 'fetch' command with args '' DEBUG - using options: {'cache_folder': None, 'algorithm': 'sha512', 'loglevel': 10, 'region': None, 'base_url': ['https://api.pub.build.mozilla.org/tooltool/'], 'visibility': None, 'manifest': 'manifest.tt', 'auth_file': None, 'message': None, 'overwrite': False, 'size': 0.0} DEBUG - materialized __main__.FileRecord(filename='LICENSE.txt', size=193, digest='054fdbe8cb55d1f7592871311c5a9da76710c7a085fd24457644d80aa0ea3c344c57e99aab3a6fb2ec7ed93c15c8f997d5b7b60692c318f9cacad6418bb359e1', algorithm='sha512', visibility=u'public') DEBUG - loaded manifest from file 'manifest.tt' DEBUG - fetching LICENSE.txt INFO - Attempting to fetch from 'https://api.pub.build.mozilla.org/tooltool/'... DEBUG - opened https://api.pub.build.mozilla.org/tooltool/sha512/054fdbe8cb55d1f7592871311c5a9da76710c7a085fd24457644d80aa0ea3c344c57e99aab3a6fb2ec7ed93c15c8f997d5b7b60692c318f9cacad6418bb359e1 for reading INFO - File LICENSE.txt fetched from https://api.pub.build.mozilla.org/tooltool/ as /home/dustin/code/relengapi/t/tooltool/x/tmphDWQYw DEBUG - hashed 'tmphDWQYw' with sha512 to be 054fdbe8cb55d1f7592871311c5a9da76710c7a085fd24457644d80aa0ea3c344c57e99aab3a6fb2ec7ed93c15c8f997d5b7b60692c318f9cacad6418bb359e1 INFO - File integrity verified, renaming tmphDWQYw to LICENSE.txt dustin@euclid ~/code/relengapi/t/tooltool/x [master*] $ ls -al total 16 drwxr-xr-x 2 dustin dustin 4096 Apr 6 11:09 . drwxr-xr-x 13 dustin dustin 4096 Apr 6 11:06 .. -rw------- 1 dustin dustin 193 Apr 6 11:09 LICENSE.txt -rw-r--r-- 1 dustin dustin 240 Apr 6 11:03 manifest.tt > * deploying to the web cluster (DONE) * fixing tooltool.py's authentication of fetches > * disabling the existing upload mechanism > * uploading the existing tooltool data to the new tooltool > * I'll try to determine appropriate visibility levels and filenames > * converting build automation to use the new client, including using a > permanent token on each host > * Allow all devs to upload to tooltool, generate user tokens
Depends on: 1151470
Depends on: 1151630
> * fixing tooltool.py's authentication of fetches (bug 1151470) > * uploading the existing tooltool data to the new tooltool > * I'll try to determine appropriate visibility levels and filenames (DONE) > * disabling the existing upload mechanism (DONE) * update the wiki page for the new upload mechanism (DONE) * blog about the change > * converting build automation to use the new client, including using a > permanent token on each host > * Allow all devs to upload to tooltool, generate user tokens * update mana
Depends on: 1151527
> * fixing tooltool.py's authentication of fetches (bug 1151470) > * blog about the change (DONE) > * converting build automation to use the new client, including using a > permanent token on each host > * Allow all devs to upload to tooltool, generate user tokens > * update mana > * fix badpenny tasks - https://github.com/mozilla/build-tooltool/issues/12
> * Allow all devs to upload to tooltool, generate user tokens (DONE) correction: vpn_tooltooleditor can upload; anyone can download public; team_moco can download internal. > * update mana (DONE) > * fix badpenny tasks - https://github.com/mozilla/build-tooltool/issues/12 (DONE) > * converting build automation to use the new client, including using a > permanent token on each host * figure out why replication is failing with "server has gone away" errors
Depends on: 1155238
> * converting build automation to use the new client, including using a > permanent token on each host but 1155238 > * figure out why replication is failing with "server has gone away" errors https://github.com/mozilla/build-tooltool/pull/16
No longer depends on: 1151527
This is now farmed out appropriately to other bugs.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.