Closed Bug 1148545 Opened 9 years ago Closed 9 years ago

Deploy build-tooltool to relengapi

Categories

(Infrastructure & Operations :: RelOps: General, task)

Product:
Infrastructure & Operations
Component:
RelOps: General
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dustin, Assigned: dustin)

References

Details

This involves:
 * landing a few more things:
   * https://github.com/mozilla/build-tooltool/pull/10
   * https://github.com/mozilla/build-relengapi/pull/196
   * https://github.com/mozilla/build-relengapi/pull/187
 * releasing a new version of relengapi
 * building a cloudformation script to create the S3 buckets
 * setting up badpenny (running badpenny-cron in a crontask on webheads)
 * deploying to the web cluster
 * documenting the new upload process (it's pretty simple but requires a token)
 * disabling the existing upload mechanism
 * uploading the existing tooltool data to the new tooltool
   * I'll try to determine appropriate visibility levels and filenames
 * converting build automation to use the new client, including using a permanent token on each host
Depends on: 1118455
Assignee: relops → dustin
(In reply to Dustin J. Mitchell [:dustin] from comment #0)
> This involves:
>  * landing a few more things:
(DONE)

>  * releasing a new version of relengapi
>  * building a cloudformation script to create the S3 buckets
(DONE)

>  * setting up badpenny (running badpenny-cron in a crontask on webheads)
(DONE - Bug 1118455)

>  * deploying to the web cluster
>  * documenting the new upload process (it's pretty simple but requires a
> token)
(DONE)

>  * disabling the existing upload mechanism
>  * uploading the existing tooltool data to the new tooltool
>    * I'll try to determine appropriate visibility levels and filenames
>  * converting build automation to use the new client, including using a
> permanent token on each host

Add:
 * Allow all devs to upload to tooltool, generate user tokens
(In reply to Dustin J. Mitchell [:dustin] from comment #0)
> This involves:
>  * releasing a new version of relengapi
(DONE)

>  * deploying to the web cluster
>  * disabling the existing upload mechanism
>  * uploading the existing tooltool data to the new tooltool
>    * I'll try to determine appropriate visibility levels and filenames
>  * converting build automation to use the new client, including using a
> permanent token on each host
>  * Allow all devs to upload to tooltool, generate user tokens
 * I need to use bucket policies to better lock down access to the buckets to something narrower than just the AWS account.  I should probably create the relengapi users with cloudformation too
The staging server appears not to be accepting the "Authorization" header, and even after switching the client to use "Authentication":

INFO - LICENSE.txt: starting upload
ERROR - LICENSE.txt: failed
Traceback (most recent call last):
  File "tooltool.py", line 778, in _s3_upload
    (resp.status, resp.reason, resp_body))
RuntimeError: Non-200 return from AWS: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>B315E8AFFCAB59B0</RequestId><HostId>FG5dturWy2xb7OqzKwrHdd9j/2uoiP9BdbwFNs+S7KwFko4iHstCuChLSv3fb+WlLdSKp+1TC6Q=</HostId></Error>
(sandbox)

Which suggests I haven't set up the user policy correctly.
For the Authorization/Authentication issue, https://github.com/mozilla/build-relengapi/issues/208
Auth issue required an upgrade to Flask-Login.  Fixed.  Now, on to the AWS permissions.
I addd a usre policy via cloudformation.  That changed the error from

RuntimeError: Non-200 return from AWS: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>56A9F1DD4087560B</RequestId><HostId>YjhTGmmHWvr6SOTOpa482sjaZg526GUB6wcHB+KzozOmeU9rNiab+QQsBjF6jin6</HostId></Error>

to 

INFO - LICENSE.txt: starting upload
https://mozilla-releng-staging-usw1-tooltool.s3-us-west-1.amazonaws.com/sha512/054fdbe8cb55d1f7592871311c5a9da76710c7a085fd24457644d80aa0ea3c344c57e99aab3a6fb2ec7ed93c15c8f997d5b7b60692c318f9cacad6418bb359e1?Signature=9KGrgB8jWVaZ%2BY7jsD82UNwcIhE%3D&Expires=1428085058&AWSAccessKeyId=AKIAILABTOAMWCDH5EBA
ERROR - LICENSE.txt: failed
Traceback (most recent call last):
  File "tooltool.py", line 778, in _s3_upload
    (resp.status, resp.reason, resp_body))
RuntimeError: Non-200 return from AWS: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>AKIAILABTOAMWCDH5EBA</AWSAccessKeyId><StringToSign>PUT

application/octet-stream
1428085058
/mozilla-releng-staging-usw1-tooltool/sha512/054fdbe8cb55d1f7592871311c5a9da76710c7a085fd24457644d80aa0ea3c344c57e99aab3a6fb2ec7ed93c15c8f997d5b7b60692c318f9cacad6418bb359e1</StringToSign><SignatureProvided>9KGrgB8jWVaZ+Y7jsD82UNwcIhE=</SignatureProvided><StringToSignBytes>50 55 54 0a 0a 61 70 70 6c 69 63 61 74 69 6f 6e 2f 6f 63 74 65 74 2d 73 74 72 65 61 6d 0a 31 34 32 38 30 38 35 30 35 38 0a 2f 6d 6f 7a 69 6c 6c 61 2d 72 65 6c 65 6e 67 2d 73 74 61 67 69 6e 67 2d 75 73 77 31 2d 74 6f 6f 6c 74 6f 6f 6c 2f 73 68 61 35 31 32 2f 30 35 34 66 64 62 65 38 63 62 35 35 64 31 66 37 35 39 32 38 37 31 33 31 31 63 35 61 39 64 61 37 36 37 31 30 63 37 61 30 38 35 66 64 32 34 34 35 37 36 34 34 64 38 30 61 61 30 65 61 33 63 33 34 34 63 35 37 65 39 39 61 61 62 33 61 36 66 62 32 65 63 37 65 64 39 33 63 31 35 63 38 66 39 39 37 64 35 62 37 62 36 30 36 39 32 63 33 31 38 66 39 63 61 63 61 64 36 34 31 38 62 62 33 35 39 65 31</StringToSignBytes><RequestId>ADAED72E986778B3</RequestId><HostId>iLPz6jhTyVr7auNEfDt+g8pUlResOM47dvbpzW7E0VYrLYvmcrdxsr+5mTqgLxNT</HostId></Error>

Which doesn't make much sense -- in my dev environment I've repeatedly signed real AWS URLs.  I'm going to try updating boto first before digging further.
Same error with boto-2.37.0.
Derp, I had a space in the secret key :(
The verification task is failing:

File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/relengapi/blueprints/tooltool/grooming.py", line 112, in verify_file_instance
    if key.storage_class != 'STANDARD':
  File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/boto/s3/key.py", line 199, in _get_storage_class
    list_items = list(self.bucket.list(self.name.encode('utf-8')))
  File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/boto/s3/bucketlistresultset.py", line 34, in bucket_lister
    encoding_type=encoding_type)
  File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/boto/s3/bucket.py", line 472, in get_all_keys
    '', headers, **params)
  File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/boto/s3/bucket.py", line 410, in _get_all
    response.status, response.reason, body)
S3ResponseError: S3ResponseError: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>16EB688054EC9362</RequestId><HostId>L/STz309bVUqzrT+WmXusPUBi/ssR1QY4dqUnAdBWbxo+z1YffvHaEPswJj+IzoJ</HostId></Error>

looks like the policy is wrong (listing keys is a bucket operation, but I've only applied it to object ARNs)
I'm not going to push this to production right now, though.
A successful fetch from production:

dustin@euclid ~/code/relengapi/t/tooltool/x [master] $ python ../tooltool.py fetch --verbose
DEBUG - processing 'fetch' command with args ''
DEBUG - using options: {'cache_folder': None, 'algorithm': 'sha512', 'loglevel': 10, 'region': None, 'base_url': ['https://api.pub.build.mozilla.org/tooltool/'], 'visibility': None, 'manifest': 'manifest.tt', 'auth_file': None, 'message': None, 'overwrite': False, 'size': 0.0}
DEBUG - materialized __main__.FileRecord(filename='LICENSE.txt', size=193, digest='054fdbe8cb55d1f7592871311c5a9da76710c7a085fd24457644d80aa0ea3c344c57e99aab3a6fb2ec7ed93c15c8f997d5b7b60692c318f9cacad6418bb359e1', algorithm='sha512', visibility=u'public')
DEBUG - loaded manifest from file 'manifest.tt'
DEBUG - fetching LICENSE.txt
INFO - Attempting to fetch from 'https://api.pub.build.mozilla.org/tooltool/'...
DEBUG - opened https://api.pub.build.mozilla.org/tooltool/sha512/054fdbe8cb55d1f7592871311c5a9da76710c7a085fd24457644d80aa0ea3c344c57e99aab3a6fb2ec7ed93c15c8f997d5b7b60692c318f9cacad6418bb359e1 for reading
INFO - File LICENSE.txt fetched from https://api.pub.build.mozilla.org/tooltool/ as /home/dustin/code/relengapi/t/tooltool/x/tmphDWQYw
DEBUG - hashed 'tmphDWQYw' with sha512 to be 054fdbe8cb55d1f7592871311c5a9da76710c7a085fd24457644d80aa0ea3c344c57e99aab3a6fb2ec7ed93c15c8f997d5b7b60692c318f9cacad6418bb359e1
INFO - File integrity verified, renaming tmphDWQYw to LICENSE.txt
dustin@euclid ~/code/relengapi/t/tooltool/x [master*] $ ls -al
total 16
drwxr-xr-x  2 dustin dustin 4096 Apr  6 11:09 .
drwxr-xr-x 13 dustin dustin 4096 Apr  6 11:06 ..
-rw-------  1 dustin dustin  193 Apr  6 11:09 LICENSE.txt
-rw-r--r--  1 dustin dustin  240 Apr  6 11:03 manifest.tt



>  * deploying to the web cluster
(DONE)

   * fixing tooltool.py's authentication of fetches
>  * disabling the existing upload mechanism
>  * uploading the existing tooltool data to the new tooltool
>    * I'll try to determine appropriate visibility levels and filenames
>  * converting build automation to use the new client, including using a
> permanent token on each host
>  * Allow all devs to upload to tooltool, generate user tokens
Depends on: 1151470
Depends on: 1151630
>  * fixing tooltool.py's authentication of fetches
(bug 1151470)

>  * uploading the existing tooltool data to the new tooltool
>    * I'll try to determine appropriate visibility levels and filenames
(DONE)

>  * disabling the existing upload mechanism
(DONE)

   * update the wiki page for the new upload mechanism
(DONE)
   * blog about the change

>  * converting build automation to use the new client, including using a
> permanent token on each host
>  * Allow all devs to upload to tooltool, generate user tokens
   * update mana
Depends on: 1151527
>  * fixing tooltool.py's authentication of fetches
(bug 1151470)
>  * blog about the change
(DONE)

>  * converting build automation to use the new client, including using a
> permanent token on each host
>  * Allow all devs to upload to tooltool, generate user tokens
>  * update mana
>  * fix badpenny tasks - https://github.com/mozilla/build-tooltool/issues/12
>  * Allow all devs to upload to tooltool, generate user tokens
(DONE)
correction: vpn_tooltooleditor can upload; anyone can download public; team_moco can download internal.

>  * update mana
(DONE)

>  * fix badpenny tasks - https://github.com/mozilla/build-tooltool/issues/12
(DONE)

>  * converting build automation to use the new client, including using a
> permanent token on each host

   * figure out why replication is failing with "server has gone away" errors
Depends on: 1155238
>  * converting build automation to use the new client, including using a
> permanent token on each host
but 1155238

>  * figure out why replication is failing with "server has gone away" errors
https://github.com/mozilla/build-tooltool/pull/16
No longer depends on: 1151527
This is now farmed out appropriately to other bugs.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.