1133851 - Create relengapi-tooltool

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

See https://gist.github.com/djmitche/5860cb6547ca33b6acd9

This should be able to track upload batches, files, and instances of files.  It should be able to provide signed S3 URLs to clients.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

This is promising!  With a manually configured bucket:

```
dmitchell@admin1a ~/tmp $ python tooltool.py fetch -v --url=http://euclid.r.igoro.us:8010/tooltool
DEBUG - processing 'fetch' command with args ''
DEBUG - using options: {'verbose': True, 'algorithm': 'sha512', 'package': None, 'base_url': ['http://euclid.r.igoro.us:8010/tooltool'], 'manifest': 'manifest.tt', 'host': None, 'quiet': False, 'user': None, 'auth_file': None, 'path': None, 'folder': None, 'size': 0.0, 'message': None, 'overwrite': False, 'cache_folder': None}
DEBUG - creating FileRecord 0xd7dfd0
DEBUG - materialized __main__.FileRecord(filename='one', size='2', digest='3abb6677af34ac57c0ca5828fd94f9d886c26ce59a8ce60ecf6778079423dccff1d6f19cb655805d56098e6d38a1a710dee59523eed7511e5a9e4b8ccb3a4686', algorithm='sha512')
DEBUG - loaded manifest from file 'manifest.tt'
DEBUG - fetching one
INFO - Attempting to fetch from 'http://euclid.r.igoro.us:8010/tooltool'...
DEBUG - opened http://euclid.r.igoro.us:8010/tooltool/sha512/3abb6677af34ac57c0ca5828fd94f9d886c26ce59a8ce60ecf6778079423dccff1d6f19cb655805d56098e6d38a1a710dee59523eed7511e5a9e4b8ccb3a4686 for reading
INFO - File one fetched from http://euclid.r.igoro.us:8010/tooltool as /home/dmitchell/tmp/tmpdakJjV
DEBUG - creating FileRecord 0xd7df90
DEBUG - hashed tmpdakJjV with sha512 to be 3abb6677af34ac57c0ca5828fd94f9d886c26ce59a8ce60ecf6778079423dccff1d6f19cb655805d56098e6d38a1a710dee59523eed7511e5a9e4b8ccb3a4686
INFO - File integrity verified, renaming tmpdakJjV to one
```
It doesn't show the redirect there, but it's getting the file data correctly, so it must be working.  The relengapi access log shows

```
2015-03-05 15:22:02,657 63.245.214.82 - - [05/Mar/2015 15:22:02] "GET /tooltool/sha512/3abb6677af34ac57c0ca5828fd94f9d886c26ce59a8ce60ecf6778079423dccff1d6f19cb655805d56098e6d38a1a710dee59523eed7511e5a9e4b8ccb3a4686 HTTP/1.1" 302 -
```

The bucket itself doesnt' allow read access:
  https://s3.amazonaws.com/mozilla-releng-use1-tooltool/sha512/3abb6677af34ac57c0ca5828fd94f9d886c26ce59a8ce60ecf6778079423dccff1d6f19cb655805d56098e6d38a1a710dee59523eed7511e5a9e4b8ccb3a4686

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

The full implementation is pending review (https://github.com/mozilla/build-tooltool/pull/10), but the blueprint is created.  Next up is to deploy it.