Closed Bug 1059941 Opened 6 years ago Closed 5 years ago

Write an S3 upload Python library that encapsulates most of blobber's functionality


(Release Engineering :: General, defect)

Not set


(Not tracked)



(Reporter: ted, Assigned: catlee)



We'd like to upload test package files to S3 directly without having to have blobber in the middle. It should be possible to extract most of the useful functionality from blobber into a Python library to make this easy. Ideally the API would be as simple as:
put(file, relative_path) -> url

The steps we'd want it to take are:
1) Hash the file
2) HEAD the S3 bucket to see if the file already exists (keyed by relative_path + hash)
3a) If the file exists, update its lifetime (so that we don't expire files we would have just uploaded)
3b) If the file doesn't exist, PUT it
4) Return the URL

catlee: did I miss anything? Assigning to catlee to either implement this or find someone to do so. I volunteered to wire it up to the build system if he got someone to write it.
Depends on: 1059943
you forgot the whiskey!

we need to check against the case where we get a 200 for our HEAD request, and then the object is deleted due to lifetime policies, and so the URL we end up with is invalid. does what we need I think.

A few tweaks maybe to make this easier to use as a library:
- have bucket passed in as a parameter
- have storage class passed in as a parameter. I can see using reduced storage class being useful for test data.
See bug 1059943 comment 42.
Closed: 5 years ago
Resolution: --- → WONTFIX
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.