1059941 - Write an S3 upload Python library that encapsulates most of blobber's functionality

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[(not currently active) Ted Mielczarek]

We'd like to upload test package files to S3 directly without having to have blobber in the middle. It should be possible to extract most of the useful functionality from blobber into a Python library to make this easy. Ideally the API would be as simple as:
put(file, relative_path) -> url

The steps we'd want it to take are:
1) Hash the file
2) HEAD the S3 bucket to see if the file already exists (keyed by relative_path + hash)
3a) If the file exists, update its lifetime (so that we don't expire files we would have just uploaded)
3b) If the file doesn't exist, PUT it
4) Return the URL

catlee: did I miss anything? Assigning to catlee to either implement this or find someone to do so. I volunteered to wire it up to the build system if he got someone to write it.

Comment 1

[Chris AtLee [:catlee]]

you forgot the whiskey!

we need to check against the case where we get a 200 for our HEAD request, and then the object is deleted due to lifetime policies, and so the URL we end up with is invalid.

Comment 2

[Chris AtLee [:catlee]]

https://github.com/catlee/blobber/blob/master/blobber/amazons3_backend.py does what we need I think.

A few tweaks maybe to make this easier to use as a library:
- have bucket passed in as a parameter
- have storage class passed in as a parameter. I can see using reduced storage class being useful for test data.

Comment 3

[(not currently active) Ted Mielczarek]

See bug 1059943 comment 42.