We'd like to upload test package files to S3 directly without having to have blobber in the middle. It should be possible to extract most of the useful functionality from blobber into a Python library to make this easy. Ideally the API would be as simple as: put(file, relative_path) -> url The steps we'd want it to take are: 1) Hash the file 2) HEAD the S3 bucket to see if the file already exists (keyed by relative_path + hash) 3a) If the file exists, update its lifetime (so that we don't expire files we would have just uploaded) 3b) If the file doesn't exist, PUT it 4) Return the URL catlee: did I miss anything? Assigning to catlee to either implement this or find someone to do so. I volunteered to wire it up to the build system if he got someone to write it.
you forgot the whiskey! we need to check against the case where we get a 200 for our HEAD request, and then the object is deleted due to lifetime policies, and so the URL we end up with is invalid.
https://github.com/catlee/blobber/blob/master/blobber/amazons3_backend.py does what we need I think. A few tweaks maybe to make this easier to use as a library: - have bucket passed in as a parameter - have storage class passed in as a parameter. I can see using reduced storage class being useful for test data.