The default bug view has changed. See this FAQ.

Write an S3 upload Python library that encapsulates most of blobber's functionality

RESOLVED WONTFIX

Status

Release Engineering
General Automation
RESOLVED WONTFIX
3 years ago
2 years ago

People

(Reporter: ted, Assigned: catlee)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

3 years ago
We'd like to upload test package files to S3 directly without having to have blobber in the middle. It should be possible to extract most of the useful functionality from blobber into a Python library to make this easy. Ideally the API would be as simple as:
put(file, relative_path) -> url

The steps we'd want it to take are:
1) Hash the file
2) HEAD the S3 bucket to see if the file already exists (keyed by relative_path + hash)
3a) If the file exists, update its lifetime (so that we don't expire files we would have just uploaded)
3b) If the file doesn't exist, PUT it
4) Return the URL

catlee: did I miss anything? Assigning to catlee to either implement this or find someone to do so. I volunteered to wire it up to the build system if he got someone to write it.
(Reporter)

Updated

3 years ago
Depends on: 1059943
(Assignee)

Comment 1

3 years ago
you forgot the whiskey!

we need to check against the case where we get a 200 for our HEAD request, and then the object is deleted due to lifetime policies, and so the URL we end up with is invalid.
(Assignee)

Comment 2

3 years ago
https://github.com/catlee/blobber/blob/master/blobber/amazons3_backend.py does what we need I think.

A few tweaks maybe to make this easier to use as a library:
- have bucket passed in as a parameter
- have storage class passed in as a parameter. I can see using reduced storage class being useful for test data.
(Reporter)

Comment 3

2 years ago
See bug 1059943 comment 42.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.