s3-copy-proxy: Disable streaming back content as part of the s3 upload.



4 years ago
3 years ago


(Reporter: jlal, Unassigned)





4 years ago
Right now we stream content back as part of the s3 upload. This means in some cases at least we have written headers and cannot redirect in the case of failure.

In the success case immediately streaming back content does not save that much time (10s~ for common case) and in the worst case we now cannot fallback to the redirect to the source url.
Note, in us-east-1 redirecting to S3 immediately is a tiny bit scary as the object may not be present
yet. Recommendation specific to this region only is poll HEAD checking if the object is visible yet.
Maybe just 5 iterations with 2, 4, 8, 16, 32 seconds in between.

This would be region specific to us-east-1.
Component: TaskCluster → General
Product: Testing → Taskcluster
Assignee: jlal → nobody
Component: General → Tools
In the new cloud mirror, this should not be a problem.  On the request which triggers the upload to a region, we will wait until the file completes the upload before redirecting.  For requests which are in the redis cache, we will wait for the same "status: present" condition as the initial request.  For requests which are in the backing storage but not the redis cache, we will do a HEAD on the resulting backend URL to see if it's there, redirecting once we find it, backfilling our redis cache as needed.

I'm going to mark this bug as complete because our solution is to deprecate the existing s3-copy-proxy.
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.