Closed Bug 1251316 Opened 8 years ago Closed 7 years ago

Migrate existing file attachments to S3

Categories

(developer.mozilla.org Graveyard :: File attachments, defect)

Product:
developer.mozilla.org Graveyard
Component:
File attachments
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: robhudson, Assigned: metadave)

References

Details

(Keywords: in-triage) 

As part of the move to S3 we need to migrate existing attachments to S3.

This bug is to investigate the various options and report back and ultimately assign to ops to perform the action.
After reviewing s3put and awscli, awscli seems like the winner. It has more options and a 'sync' command to synchronize folders that can be run multiple times to keep a folder in sync.

See: http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

It's built on top of botocore and written in Python.

The command would look something like:

  aws s3 sync --acl public-read media/attachments s3://<bucket>

There are other options we may consider, like:

  --no-guess-mime-type
  --content-type
  --cache-control or --expires
  --delete
Dave Parfitt worked on this in 2017, probably duplicating Rob Hudson's work:

https://github.com/mozmeao/infra/issues/408

He worked with webops to sync using s3cmd sync.  The sync has been tested several times, and we have a working process for the AWS "go live" date, tracked in the production release plan.

https://github.com/mozmeao/infra/issues/410
Assignee: nobody → dparfitt
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Product: developer.mozilla.org → developer.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.