Migrate existing file attachments to S3

RESOLVED FIXED

Status

RESOLVED FIXED
3 years ago
a year ago

People

(Reporter: robhudson, Assigned: metadave)

Tracking

(Depends on: 1 bug, {in-triage})

Details

(Reporter)

Description

3 years ago
As part of the move to S3 we need to migrate existing attachments to S3.

This bug is to investigate the various options and report back and ultimately assign to ops to perform the action.
(Reporter)

Comment 1

3 years ago
After reviewing s3put and awscli, awscli seems like the winner. It has more options and a 'sync' command to synchronize folders that can be run multiple times to keep a folder in sync.

See: http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

It's built on top of botocore and written in Python.

The command would look something like:

  aws s3 sync --acl public-read media/attachments s3://<bucket>

There are other options we may consider, like:

  --no-guess-mime-type
  --content-type
  --cache-control or --expires
  --delete
Keywords: in-triage
Dave Parfitt worked on this in 2017, probably duplicating Rob Hudson's work:

https://github.com/mozmeao/infra/issues/408

He worked with webops to sync using s3cmd sync.  The sync has been tested several times, and we have a working process for the AWS "go live" date, tracked in the production release plan.

https://github.com/mozmeao/infra/issues/410
Assignee: nobody → dparfitt
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.