1292193 - Create a secure AWS S3 bucket specifically for storing BMO attachments instead of the database

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Infrastructure, defect)

Description

[David Lawrence [:dkl]]

As bug 1160929 has been implemented and deployed for quite some time now, I would like to come up with a plan for migration. We will need the S3 bucket to be created. Then we can plan on doing the actual migration during a tree closure window as it will take a while. Currently we have ~135GB of attachment data so the bucket would need to be large enough to handle that and future growth.

We could do a test migration of the attachment data from bugzilla-dev.allizom.org first and get somewhat of an idea on how much time it will take to upload the data. So we should have a production, stage, and devel buckets for attachments.

We would need to clear the attach_data.data column to realize the space savings in the database. This will speed up replication some and possibly improve performance slightly on the DB side.

Thanks
dkl

Comment 1

[Kendall Libby [:fubar] (he/him)]

NI on gozer for setting up the bucket (presumably in the new acct, if it doesn't already exist) and for getting us the creds.

Comment 2

[Kendall Libby [:fubar] (he/him)]

(helps if I actually set the NI...)

Comment 3

[Philippe M. Chiasson (:gozer)]

Question for ya, do you want seperate credentials for this bucket, or is re-using the same credentials as for the data/ bucket a possiblilty?

Either way, I'll need to do some small amount of CF hackery.

Comment 4

[Philippe M. Chiasson (:gozer)]

Adding permissions to the attachment bucket to the existing data keys, should deploy shortly

https://github.com/mozilla-bteam/bmo-nubis/commit/48d0d99673af81b7baa4afbb7745f5bec5eec1a5

Comment 5

[David Lawrence [:dkl]]

(In reply to Philippe M. Chiasson (:gozer) from comment #4)
> Adding permissions to the attachment bucket to the existing data keys,
> should deploy shortly
> 
> https://github.com/mozilla-bteam/bmo-nubis/commit/
> 48d0d99673af81b7baa4afbb7745f5bec5eec1a5

Just to note for clarity, we store the S3 access key and secret in data params when using S3 for data storage. So it would be good to not use the same keys as some other service in case the data/params file gets lifted somehow. It should be a specific key set for Bugzilla attachments IMO.

If that is what is happening anyway, disregard all I just said :)

dkl

Comment 6

[Kendall Libby [:fubar] (he/him)]

/me agrees with :dkl, separate is preferred.

Comment 7

[Philippe M. Chiasson (:gozer)]

Done, credentials sent to :fubar by email. Enjoy!