Create a secure AWS S3 bucket specifically for storing BMO attachments instead of the database

RESOLVED FIXED

Status

()

bugzilla.mozilla.org
Infrastructure
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: dkl, Assigned: gozer)

Tracking

(Blocks: 1 bug)

Production

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

2 years ago
As bug 1160929 has been implemented and deployed for quite some time now, I would like to come up with a plan for migration. We will need the S3 bucket to be created. Then we can plan on doing the actual migration during a tree closure window as it will take a while. Currently we have ~135GB of attachment data so the bucket would need to be large enough to handle that and future growth.

We could do a test migration of the attachment data from bugzilla-dev.allizom.org first and get somewhat of an idea on how much time it will take to upload the data. So we should have a production, stage, and devel buckets for attachments.

We would need to clear the attach_data.data column to realize the space savings in the database. This will speed up replication some and possibly improve performance slightly on the DB side.

Thanks
dkl
NI on gozer for setting up the bucket (presumably in the new acct, if it doesn't already exist) and for getting us the creds.
(helps if I actually set the NI...)
Flags: needinfo?(gozer)
(Assignee)

Comment 3

2 years ago
Question for ya, do you want seperate credentials for this bucket, or is re-using the same credentials as for the data/ bucket a possiblilty?

Either way, I'll need to do some small amount of CF hackery.
Flags: needinfo?(gozer)
(Assignee)

Updated

2 years ago
Assignee: nobody → gozer
(Assignee)

Updated

2 years ago
Flags: needinfo?(klibby)
(Assignee)

Comment 4

2 years ago
Adding permissions to the attachment bucket to the existing data keys, should deploy shortly

https://github.com/mozilla-bteam/bmo-nubis/commit/48d0d99673af81b7baa4afbb7745f5bec5eec1a5
(Reporter)

Comment 5

2 years ago
(In reply to Philippe M. Chiasson (:gozer) from comment #4)
> Adding permissions to the attachment bucket to the existing data keys,
> should deploy shortly
> 
> https://github.com/mozilla-bteam/bmo-nubis/commit/
> 48d0d99673af81b7baa4afbb7745f5bec5eec1a5

Just to note for clarity, we store the S3 access key and secret in data params when using S3 for data storage. So it would be good to not use the same keys as some other service in case the data/params file gets lifted somehow. It should be a specific key set for Bugzilla attachments IMO.

If that is what is happening anyway, disregard all I just said :)

dkl
/me agrees with :dkl, separate is preferred.
Flags: needinfo?(klibby)
(Assignee)

Comment 7

2 years ago
Done, credentials sent to :fubar by email. Enjoy!
(Assignee)

Updated

2 years ago
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Reporter)

Updated

2 years ago
Blocks: 1309706
You need to log in before you can comment on or make changes to this bug.