Closed Bug 1309706 Opened 8 years ago Closed 4 years ago

[meta] Tracking bug for migration of BMO attachments from the database to AWS S3

Categories

(bugzilla.mozilla.org :: General, task)

Production
task
Not set
normal

Tracking

()

RESOLVED FIXED

People

(Reporter: dkl, Assigned: dkl)

References

Details

(Keywords: bmo-goal)

User Story

Production Steps

1. Block access to BMO using standard hardhat procedure.
2. Suspend execution of all helper daemons such as push, jobqueue, httpd, etc.
3. Backup the database by detaching master from slaves and disabling replication.
4. Obtain S3 bucket credentials and store them in the proper BMO configuration params.
5. Execute migration script to move the attachments from the database to S3.
   - perl scripts/migrate-attachments.pl --copy database s3
   - This may take a bit depending on the amount of data (will get time estimates before day of migration).
6. Verify proper operation by storing, updating attachment meta data, and retrieving stored attachments of various file types.
7. If any issues occur, remove AWS credentials from params and change back to database access. 
8. If all is well, remove the attachment data from the database by executing the migration script:
   - perl scripts/migrate-attachments.pl --delete database
   UPDATE: We will leave the old attachment data in place for one week to make sure everything is working
   properly.
9. Re-enable execution of helper daemons and re-enable replication to the database slaves.
10. Unblock access to BMO by reversing hardhat procedure.
11. Run some final sanity tests to further verify proper operation.
12. Drink.
> Summary

Migrate stored attachment data currently in the bugzilla.mozilla.org database to AWS S3, saving space and streamlining replication.

> Description

Bugzilla.mozlla.org (BMO) has code allowing storage of bug attachments in differnet locations such as filesystem, database, and AWS S3 buckets. Currently BMO is configured to use the database for attachment storage. This uses a substantial amount of disk space on the database server as well as the other slaves which replicate the attachment data. Attachment attribute to a large portion of the database. BMO has a failover cluster in AWS which also needs to have a replicated copy of all of the data including attachments. BMO needs to start using S3 as a repository for storing attachment data which will allow the amount of storage space for the rest of the system data to be substantially less. This will simplify what needs to be replicated as well as make the process faster. End users of BMO should not see any difference as the storage and retrieval of attachment data will happen on the backend.

Will add full production steps to the user story.

dkl
User Story: (updated)
Depends on: 1310041
Hi David, 
I just sent you an email regarding this about the bucket (resource) policies and iam policies that you're planning to use to govern access for bugzilla users to read attachments and for the bugzilla app to add/modify attachments.
Flags: needinfo?(dkl)
Thanks for the response! marking NI complete
Flags: needinfo?(dkl)
Blocks: 1313152
User Story: (updated)
Depends on: 1315326
Depends on: 1318748
Component: Infrastructure → General
QA Contact: mcote
Type: defect → task
Assignee: nobody → dkl
Status: NEW → ASSIGNED
Depends on: 1588221
Depends on: 1588227
Depends on: 1588229
Keywords: bmo-goal
No longer depends on: 1588229

The migration of attachments with size over 20k to S3 was completed in late May 2020. Closing. Any new changes will be done in new separate bug reports.

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.