Closed Bug 432450 Opened 12 years ago Closed 11 years ago

Design and deploy new partitioning scheme and associated scripts

Categories

(Socorro :: General, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: morgamic, Assigned: lars)

References

Details

For housecleaning and automation, we need to write a script that:
* detects partition size and automatically creates a new partition when the reports partition reaches a reasonable size -- like 250000 or 500000 records or 15gb -- whichever comes first (something like that, it'll change based on actual results)
* deletes reports that are older than 3 months or at least creates a constraint to exclude old partitions from all queries
* adds new date constraints on old partitions
Blocks: 422908
Assignee: nobody → lars
Depends on: 434883
Target Milestone: 0.5 → 0.6
No longer blocks: 422908
Duplicate of this bug: 432449
Depends on: 454414
Duplicate of this bug: 434883
The new partition scheme makes new partitions weekly.  Inserts redirect automatically to the appropriate partitions of the reports, dumps, frames (and extensions) tables.

There's a script to by run by cron weekly to create the new partitions.

Just waiting for a testing environment...
Summary: Create script to auto-archive old records and create new partitions when needed → Design and deploy new partitioning scheme and associated scripts
Blocks: 444106
targeting this for the WebDev on site Nov 11-15
Blocks: 454438
Status: NEW → ASSIGNED
Blocks: 444749
Blocks: 465781
Update - we're staging Wednesday Dec 3, pushing within next couple of weeks.
Duplicate of this bug: 465781
Blocks: 468405
Blocks: 465632
No longer blocks: 468405
morgamic: status?
this merged to trunk today in preparation for staging and deployment - expect deployment within a few days
finally completed after several false starts and rollbacks.  

The database did not have the capacity to run the algorithm that took the current data and repartition it into smaller chunks.  Tests on smaller databases indicated that scaling would happen with O(n).  Reality on the full dataset showed a non linear scaling that did not work in our favor.

This was finally finished using a "from this point forward" scheme.  The old data was installed as a giant partition.  All new data arriving as of Monday, January 9 started going into new weekly partitions.  The full benefits of partitioning will not be felt for several weeks as the new partitions fill and the old giant partition ages into irrelevancy.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Component: Socorro → General
Product: Webtools → Socorro
You need to log in before you can comment on or make changes to this bug.