Design and deploy new partitioning scheme and associated scripts

RESOLVED FIXED in 0.6

Status

Socorro
General
RESOLVED FIXED
9 years ago
6 years ago

People

(Reporter: morgamic, Assigned: lars)

Tracking

Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

9 years ago
For housecleaning and automation, we need to write a script that:
* detects partition size and automatically creates a new partition when the reports partition reaches a reasonable size -- like 250000 or 500000 records or 15gb -- whichever comes first (something like that, it'll change based on actual results)
* deletes reports that are older than 3 months or at least creates a constraint to exclude old partitions from all queries
* adds new date constraints on old partitions

Updated

9 years ago
Blocks: 422908
(Reporter)

Updated

9 years ago
Assignee: nobody → lars
(Reporter)

Updated

9 years ago
Depends on: 434883
(Reporter)

Updated

9 years ago
Target Milestone: 0.5 → 0.6
(Reporter)

Updated

9 years ago
No longer blocks: 422908
(Reporter)

Updated

9 years ago
Duplicate of this bug: 432449
(Reporter)

Updated

9 years ago
Depends on: 454414
(Reporter)

Updated

9 years ago
Duplicate of this bug: 434883
(Assignee)

Comment 3

9 years ago
The new partition scheme makes new partitions weekly.  Inserts redirect automatically to the appropriate partitions of the reports, dumps, frames (and extensions) tables.

There's a script to by run by cron weekly to create the new partitions.

Just waiting for a testing environment...
(Assignee)

Updated

9 years ago
Summary: Create script to auto-archive old records and create new partitions when needed → Design and deploy new partitioning scheme and associated scripts
(Assignee)

Updated

9 years ago
Blocks: 444106
(Assignee)

Comment 4

9 years ago
targeting this for the WebDev on site Nov 11-15
(Assignee)

Updated

9 years ago
Blocks: 454438
(Reporter)

Updated

9 years ago
Status: NEW → ASSIGNED
(Reporter)

Updated

9 years ago
Blocks: 444749
(Reporter)

Updated

9 years ago
Blocks: 465781
(Reporter)

Comment 5

9 years ago
Update - we're staging Wednesday Dec 3, pushing within next couple of weeks.
(Reporter)

Updated

9 years ago
Duplicate of this bug: 465781
(Reporter)

Updated

9 years ago
Blocks: 468405
(Reporter)

Updated

9 years ago
Blocks: 465632
No longer blocks: 468405

Comment 7

9 years ago
morgamic: status?
(Assignee)

Comment 8

9 years ago
this merged to trunk today in preparation for staging and deployment - expect deployment within a few days
(Assignee)

Comment 9

9 years ago
finally completed after several false starts and rollbacks.  

The database did not have the capacity to run the algorithm that took the current data and repartition it into smaller chunks.  Tests on smaller databases indicated that scaling would happen with O(n).  Reality on the full dataset showed a non linear scaling that did not work in our favor.

This was finally finished using a "from this point forward" scheme.  The old data was installed as a giant partition.  All new data arriving as of Monday, January 9 started going into new weekly partitions.  The full benefits of partitioning will not be felt for several weeks as the new partitions fill and the old giant partition ages into irrelevancy.
Status: ASSIGNED → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
Component: Socorro → General
Product: Webtools → Socorro
You need to log in before you can comment on or make changes to this bug.