Last Comment Bug 432450 - Design and deploy new partitioning scheme and associated scripts
: Design and deploy new partitioning scheme and associated scripts
Status: RESOLVED FIXED
:
Product: Socorro
Classification: Server Software
Component: General (show other bugs)
: Trunk
: All All
: -- normal (vote)
: 0.6
Assigned To: K Lars Lohn [:lars] [:klohn]
: socorro
Mentors:
: 432449 434883 465781 (view as bug list)
Depends on: 434883 454414
Blocks: 444106 444749 454438 465632 465781
  Show dependency treegraph
 
Reported: 2008-05-06 09:24 PDT by Michael Morgan [:morgamic]
Modified: 2011-12-28 10:40 PST (History)
3 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description Michael Morgan [:morgamic] 2008-05-06 09:24:03 PDT
For housecleaning and automation, we need to write a script that:
* detects partition size and automatically creates a new partition when the reports partition reaches a reasonable size -- like 250000 or 500000 records or 15gb -- whichever comes first (something like that, it'll change based on actual results)
* deletes reports that are older than 3 months or at least creates a constraint to exclude old partitions from all queries
* adds new date constraints on old partitions
Comment 1 Michael Morgan [:morgamic] 2008-09-10 11:18:57 PDT
*** Bug 432449 has been marked as a duplicate of this bug. ***
Comment 2 Michael Morgan [:morgamic] 2008-09-10 11:20:13 PDT
*** Bug 434883 has been marked as a duplicate of this bug. ***
Comment 3 K Lars Lohn [:lars] [:klohn] 2008-10-06 14:27:30 PDT
The new partition scheme makes new partitions weekly.  Inserts redirect automatically to the appropriate partitions of the reports, dumps, frames (and extensions) tables.

There's a script to by run by cron weekly to create the new partitions.

Just waiting for a testing environment...
Comment 4 K Lars Lohn [:lars] [:klohn] 2008-10-29 16:15:56 PDT
targeting this for the WebDev on site Nov 11-15
Comment 5 Michael Morgan [:morgamic] 2008-12-01 13:14:28 PST
Update - we're staging Wednesday Dec 3, pushing within next couple of weeks.
Comment 6 Michael Morgan [:morgamic] 2008-12-04 00:32:36 PST
*** Bug 465781 has been marked as a duplicate of this bug. ***
Comment 7 Marc Bejarano 2009-01-20 12:44:38 PST
morgamic: status?
Comment 8 K Lars Lohn [:lars] [:klohn] 2009-01-20 12:53:00 PST
this merged to trunk today in preparation for staging and deployment - expect deployment within a few days
Comment 9 K Lars Lohn [:lars] [:klohn] 2009-02-10 09:55:24 PST
finally completed after several false starts and rollbacks.  

The database did not have the capacity to run the algorithm that took the current data and repartition it into smaller chunks.  Tests on smaller databases indicated that scaling would happen with O(n).  Reality on the full dataset showed a non linear scaling that did not work in our favor.

This was finally finished using a "from this point forward" scheme.  The old data was installed as a giant partition.  All new data arriving as of Monday, January 9 started going into new weekly partitions.  The full benefits of partitioning will not be felt for several weeks as the new partitions fill and the old giant partition ages into irrelevancy.

Note You need to log in before you can comment on or make changes to this bug.