574998 - Archive old format crash report records in HBase to reduce region load

Status: RESOLVED FIXED

(Mozilla Metrics :: Hadoop/HBase Operations, defect)

Description

[Daniel Einspanjer [:dre] [:deinspanjer]]

We are currently at ~43k online regions, and this is a few too many for the number of servers we have (17).  Since we just lost a server on Saturday morning, it is urgent that we relieve some of this load so we don't have stability issues next week.

Before the Socorro 1.7 push, raw crash reports were being stored in HBase in a format that is not very useful to the Socorro system anyway.  This makes these records a prime target for archival.

We will be using a script custom written by HBase devs for us.  I've tested it on staging and it works as advertised.  Eventually, we can write a separate script that will be able to resurrect these regions so we can do something with the data, but for now, they will just be sitting in archive storage.

Comment 1

[Daniel Einspanjer [:dre] [:deinspanjer]]

Attachment: Ganglia results of region archival

Operation completed without incident.  We now have only 6k regions online which puts us far out of any danger zone.  We didn't have to archive that much, but the data isn't very useful until we migrate it to the new format anyway so this seemed like a reasonable plan.

[deinspanjer@cm-hadoop-adm01 bin]$ ls -lh /home/deinspanjer/meta*
-rw-r--r-- 1 deinspanjer users 1.8M Jun 26 14:53 /home/deinspanjer/meta_after_excise_apr.txt.gz
-rw-r--r-- 1 deinspanjer users 3.8M Jun 26 14:21 /home/deinspanjer/meta_after_excise_feb.txt.gz
-rw-r--r-- 1 deinspanjer users 3.0M Jun 26 14:37 /home/deinspanjer/meta_after_excise_mar.txt.gz
-rw-r--r-- 1 deinspanjer users 576K Jun 26 15:03 /home/deinspanjer/meta_after_excise_may.txt.gz
-rw-r--r-- 1 deinspanjer users 3.8M Jun 26 14:16 /home/deinspanjer/meta_before_excise.txt.gz