693481 - Socorro - adopt orphan crashes

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[K Lars Lohn [:lars] [:klohn]]

On Thursday, Sept 6, there were some troubles with HBase that resulted in a lot of crashes being orphaned in the collectors' primary store.  The bug that spawned these wretched orphans was repaired in Socorro 2.3.  Now it is time to send those orphans packing.

I'd like this procedure to be done when I am available to watch the results live.  So during the working day Pacific time would be best.  Reach me in #breakpad when you're ready to proceed.

On one of the collectors, starting with just one, run the script: 

.../scripts/orphanSubmitter.py --searchRoot=/home/socorro/primaryStorage/20110906/name 

Iff that completes successfully, then proceed with doing the same on the rest of the collectors.  I will advise as I see the results.

Comment 1

[Justin Dow [:jabba]]

After some help from lars getting the python path set right, I was able to successfully run this on all collectors:

[[e-mail]1 ~]$ export PYTHONPATH=/home/socorro/python_extras/lib:/data/socorro/application:/data/socorro/thirdparty

[[e-mail]1 ~]$ /data/socorro/application/scripts/orphanSubmitter.py --searchRoot=/home/socorro/primaryCrashStore/20111006 --hbaseHost=10.8.81.210 --syslogLineFormatString="Orphans (pid %(process)d): %(asctime)s %(levelname)s - %(threadName)s - %(message)s" --syslogFacilityString=local2 >/dev/null