607335 - Migrate old data from PostgreSQL into HBase or some other archive

Status: RESOLVED WONTFIX

(Mozilla Metrics :: Data/Backend Reports, defect)

Description

[Laura Thomson :laura]

We'd like to migrate old data out of PostgreSQL.  We have stuff going back to 2008 which can't be queried from the webapp.  To make this available it should migrate into HBase.

Comment 1

[Daniel Einspanjer [:dre] [:deinspanjer]]

Would it be possible to dump this data in a plain text format and tarball it in some fashion?  Would make it easier to put in cold storage if we don't have to slurp directly from Postgres.

Comment 2

[Aravind Gottipati [:aravind]]

We can clone the current production database into dm-breakpad-devdb, and you can massage it into whatever format works best for you.  Dumping it all out to plain text may not be possible (afaik, we have old archived reports in there?)

Comment 3

[Daniel Einspanjer [:dre] [:deinspanjer]]

We discussed this bug in triage today and came up with an alternate solution for which we are curious to get feedback regarding viability.
It is to write a simple python script that queried a Postgres DB for all the old processed crashes and extracts the processed json and uses the existing hbaseClient.py call to insert the data into the HBase processed_data:json cell.

Thoughts?

Comment 4

[Annie Elliott]

CLosing all old (created < 1-1-2011 ) cases still in NEW state.