566340 - Need Map Reduce job to clean up pre-1.7 data

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Daniel Einspanjer [:dre] [:deinspanjer]]

Currently the rowkeys are not salted.  We need to have a job that fixes the old data so it can be retrieved via the 1.7 codebase.

Comment 1

[Daniel Einspanjer [:dre] [:deinspanjer]]

Also, we'll want to draw out the new attributes such as ids:ooid and timestamps:*.
One big MR that is finished before 2.0 is the best way to go for this.

Comment 2

[Anurag Phadke[:[e-mail]]]

https://wiki.mozilla.org/Socorro_hbase_cleanup

The wiki page contains details on all the required changes that need to happen before 1.8 release.

Comment 3

[Laura Thomson :laura]

Daniel, did this get done?

If not, is it still relevant?  I'm guessing it's no longer critical.

Comment 4

[Daniel Einspanjer [:dre] [:deinspanjer]]

Has not been done yet, still needs to be done, not critical at the moment.

Comment 5

[Xavier Stevens [:xstevens]]

Not really trying to argue against this, but aren't we also talking about removing data that is over 6 months old? Because if that was the case we might as well delete these as part of that process rather than update them.

Comment 6

[Daniel Einspanjer [:dre] [:deinspanjer]]

The pre-1.7 data isn't inside hbase at all, it is all the region directories that have no corresponding row in .META. whatever our TTL strategy is won't affect them.

Comment 7

[Xavier Stevens [:xstevens]]

I don't think we'll need a MapReduce job for this at all. I've written some code that will give us all of the region directories in HDFS that .META. doesn't know about.

Comment 8

[Xavier Stevens [:xstevens]]

HBase actually now detects this automatically. I moved the directories out of the way on secondary and everything seems to work fine.

Comment 9

[Daniel Einspanjer [:dre] [:deinspanjer]]

Cool.  How much disk space does the offline data represent?  If possible, we should try to pull it out of HDFS and onto tape so we can archive it until we are given clearance to delete entirely.

Comment 10

[Xavier Stevens [:xstevens]]

6.8TB unreplicated

Comment 11

[Daniel Einspanjer [:dre] [:deinspanjer]]

could you file an IT bug and cc laura asking IT if they have any ideas for how we could archive this to tape?

Comment 12

[T [:tmary] Meyarivan]

Existing archive has been deleted after socorro-dev verified that it is not needed.

--