544381 - Need job that can import Socorro crash report .jsonz files into HBase

Status: RESOLVED FIXED

(Mozilla Metrics :: Hadoop/HBase Operations, defect)

Description

[Daniel Einspanjer [:dre] [:deinspanjer]]

Input should be a list of jsonz files to import

Processing should unzip them and insert the data into a new or existing crash report record as part of an appropriate column family

Comment 1

[Tristan Buckner]

The cooked json files outputted by the processor differ in both content and key names.  I'm going to created a new column called processed_data:jsonz or something similar.

A potential side effect could be that any new processing rules for the eventual background processor that we might want to apply to all existing reports won't affect processed records that came in via thrit.  We could also attempt to recover the raw json metadata from the jsonz file, though I'm not sure that's entirely possible.

Comment 2

[Tristan Buckner]

Checked-in in socorro trunk/tools/loadjsonz.py

I'm having some path problems when I execute it as a script rather than importing it as a module. Any ideas ozten?

Comment 3

[Austin King [:ozten]]

(In reply to comment #2)
We are good here, right?

Also mentioned in the meeting - much of the data from report/index is populated from the jsonz file.

http://crash-stats.mozilla.com/report/index/ec144062-6d72-4adc-8582-b18022100210

(some data like extension urls, bugzilla ids, etc are not directly in the jsonz)

Comment 4

[Tristan Buckner]

We are good.  If we can get some files of the form uuid TAB jsonz_file_path we can start loading.

Do you know what all we need besides the jsonz?