580723 - hbaseClient.py need an iterator for processed legacy crashes

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[K Lars Lohn [:lars] [:klohn]]

we need an iterator that will give a sequence of ooids that are for processed legacy crashes.  Should the iterator be perpetual and blocking?  Not sure.  Probably not.

Comment 1

[Daniel Einspanjer [:dre] [:deinspanjer]]

New table added to HBase staging: crash_reports_index_legacy_processed
It contains two columns, ids:ooid, and also processed_data:json.  The json is a shortcut so the feeder doesn't have to make a separate request to the crash_reports table.  We are storing an extra copy of the json, but since life in this queue is short, it shouldn't make a difference.

New methods added to hbaseClient in rev 2239:
insert_to_legacy_db_feeder_queue is called by put_processed_json.

iterator_for_legacy_db_feeder_queue is the method to retrieve items. There are TODO comments in there.  One that I didn't add was that you should pass a limit to limited_iteration if you don't want the default(whatever that is)

At the moment, iterator_for_legacy_db_feeder_queue calls delete_from_legacy_db_feeder_queue as it retrieves items.  That could be called separately if needed.

Comment 2

[Daniel Einspanjer [:dre] [:deinspanjer]]

Also added a new table crash_reports_index_legacy_priority_processed
that can be used for a separate queue of priority items so they can always be feed first.

Comment 3

[K Lars Lohn [:lars] [:klohn]]

minor point: "crash_reports_index_legacy_priority_processed" does not need to contain the word "legacy".  The state of being a priority job has nothing to do with legacy status.  A non-legacy crash (a crash that was throttled) can be a priority job.

Comment 4

[Daniel Einspanjer [:dre] [:deinspanjer]]

I think I was abusing the term legacy. I was thinking more about it being code that was only to support 1.8 and going away in 1.9

Comment 5

[Daniel Einspanjer [:dre] [:deinspanjer]]

This was implemented and is in 1.8 staging

Comment 6

[Laura Thomson :laura]

Not relevant for 1.7.x.