hbaseClient.py need an iterator for processed legacy crashes

RESOLVED FIXED in 1.8

Status

RESOLVED FIXED
8 years ago
7 years ago

People

(Reporter: lars, Assigned: dre)

Tracking

x86_64
Linux

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

8 years ago
we need an iterator that will give a sequence of ooids that are for processed legacy crashes.  Should the iterator be perpetual and blocking?  Not sure.  Probably not.
(Reporter)

Updated

8 years ago
Assignee: nobody → deinspanjer
Blocks: 580706
(Reporter)

Updated

8 years ago
Target Milestone: --- → 1.8
New table added to HBase staging: crash_reports_index_legacy_processed
It contains two columns, ids:ooid, and also processed_data:json.  The json is a shortcut so the feeder doesn't have to make a separate request to the crash_reports table.  We are storing an extra copy of the json, but since life in this queue is short, it shouldn't make a difference.

New methods added to hbaseClient in rev 2239:
insert_to_legacy_db_feeder_queue is called by put_processed_json.

iterator_for_legacy_db_feeder_queue is the method to retrieve items. There are TODO comments in there.  One that I didn't add was that you should pass a limit to limited_iteration if you don't want the default(whatever that is)

At the moment, iterator_for_legacy_db_feeder_queue calls delete_from_legacy_db_feeder_queue as it retrieves items.  That could be called separately if needed.
Also added a new table crash_reports_index_legacy_priority_processed
that can be used for a separate queue of priority items so they can always be feed first.
(Reporter)

Comment 3

8 years ago
minor point: "crash_reports_index_legacy_priority_processed" does not need to contain the word "legacy".  The state of being a priority job has nothing to do with legacy status.  A non-legacy crash (a crash that was throttled) can be a priority job.
I think I was abusing the term legacy. I was thinking more about it being code that was only to support 1.8 and going away in 1.9
This was implemented and is in 1.8 staging
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED

Comment 6

8 years ago
Not relevant for 1.7.x.
Component: Socorro → General
Product: Webtools → Socorro
You need to log in before you can comment on or make changes to this bug.