574753 - processors need to be able to work around isolated errors interacting with hbase

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Daniel Einspanjer [:dre] [:deinspanjer]]

If a region is down and that causes one or two items to be inaccessible, the processor should set them aside in an exceptions table and continue working.

Comment 1

[Laura Thomson :laura]

More info:
The current behavior for a processor is to retry, then fail, then die.  Then the next processor tries to pick up the next job from the monitor but it's the same job, same problem, next processor dies.  Repeat until we don't have any processors.

Comment 2

[K Lars Lohn [:lars] [:klohn]]

This is a major change to the architecture of the current monitor/processor system.  There would have to be an new database table to save these "poisonous" jobs and a new system within monitor to deal with them.

It may be better to have the processor just take the poisonous job and tack it back onto the end of its own queue to try again later rather than committing suicide.  That would be a simpler solution that wouldn't involve making new communication channels with the monitor.  Though, I suppose that some jobs could end up repeatedly looping through the queue until the hbase issue was resolved.  

Do we really want to make a major change to the 1.7 system considering its expected life?  Surely any system that resolved the problem for 1.7 would hardly be applicable to 1.8 since that uses a radically different job allocation scheme.

Comment 3

[Daniel Einspanjer [:dre] [:deinspanjer]]

I marked this as a post 1.8 bug.  Don't think we should attempt to undertake this for 1.7.x.

Even post 1.8, the processors could still run into this type of problem.  With the new processors though, we can add an hbase table for storing the exception ooids and implement my strategy in comment #0 fairly easily.

Comment 4

[Laura Thomson :laura]

Since processors now recover, this is no longer critical.

Comment 5

[K Lars Lohn [:lars] [:klohn]]

resolved with modern processor