929669 - Send a copy of production data to stage through PolyCrashStore

Status: RESOLVED INVALID

(Socorro :: Backend, task)

Description

[Selena Deckelmann :selenamarie :selena]

Currently, we have a job that runs every 4 hours or so to copy Hbase data from prod to stage.

I propose we change this so that we're sending data directly to stage from the crashmovers. 

I suggest we also change the retry functions to only try two or three times before timing out/totally failing. Goal is to get a reasonable amount of data into stage, rather than perfection.

Comment 1

[Selena Deckelmann :selenamarie :selena]

This change will require a configuration change for our crashmovers. 

I don't think we need a netflow, since we already are sending crashes with the submitter.

Comment 2

[T [:tmary] Meyarivan]

(In reply to Selena Deckelmann :selenamarie :selena from comment #2)
> I suggest we also change the retry functions to only try two or three times
> before timing out/totally failing. Goal is to get a reasonable amount of
> data into stage, rather than perfection.

Actually "perfection" is a requirement here - I have been asked many a times re crashes that exist in prod but not on stg

--

Comment 3

[K Lars Lohn [:lars] [:klohn]]

is it a requirement that the crash_id match between production and stage?

Comment 4

[T [:tmary] Meyarivan]

(In reply to K Lars Lohn [:lars] [:klohn] from comment #3)
> is it a requirement that the crash_id match between production and stage?

If you meant to refer to the HBase table key, *yes*

--

Comment 5

[Selena Deckelmann :selenamarie :selena]

:lars - was this for the code change, or did the config change happen?

Comment 6

[K Lars Lohn [:lars] [:klohn]]

removing the tag - we cannot do this until we have all the hardware in place.  Looking for the hardware bug to block this.

Comment 7

[Lonnen :lonnen]

we implemented this with aws lambda instead