1229507 - Make UT Heartbeat Packets Easily Accessible in Spark

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P3)

Description

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

So, HB in UT is coming.  See #1193535.

This is the 'rare packet, but we need them all' use case.  

Possible by Orlando?  PLZ?!

Comment 1

[Mark Reid [:mreid]]

Filed https://github.com/mozilla-services/puppet-config/pull/1678 to add the "heartbeat" document type (so that it doesn't get bucketed into "OTHER".

Comment 2

[Mark Reid [:mreid]]

Gregg, do these really need to go into Redshift?  If so, what do you want done with all the nested environment stuff?

Comment 3

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

This is still waiting on packets coming to UT.  

I don't care about redshift vs some other solution.  Something very very similar to the executive summary vars would be awesome.  (+ crashes).

Let's talk this out again soon.

(In my head, i have this fantasy 'lifespan' data row, and a 'day-person' row form, but they are vague.  It's the same stuff we talke about with churn and other things.)

Comment 4

[Thomas Huelbert]

will need to size this when we know more about the package.

Comment 5

[Mark Reid [:mreid]]

The "heartbeat" records are now flowing in to the pipeline, and are accessible via Spark using the 'get_pings' functionality, similar to "main" pings (specifying doc_type="heartbeat").

If we need a specific derived dataset, let's file a new bug to specify/implement it.