1225140 - Further parallelize executive report

Status: RESOLVED WONTFIX

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Mark Reid [:mreid]]

Currently the executive report needs to ingest data in order, one day at a time.

This makes it very time consuming to run the report on a large date range, since it can't easily be parallelized across analysis machines (beyond parallelizing by reporting period).

I would like to be able to run each day's report to produce a cuckoo filter containing that day's summary, then combine these summaries into a report.

This would mean that all the days could be run at the same time in parallel, hopefully speeding things up a lot.

Comment 1

[Mike Trinkala [:trink]]

There is a custom one-off cuckoo filter version I provided to mreid.  However, custom one-offs will not scale well so mreid is testing out longitudinal analysis using the Redshift databases.

Comment 2

[Mark Reid [:mreid]]

The change to use Redshift / SQL to power the exec report renders this obsolete.