Closed Bug 1312747 Opened 8 years ago Closed 8 years ago

Refactor EMR scripts

Categories

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Product:
Cloud Services Graveyard
Component:
Metrics: Pipeline
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED INVALID

People

(Reporter: rvitillo, Assigned: jezdez)

References

Details

User Story

Currently airflow and atmo are using two different EMR steps [1] [2] for  almost the same logic. We should refactor those into a single script and add that directly to the telemetry-analysis-service repository so that we can have different steps in different environments, like staging and production.

The bootstrap script [3] and the Spark configuration [4] should also be moved to telemetry-analysis-service.

[1] https://github.com/mozilla/emr-bootstrap-spark/blob/master/ansible/files/batch.sh
[2] https://github.com/mozilla/telemetry-airflow/blob/master/ansible/files/spark/airflow.sh
[3] https://github.com/mozilla/emr-bootstrap-spark/blob/master/ansible/files/telemetry.sh
[4] https://github.com/mozilla/emr-bootstrap-spark/blob/master/ansible/files/configuration.json
No description provided.
Points: --- → 2
Priority: -- → P2
Blocks: 1315355
Assignee: nobody → jezdez
Status: NEW → ASSIGNED
Priority: P2 → P1
Moved to https://github.com/mozilla/telemetry-analysis-service/issues/203
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → INVALID
Product: Cloud Services → Cloud Services Graveyard
You need to log in before you can comment on or make changes to this bug.