1309215 - ATMO V2: Remove workers

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Jannis Leidel [:jezdez]]

The workers feature can be removed following advise from :rvitillo

Comment 1

[GitHub Autolander Bot]

Attachment: [telemetry-analysis-service] mozilla:worker-removal > mozilla:master

Comment 2

[Mark Reid [:mreid]]

The worker feature is still in use for one of the jobs[1] that powers the Firefox dashboard[2].

Bug 1272388 would allow us to relatively easily port that job to Spark, but otherwise we're paying the EMR tax for no good reason.


[1] https://github.com/mozilla-services/data-pipeline/tree/master/reports/executive_summary
[2] https://metrics.services.mozilla.com/firefox-dashboard/

Comment 3

[Jannis Leidel [:jezdez]]

:mreid The feature was already removed from the atmo v2 code base

:rvitillo Can you elaborate how to proceed with this?

Comment 4

[Roberto Agostino Vitillo (:rvitillo)]

I am not keen to maintain a feature that is used by only one job. We will address the EMR tax when we move away from EMR (which is part of next year's plan).

Comment 5

[Mark Reid [:mreid]]

As long as we can run the job script from atmo v2 (on a single-node Spark cluster), then I can live with this.

I filed Bug 1309574 to port the job over to use main_summary, which will also avoid the dependency on spark-redshift.

Comment 6

[Jannis Leidel [:jezdez]]

Closing this since I think this is okay now