1485583 - Add metrics source pluggable to Spark metrics system

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Arkadiusz Komarzewski [:akomar]]

There is a need for monitoring custom metrics in our Spark applications.

We can extend Spark's metric system to collect application-level metrics together with framework ones. The easiest way seems to be using accumulators for aggregation, providing them to a custom metrics source.

Comment 1

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/telemetry-streaming/pull/179

Comment 2

[Jeff Klukas [:klukas] (UTC-4)]

Have we considered https://github.com/groupon/spark-metrics ?

Comment 3

[Arkadiusz Komarzewski [:akomar]]

Yes, we did - it's a little bit more complicated than our solution and looks abandoned (targets Spark 2.0, last commit 2 years ago).

Comment 4

[Jeff Klukas [:klukas] (UTC-4)]

Tha(In reply to akomarzewski from comment #3)
> Yes, we did - it's a little bit more complicated than our solution and looks
> abandoned (targets Spark 2.0, last commit 2 years ago).

Indeed, a library without commits for 2 years sounds like a risky bet.