1501757 - Broken Datadog metrics on Databricks

Status: RESOLVED FIXED

(Data Platform and Tools Graveyard :: Operations, defect)

Description

[Arkadiusz Komarzewski [:akomar]]

Metrics sent to Datadog from streaming jobs on Databricks seem to have lost their tags in July:

https://app.datadoghq.com/dash/838007/spark-streaming---databricks?live=true&page=0&is_auto=false&from_ts=1532624221611&to_ts=1540400221611&tile_size=m&tpl_var_env=*&tpl_var_TelemetryJobName=error_aggregator

I have just started a staging job that also isn't submitting tagged metrics:
https://dbc-caf9527b-e073.cloud.databricks.com/#job/630

Its cluster configuration with tags can be checked here: https://dbc-caf9527b-e073.cloud.databricks.com/#/setting/clusters/1024-162753-owlet59/configuration

Comment 1

[Wesley Dawson [:whd]]

The datadog tag logic (dbfs:/databricks/init/startDatadogTags.sh) was not written to be resilient to complex tag content (specifically, the inclusion of slashes in the Jar tag which was introduced in July). I've fixed it, assuming we don't put '|'s in our tags.

I've verified in the referenced datadog dashboard that minimally TelemetryJobName is available. :akomar if things look good to you please close this.

Comment 2

[Arkadiusz Komarzewski [:akomar]]

Thank you, looks good.