1539249 - Create an Uptake Telemetry to Datadog Streaming job

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P2)

Description

[bugzilla]

Per the Normandy sign error postmortem, it wouldn't not be too difficult to change the NormandyToDatadog job to send the new uptake telemetry events, allowing product delivery to create telemetry-based monitors on their systems, and this would have allowed us to detect and respond quicker to the sign error issue.

See Bug 1517469 for details on the new uptake events -- they are currently riding 67 (release channel will be sampled to 1%) but even the beta population is enough to start detecting these problems.

Comment 1

[bugzilla]

ni? ethan: what are the metrics you'd like to see re: uptake?

Comment 2

[Ethan Glasser-Camp (:glasserc)]

These are off the cuff and haven't really been tested yet, but:

• How many Normandy enroll/unenroll events are we seeing overall? How much uptake telemetry are we seeing overall?
• How many exceptions are we seeing (*_error statuses)?
• For each client, what is the most recent recipe that it reports for which a status was recorded? What is the earliest recipe?
  • The goal here is to be able to verify that clients are "seeing" the correct set of recipes (i.e. the ones that should be seen on the server). I imagine some code might be necessary here to compare with what's on the server.
• For each recipe, how many clients are "seeing" it (all statuses)?
• For each recipe, what is the ratio of "successful" applications (success + backoff) to "unsuccessful" ones (*_error)?

I think all the above metrics could also be useful narrowed down to just Nightly, on the assumption that Nightly might have different client code that would trigger errors that we don't expect.

Comment 3

[Mathieu Leplatre [:leplatrem]]

With regards to Remote Settings, we could start with something simple.

We don't need everything of course. Sampling (eg. 1% or Nightly) and/or aggregating events (eg. 5min) would be fine.

As a start, the following metrics from events of category uptake.remotecontent.result could help us:

• Success: count of events filtered on event_map_values['source'] = 'settings-sync' AND event_string_value = 'success'
• Errors: count of events filtered on event_map_values['source'] = 'settings-sync' AND event_string_value = 'sync_error'

The following would be useful too, but less crucial:

• Duration: int(event_map_values["duration"]) filtered on event_map_values['source'] = 'settings-sync'
• Age of data: int(event_map_values["age"]) filtered on event_map_values['source'] = 'settings-changes-monitoring' AND event_map_values['trigger'] = 'broadcast' AND event_string_value = 'success'
• Errors Distribution: event_string_value filtered on event_map_values['source'] NOT LIKE 'normandy/%' AND event_string_value NOT IN ('up_to_data', 'success', 'network_error')

Does this look feasible Sunah?

Comment 4

[bugzilla]

Interesting -- the Success/Errors count we can definitely do, and I'll slate those for next week. The other distributions seem like they'd be doable but might take a little more time, so no promises on timeline there at the moment.

Comment 5

[Ethan Glasser-Camp (:glasserc)]

Hi! Mat suggested I frame my requests in the same format.

I think the simplest metrics that could be useful would be:

• Enrollments: a scalar count of all enroll events. I guess we have this already for addon_study but other types of enrolls would be good too. [Edit: :sunahsuh points out that we actually have all the enrollments already.]
• Errors: a scalar count of all statuses that match *_error
• Successes: a scalar count of all statuses that are not *_error. This should only be success and backoff

The following would be useful too, but less crucial:

• Recipe visibility: something like count(*) group by event_map_values['source']
• Recipe status: something like count(event_map_string NOT IN ('success', 'backoff')) group by event_map_values['source']

I previously wrote about age of data; I think this is unfortunately outside the scope of the datadog connection as it requires joining with data in Normandy.

Comment 6

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/telemetry-streaming/pull/243

Comment 7

[bugzilla]

Okay, so, we have a new version of the job deployed and running, and I create an example timeboard with some of the new metrics being emitted (it's not exhaustive though):
https://app.datadoghq.com/dashboard/piq-rtw-fk9?from_ts=1559246361848&live=true&tile_size=m&to_ts=1559249961848

This includes adding the duration/age data as timers (which gives us avg, median, max, p95 metrics in datadog.)

I decided to handle sources by splitting them by / and sticking each "level" into a separate tag (if it exists): source_type, source_subtype and source_details. That lets you do all the things requested in the bug, I think.

Let me know if you'd like tweaks to the data format in any way. It'd be pretty easy to change now that we know the data is flowing in correctly. Otherwise, feel free to close out this bug if you've gotten everything you need.

Comment 8

[Mathieu Leplatre [:leplatrem]]

Thank you Sunah! That's fantastic! 🎉🎉🎉

I will spend some time to build a dashboard with the metrics I mentioned. I'm no Datadog guru, but from what I could see it looks like everything is there :) And live! I really like how easy it is to explore different scales of time and see trends.

I propose that we close this once Ethan confirms that it works for him too.

Thanks a lot for your efforts on this 🙏

Comment 9

[bugzilla]

I haven't heard from Ethan yet so I'm gonna close this out for now to tie things up for the quarter. Ethan, please reopen or file a new bug if you determine that the data format needs tweaking to meet your needs!