1568128 - Create Glean error reporting dashboards

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P3)

Description

[Alessio Placitelli [:Dexter]]

In bug 1552507 Chris found that the 'metrics' ping was reporting errors about some important metrics: they were not getting recorded or they were getting partial data.

Whenever there's a problem with collecting some data, the Glean SDK reports it through its error reporting mechanism.

It would be great if there were per-product dashboards to allow teams to monitor such error stream.

I could imagine this dashboard having one or more plots with the number of errors per metric over time/release, to easily capture error trends introduced by erroneous collection or other bugs.

Comment 1

[Frank Bertsch [:frank]]

Alessio, I've made an example query here with two plots: https://sql.telemetry.mozilla.org/queries/64407/source#164348

Some questions:

• Should we break up into a plot for invalid values and a plot for invalid labels? We would then expand this for other errors when you include them
• Is this what you were thinking for "metric over time"?

Comment 2

[Alessio Placitelli [:Dexter]]

(In reply to Frank Bertsch [:frank] from comment #1)

Alessio, I've made an example query here with two plots: https://sql.telemetry.mozilla.org/queries/64407/source#164348

This looks great!!

Some questions:

• Should we break up into a plot for invalid values and a plot for invalid labels? We would then expand this for other errors when you include them

Mh, that would "force" us to expand, as you say, in the future. Ideally, this plot should not contain any error. As soon as something becomes trending, we should investigate.
If you want to break up by type, I'd consider adding a "other" category to each plot. This way, if we forget to add a new type, we can still track the overall volume of errors.

Honestly, I'd keep it simple and just have one table, so that we can make checking this part of our glean meeting.

• Is this what you were thinking for "metric over time"?

Exactly! This is 100% what I had in mind, thanks for doing this! We can already see a few weird things:

1. The uptake of the 'baseline.duration' fix is REALLY SLOW :| Maybe this is because we're targeting all versions? Maybe we really need 2 dashboards: one for all the versions and one just for the latest couple of Fenix releases?
2. The search* metrics still seem to have some bug. Gah.

Mike, thoughts on comment 1?

Comment 3

[Michael Droettboom [:mdroettboom]]

I agree with :Dexter here that I think having separate tables for different error types is probably overkill. I like everything being together -- but the separate error types on different lines (columns) is nice to help us determine the nature of the error.

Not a re:dash expert, but it would possible to have the y-axis be a percentage (of all pings)? It's hard to get a sense of the "severity" of an error with absolute numbers. The may also help with what :Dexter suggested -- if we could have both "all releases" and "specific releases (for last n releases)" dashboards, if we had percentages they'd be more directly comparable.

Comment 4

[Frank Bertsch [:frank]]

Okay, I've updated the chart to show fraction of clients affected: https://sql.telemetry.mozilla.org/queries/64407/source#164349

I thin that's a better metric than SUM(errors), since one heavily-affected client won't skew the plot. See it right now for an example of that, where search.default_engine.submission_url has spiked in Fenix Error Counts, but is trending steady in Fenix Errors Affected Clients.

For now we'll have to manually add new errors here, when you add them to the client.

Comment 6

[Frank Bertsch [:frank]]

Alessio, Mike, if you need something else here, let me know. Closing as resolved.