1711790 - Add support of metric level metadata

Status: RESOLVED INVALID

(Data Platform and Tools Graveyard :: Glean.js, enhancement)

Description

[Evgeny Pavlov]

The idea is to provide API to record extra metrics together with the main recorded metrics as metadata, similar to client info. It would allow adding custom filters as columns in BigQuery.

It is useful when we need custom fields as filters for many application metrics.

Currently, there are two workarounds:

1. Use events with extra keys. This one limits the usage of the Glean platform significantly. Also according to Glean book: "events are the most expensive metric type to record, transmit, store and analyze, so they should be used sparingly, and only when none of the other metric types are sufficient for answering your question."
2. Use regular string/timespan/quantity etc metrics as metadata. It works only for single thread scenarios and with frequent sending of custom pings. Multi-thread can mess up metadata, which is the case for multi-tab usage in the browser.

Comment 1

[Alessio Placitelli [:Dexter]]

Hi Evgeny,

thank you for filing this bug.

As documented in the Glean book, if you feel like the current APIs do not serve your needs you should follow this process to suggest changes.

While it would be good to leave the in depth conversation within the boundaries of that specific process, let me tackle it at high level below.

(In reply to Evgeny Pavlov from comment #0)

The idea is to provide API to record extra metrics together with the main recorded metrics as metadata, similar to client info. It would allow adding custom filters as columns in BigQuery.

This is currently possible in the existing Glean model by using custom pings. I think we already touched this

It is useful when we need custom fields as filters for many application metrics.

Currently, there are two workarounds:

1. Use events with extra keys. This one limits the usage of the Glean platform significantly.

How so? Can you provide a concrete example?

Also according to Glean book: "events are the most expensive metric type to record, transmit, store and analyze, so they should be used sparingly, and only when none of the other metric types are sufficient for answering your question."

This is a general advice. Firefox sends events on a regular cadence and it's not a problem: as long as the decision is intentional and well-thought, it's not a problem.

2. Use regular string/timespan/quantity etc metrics as metadata. It works only for single thread scenarios and with frequent sending of custom pings. Multi-thread can mess up metadata, which is the case for multi-tab usage in the browser.

The specific Bergamot case was discussed in bug 1700902 and further analysed in the notes document attached to that bug. Did the situation change?

Comment 2

[Beatriz Rizental [:brizental]]

(In reply to Evgeny Pavlov from comment #0)

The idea is to provide API to record extra metrics together with the main recorded metrics as metadata, similar to client info. It would allow adding custom filters as columns in BigQuery.

It is useful when we need custom fields as filters for many application metrics.

The client_info is a section included en every ping which contains a set of baseline metrics. There is no major difference, for analysis purposes, between a metric inside the client_info section of a ping and a metric inside the metrics section of a ping, aside from the fact that the ones in the client_info (and ping_info for that matter) are included opaquely by Glean itself in every ping. Nevertheless, Bergamot has access to the complete list of pings sent by itself, so it would be possible to include a list of all pings in a metric send_in_pings list, if necessary. Can you provide more detail on why that doesn't cover Bergamot's use case?

Note that discussion related to this, has already happened over on this proposal and the current specification is as follows:

We freeze the current list of metrics in client_info. See the client_info section's documentation for the current list. We reject adding new (platform- or application-specific) data points.

Still, as Alessio mentions, it would be great to have a concrete use case for the feature requested. No decision is set in stone and if necessary we will definitely re-discuss the spec above.

Currently, there are two workarounds:

1. Use events with extra keys. This one limits the usage of the Glean platform significantly. Also according to Glean book: "events are the most expensive metric type to record, transmit, store and analyze, so they should be used sparingly, and only when none of the other metric types are sufficient for answering your question."

Using events is not a workaround, it is ordinary usage of Glean. I went further on this topic over on https://github.com/mozilla-extensions/bergamot-browser-extension/issues/101#issuecomment-824637453 a while back and I think that use case is related to what is being discussed here.

2. Use regular string/timespan/quantity etc metrics as metadata. It works only for single thread scenarios and with frequent sending of custom pings. Multi-thread can mess up metadata, which is the case for multi-tab usage in the browser.

The background script of a webextension (i.e. where Glean.js API is called), does not spawn a new thread per tab. The issue for Bergamot as I remember it is that translation actions are asynchronous actions and may happen simultaneously, the fact that they happen in different tabs should be irrelevant for the purposes of this discussion though.

There is no way for Glean to know about such specific requiremenets for every consumer in order to provide generic scheduling based on the activities performed by each consumer. That is the use case covered by custom pings.

On the notes doc Alessio points to I go further in this topic. The example in the conclusion of that docs is a better ilustration of the challenges of asynchronous JS code than anything I can describe here. Redesigning the data collection with a clear picture of the environment and tools that will be used for such collection is probably the best solution for this problem. Note that we are open to help with designing scheduling of pings, we only need to know exactly what is the use case in order to provide such help.

Comment 3

[Evgeny Pavlov]

Thanks for the detailed response! I'm definitely not ready to prepare a proposal, I can only explain my use case.

I want to utilize "native" Glean types (Quantity, Timespan etc.) as much as possible because they allow building powerful dashboards in Looker out of the box (like this one https://mozilla.cloud.looker.com/dashboards-next/84). Using events metadata would require unnesting, types conversion and writing custom code for this). The same for regular queries in Redash or BigQuery. So they are definitely more expensive to analyze.

Our main struggle is that we have to group metrics in custom ping manually. Because of parallel usage from another tab, we can't just record metrics separately and send the ping decoupled from metrics collection. We have to group, collect and send them based on custom metadata and tab id using an internal queue. (I see that this problem is already extensively analyzed in https://bugzilla.mozilla.org/show_bug.cgi?id=1700902, thank you for this!).

The ideal solution for this would be to provide separate Glean instances per tab, so that we can safely record and send metrics in a decoupled way for every tab.

I tried to come up with API example which can solve our use case and it all looks too complex. I'm closing this bug then.