1659138 - Thunderbird on Glean

Status: NEW

(Data Platform and Tools :: Glean: SDK, defect, P4)

Description

[Alessio Placitelli [:Dexter]]

Today I had a very interesting conversation with :mkmelin about Thunderbird: they are currently building their own pipeline (AWS-based) for legacy telemetry, since that's all they can use in Gecko/Thunderbird.

However, this will change by the end of this year: FOG is still a priority and FOG will be finished in H2. Which means Thunderbird will be able to use the FOG-exposed JS API with the Glean SDK.

Things to talk about:

• Will it be easy-ish to use Glean in Thunderbird given FOG?
• Can Thunderbird infra folks deploy Glean-pipeline tools?
• Can Glean-tools (GLAM, Glean Dictionary) be deployed there, too?
• Do we have docs for the Glean-pipeline pieces?

Mark, thoughts on the above?

Comment 1

[Mark Reid [:mreid]]

The current iteration of the pipeline is closely tied to GCP, so wouldn't be deployable on AWS. Is an AWS-based stack a hard requirement?

Will it be easy-ish to use Glean in Thunderbird given FOG?

I'm not totally sure, this is probably a better question for :chutten.

Can Thunderbird infra folks deploy Glean-pipeline tools?

Again, things are pretty GCP-specific right now, so not without possibly significant work.

Can Glean-tools (GLAM, Glean Dictionary) be deployed there, too?

Possibly, but there's quite a bit of heavy lifting done to get data prepared and available to these tools, so the work involved will be significant.

Do we have docs for the Glean-pipeline pieces?

The Glean dictionary is still in prototype phase, but documentation will be part of it as it matures. Glam has some documentation in the git repo.

It would be much easier (even moreso if Thunderbird moves to Glean!) to use the existing pipeline, but I'm not sure whether that's feasible for us or even desirable for Thunderbird.

Comment 2

[Jeff Klukas [:klukas] (UTC-4)]

It would be much easier (even moreso if Thunderbird moves to Glean!) to use the existing pipeline, but I'm not sure whether that's feasible for us

From a technical standpoint, this definitely seems like the only way forward that wouldn't involved a large amount of work. We have documentation such that I expect Thunderbird folks would be able to define Glean metrics, and we'd be able to include that in probe-scraper so the pipeline can accept submissions. So that would be a story for getting the data from Thunderbird to ping tables in Mozilla Corp's shared-prod project. If relevant Thunderbird folks are NDA'd Mozillians, we could feasibly give them access to tools on the Mozilla Corp side, although there would likely be some burden to make sure Thunderbird data showed up in those tools.

If Thunderbird folks were willing to set up a GCP account, it would also be feasible to allow access to just Thunderbird pings to a service account on their side to use with arbitrary analysis tools. But it sounds like they want more processed data and tooling rather than straight pings.

Comment 3

[Chris H-C :chutten]

(In reply to Mark Reid [:mreid] from comment #1)

Will it be easy-ish to use Glean in Thunderbird given FOG?

I'm not totally sure, this is probably a better question for :chutten.

Oof, could I answer an easy question instead? : D

In short: Yes. With FOG there'll be build system integration for the generation of APIs for metrics from many different definitions files. Thunderbird can own and maintain its own list of metrics it wants to collect and include it somewhere around here only when building for Thunderbird.

This'll give you Rust, C++, and JS APIs you can then call to collect data, persistent storage, a lifecycle management scheme, autodocs for the data, an uploader (providing viaduct is present and has an implemented backend on Thunderbird, which I presume it will since it uses necko), and all the other bells and whistles.

If your data ingestion pipeline isn't incoming.tmo you'll need to swap out the server here, but we can work with you to provide an ergonomic way to do this. (it'll be at compile-time. No runtime switching of telemetry servers anymore as that's been used as an abuse vector).