1811648 - Migrate stub installer telemetry ping to modern structured ingestion

Status: NEW

(Firefox :: Installer, enhancement, P3)

Description

[Nick Alexander :nalexander [he/him]]

I was surprised to discover today that the stub installer reports its telemetry via HTTP GET to download-stats.mozilla.org (https://searchfox.org/mozilla-central/rev/eb00c7365e72ac422acd97f6eee3c26926b786cd/browser/installer/windows/nsis/defines.nsi.in#92). mhowell confirms that this is simply because the stub telemetry predates the modern mechanism. This ticket tracks updating the stub installer to use the modern approach, and to bring the back-end ingestion pipeline along for the ride.

Comment 1

[Wesley Dawson [:whd]]

Some notes for the back-end ingestion pipeline changes involved.

I've linked bugs where we migrated certain aspects of the DSMO pipeline to standardized infrastructure. The current state is that download-stats.mozilla.org is CNAMEd to the same endpoint as incoming.tmo and has its own (unused) HTTPS cert. We have custom routines for parsing the URI and a special case to convert stub URIs into the standard representation, but the code base is the same: https://github.com/mozilla/gcp-ingestion/blob/main/ingestion-beam/src/main/java/com/mozilla/telemetry/decoder/ParseUri.java#L158

That standard representation is captured in pipeline logic as a structured ping called firefox-installer/install, which has a json schema. This is shared with the "full installer" and pings are differentiated by the installer_type field: https://github.com/mozilla-services/mozilla-pipeline-schemas/blob/main/schemas/firefox-installer/install/install.1.schema.json#L101.

So I think some material considerations are:

1. whether to switch to download-stats.mozilla.org to incoming.telemetry.mozilla.org

This is completely optional and no backend change should be required since the two backends are already the same endpoint.

2. whether to use HTTPS

We've had various issues with certificate support when using non-gecko networking stacks such as https://mozilla-hub.atlassian.net/browse/DSRE-1105. DSMO's cert is currently provisioned via pki.goog, which we could switch to letsencrypt, but incoming.tmo itself is currently provisioned via digicert.

Note also that *.tmo is listed in https://searchfox.org/mozilla-central/source/security/manager/ssl/StaticHPKPins.h while DSMO is not, though *.tmo is in "test mode". Suffice it to say, switching to HTTPS may require some additional investigation.

3. confirm whether any code changes are required to handle stub installer pings via structured ingestion

In particular, verify that simply sending a structured message with installer_type set to stub works, and that the fields where stub vs. full are called out in https://github.com/mozilla-services/mozilla-pipeline-schemas/blob/main/schemas/firefox-installer/install/install.1.schema.json can all be handled client side.

In other words, I think it's possible for there to be no ingestion pipeline changes required to start processing these via POST requests, as long as they are sent in the format matching existing schemas. New fields could then be added to the existing schema and would only be supported via the new structured-based stub installer.

:relud is point on DSMO and can confirm whether my representations here are accurate.

Comment 2

[Daniel Thorn [:relud]]

can confirm :whd got that all correct.