1400931 - [meta] Add Direct-to-parquet for telemetry.error & telemetry.duplicates streams

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P3)

Description

[Frank Bertsch [:frank]]

These two streams need not store the data in the ping, just some metadata about the ping - docType, appVersion, submissionDate, etc. It would make it easy for users to check if their pings are having problems, and would be accessible in re:dash.

Comment 1

[Mark Reid [:mreid]]

Once the dependent bugs are resolved, we should have this info available in parquet.

We actually (kind of) already have this information in parquet form, but recent refactors have made it suboptimal.

Once the required changes are made we will have three parquet outputs:

mozingest_errors (for url-parsing failures)
telemetry_errors (for telemetry-specific failures)
telemetry_duplicates (for duplicate telemetry docids)

Comment 2

[Wesley Dawson [:whd]]

This should land on Monday.

Comment 3

[Wesley Dawson [:whd]]

These datasets are available via re:dash as telemetry_duplicates_parquet, telemetry_errors_parquet, and moz_ingest_uri_errors_parquet. Bug #1400935 is a separate but related concern so I've removed its blocking status and am closing this bug.