Move FirefoxOS pings to `UnwantedDataException`
Categories
(Data Platform and Tools :: General, task, P2)
Tracking
(Not tracked)
People
(Reporter: klukas, Assigned: ascholtz)
Details
(Whiteboard: [dataquality])
We see ~10 million messages per day coming through the telemetry pipeline family that end up in errors with message:
com.mozilla.telemetry.ingestion.core.schema.SchemaNotFoundException: Element was assumed to be a telemetry message because it contains no document_version attribute, but the payload does not include the top-level 'version' or 'v' field expected for a telemetry document
The vast majority of these are FirefoxOS pings having URI like
/submit/telemetry/<uuid>/ftu/FirefoxOS/34.0/default/20191216212948
These match the telemetry uri pattern enough to make it to ParsePayload, but they otherwise do not conform.
These should minimally be marked with UnwantedDataException but we perhaps want to drop them entirely.
Comment 1•6 years ago
|
||
We have prior art for filtering to FirefoxOS pings based on URI and I would say it's safe to completely drop any pings with URIs matching /FirefoxOS/ (note that the edge filter is specifically including them but that was for the heka->hindsight migration so many years ago). The new UnwantedDataException mechanisms should largely replace edge filtering except where we have a compelling reason to drop submissions entirely before they hit pubsub.
| Reporter | ||
Comment 2•6 years ago
|
||
We talked about this briefly in the weekly platform health check meeting. We'll mark these as UnwantedData, but we'll send out an email to #fx-data-dev to let folks know.
Updated•6 years ago
|
| Assignee | ||
Updated•6 years ago
|
Updated•6 years ago
|
| Assignee | ||
Updated•6 years ago
|
| Assignee | ||
Updated•3 years ago
|
Description
•