Closed Bug 1618684 Opened 6 years ago Closed 6 years ago

Move FirefoxOS pings to `UnwantedDataException`

Categories

(Data Platform and Tools :: General, task, P2)

task

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: klukas, Assigned: ascholtz)

Details

(Whiteboard: [dataquality])

We see ~10 million messages per day coming through the telemetry pipeline family that end up in errors with message:

com.mozilla.telemetry.ingestion.core.schema.SchemaNotFoundException: Element was assumed to be a telemetry message because it contains no document_version attribute, but the payload does not include the top-level 'version' or 'v' field expected for a telemetry document

The vast majority of these are FirefoxOS pings having URI like

/submit/telemetry/<uuid>/ftu/FirefoxOS/34.0/default/20191216212948

These match the telemetry uri pattern enough to make it to ParsePayload, but they otherwise do not conform.

These should minimally be marked with UnwantedDataException but we perhaps want to drop them entirely.

We have prior art for filtering to FirefoxOS pings based on URI and I would say it's safe to completely drop any pings with URIs matching /FirefoxOS/ (note that the edge filter is specifically including them but that was for the heka->hindsight migration so many years ago). The new UnwantedDataException mechanisms should largely replace edge filtering except where we have a compelling reason to drop submissions entirely before they hit pubsub.

We talked about this briefly in the weekly platform health check meeting. We'll mark these as UnwantedData, but we'll send out an email to #fx-data-dev to let folks know.

Whiteboard: [data-platform-health] → [data-quality]
Assignee: nobody → ascholtz
Summary: FirefoxOS pings lead to large volume of SchemaNotFoundException → Move FirefoxOS pings to `UnwantedDataException`
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Whiteboard: [data-quality] → [dataquality]
You need to log in before you can comment on or make changes to this bug.