Closed Bug 1941442 Opened 1 month ago Closed 24 days ago

Backfill firefox_installer_stable.install_v1 with errored pings starting on 2025-01-07

Categories

(Data Platform and Tools :: General, task)

task

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: benwu, Assigned: benwu)

References

Details

(Whiteboard: [dataquality])

In bug 1934302, it was discovered that nearly all pings for firefox_installer_stable.install_v1 were being dropped due to validation errors. To correct the historical data, we'll need to backfill these errors from payload_bytes_error. The errors started with Firefox 134 which reached release on 2025-01-07 but we may want to backfill previous dates as well.

The most recent similar backfill is https://github.com/mozilla/bigquery-backfill/tree/main/backfill/2024-11-06-metrics-pings-metric-labels-over-max-length so that can be used as a guideline.

The fix is already in Firefox so we can start the backfill when we confirm most new telemetry is fixed. In order to backfill the errored pings, we need to modify the payloads and metadata so the pings pass validation. We can do these transformations in the decoder. I put up a PR https://github.com/mozilla/gcp-ingestion/pull/2720 to do this. I don't think this needs to deployed to production since the client-side fix is in place. We can just do a one-time backfill using this code and accept that errors from specific versions of Firefox will linger.

firefox_installer_stable.install_v1 receives data from two different pings:

  • firefox-installer.install sent through structured telemetry so errors end up in payload_bytes_error.structured
    • error message is org.everit.json.schema.ValidationException: #/windows_ubr: expected type: Integer, found: String
  • firefox-installer.install sent to the stub installer end point so errors end up in payload_bytes_error.stub_installer
    • error message is com.mozilla.telemetry.decoder.ParseUri$UnexpectedPathElementsException: Found 2 more path elements in the URI than expected for this endpoint

Example queries to get the errored pings:

stub installer:

SELECT
  *
FROM
  `moz-fx-data-shared-prod.payload_bytes_error.stub_installer`
WHERE
  DATE(submission_timestamp) = "2025-01-13"
  AND error_message LIKE "com.mozilla.telemetry.decoder.ParseUri$UnexpectedPathElementsException: Found 2 more path elements in the URI than expected for this endpoint"
  AND STARTS_WITH(uri, "/stub/v9/")

full installer:

SELECT
  *
FROM
  `moz-fx-data-shared-prod.monitoring.payload_bytes_error_structured`
WHERE
  DATE(submission_timestamp) = "2025-01-13"
  AND document_namespace = "firefox-installer"
  AND document_type = "install"
  AND error_message = "org.everit.json.schema.ValidationException: #/windows_ubr: expected type: Integer, found: String"
Assignee: nobody → bewu

Fix is out in Firefox 134.0.2 which has reached 100% rollout so starting backfill now. WIP documentation in https://github.com/mozilla/bigquery-backfill/pull/58

Backfill is now complete. Ping count by version looks correct https://sql.telemetry.mozilla.org/queries/104802/source#258025

Status: NEW → RESOLVED
Closed: 24 days ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.