Backfill firefox_installer_stable.install_v1 with errored pings starting on 2025-01-07
Categories
(Data Platform and Tools :: General, task)
Tracking
(Not tracked)
People
(Reporter: benwu, Assigned: benwu)
References
Details
(Whiteboard: [dataquality])
In bug 1934302, it was discovered that nearly all pings for firefox_installer_stable.install_v1
were being dropped due to validation errors. To correct the historical data, we'll need to backfill these errors from payload_bytes_error
. The errors started with Firefox 134 which reached release on 2025-01-07 but we may want to backfill previous dates as well.
The most recent similar backfill is https://github.com/mozilla/bigquery-backfill/tree/main/backfill/2024-11-06-metrics-pings-metric-labels-over-max-length so that can be used as a guideline.
The fix is already in Firefox so we can start the backfill when we confirm most new telemetry is fixed. In order to backfill the errored pings, we need to modify the payloads and metadata so the pings pass validation. We can do these transformations in the decoder. I put up a PR https://github.com/mozilla/gcp-ingestion/pull/2720 to do this. I don't think this needs to deployed to production since the client-side fix is in place. We can just do a one-time backfill using this code and accept that errors from specific versions of Firefox will linger.
firefox_installer_stable.install_v1
receives data from two different pings:
firefox-installer.install
sent through structured telemetry so errors end up inpayload_bytes_error.structured
- error message is
org.everit.json.schema.ValidationException: #/windows_ubr: expected type: Integer, found: String
- error message is
firefox-installer.install
sent to the stub installer end point so errors end up inpayload_bytes_error.stub_installer
- error message is
com.mozilla.telemetry.decoder.ParseUri$UnexpectedPathElementsException: Found 2 more path elements in the URI than expected for this endpoint
- error message is
Example queries to get the errored pings:
stub installer:
SELECT
*
FROM
`moz-fx-data-shared-prod.payload_bytes_error.stub_installer`
WHERE
DATE(submission_timestamp) = "2025-01-13"
AND error_message LIKE "com.mozilla.telemetry.decoder.ParseUri$UnexpectedPathElementsException: Found 2 more path elements in the URI than expected for this endpoint"
AND STARTS_WITH(uri, "/stub/v9/")
full installer:
SELECT
*
FROM
`moz-fx-data-shared-prod.monitoring.payload_bytes_error_structured`
WHERE
DATE(submission_timestamp) = "2025-01-13"
AND document_namespace = "firefox-installer"
AND document_type = "install"
AND error_message = "org.everit.json.schema.ValidationException: #/windows_ubr: expected type: Integer, found: String"
Updated•1 month ago
|
Assignee | ||
Updated•1 month ago
|
Assignee | ||
Comment 1•26 days ago
|
||
Fix is out in Firefox 134.0.2 which has reached 100% rollout so starting backfill now. WIP documentation in https://github.com/mozilla/bigquery-backfill/pull/58
Assignee | ||
Comment 2•24 days ago
|
||
Backfill is now complete. Ping count by version looks correct https://sql.telemetry.mozilla.org/queries/104802/source#258025
Description
•