1901256 - Reduced Metadata Pings are failing to process

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Tom Ritter [:tjr]]

I enabled the reduced metadata ping (Bug 1866559 ), and landed it on 5/20 in Bug 1897506. I broke a thing in Bug 1898105 and it got fixed but I don't know anything about that.

I am select *-ing from the data table. I have rows with build dates of 5/22 and up, and they still have data in the client_info and ping_info columns.

Chutten helped me look at errors:

SELECT
    DATE(submission_timestamp) AS received_day,
    COUNT(*) AS errored_payload_count,
FROM `mozdata.monitoring.payload_bytes_error_all`
WHERE
    submission_timestamp > '2024-05-01'
    AND document_type = 'user-characteristics'
GROUP BY ALL

and it does indeed show lots of errors. (This is a low-volume ping because it's just mozilla opt-in devs right now.) He cited the error org.everit.json.schema.ValidationException: #: required key [ping_info] not found

Related or unrelated, the glean debug-ping viewer is also broken because of this: https://github.com/mozilla/debug-ping-view/issues/168 e.g. https://debug-ping-preview.firebaseapp.com/pings/corvette

Chutten mentioned that after this gets fixed, the pings can be reprocessed?

Comment 1

[Wil Stuckey]

fixed in #1898105. old data had to be deleted as we don't plan on supporting a conversion from include_info_sections: true to include_info_sections: false in the future. This allows us to keep the schemas distinct and makes it clear that the table cannot contain ping_info or client_info data.