1402000 - New field ("disabled_addons_ids") showing up in the parquet file but not in re:dash

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Alessio Placitelli [:Dexter]]

I recently added [1] the "disabled_addons_ids" field to the main_summary in bug 1390814 (14th of September). Unfortunately, if I try to query this field in re:dash [2] it will fail saying that it doesn't know the column. Reading a parquet file in pyspark (from telemetry-parquet/main_summary/v4/submission_date_s3=20170920/) shows the field in the schema and allows me to see some data.

[1] - https://github.com/mozilla/telemetry-batch-view/pull/295#pullrequestreview-61600255
[2] - https://sql.telemetry.mozilla.org/queries/40380/source

Comment 1

[Frank Bertsch [:frank]]

main_summary is still missing "UPTAKE_REMOTE_CONTENT_RESULT_1" as well - Blake, can you take a look?

[0] https://github.com/mozilla/telemetry-batch-view/commit/812a1f7a65100937c5fce481c762c9fda724aa93

Comment 2

[Frank Bertsch [:frank]]

Jason, can you help us out here?

Comment 3

[Blake Imsland [:robotblake]]

Ugh, updated the wrong bug, these got manually added last Friday so I'm confused why they're not working, looking into it again.

Comment 4

[Blake Imsland [:robotblake]]

This has been fixed but still working to understand the underlying cause, closing for now but will add a note once I figure that out.