bigquery_table_storage_timeline_daily has some wild swings in numbers
Categories
(Data Platform and Tools :: General, defect)
Tracking
(Not tracked)
People
(Reporter: chutten|PTO, Unassigned)
References
Details
(Whiteboard: [dataplatform])
I was looking at the change in sizes of use counter tables (to try and figure out what size the partitions are, in case you're wondering. I don't have access to INFORMATION_SCHEMA.PARTITIONS inside telemetry_stable) and found some pretty wild swings.
And for once the problem doesn't appear to be in my use of window functions. Those are accurate measurements of changes in the values from one day to another.
This seems like a bug, but I'd be happy to be informed that I'm wrong.
|
||
Updated•2 years ago
|
|
||
Comment 1•2 years ago
|
||
cc :wichan: maybe you have some insight on why this might be happening?
|
||
Comment 2•2 years ago
|
||
I ran a query against the source table and I see fluctuations on total_physical_bytes around the same dates: https://sql.telemetry.mozilla.org/queries/97442/source#240532
|
||
Comment 3•11 months ago
|
||
The reason for this is that active_physical_bytes includes time travel bytes. Each time shredder runs on the table, that value will double for a week. The value that represents what's actually in the table is current_physical_bytes but that's not available in this table.
The information schema view this table is using is deprecated anyway so this table should be removed. monitoring_derived.bigquery_table_storage_v1 and monitoring_derived.table_storage_v1 can be used instead. I created https://mozilla-hub.atlassian.net/browse/DENG-9208 for that
Description
•