The primary affected window is August 30th to September 3rd UTC, with increasing numbers of submissions affected over the course of the week as traffic was shifted. Per https://bugzilla.mozilla.org/show_bug.cgi?id=1666498#c24 the affected window for telemetry and structured data includes a limited window (< 1/251) from August 23-29. From discussion with :klukas we think it's best to avoid doing a full backfill for those days as the cost and time would be substantial, and instead make a note of it on https://github.com/mozilla/data-docs/blob/main/src/concepts/analysis_gotchas.md#notable-historic-events. NI :mreid for approval. The main issue here is that a new static value (the GCLB IP) is now added via nginx to the XFF proxy chain, and the logic we use in gcp-ingestion didn't account for it. This was caused by a logging change at some point from the cloudops-infra skeleton that I did not properly test for in later iterations of the edge deployment. The resolution we're planning to make on Tuesday is to perform the geoip lookup using the same rule as if `x_pipeline_proxy` is set if a static value `35.227.207.240` is the second to last entry of XFF. Essentially, the presence of `x_pipeline_proxy` means we should skip the an entry in XFF (that's the AWS tee), and the presence of `35.227.207.240` means we should also skip an entry (this is the GCLB and wasn't logged on the old GCP edge). stub installer will need a special case to backfill since the namespace it writes to `firefox-installer` is shared with the full installer ping from the `structured` pipeline family. We need to reprocess from August 17th to September 3rd for this table, but only from August 30th from `payload_bytes_raw.structured`. We should be able to select pings from the decoded table before that that have `installer_type` = `full` and combine them with data from `stub_installer_snapshot_bug1729069`. I will separately re-verify ctxsvc iprepd logic later today.
Bug 1729069 Comment 8 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
The primary affected window is August 30th to September 3rd UTC, with increasing numbers of submissions affected over the course of the week as traffic was shifted. Per https://bugzilla.mozilla.org/show_bug.cgi?id=1666498#c24 the affected window for telemetry and structured data includes a limited window (< 1/251) from August 23-29. From discussion with :klukas we think it's best to avoid doing a full backfill for those days as the cost and time would be substantial, and instead make a note of it on https://github.com/mozilla/data-docs/blob/main/src/concepts/analysis_gotchas.md#notable-historic-events. NI :mreid for approval. The main issue here is that a new static value (the GCLB IP) is now added via nginx to the XFF proxy chain, and the logic we use in gcp-ingestion didn't account for it. This was caused by a logging change at some point from the cloudops-infra skeleton that I did not properly test for in later iterations of the edge deployment. The resolution we're planning to make on Tuesday is to perform the geoip lookup using the same rule as if `x_pipeline_proxy` is set if a static value `35.227.207.240` is the second to last entry of XFF. Essentially, the presence of `x_pipeline_proxy` means we should skip the an entry in XFF (that's the AWS tee), and the presence of `35.227.207.240` means we should also skip an entry (this is the GCLB and wasn't logged on the old GCP edge). EDIT: this value should probably be configurable to account for multiple deployments such as stage or another migration, but in practice we will never want to backfill stage. stub installer will need a special case to backfill since the namespace it writes to `firefox-installer` is shared with the full installer ping from the `structured` pipeline family. We need to reprocess from August 17th to September 3rd for this table, but only from August 30th from `payload_bytes_raw.structured`. We should be able to select pings from the decoded table before that that have `installer_type` = `full` and combine them with data from `stub_installer_snapshot_bug1729069`. I will separately re-verify ctxsvc iprepd logic later today.