1768507 - Monitor rate of duplicates in live tables

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P2)

Description

[Jeff Klukas [:klukas] (UTC-4)]

As discussed in https://bugzilla.mozilla.org/show_bug.cgi?id=1767545 we should add a monitor for the rate of duplicates in each live table.

Comment 1

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/bigquery-etl/pull/2953

Comment 2

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/bigquery-etl/pull/2958

Comment 3

[Jeff Klukas [:klukas] (UTC-4)]

PR has been merged and the job is working. You can now run queries like:

SELECT *, SAFE_DIVIDE(live_nondistinct, live) AS dupe_ratio
FROM
  `moz-fx-data-shared-prod.monitoring.structured_distinct_docids`
  where submission_date = '2022-05-17'
LIMIT
  1000

We still need to add an explore to the Looker dashboard, though.

Comment 4

[Jeff Klukas [:klukas] (UTC-4)]

Unassigning to make this available for others to take over the finish line with a new Looker explore.

Comment 5

[Marlene Hirose]

Added: https://mozilla.cloud.looker.com/looks/1248

Added the above Look to the Platform Health Check dash: https://mozilla.cloud.looker.com/dashboards/387?Submission+Date=90+day+ago+for+90+day