1803772 - Airflow task bqetl_search_terms_daily .search_terms_derived__suggest_impression_sanitized__v2 failing on 2022-12-01

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Arkadiusz Komarzewski [:akomar]]

Airflow task bqetl_search_terms_daily .search_terms_derived__suggest_impression_sanitized__v2 failing on 2022-12-01

Comment 1

[Wesley Dawson [:whd]]

Because the DAG docs indicate this job's failure should be addressed urgently (since the upstream data has a very short expiration), :akomar reached out to me directly.

Preliminary investigation indicates that there is a dependency on copy-deduplicate for the scheduled query that runs
https://github.com/mozilla/bigquery-etl/blob/main/sql/moz-fx-data-shared-prod/search_terms_derived/suggest_impression_sanitized_v2_external/query.sql#L40. That query is defined here: https://github.com/mozilla-services/cloudops-infra/blob/master/projects/merino/tf/modules/log-routing/main.tf#L304, running at 2AM UTC. copy-deduplicate had some issues for today UTC's run and was delayed beyond that time.

:jbuck is going to try to manually trigger the scheduled query so that :wil can re-run the Airflow DAG.

It's worth nothing that this is a _v2 of a dataset for which a _v3 exists, and the latter appears to not have a dependence on stable table data, so it's perhaps worth investigating whether the workflow and scheduled query can be deprecated in its entirely. Absent that, making sure the dependence is well understood and documented in the DAG docs might be a useful remediation for when we have copy-deduplicate issues.

Comment 2

[Wil Stuckey]

A retry was run after jbuck manually triggered the scheduled query in the merino-logs project and the job + downstream completed successfully. Closing this out.

Regarding whether we can decommission this scheduled query. I think the answer to that is ultimately yes. However, we need to do the work to move adM reporting over to the v3 table before we can do that.