1867804 - Airflow tasks dim_active_users_aggregates_mobile.*_dim_check failed for exec_date 2023-12-01

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Mike Williams [:mwilliams]]

Airflow tasks dim_active_users_aggregates_mobile.*_dim_check failed for exec_date 2023-12-01

Task link:
https://workflow.telemetry.mozilla.org/dags/dim_active_users_aggregates_mobile/grid?dag_run_id=scheduled__2023-11-30T04%3A15%3A00%2B00%3A00&task_id=fenix_active_users_aggregates_dim_check&tab=logs

Log extract:

Some dim checks failed during the run. Please check run_id: 517007e2-93d4-4a67-a3a8-e177345f5d57 inside dim_run_history table for more information.

Comment 1

[Lucia Vargas]

Attachment: bqetl_analytics_aggregations_status.png

A failure in firefox_android_clients which is a dependency for all mobile apps update resulted in this aggregates not updating on Nov 29, Dec 1 & 2.

Comment 2

[Lucia Vargas]

Attachment: bqetl_analytics_tables.png

The failure of the dependency is affecting the firefox_android_tables..

Comment 3

[Lucia Vargas]

Nov 29 is resolved.
Dec 1 and 2 pending.

Comment 4

[Lucia Vargas]

Attachment: dau_in_aggregates.png

Data is only available for Firefox Desktop on expected dates (until Dec 3).
Missing data for all mobile apps on Dec 1, 2 (expected) except for Firefox Android on Dec 1.

Comment 5

[Lucia Vargas]

@kignasiak There is a constant failure in firefox_android_clients task in the last 3 days.

Comment 6

[Krzysztof Ignasiak [:kik]]

I'm having a look and cannot see any failures of the firefox_android_clients task in the past few days, this includes the screenshot you shared above. I can see other task failures, but those are related to upstream task failures. This is something that the triage engineer should look into and if necessary involve the DAG owner to resolve.

For example:

I can currently see wait_for_checks__fail_org_mozilla_fenix_nightly_derived__baseline_clients_last_seen__v1 task having failed, but it is unrelated to this DAG but instead represents a failure in the upstream DAG's failure (bqetl_glean_usage) which is owned by ascholtz. As much as I'd like to help with all issue that come up, I am unable to do so. I expect this specific DAG (bqetl_analytics_aggregations) to become healthy again once the upstream issue is resolved.

Related to DIM tasks, there's an epic currently open for deprecating DIM: https://mozilla-hub.atlassian.net/browse/DENG-1990 which we plan to complete this year and DIM related runs should no longer be involved in any of the workflows.

Comment 7

[Lucia Vargas]

The checks for firefox_android_clients_v1 still return 0 records therefore block the downstream update.

Since the upstream view baseline_clients_daily has data, I am rerunning the Airflow task bqetl_analytics_tables.firefox_android_clients starting on Nov 30, 2023 where data began to be missing.

This run includes downstream dependencies. to unblock the update of active users and funnel aggregates.

Comment 8

[Lucia Vargas]

Data is now up to date in firefox_android clients and active_users_aggregates for Mobile.