2023732 - Airflow task `peopleintegration-workday-get-report.peopleintegration-workday-to-bigquery` failed for run `scheduled__2026-03-15T13:50:00+00:00`

Status: NEW

(Data Platform and Tools :: General, defect)

Description

[cbeck]

Airflow task peopleintegration-workday-get-report.peopleintegration-workday-to-bigquery failed for run scheduled__2026-03-15T13:50:00+00:00.

Task link:
https://workflow.telemetry.mozilla.org/dags/peopleintegration-workday-get-report/grid?dag_run_id=scheduled__2026-03-15T13%3A50%3A00%2B00%3A00&task_id=peopleintegration-workday-to-bigquery

Log extract:

<extract of error log>

Comment 1

[Wesley Dawson [:whd]]

As of https://mozilla-hub.atlassian.net/browse/DENG-9313?focusedCommentId=1132689 and https://mozilla-hub.atlassian.net/browse/SVCSE-3016?focusedCommentId=1127778 this job is DE-owned, and any changes related to it need to be managed by DE. The logs make it look like something changed on the Workday side but it will require more investigation to resolve.

My understanding of https://mozilla-hub.atlassian.net/browse/DENG-9313?focusedCommentId=1114862 is that DE doesn't have any modern exports that might reasonably be used to replace this. As such the current state is that we don't have a reliable export of “active employees” to BigQuery until this job is fixed or a replacement is set up by DE.

Comment 2

[Wesley Dawson [:whd]]

After discussing with Alekhya I filed https://mozilla-hub.atlassian.net/browse/DO-2133 since it's clear DE does not want to support the existing pipeline.

Comment 3

[Alekhya Kommasani [:Alekhya]]

Hi Wesley, thanks for the detailed context here.

Given the current state and understanding that we’ll focus on evaluating and defining a new approach that DE can support long-term for providing “active employees” data in BigQuery.

In the meantime, is there any workaround or stopgap solution available to access reasonably up-to-date data while we work through the new request? Want to make sure downstream consumers aren’t blocked. Kiran will be looking into this

Happy to coordinate on next steps.

Comment 4

[Curtis Morales [:curtis]]

Sorry I missed this -- I flagged this last week with the users/maintainers of this pipeline, and they, specifically, @jmoscon and @prestonjackson, indicated that they have fully replaced it with their own solution. So we can simply delete this DAG. I'm opening up a PR to remove it now.

See Slack thread https://mozilla.slack.com/archives/C01E8GDG80N/p1773416069998719?thread_ts=1773332717.891209&cid=C01E8GDG80N

Comment 5

[Wesley Dawson [:whd]]

indicated that they have fully replaced it with their own solution

What solution is this? The dataset mozdata.taskclusteretl.person_mozilla_com remains a user-facing view. Its implementation can and should change based on what DE's willing to support but until that view has the properties it used to have or it is replaced by DE with a different way to get access to that data I don't consider deleting the existing DAG to be sufficient resolution to https://mozilla-hub.atlassian.net/browse/DO-2133 which is the upstream ticket I filed.

Comment 6

[Wesley Dawson [:whd]]

See https://mozilla-hub.atlassian.net/browse/DO-2133?focusedCommentId=1331696. If that approach works out I am fully in favor of deleting this DAG.

Comment 7

[kbammarito]

Per Wesley, we're pausing this broken DAG until we can regroup on a long-term solution (see above comment). See more info in this ticket.