1768419 - Airflow task bqetl_subplat .stripe_external__itemized_payout_reconciliation__v5 failed on 2022-05-08

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Krzysztof Ignasiak [:kik]]

The Airflow task bqetl_subplat .stripe_external__itemized_payout_reconciliation__v5 failed on 2022-05-08.

DAG link: https://workflow.telemetry.mozilla.org/task?dag_id=bqetl_subplat&task_id=stripe_external__itemized_payout_reconciliation__v5&execution_date=2022-05-08T01%3A45%3A00%2B00%3A00

Error message from the failing pod (stripe-external--itemized-payout-reconciliation--v5.e1b80e7e7e08422ea03ac0c47052ce53):

2022-05-09T08:49:01.274526237Z
Error: Request req_OTssZ0Zv1cEmmm: Data for the report type payout_reconciliation.itemized.5 is only available through 2022-05-08 12:00:00 UTC; you requested `interval_end` = 2022-05-09 00:00:00 UTC. 

Link: https://console.cloud.google.com/kubernetes/pod/us-central1-a/bq-load-gke-1/default/stripe-external--itemized-payout-reconciliation--v5.e1b80e7e7e08422ea03ac0c47052ce53/logs?project=moz-fx-data-derived-datasets

My first thought this might just be an issue of scheduling from our side. However, I tried to re-run the DAG now and I still experienced the same error message. Perhaps there is a delay on Stripe's end in making the data available?

Comment 1

[Daniel Thorn [:relud]]

the arrival time of the report data is highly variable, and it's intended to retry until data becomes available.

Comment 2

[Krzysztof Ignasiak [:kik]]

Gotcha, thanks for the info. If it's okay I'd like to make one suggestion. Do you think it would be worth adding this as a note to the DAG's docs?

Comment 3

[Daniel Thorn [:relud]]

yeah, i'll file a PR to update the docs

Comment 4

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/bigquery-etl/pull/2959