burnham failures for execution date 2021-02-28
Categories
(Data Platform and Tools :: Glean: SDK, defect, P1)
Tracking
(Not tracked)
People
(Reporter: raphael, Assigned: raphael)
References
Details
All of the sensors of the burnham Airflow DAG failed last night:
https://workflow.telemetry.mozilla.org/graph?dag_id=burnham&root=&execution_date=2021-02-28T00
When I run sensor query for the discovery
table manually, I seem to get 10 rows. This matches our sensor success condition (see https://github.com/mozilla/telemetry-airflow/blob/master/dags/burnham.py#L603):
SELECT
*
FROM
`moz-fx-data-shared-prod.burnham_live.discovery_v1`
WHERE
submission_timestamp BETWEEN TIMESTAMP_SUB("2021-03-01T00:00:22.293900+00:00", INTERVAL 1 HOUR)
AND TIMESTAMP_ADD("2021-03-01T00:00:22.293900+00:00", INTERVAL 3 HOUR)
AND metrics.uuid.test_run = "ed419cf0-30da-41dd-afbe-177f249de010"
AND metrics.string.test_name = "test_burnham"
Comment 1•4 years ago
|
||
This error case looks pretty different from what I've seen before. These "wait_for_" tasks appear as if they never actually ran. There are no logs registered in Airflow for these runs.
Comment 2•4 years ago
|
||
I am pretty convinced that this failure is unrelated to https://bugzilla.mozilla.org/show_bug.cgi?id=1695230 so my recommendation would be to close this as WONTFIX for now. If we see a similar scenario in the future (all tasks fail, with no tries actually logged), it will be easier to justify prioritizing more investigation.
Assignee | ||
Comment 3•4 years ago
|
||
That sounds reasonable to me. Does that work for you, Jan-Erik?
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Updated•4 years ago
|
Description
•