Closed Bug 1695667 Opened 4 years ago Closed 4 years ago

burnham failures for execution date 2021-02-28

Categories

(Data Platform and Tools :: Glean: SDK, defect, P1)

defect

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: raphael, Assigned: raphael)

References

Details

All of the sensors of the burnham Airflow DAG failed last night:

https://workflow.telemetry.mozilla.org/graph?dag_id=burnham&root=&execution_date=2021-02-28T00

When I run sensor query for the discovery table manually, I seem to get 10 rows. This matches our sensor success condition (see https://github.com/mozilla/telemetry-airflow/blob/master/dags/burnham.py#L603):

SELECT
  *
FROM
  `moz-fx-data-shared-prod.burnham_live.discovery_v1`
WHERE
  submission_timestamp BETWEEN TIMESTAMP_SUB("2021-03-01T00:00:22.293900+00:00", INTERVAL 1 HOUR)
  AND TIMESTAMP_ADD("2021-03-01T00:00:22.293900+00:00", INTERVAL 3 HOUR)
  AND metrics.uuid.test_run = "ed419cf0-30da-41dd-afbe-177f249de010"
  AND metrics.string.test_name = "test_burnham"

This error case looks pretty different from what I've seen before. These "wait_for_" tasks appear as if they never actually ran. There are no logs registered in Airflow for these runs.

I am pretty convinced that this failure is unrelated to https://bugzilla.mozilla.org/show_bug.cgi?id=1695230 so my recommendation would be to close this as WONTFIX for now. If we see a similar scenario in the future (all tasks fail, with no tries actually logged), it will be easier to justify prioritizing more investigation.

That sounds reasonable to me. Does that work for you, Jan-Erik?

Flags: needinfo?(jrediger)

Works for me.

Flags: needinfo?(jrediger)
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → WONTFIX
Assignee: nobody → rpierzina
See Also: → 1700121
You need to log in before you can comment on or make changes to this bug.