Closed Bug 1880272 Opened 2 years ago Closed 2 years ago

Airflow task ltv_daily.ltv_daily failed for exec_date 2024-02-14

Categories

(Data Platform and Tools :: General, defect)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: lvargas, Unassigned)

Details

(Whiteboard: [airflow-triage])

Airflow task ltv_daily.ltv_daily failed for exec_date 2024-02-14

Task link:
https://workflow.telemetry.mozilla.org/dags/ltv_daily/grid?dag_run_id=scheduled__2024-02-13T04%3A00%3A00%2B00%3A00&task_id=ltv_daily&tab=logs

Log extract:

[2024-02-14, 05:37:32 UTC] {taskinstance.py:1937} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 1518, in _run_raw_task
    self._execute_task_with_callbacks(context, test_mode, session=session)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 1684, in _execute_task_with_callbacks
    self.task.post_execute(context=context, result=result)  # type: ignore[union-attr]
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/operators/subdag.py", line 190, in post_execute
    raise AirflowException(f"Expected state: SUCCESS. Actual state: {dag_run.state}")

Error here was of model convergence:

24/02/14 13:14:20 WARN org.apache.spark.util.Utils: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.debug.maxToStringFields' in SparkEnv.conf.
      fun: -150.67940576586616
 hess_inv: array([[1.65548971, 1.7232383 , 0.18719648, 0.0533623 ],
       [1.7232383 , 6.35185852, 0.25046752, 0.04126487],
       [0.18719648, 0.25046752, 4.17361378, 1.80086176],
       [0.0533623 , 0.04126487, 1.80086176, 3.73694248]])
      jac: array([-1.14056470e-07, -3.88347797e-08,  5.38994670e-08, -4.93878932e-08])
  message: 'Desired error not necessarily achieved due to precision loss.'
     nfev: 86
      nit: 20
     njev: 74
   status: 2
  success: False
        x: array([-1.46818669, -5.88405973, -2.54970807, -1.41937746])
Traceback (most recent call last):
  File "/tmp/ltv-daily_83a4891d/ltv_daily.py", line 252, in <module>
    main(**vars(args))
  File "/tmp/ltv-daily_83a4891d/ltv_daily.py", line 138, in main
    model_perf, model = train_metric(search_rfm_ds, metric, plot=False, penalty=0.8)
  File "/tmp/ltv-daily_83a4891d/ltv_daily.py", line 30, in train_metric
    bgf.fit(train[frequency], train[recency], train[T])
  File "/opt/conda/default/lib/python3.6/site-packages/lifetimes/fitters/beta_geo_fitter.py", line 129, in fit
    **kwargs
  File "/opt/conda/default/lib/python3.6/site-packages/lifetimes/fitters/__init__.py", line 119, in _fit
    """
lifetimes.utils.ConvergenceError: 
The model did not converge. Try adding a larger penalizer to see if that helps convergence.

From the logs here: https://console.cloud.google.com/dataproc/jobs/ltv-daily_83a4891d/monitoring?project=airflow-dataproc&region=us-west1&rapt=AEjHL4NAiKMANfjAZwPFBfQDyr97k-b2JQOAUiIhpz5aVg_O98lNp1d1qcO050hY-MoJfSP9bn9fNKU1Ej2vsXUmqg2ETq_RtsTHAk3Nfqhh4B9ZjX1dqk8

SubDAG ltv_daily.ltv_daily is showing No Status for all tasks, and overall status running on 2024-02-15.

Task Link:
https://workflow.telemetry.mozilla.org/dags/ltv_daily.ltv_daily/grid?execution_date=2024-02-14T04%3A00%3A00%2B00%3A00

Depends on past = True. Further runs are blocked until the issue is resolved.

still blocked

Restarting fixed this.

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.