Closed Bug 1866892 Opened 2 years ago Closed 2 years ago

Airflow task taar_daily..taar_similarity failed for exec_date 2023-11-27

Categories

(Data Platform and Tools :: General, defect)

Product:
Data Platform and Tools
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED INACTIVE

People

(Reporter: mwilliams, Unassigned)

References

Details

(Whiteboard: [airflow-triage])

Airflow task taar_daily..taar_similarity failed for exec_date 2023-11-27

Task link:
https://workflow.telemetry.mozilla.org/dags/taar_daily/grid?dag_run_id=scheduled__2023-11-26T04%3A00%3A00%2B00%3A00&task_id=taar_similarity&tab=logs

Log extract:

[2023-11-27, 04:00:12 UTC] {subdag.py:172} INFO - Created DagRun: scheduled__2023-11-26T04:00:00+00:00
[2023-11-27, 04:32:13 UTC] {base.py:255} INFO - Success criteria met. Exiting.
[2023-11-27, 04:32:13 UTC] {subdag.py:187} INFO - Execution finished. State is failed
[2023-11-27, 04:32:13 UTC] {taskinstance.py:1824} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 1407, in _run_raw_task
    self._execute_task_with_callbacks(context, test_mode)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 1560, in _execute_task_with_callbacks
    self.task.post_execute(context=context, result=result)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/operators/subdag.py", line 190, in post_execute
    raise AirflowException(f"Expected state: SUCCESS. Actual state: {dag_run.state}")
airflow.exceptions.AirflowException: Expected state: SUCCESS. Actual state: failed
[2023-11-27, 04:32:13 UTC] {taskinstance.py:1345} INFO - Marking task as FAILED. dag_id=taar_daily, task_id=taar_similarity, execution_date=20231126T040000, start_date=20231127T040011, end_date=20231127T043213
[2023-11-27, 04:32:13 UTC] {warnings.py:109} WARNING - /home/airflow/.local/lib/python3.10/site-packages/airflow/utils/email.py:153: RemovedInAirflow3Warning: Fetching SMTP credentials from configuration variables will be deprecated in a future release. Please set credentials using a connection instead.
  send_mime_email(e_from=mail_from, e_to=recipients, mime_msg=msg, conn_id=conn_id, dryrun=dryrun)
[2023-11-27, 04:32:13 UTC] {email.py:269} INFO - Email alerting: attempt 1
[2023-11-27, 04:32:14 UTC] {email.py:281} INFO - Sent an alert email to ['telemetry-alerts@mozilla.com', 'hwoo@mozilla.com', 'epavlov@mozilla.com']
[2023-11-27, 04:32:14 UTC] {standard_task_runner.py:104} ERROR - Failed to execute job 1921048 for task taar_similarity (Expected state: SUCCESS. Actual state: failed; 1046388)
[2023-11-27, 04:32:14 UTC] {local_task_job_runner.py:225} INFO - Task exited with return code 1
Flags: needinfo?(hwoo)
Flags: needinfo?(epavlov)

Theres not really an owner for this since epavlov has moved to pocket. But the spark job logs seem to indicate that this may be due to the v4->v5 switch? Or maybe thats just a coincidence since it's an api version and not related to main v4/v5

23/12/13 05:47:26 INFO com.mozilla.telemetry.ml.AddonRecommender$: Training - using clients_daily from 20230914
23/12/13 05:47:26 INFO com.mozilla.telemetry.ml.AMODatabase$: Downloading AMO data from https://addons.mozilla.org/api/v4/addons/search/
23/12/13 05:47:26 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&sort=created&type=extension
23/12/13 05:47:27 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=2&sort=created&type=extension
23/12/13 05:47:27 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=3&sort=created&type=extension
23/12/13 05:47:27 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=4&sort=created&type=extension
23/12/13 05:47:27 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=5&sort=created&type=extension
23/12/13 05:47:27 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=6&sort=created&type=extension
23/12/13 05:47:28 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=7&sort=created&type=extension
23/12/13 05:47:28 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=8&sort=created&type=extension
23/12/13 05:47:28 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=9&sort=created&type=extension
23/12/13 05:47:28 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=10&sort=created&type=extension
23/12/13 05:47:28 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=11&sort=created&type=extension
23/12/13 05:47:29 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=12&sort=created&type=extension
23/12/13 05:47:29 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=13&sort=created&type=extension
23/12/13 05:47:29 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=14&sort=created&type=extension
23/12/13 05:47:29 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=15&sort=created&type=extension
23/12/13 05:47:29 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=16&sort=created&type=extension
23/12/13 05:47:30 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=17&sort=created&type=extension
23/12/13 05:47:30 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=18&sort=created&type=extension
23/12/13 05:47:30 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=19&sort=created&type=extension
23/12/13 05:47:30 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=20&sort=created&type=extension
23/12/13 05:47:30 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=21&sort=created&type=extension
23/12/13 05:47:30 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=22&sort=created&type=extension
23/12/13 05:47:31 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=23&sort=created&type=extension
23/12/13 05:47:31 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=24&sort=created&type=extension
23/12/13 05:47:31 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=25&sort=created&type=extension
23/12/13 05:47:31 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=26&sort=created&type=extension
23/12/13 05:47:31 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=27&sort=created&type=extension
23/12/13 05:47:32 INFO com.mozilla.telemetry.ml.AMODatabase$: Fetching https://addons.mozilla.org/api/v4/addons/search/?app=firefox&page=28&sort=created&type=extension
Exception in thread "main" com.fasterxml.jackson.databind.JsonMappingException: No content to map due to end-of-input
 at [Source: ; line: 1, column: 1]
	at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:148)
	at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3781)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3721)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2726)
	at org.json4s.jackson.JsonMethods$class.parse(JsonMethods.scala:20)
	at org.json4s.jackson.JsonMethods$.parse(JsonMethods.scala:50)
	at com.mozilla.telemetry.ml.AMODatabase$.fetchAMOPage$1(AMODatabase.scala:68)
	at com.mozilla.telemetry.ml.AMODatabase$.fetchAddonsDatabase(AMODatabase.scala:82)
	at com.mozilla.telemetry.ml.AMODatabase$.getDatabase(AMODatabase.scala:101)
	at com.mozilla.telemetry.ml.AMODatabase$.getAddonMap(AMODatabase.scala:114)
	at com.mozilla.telemetry.ml.AddonRecommender$.train(AddonRecommender.scala:191)
	at com.mozilla.telemetry.ml.AddonRecommender$.main(AddonRecommender.scala:325)
	at com.mozilla.telemetry.ml.AddonRecommender.main(AddonRecommender.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:890)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:192)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:217)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Flags: needinfo?(hwoo)

Ok this started failing 12/06 which aligns with when we migrated AMO from AWS to GCP

See Also: → 1868684

Comments above are related to Bug 1868684, taar_daily.taar_similarity hasn't failed since 2023-11-27

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → INACTIVE

I believe ownership of this service was transferred to data engineering/AMO teams more than a year ago. All the issues are likely due to migrations of data sources. It shouldn't break the service though, the recommendations will be based on old predictions.

Flags: needinfo?(epavlov)
You need to log in before you can comment on or make changes to this bug.