1738919 - Airflow bqetl_subplat.mozilla_vpn_external__devices__v1 failing on 2021-11-01

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Anna Scholtz [:ascholtz]]

bqetl_subplat.mozilla_vpn_external__devices__v1 has been failing on 2021-11-01:

[2021-11-02 02:45:56,686] {{pod_launcher.py:149}} INFO - Error in query string: Error processing job 'moz-fx-data-shared-
[2021-11-02 02:45:56,686] {{pod_launcher.py:149}} INFO - prod:bqjob_r4e9fb34b9b86a786_0000017cde89ef4e_1': No matching signature for
[2021-11-02 02:45:56,687] {{pod_launcher.py:149}} INFO - function IF for argument types: BOOL, STRUCT<id INT64, user_id INT64, name
[2021-11-02 02:45:56,687] {{pod_launcher.py:149}} INFO - STRING, ...>, STRUCT<id INT64, user_id INT64, name STRING, ...>. Supported
[2021-11-02 02:45:56,687] {{pod_launcher.py:149}} INFO - signature: IF(BOOL, ANY, ANY) at [2:3]
[2021-11-02 02:45:56,885] {{pod_launcher.py:149}} INFO - Traceback (most recent call last):
[2021-11-02 02:45:56,885] {{pod_launcher.py:149}} INFO -   File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
[2021-11-02 02:45:56,885] {{pod_launcher.py:149}} INFO -     return _run_code(code, main_globals, None,
[2021-11-02 02:45:56,886] {{pod_launcher.py:149}} INFO -   File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
[2021-11-02 02:45:56,886] {{pod_launcher.py:149}} INFO -     exec(code, run_globals)
[2021-11-02 02:45:56,886] {{pod_launcher.py:149}} INFO -   File "/app/bigquery_etl/run_query.py", line 107, in <module>
[2021-11-02 02:45:56,886] {{pod_launcher.py:149}} INFO -     main()
[2021-11-02 02:45:56,886] {{pod_launcher.py:149}} INFO -   File "/app/bigquery_etl/run_query.py", line 97, in main
[2021-11-02 02:45:56,886] {{pod_launcher.py:149}} INFO -     run(
[2021-11-02 02:45:56,886] {{pod_launcher.py:149}} INFO -   File "/app/bigquery_etl/run_query.py", line 91, in run
[2021-11-02 02:45:56,886] {{pod_launcher.py:149}} INFO -     subprocess.check_call(["bq"] + query_arguments, stdin=query_stream)
[2021-11-02 02:45:56,887] {{pod_launcher.py:149}} INFO -   File "/usr/local/lib/python3.8/subprocess.py", line 364, in check_call
[2021-11-02 02:45:56,887] {{pod_launcher.py:149}} INFO -     raise CalledProcessError(retcode, cmd)
[2021-11-02 02:45:56,887] {{pod_launcher.py:149}} INFO - subprocess.CalledProcessError: Command '['bq', 'query', '--project_id=moz-fx-data-shared-prod', "--parameter=external_database_query:STRING:SELECT * FROM devices WHERE DATE(updated_at) = DATE '2021-11-01'", '--dataset_id=mozilla_vpn_external', '--destination_table=devices_v1']' returned non-zero exit status 1.
[2021-11-02 02:45:58,988] {{pod_launcher.py:198}} INFO - Event: mozilla-vpn-external--devices--v1.b7560f6785d941c0b872dd9c2970c9e0 had an event of type Failed
[2021-11-02 02:45:58,989] {{pod_launcher.py:308}} ERROR - Event with job id mozilla-vpn-external--devices--v1.b7560f6785d941c0b872dd9c2970c9e0 Failed
[2021-11-02 02:45:59,032] {{pod_launcher.py:198}} INFO - Event: mozilla-vpn-external--devices--v1.b7560f6785d941c0b872dd9c2970c9e0 had an event of type Failed
[2021-11-02 02:45:59,032] {{pod_launcher.py:308}} ERROR - Event with job id mozilla-vpn-external--devices--v1.b7560f6785d941c0b872dd9c2970c9e0 Failed
[2021-11-02 02:45:59,046] {{taskinstance.py:1501}} ERROR - Task failed with exception

Comment 1

[Daniel Thorn [:relud]]

added new field unique_id of type STRING to the schema and this is working again.