1803344 - Airflow task glam_fog.query_firefox_desktop_glam_nightly__histogram_percentiles_v1 failing on 2022-11-30

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Arkadiusz Komarzewski [:akomar]]

Airflow task glam_fog.query_firefox_desktop_glam_nightly__histogram_percentiles_v1 failing on 2022-11-30

Probably caused by some weird data coming from nightly:

[2022-11-30 10:27:29,092] {{pod_launcher.py:149}} INFO - + bq query --max_rows=0 --use_legacy_sql=false --replace --project_id=moz-fx-data-glam-prod-fca7 --dataset_id=glam_etl --destination_table=firefox_desktop_glam_nightly__histogram_percentiles_v1 --parameter=submission_date:DATE:2022-11-29 --parameter=min_sample_id:INT64:0 --parameter=max_sample_id:INT64:99 --parameter=sample_size:INT64:10 ''
[2022-11-30 10:27:34,326] {{pod_launcher.py:149}} INFO - 
Waiting on bqjob_r39f32313cf69a8de_00000184c813cf68_1 ... (0s) Current status: RUNNING
                                                                                      
Waiting on bqjob_r39f32313cf69a8de_00000184c813cf68_1 ... (0s) Current status: DONE
[2022-11-30 10:27:34,342] {{pod_launcher.py:149}} INFO - Error in query string: Error processing job 'moz-fx-data-glam-prod-
[2022-11-30 10:27:34,342] {{pod_launcher.py:149}} INFO - fca7:bqjob_r39f32313cf69a8de_00000184c813cf68_1': TypeError: Reduce of empty
[2022-11-30 10:27:34,342] {{pod_launcher.py:149}} INFO - array with no initial value at UDF$1(FLOAT64, ARRAY<STRUCT<key STRING, value
[2022-11-30 10:27:34,342] {{pod_launcher.py:149}} INFO - FLOAT64>>, STRING) line 7, columns 21-22

Comment 1

[Eduardo Filho [:efilho]]

Querying the source table firefox_desktop.metrics I can see the metric in question has data. So working with the assumption the ETL likely has a bug during Extraction/Transform.

Comment 2

[Eduardo Filho [:efilho]]

Attachment: pr 3386: [Bug 1803344] GLAM - use firefox-desktop Glean metadata to build fog queries

Comment 3

[Eduardo Filho [:efilho]]

Landed the PR, re-run the bucket_counts step onward for both Nov 29 and Nov 30, then imported the aggregations into GLAM.
The import DAG failed a few times saying psycopg2.errors.UndefinedTable: table "tmp_import_glean_nightly" does not exist but that's probably because of race conditions (two dags trying to import data at the same time). We then killed the Nov 29 import which made the other import succeed.
I also checked a couple of Glean probes in GLAM and we have data for Nov29 and Nov30.
Probes checked:

• https://glam.telemetry.mozilla.org/fog/probe/glean_upload_pending_pings_directory_size/explore?aggType=avg&currentPage=1&ping_type=*
• https://glam.telemetry.mozilla.org/fog/probe/glean_database_size/explore?aggType=avg&currentPage=1&ping_type=*

I'm convinced the bug has been fixed and the data has caught up.